linkcheck.cache.urlqueue

Handle a queue of URLs to check.

Classes

UrlQueue([max_allowed_urls])

A queue supporting several consumer tasks.

Exceptions

Empty	Exception raised by get().
Timeout	Raised by join()

exception linkcheck.cache.urlqueue.Empty

Bases: Exception

Exception raised by get().

exception linkcheck.cache.urlqueue.Timeout

Bases: Exception

Raised by join()

class linkcheck.cache.urlqueue.UrlQueue(max_allowed_urls=None)

Bases: object

A queue supporting several consumer tasks. The task_done() idea is from the Python 2.5 implementation of Queue.Queue().

Initialize the queue state and task counters.

cleanup()

Move cached elements to top.

do_shutdown()

Shutdown the queue by not accepting any more URLs.

empty()

Return True if the queue is empty, False otherwise. Result is thread-safe, but not reliable since the queue could have been changed before the result is returned!

get(timeout=None)

Get first not-in-progress url from the queue and return it. If no such url is available return None.

join(timeout=None)

Blocks until all items in the Queue have been gotten and processed.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate the item was retrieved and all work on it is complete.

When the count of unfinished tasks drops to zero, join() unblocks.

put(item)

Put an item into the queue. Block if necessary until a free slot is available.

qsize()

Return the approximate size of the queue (not reliable!).

status()

Get tuple (finished tasks, in progress, queue size).

task_done(url_data)

Indicate that a formerly enqueued task is complete.

Used by Queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items placed in the queue.