linkcheck.director.aggregator

Aggregate needed object instances for checker threads.

Functions

new_request_session(config, cookies)

Create a new request session.

Classes

Aggregate(config, urlqueue, robots_txt, ...)

Store thread-safe data collections for checker threads.

class linkcheck.director.aggregator.Aggregate(config, urlqueue, robots_txt, plugin_manager, result_cache)

Bases: object

Store thread-safe data collections for checker threads.

Store given link checking objects.

abort()

Print still-active URLs and empty the URL queue.

add_downloaded_bytes(**kwargs)

Add to number of downloaded bytes.

add_request_session(**kwargs)

Add a request session for current thread.

cancel()

Empty the URL queue.

end_log_output(**kwargs)

Print ending output to log.

finish(**kwargs)

Wait for checker threads to finish.

get_check_threads(**kwargs)

Return iterator of checker threads.

get_request_session(**kwargs)

Get the request session for current thread.

is_finished(**kwargs)

Determine if checking is finished.

print_active_threads(**kwargs)

Log all currently active threads.

remove_stopped_threads(**kwargs)

Remove the stopped threads from the internal thread list.

set_maxrated_for_host(**kwargs)

Remove the limit on the maximum request rate for a host.

start_threads(**kwargs)

Spawn threads for URL checking and status printing.

visit_loginurl()

Check for a login URL and visit it.

wait_for_host(**kwargs)

Throttle requests to one host.

wait_time_max_default = 0.6

wait_time_min_default = 0.1

linkcheck.director.aggregator.new_request_session(config, cookies)

Create a new request session.