Cache robots.txt contents.
Classes
RobotsTxt(useragent)
RobotsTxt
Thread-safe cache of downloaded robots.txt files.
Bases: object
object
Thread-safe cache of downloaded robots.txt files. format: {cache key (string) -> robots.txt content (RobotFileParser)}
Initialize per-URL robots.txt cache.
Add sitemap URLs to queue.
Ask robots.txt allowance.
Return lock for robots.txt url.