User-agent: msnbot-media Crawl-delay: 720 Disallow: /
User-agent: msnbot-news Crawl-delay: 720 Disallow: /
User-agent: msnbot-products Crawl-delay: 720 Disallow: /
User-agent: psbot Crawl-delay: 720 Disallow: /
User-agent: MSNBot Crawl-delay: 720 Disallow: /
User-agent: msnbot/1.1 Crawl-delay: 720 Disallow: /
User-agent: Twiceler-0.9 Crawl-delay: 720 Disallow: /
User-agent: yahoo-slurp Crawl-delay: 10
User-agent: Yahoo-Blogs/v3.9 Crawl-delay: 20
User-agent: * Disallow: /log Disallow: /newticket
Last modified
15 years ago
Last modified on Dec 22, 2008, 10:37:06 AM
Note:
See TracWiki
for help on using the wiki.
