wiki:RobotsTxt

User-agent: msnbot-media Crawl-delay: 720 Disallow: /

User-agent: msnbot-news Crawl-delay: 720 Disallow: /

User-agent: msnbot-products Crawl-delay: 720 Disallow: /

User-agent: psbot Crawl-delay: 720 Disallow: /

User-agent: MSNBot Crawl-delay: 720 Disallow: /

User-agent: msnbot/1.1 Crawl-delay: 720 Disallow: /

User-agent: Twiceler-0.9 Crawl-delay: 720 Disallow: /

User-agent: yahoo-slurp Crawl-delay: 10

User-agent: Yahoo-Blogs/v3.9 Crawl-delay: 20

User-agent: * Disallow: /log Disallow: /newticket

Last modified 8 years ago Last modified on Dec 22, 2008, 10:37:06 AM