Changes between Initial Version and Version 1 of RobotsTxt


Ignore:
Timestamp:
Dec 22, 2008, 10:29:55 AM (15 years ago)
Author:
Bruno Cornec
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • RobotsTxt

    v1 v1  
     1User-agent: *
     2Disallow: /log
     3Disallow: /newticket
     4
     5User-agent: msnbot-media
     6Crawl-delay: 720
     7Disallow: /
     8
     9User-agent: msnbot-news
     10Crawl-delay: 720
     11Disallow: /
     12
     13User-agent: msnbot-products
     14Crawl-delay: 720
     15Disallow: /
     16
     17User-agent: psbot
     18Crawl-delay: 720
     19Disallow: /
     20
     21User-agent: MSNBot
     22Crawl-delay: 720
     23Disallow: /
     24
     25User-agent: msnbot/1.1
     26Crawl-delay: 720
     27Disallow: /
     28
     29User-agent: yahoo-slurp
     30Crawl-delay: 10
     31
     32User-agent: Yahoo-Blogs/v3.9
     33Crawl-delay: 20