Changes between Version 1 and Version 2 of RobotsTxt


Ignore:
Timestamp:
Dec 22, 2008, 10:34:05 AM (15 years ago)
Author:
Bruno Cornec
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • RobotsTxt

    v1 v2  
    1 User-agent: *
    2 Disallow: /log
    3 Disallow: /newticket
    4 
    51User-agent: msnbot-media
    62Crawl-delay: 720
    … …  
    3228User-agent: Yahoo-Blogs/v3.9
    3329Crawl-delay: 20
     30
     31User-agent: *
     32Disallow: /log
     33Disallow: /newticket