Changes between Version 2 and Version 3 of RobotsTxt


Ignore:
Timestamp:
Dec 22, 2008, 10:37:06 AM (15 years ago)
Author:
Bruno Cornec
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • RobotsTxt

    v2 v3  
    2323Disallow: /
    2424
     25User-agent: Twiceler-0.9
     26Crawl-delay: 720
     27Disallow: /
     28
    2529User-agent: yahoo-slurp
    2630Crawl-delay: 10