my robots.txt is correct?

Discussion in 'Google Sitemaps' started by anhthaiduong, Apr 7, 2008.

0
  1. #1
    here's my robots.txt just allowing yahoo and google crawler

    User-agent:slurp
    dissallow:

    User-agent:googlebot
    dissallow:

    User-agent:*
    dissallow:/

    sitemap:htpp://.......
    --------------------------
    is it correct?
    what should i choose to state in robots.txt ? slurp or yahoo!slurp
     
    anhthaiduong, Apr 7, 2008 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    This will work fine as per your specifications...
    And there is no difference in slurp and yahoo!slurp..both do the same job..
     
    manish.chauhan, Apr 7, 2008 IP
  3. mistoovrool

    mistoovrool Banned

    Messages:
    202
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    :D:D
    Using this robots.txt you disallow all crawler to crawl your site.

    ------------------------------------------------------
    User-agent:slurp
    dissallow:

    By this 2 line code you allow yahoo bot to crawl all your site and disallow nothing.
    --------------------------------------------------
    User-agent:googlebot
    dissallow:

    By this 2 line code you allow google bot to crawl all your site and disallow nothing.
    -------------------------------------------------------
    User-agent:*
    dissallow:/

    User-agent:* mean all search engine crawler and
    And this dissallow:/ will disallow your entire website.
    -------------------------------------------------------------------
    Remove that dissallow:/ code other wise none of search engine crawler crawl your site.
     
    mistoovrool, Apr 7, 2008 IP