1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Hello, I am new, and need some help on robots.txt

Discussion in 'robots.txt' started by abrahamgold, Oct 21, 2005.

0
  1. #1
    I found a robots.txt file on the root directory of my website, I did not create this file ( code below). My store and website is hosted in yahoo, Can anyone help me understand if this file is preventing all SE to index my pages ?
    Thanks in advance for your help


    User-agent: *
    Disallow: /cgi-bin
    Disallow: /RT
    Disallow: /TG
    Disallow: /QS
    Disallow: /Img
    Disallow: /I
    Disallow: /wgl
    Disallow: /P
    Disallow: /ymix

    # Rover is a bad dog <http://www.roverbot.com>
    User-agent: Roverbot
    Disallow: /
     
    abrahamgold, Oct 21, 2005 IP
  2. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #2
    Hard to say. What is in all those directories on your site?

    cgi-bin is the standard directory for scripts - usually disallowed
    I'd guess that /Img is images - good to disallow

    No idea what the rest are.

    But if you didn't put it there, who did? Anyone else working on your site?
     
    minstrel, Oct 22, 2005 IP
  3. The Webmaster

    The Webmaster IdeasOfOne

    Messages:
    9,516
    Likes Received:
    718
    Best Answers:
    0
    Trophy Points:
    360
    #3

    It is not disallowing Bots to Crawl Main site. just the script directory, Image directory and few other special directories. so nothing to worry. untill you want your images to appear in Google image search.
    if you want your images to appear in Search, remove the line
    Disallow: /Img
    from the file and upload it to its previous location.
    If this is Yahoo store site, Yahoo puts it automaticaly along with other scripts they put on the webserver.
     
    The Webmaster, Oct 22, 2005 IP
  4. RaZoRLeGaCy

    RaZoRLeGaCy Well-Known Member

    Messages:
    189
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #4
    everything seems of the norm so don't worry.

    They will not hinder your results in the slightest.
     
    RaZoRLeGaCy, Oct 25, 2006 IP
  5. Manthem

    Manthem Well-Known Member

    Messages:
    432
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    110
    #5
    I'm sure he's still worrying about this. Thanks Razor.
     
    Manthem, Oct 25, 2006 IP
  6. Tal Kedar

    Tal Kedar Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    This code is generated by the Yahoo! engine, disallowing crawling over parts of the site irrelevant for SEs (checkout pages, for example). The relevant parts of the site will indeed be crawled with these definitions.

    Tal Kedar
    Research
     
    Tal Kedar, Oct 30, 2006 IP
  7. hasitruparel

    hasitruparel Peon

    Messages:
    60
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    we cannot edit the Yahoo store robots.txt, this is CLEARLY mentioned on yahoostore help page.

    http://help.yahoo.com/help/us/store/promote/mail/mail-42.html
     
    hasitruparel, Nov 28, 2006 IP
  8. davidannis

    davidannis Greenhorn

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #8
    I noticed another site, topiccraze.com that has roverbot disallowed in their robots.txt with the following:
    Any idea why these websites hate roverbot so much? More to the point does it make sense to disallow it from my site?
     
    davidannis, Mar 18, 2008 IP
  9. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #9
    What I know about Roverbot, it is basically Email Gathering Robot...So every website disallow this robot...
     
    manish.chauhan, Apr 4, 2008 IP
  10. Yankee85

    Yankee85 Peon

    Messages:
    1,067
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Yes, it collects emails from websites
     
    Yankee85, Apr 19, 2008 IP
  11. ankitsoldak

    ankitsoldak Well-Known Member

    Messages:
    343
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    108
    #11
    when ever i use to login in gmail 1 robot.txt file pops up.

    i don't whats that .....

    plz give some suggestion
     
    ankitsoldak, May 17, 2008 IP
  12. riverrun

    riverrun Active Member

    Messages:
    233
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    60
    #12
    I think ill disallow the rover bot after reading this.
     
    riverrun, Jun 28, 2008 IP
  13. dkextbmx

    dkextbmx Member

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #13
    Rover has been a bad dog indeed!
     
    dkextbmx, Feb 20, 2012 IP
  14. bdthanh

    bdthanh Peon

    Messages:
    28
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #14
    It's normal Yahoo robots file
     
    bdthanh, Mar 4, 2012 IP
  15. SPA assurance

    SPA assurance Peon

    Messages:
    93
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    plz edit all robot.txt file and create New file

    User-agent: *
    Disallow: /

    I think it is best option for you..............
     
    SPA assurance, Mar 8, 2012 IP