Hello, I am new, and need some help on robots.txt

[abrahamgold]

I found a robots.txt file on the root directory of my website, I did not create this file ( code below). My store and website is hosted in yahoo, Can anyone help me understand if this file is preventing all SE to index my pages ?
Thanks in advance for your help


User-agent: *
Disallow: /cgi-bin
Disallow: /RT
Disallow: /TG
Disallow: /QS
Disallow: /Img
Disallow: /I
Disallow: /wgl
Disallow: /P
Disallow: /ymix

# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /

[minstrel]

Hard to say. What is in all those directories on your site?

cgi-bin is the standard directory for scripts - usually disallowed
I'd guess that /Img is images - good to disallow

No idea what the rest are.

But if you didn't put it there, who did? Anyone else working on your site?

[The Webmaster]

I found a robots.txt file on the root directory of my website, I did not create this file ( code below). My store and website is hosted in yahoo, Can anyone help me understand if this file is preventing all SE to index my pages ?
Thanks in advance for your help


User-agent: *
Disallow: /cgi-bin
Disallow: /RT
Disallow: /TG
Disallow: /QS
Disallow: /Img
Disallow: /I
Disallow: /wgl
Disallow: /P
Disallow: /ymix

# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /


It is not disallowing Bots to Crawl Main site. just the script directory, Image directory and few other special directories. so nothing to worry. untill you want your images to appear in Google image search.
if you want your images to appear in Search, remove the line
Disallow: /Img
from the file and upload it to its previous location.
But if you didn't put it there, who did? Anyone else working on your site?

If this is Yahoo store site, Yahoo puts it automaticaly along with other scripts they put on the webserver.

[RaZoRLeGaCy]

everything seems of the norm so don't worry.

They will not hinder your results in the slightest.

[Manthem]

I'm sure he's still worrying about this. Thanks Razor.

[Tal Kedar]

This code is generated by the Yahoo! engine, disallowing crawling over parts of the site irrelevant for SEs (checkout pages, for example). The relevant parts of the site will indeed be crawled with these definitions.

Tal Kedar
Research

[hasitruparel]

I found a robots.txt file on the root directory of my website, I did not create this file ( code below). My store and website is hosted in yahoo, Can anyone help me understand if this file is preventing all SE to index my pages ?
Thanks in advance for your help


User-agent: *
Disallow: /cgi-bin
Disallow: /RT
Disallow: /TG
Disallow: /QS
Disallow: /Img
Disallow: /I
Disallow: /wgl
Disallow: /P
Disallow: /ymix

# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /
we cannot edit the Yahoo store robots.txt, this is CLEARLY mentioned on yahoostore help page.

http://help.yahoo.com/help/us/store/promote/mail/mail-42.html

[davidannis]

I noticed another site, topiccraze.com that has roverbot disallowed in their robots.txt with the following:
# Rover is the spawn of satan <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /

Any idea why these websites hate roverbot so much? More to the point does it make sense to disallow it from my site?

[manish.chauhan]

I noticed another site, topiccraze.com that has roverbot disallowed in their robots.txt with the following:

Any idea why these websites hate roverbot so much? More to the point does it make sense to disallow it from my site?

What I know about Roverbot, it is basically Email Gathering Robot...So every website disallow this robot...

[Yankee85]

Yes, it collects emails from websites