Hello! Great forum. I just submitted my feed and of course I have multiple errors (too few tabs). One thing that concerns me is the robots.txt file. Can someone please take a look at my site and see if it will allow web crawlers or if I need to change something. Thanks in advance. http://www.petquarter.com Cherie
Can you tell me how to create one? Step by step, please. By the way, what sort of problems could this cause (not having a robots.txt file)?
wrkalot, I don't think there is a need for a robot.txt file for froogle, you just apply with froogle and they will send an e-mail. You will then have to put together a feed and they will give you a ftp server to send it to. Not complex, but building the feeds manually kind of sucks....
Okay, So if I understand correctly, the robots.txt file is not needed, right? I don't want to do anything that would prevent the spiders from crawling my site. Would you all suggest having one or not? Do your sites have them? Thanks!
I have robots.txt files for some of my sites but they are more for telling certain robots NOT to crawl my site or certain directories within my site (not that they always follow the robots.txt instructions... but it makes me feel better ). They are NOT needed and I wouldn't worry about it if I were you.
Froggle is my #7 overall referrer. And the conversion rate is sky high!! Anyways, robots.txt doesn't tell a robot to do anything. It asks them! robots.txt is not needed for Froogle. Some people like to keep google from spidering their images, but Froogle says that you need to allow google to crawl your images if you want them to show up in Froogle.
Froogle does not require a robot.txt file - only a data feed to be submitted. Apply for an account - wait - create your feed - wait - submit your feed - wait - see listings - wait - re-submit - see improvement etc...