Robots.txt File Format is Right or wrong?

Hello Frinds:

This is my website.. http://www.ibazaarindia.com/robots.txt Please check and let me know this robots.txt file format is right or wrong?

Pls suggest me if have any problem..


Thanks

 

It seems a bit bloated, but I can't see anything major wrong with it - unless being that bloated is a problem. I would suggest going to Google's Webmaster Tools and where you can test your robots file.

 

you could also try generating it

 

Are you facing any problem with that robot file? If yes then what problem are you facing? I have checked your file and I think its ok.

 

have you tried a check on the file to check validation? my webhost does that

 

It's too long and ugly. Use wildcards instead of listing all those entries individually.

 

Instead of making a useless and meaningless post, why don't you provide useful help and explain what the problem is?

@yogeshseo, the robots.txt is long, but I assume you have all in there that you want. You do not need:

Allow: /

Code (markup):

Not all robots are equal, so best remove it to avoid any of them getting confused and crawling areas you don't want them to.

You might want to make use of the robots meta tag for pages you're wanting to specifically exclude from being crawled.

However, I noticed some of these pages you're blocking with robots.txt don't exist any more (I didn't check all of them). This is wrong really. Instead of blocking them, you should redirect to an alternative product, category or your home using .htaccess and a 301. I'm guessing you used robots.txt to block these URLs due to crawl errors. The cleaner way is the 301 redirect, plus any backlinks you have to these pages are then not lost and still directing traffic to other relevant pages (or your home page).

Good luck.

 

Looks okay but too many "disallow" /products/ and that makes it bloated. Wouldn't it be easier to simply write one line:

Disallow: /products

Code (markup):