Robots.txt

I've never used robots.txt and wonder if i'm missing out on some seo trick. Does anyone else use it? Have you noticed if its helped with google or other search engines?

 

It helps if you want pages to not be spidered.

 

I use it to filter out redirect links...

 

It can, By it's very nature, Not help at all for SEO in any way.

 

Hi!
Nop, no SEO trick....

Best
Jakomo

 

Not all the bots follow the same rules.

 

it doesn't help but having a blank one can help if the search engines are being a bit pissey so they receive a 200 status rather than a 404 when they request it (and they request it practically everytime they come to your site).

So apart from that ever so minor there's no point doing it tip, it's used solely to stop the search engines from going to parts of your site

 

I have directories on my site that are not meant for web users but they are not linked from anywhere on my site so search engines dont index them. So i dont think i need them.

Dudibob, I will take up your advice and add a blank one to avoid the 404 errors!