Robots.txt?

[rockinaway]

well my site is located at http://www.etcworld.co.uk

and well i don't know whether to have this file.. I have a forum which has imporatant folder but i dont know which ones to block and which ones not too, and my sie is also online open-source.. so what should I do?

[BILZ]

the only need for it is to tell the SEs not to visit a specific area or page. If you have areas you dont want the SEs to goto than use it. Otherwise, spend your time on something else more useful.

[rockinaway]

so really no point?.. what kind of places wouldn't I want it to visit?

[BILZ]

I've never really needed it. I usually just put stuff in a password secured directory if i dont want the SE to find it.

But some people find it a useful tool.

[ServerUnion]

nice if you dont want login screens or non-content related pages indexed. Even if I do not use the function I still place an empty robots.txt file, this way I dont see 404 errors in the logs when the search engines come by.

[rockinaway]

so would i block places like /admin/

[ServerUnion]

so would i block places like /admin/

Since we aren't really sure about your site, we cant say. Just think if you really want the search engines to read that content, or should users of the search engine be able to view it from searches. good luck

[rockinaway]

ok thanks..