robots.txt?

Hi guys!

I saw a lot of people here that are talking about a certain file called robots.txt...
Why is it necessary?
What should I write there?
Where should I put it?

Thanks,
Vlad

 

It manages how search engines crawl your pages. Google it and you'll find tons of info on format, contents of it an so on, and so forth.

Regards.

 

Vlad,

The robots.txt file can come in handy in many ways, but the main benefit in my opinion is that it allows you to prevent SE spiders from crawling certain pages in your website. By doing so, you can minimize duplicate content and keep unnecessary pages from being indexed.

Reading this site helped me a ton: http://www.robotstxt.org/

 

Robotz.txt tells search engines which files not to crawl.

 

Robotz.txt??? i think it should be robots.txt

Why is it necessary?

To tell Search Engine Bots which files to access

What should I write there?

Mostly like this

User-agent:  *
Disallow:

Code (markup):

Where should I put it?

In the root directory of your domain (i mean website!)

 

http://en.wikipedia.org/wiki/Robots.txt

 

hi

u can must see this site - http://www.robotstxt.org/

 

Hi,

if u have some duplicate pages in your websites you can write them in robot file, search engine will not crawl the specified links

or you can do same thing by no follow attribute in your html file

thanks

 

Also - a nice new feature of the robots.txt is that you can add a link to your sitemap.xml file to help the SEs find all your pages.

 

So, I followed the links that you guys provided and I decided to add this in my robots.txt file:

User-agent: *
Disallow:

Code (markup):

To allow all the bots to crawl my website

sitemap.xml ?
I found www.xml-sitemaps.com which apparently generates a sitemap.xml page, is it ok to save that file in my root directory or do I need to modify it ?

 

It is a set of instructions for bots that visit your site. You need one because bots will be looking for it. You should create one and save it in your root directory.

Try this link http://www.seochat.com/seo-tools/robots-generator/

 

It filters bots from crawling a certain page.

User-agent: robotname
Disallow: /pagename

PHP:

Put it on notepad and upload it in your web FTP server.

 

Check out the post on...

SEO : Help Search Engines Find All Your Pages

...This explains how to add your sitemap xml file to your robots.txt file.