Do you use Robots.txt File? Here is the Importance.

Hi, Not much of the webmasters are known to the role of Robots.txt. SO here is my try, please also share your views about this one.
Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through.

The robots.txt file which can be created simply in a notepad can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats and the files that you are going to provide the visitors to download after making the payment etc.

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content, for example you might have also uploaded those files which are in progress and not fully developed and completed. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.
What are your ideas about this file?

Feedback Appreciated!!

 

It is very important especially to disallow something,a file or file extension.

 

this is very important to our sites but i didn't use to our sites yet.

 

On all of our sites, we use the robots.txt file. It keeps the SEs from indexing the junk on your sites.

 

Yah but some SE can still index the files robot.txt tells you not to.
Most of are like hacking spiders tho...lol

 

I don't think I have ever used a Robots.txt file.

 

Really?!?! I have always considered it a must, it is one of the first steps I take on any on site optimization. It doesn't rank a site by itself but I consider it still is a good thing to have.

 

Google by default caches pages, so meta name="googlebot" content="noarchive"

* NOINDEX tag tells Google not to index a specific page
* NOFOLLOW tag tells Google not to follow the links on a specific page
* NOARCHIVE tag tells Google not to store a cached copy of your page
* NOSNIPPET tag tells Google not to show a snippet (description) under your Google listing, it will also not show a cached link in the search results

Although thats not really SEO, most SEO would want google to crawl over everything!

If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt

Google also rely on the following two robots, deepbot and freshbot.. so you can "deepbot" content="noarchive" etc.

 

Hello,
Where do I place this code in a wordpress blog?

Thanks

 

thanks for sharing this info. i am completely new to robots.txt. i will create a robots.txt file for my site.

 

NOARCHIVE tag tells Google not to store a cached copy of your page

When/why would you not want Google to store a cached copy?

Thanks,
Zeek

 

You can also direct search engines to your sitemap with a robots.txt file by placing Sitemap: 'sitemap link' in it.

 

Thanks buddy for providing such information. Actually, I don't know about robots.txt.
But, after reading your thread i get idea about robots.txt and its importance.

 

We need it in that case, when we plan to change the image of our website and its graphic very frequently.

 

Of course the robots.txt is very important for us,if you do not want the search engine to crawl your page,you need it.

 

Hi Rogi
Can we tell from a site's "view source code" if the file is attached?
Does this file help in optimising in some way as well?

Regards

 

I was totally unaware about this,thanks for showing importance i will try it!

 

I agree with everyone, robot.txt file plays an important role in SEO if you want to keep the spiders from finding pages that aren't really related to your site, but to my understanding having one that doesn't disallow anything helps your SE rankings a little too.

 

oh my.. i m shocked to see how many people are not aware of robots.txt. I suppose every matured and experienced webmaster do use robots.txt in the times of need. If your website having any data that you don't want to be get listed like user information. Robots.txt help you to prevent SE bot to crawl such pages.

 

robots.txt is a file that you use to define your targeted users. you mention those bots as nofollow in this file to whom you don't want to show your site.