Robots.txt

Do you know anything about robots.txt?

 

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.For example, you may not want Google to crawl the images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth."Robots.txt" lets you tell Google just that.

 

This file does help to tell search engine spiders to allow/disallow page which you want to take action. If there are any contents on your website which would harmful for website ranking then you can disallow that page in robots file instead of to remove that page.

 

I Google it for you - http://www.robotstxt.org/robotstxt.html

 

Robots.txt is just allow or disallow the search engine spider to index your website pages, folder, css, images etc by write a simple code in robots.txt. For example if you want to allow search engine your all website than code is-:

For Allow-:
---------------
User-agent: *
Allow: /

If you want to disallow search engine to index certain page then code is

User-agent: *
Disallow: /services
Disallow: /services.php

Or your want to disallow folder then code is -:

User-agent: *
Disallow: /search
Disallow: /catalogues

 

txt file gives the instruction to goolge crawler to not indexed some specific data
take the help from here.
www.robotstxt.org/robotstxt.html

 

robots.txt is a text you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

 

Robots.txt is used for tell to crawler to index the pages of your sites or keep away some pages from indexing the sites.

for eg.


User-agent: *
Allow:/

Or

User-agent: *
Disallow: /abc.html

 

what does robots.txt do?

 

Robots.txt file in root directory ,helping to securing purpose.you have saying to search engines ,follow this files and folders etc.

 

It's especially good, if you have a print preview for each of your sites. So to avoid dublicate contant you can determine with the robots.txt, that all the print previews shouldn't be crawled.

 

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.

 

Update the coding of robot.txt in your website root directory...
It will let google not crawl the content on ur website which you dnt want it to do...
update it if its very neccesary...

 

Hello

Through robot.txt file website owners gives instructions to web robots about that website. This file serves as a request that specified robots ignore specified files or directories when crawling a site.