HELP - Creating and Using a robots.txt File For Crawlers Daily

Oaky DP - Here sThe Issue With More Relevant Web Crawlers, Google, Yahoo, MSN - If anyone can admit it or denie it im all EARS - until then - do really need a robot text file on evvery domain site website??? ((( Creating and Using a robots.txt File For Crawlers Daily)))) Is this really helpful??

if so how easy is it to do ... kudos for help thanks just need more detailed instructions

 

Well, can you explain what you want to do??

 

by creating robot.txt file you never call Crawler daily. robot file is used to specify whether crawler need to crawl your site or don't

 

since youre the Open seo - Im asking for help creating the file - CAN YOU HELP???

 

Hi, You can easily create a robots.txt file from google webmaster tool.

 

thanks but realy is that helpful .... just wanted a tutorial maybe :0

 

There is no requirement that you have robots.txt file on your site. You really only need one if you want to keep crawlers from indexing certain portions of your site.

The only other reason you would possibly need a robots.txt is to specify the location where the crawler can find your sitemap.xml. Of course, sitemap.xml files are also not needed unless:

1) your site has crawlability issues due to the way its coded preventing search engines from discovering your URLs naturally when crawling the site or
2) your site has hundreds of thousands or millions of URLs and you want to prioritize the pages to indicate the order in which you'd like them indexed using the <priority> element.

Other than those reasons (or trying to get a web site with no back links indexed which is itself totally worthless), sitemap.xml files are worthless.

 

Okay - nice really nice - thank you very much --------- in sure free automated classified poster can help

 

Hi,

Robots.txt is a simple text file that sits on the server with your web site. It's basically your web site's way of giving instructions to search engines about what how they index your web site.

Once you've written your robots.txt file to reflect the directions you want to give the search engines, you simply save the text file as "robots.txt" and upload it to the root folder of your web site.

 

pm me - the details - PLEASE - im looking for that kind of HELP --- kudos and green reps --- please pm me the sample of that one okay

 

well when google search engine crawl your website it looks for robots.txt where it is defined that which pages to be crawl by search engine that are relevant for the site and which pages to ignore such as images etc pages which are ignore by search engine.

 

indiamart Pm me the full details OKAY