hi friend, how to create the website robot file and my site some page not robot spider touch how????????.
Creating a robots.txt file will not improve your search engine positioning, but it does provide robots with information concerning which files you will not allow to be crawled and indexed in the search engines. When a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site. But since you asked about how to create that file, then here's how: http://www.webmasters-central.com/wp/se/robotstxt.shtml
Generally these are used for pages for example admin pages which you don't want Google to display when searching for a particular keyword. These maybe something secure, or just something that isn't worth cache-ing by Google.
Thanx for your nice information but i want more information plz suggest for you and my site is yossum.com and i want this page http://www.yossum.com/our_concept/our_concept.php doesn't find google crewl how?????? plz made this page robot.txt file and where upload file.
try to submit sitemap.xml to google webmaster... this will help the googlebot in crawling and indexing your site.
hi i am already submit sitemap.xml to google webmaster tool i these site not me for a example but not know how to create robot.txt file and i want not crewl google robot this page http://www.yossum.com/our_concept/our_concept.php for a example plz send how to create robot file and where upload this file.
check out this link you will find good information about creation of robots.txt file http://www.seoresource.net/robots-metatags.htm
The official website of the Robots.txt "organisation" (?) contains all the information you need. Besides the "disallow" command there is also the "allow" command, which isn't mentioned on the website.
To submit robot.txt in your website route directory Folder. If you webpage are not going crawle by google then submit your xml site map in google.
Thanx for your nice post but robot file only one time route upload/ one by one page in robot.txt file.<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"> only this line page add or any page googe spider doesn't crewl this:--<META NAME="ROBOTS" CONTENT="INDEX, Not FOLLOW"> right /no. plz suggest me.
I wrote a small tutorial about robots.txt some time ago. Pay it a visit, maybe it'll clear all your doubts. Here's the link: Why you should have a robots.txt in your sites | Rauru.com Hope that helps
When doing your robots.txt file remember to include where your XML sitemap is located so search engines other than google can easily locate and crawl it.
robot.txt file is most important for any site. it is text file. it created by webmaster tool,and we must we specified disallow site and save in root directory. and create mata tag like <meta name="robots" content="index, follow" />
What is a Robots.txt file ? Robots.txt is a simple text file without any HTML code in it and is placed in the root directory like http://www.mywebsite.com/robots.txt Purpose of Robots.txt File When ever a web spider or crawler comes to your site for the purpose of crawling then it first of all search out robots.txt. From this file the spider gets this information about which pages of your site to be indexed and which to ignore. Contents of Robots.txt file: As its a text file so it can be created in any text editor. Robots.txt consist of records which contain information for search engines. Each record consist of two fields the user agent line and one or more of the following directives; Disallow Allow $wild card support *wile card support site map location Example of the Pattern: The fields in the record are written in the following pattern User-agent: googlebot Disallow: /support This robots.txt file would allow the "googlebot", which is the search engine spider of Google, to retrieve every page from your site except for files from the "support" directory. All files in the "support" directory will be ignored by googlebot. Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file. From an SEO professional point of view like SEO London its recommended to include robots.txt file with in your website.
If you want to create robots.txt file go to google webmaster tool in sitemaps & crawler access so you can generate robots.txt file