Help needed - robts.txt and sitemap

Interesting tips on sitemaps and robots.txt, thx for the info.

 

i did same but still not index page

click here to check my robot file and site

Thanks

 

a must for your sites. make your you update them each time you add a page. Otherwise indexing issues!

 

Do you have a blogger blog or wordpress one?

 

That file's name is robot.txt. It should be robots.txt. Otherwise SE's won't find it.
Best regards!

 

Thanks for the tips on Robot.txt not Robots.txt

 

Robot.txt file is used for two reasons :
1. To not allow some search engine spiders to crawl your site or some pages of your site that you don't want to get indexed.
2. To save your bandwidth and privacy. Search engine crawlers,keep crawling your website and hence wastes your bandwidth and indexes not useful content also.

A simple robot.txt file can b made just by writing some lines in it. Just put the directory or file names in / to stop crawler to read that file.

User-agent: *
Disallow: /js/
Disallow: /ext/
Disallow: /admin/
Disallow: /css/
Disallow: /includes/
Disallow: /admin/

Sitemap: http://www.yousite.com/sitemap.xml

 

ya one more thing,. the file should be name robots.txt. You can check even facebook for the same thing. Robots.txt is available there ,and robot.txt is not there.

http://www.facebook.com/robots.txt

 

go to free xml maps. that is what i use

 

Sitemap: your sitemap goes here
User-agent: *
Disallow:

This should be a simple robots.txt file that might solve the search engines problem of not having a robots file.
But unless you want to nofollow some directories in your site, robots file is not much of an issue.

 

Understand the simple concept...

You don't want bots to crawl your admin area or area which meant useful to you but not for Google search result like admin pages..
so just to say bots not to crawl and index these pages you use robots.txt
In case if you don't use Robots.txt and use sitemap as said by Mr. Kumar ...is acceptable..
Because even a minor error can make your website go down in Search result.

 

i read somewhere that a good practise for robots files is to simple insert http://www.yoursite/sitemap.xml into your robots.txt file and upload to your server. This is supposed to encourage bots to find your sitemap and index your pages. Can anyone comment on this?

 

Thanks for the sitemap tips, now that i understand

 

Please add this code to your Robots.txt file:

Sitemap: http://URL TO YOUR SITEMAP

Code (markup):

 

I had two sites running without the robot text files for a long time. I received a lot of traffic, BUT when I decided to use a robot on my site I was a little concerned. I found that by using a robot to set up links automatically created much more traffic to my site and sales went up. I bought Zeus for setting up links to boost exposure for my site and establish a better page ranking for my sites. You can obtain good traffic by using good quality information on your site for the search engines to find and receive a good position on the search engines but it takes time that way. DO NOT give up on what you are doing as it will pay off in the end. Your Meta tag information would be considered one of the most important information points for the search engines. Should these not be well set your site will not obtain a good ranking. Most say DO NOT place your meta information in capital letters, my experience says, BULL as capital letters show the word to be very important to the site. You must also make sure these words are also in capital letters on the pages to show there importance. You should also use good key word phrases in the meta tag like, SPUN GLASS ROSE. Here you can see I placed it all in capital letters to show the phrase was very important to that page and the site. DO NOT give up as anything one goes after takes time to complete. I hope this has been of some help for you.

 

This is a very nice tips, thanks

 

so robots .txt not be good nowadays or what

 

Yes it's gooddddddddd

 

I am very new to this technical words, but still I think these are very important

 

Thanks for the info web robots looks like it has the answer I'm looking for.