Hello everyone, I have a new site(2 months) and I am new in this webmaster thingy as well. I wouldn't have started this thread but now I think I must becoz I m confused and my problem is not similar to other threads. Things I haven't done yet for my site: 1. Haven't placed the robots.txt file in my web sites root. 2. Have not uploaded any sitemap on googles webmaster tools. (i do have a plugin activated for creating sitemaps but I never used it) Still I use webmaster tools and see no errors except for 1 crawl error (not found 1), I guess but not sure that this error may be reported bcoz of server problem at the time googlebot was crawling my page and thus my archieve page is not found by the bot. Btw, it also tells me that googlebot can crawl my site. I am also using Google Analytics and it shows me all the information properly. Now my question is how come everything is working fine without the robots.txt and the sitemap? My site is found in the search engines too. Like in the first month my site was found in the 5th/7th page in google with the title keywords "Symbian Smartphone". Now it is in ranked 5th place in the first page so that means google is ranking me up as I m adding more content, right? My site is ranked 1st in google for keywords "Boom blox ngage"(a content of my site). Now don't get me wrong thinking that I mistakenly didn't submit sitemap, I still have some pages yet to add and thats the only reason I didn't submit sitemap. I don't know how the sitemap update and pings work so I wanted to submit all my pages at once(dumb/scared me). Please answer the followings: 1. why is everything ok with my site and why should I submit sitemap and get the robots.txt in my root? 2. what changes will it make if i submit and place robots.txt? I know I hv good answers for those 2 questions so please answer these also to help me farther, 3. shud I generate robots.txt file from webmaster tools or make it myself?(if u can please upload the simplest robots.txt for me that will allow bots to crawl all my pages). 4. shud I submit sitemap now or when I am finished adding all the pages? And how does the sitemap updates and pings work actually? 5. Any other suggestions that goes well for my site( included in the sygnature). Thanks all. It seems like as everything is going more fine, I am getting more confused.lol
I have an excellent article on my site with a video for wordpress users that are using the XML Sitemap plugin. It explains all about robot.txt and sitemaps. The video is for wordpress users to set up the plugin. The plugin should run all the time as it builds your sitemap automatically when you post new content. The benefit you will gain is that it will ping after it has rebuilt each time. This is perfect because that is when you want to let the search engines know that you have new content. I usually see the googlebot on my site within 15 minutes of posting a new article. You can view the article at...businessopp4you.com/478/kill-duplicate-content/
Thnx. I am expecting more answers from highly professional webmasters so that I don't make any mistakes with my new site.
Hi Black Mamba, First off, I'm glad everything is going well with your site. Regarding the sitemaps, as a rule of thumb, Google and other search engines will still find your site, even without a sitemap. The sitemap, and Webmaster Tools, are more there for your own use - to see what Google is and is not indexing and why, so you can fix problems with your site. The robots.txt file is simply there to guide the search bots as to what not to index. If you want everything on your site indexed by the search engines, you don't need a robots.txt file.
According to Google Webmaster Central Blog You can follow the link for FAQs about Google Sitemaps: http://googlewebmastercentral.blogspot.com/2008/01/sitemaps-faqs.html Also, Google webmaster tools allow you following tools: * Tools o Crawl rate + How often does Googlebot access my site? + Changing Google's crawl rate o Custom 404 widget + Enhance your custom 404 page + Customizing the 404 widget o Preferred domain - www or non-www + Preferred domain (www or non-www) + Specifying www or non-www + Reverting preferred domain o robots.txt analysis + Checking robots.txt + Creating a robots.txt file + robots.txt analysis results + Syntax not understood o Generating a robots.txt file + About the robots.txt generator + Generating a robots.txt file + Pattern matching + Blocking spammers and scrapers o Other tools + Enhanced image search + Geotargeting For these you don't have to submit sitemap, you can use them by verifying your authority towards the site. You can resubmit your updated sitemap through an HTTP request. Follow this link to know more: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=34609 I have seen websites doing good even without a robots.txt Robots.txt can be used to stop a duplicate page problem, pages that you don't want to show for privacy reason etc... It seems that the spiders will always look for the robots.txt file first and if they don't get, an error message is ensued . Having a robots.txt does stop your error logs being filled with messages Apart from these, nothing you can do in robots.txt to improve your ranking.
if you have no robots.txt means your website is open for crawling. google may still index your pages. submission of sitemap to google is just to control over the situations. like for example if you do not want a particular page to be indexed or do not want a particular robot to be visiting. hope this helps.
Thanks a lot for your help guys. Now I understand what exactly a sitemap and robots.txt file does. I have uploaded my sitemap, do I need to reupload everytime I add new content or pages? Haven't placed robots.txt bcoz I am not sure what pages webmasters usually need to restrict from bots and don't have any idea how to code robots.txt.