Snippet 1: added my first google site map and have seen 2 new bots (Gigabot and some Java thingy - helping to confirm the side effects of sitemaps...oh, make that 3, I just spotted another one go by as I am typing -- yes, watching your logs in realtime is a little geeky but really worth it -- hello Slurp! -- hang on, make that 4, some Polish bot... Snippet 2: Firefox extension "Search Status" v1.9 - really useful. Allows you to add Pagerank and Alexa Rank to your browser (in a choice of places) - instantly visible for any site you are on. Bonus: color code "no-follow" links -- quite an eye opener! Example: I can see that the seach link in the DP toolbar and the contact us at the foot of this page are no follow links... nice for doing investigative research. https://addons.mozilla.org/extensions/moreinfo.php?id=321 HTH Steve
No surprise there. It's a straightforward format that Google's made public, you can bet that other bots are looking for it to figure out which pages to spider.
arjunprabhuc - I noticed a few postings about "I posted a sitemap and Bot X stopped by" and I saw a similar thing happen and I thought it would be helpful to mention it so others can have the information to decide whether sitemaps is worth the effort... to me it has been as traffic seems to have picked up a bit too.
Actually when you run googles python script it says "notifying search engines" at the end. I only see it send data to google but perhaps google pings other sources?
I'd really like to see Google's XML sitemaps become a "standard" for use by all search engines. There's nothing proprietry in there (like api codes) so no reason for the others not to pick it up. Finding them is another matter. Why, though, is this posted under Adsense?
It's an xml file listing each of the pages on your site which allows google to get a clear picture of the site and of changes without having to check each page for new content and draining your bandwidth. You don't get priority treatment but you do ensure that Google reaches into the farthest corners of your site. [more]
I did the sitemap thing, took a long time for it to create it...... sent it to googles sitemap submission.... nothing ! no bots visited, no SER changes etc. seemed to do nothing visible anyway for my site(s)
Go back to Google and check the status of the sitemap. You may have errors in it. & maybe post the link to the map here for a review.
So far sitemap has been an interesting exercise for me. I created a sitemap and omitted as many duplicative or useless type urls, like images, popup windows, etc. and still ended up with over 200K urls in the list, even though my site only has a couple thousand pages. Anyway, I uploaded the sitemap and have since verified that G is looking at it and it has no errors. My indexed page count has subsquently gone DOWN by nearly 100 pages. This is not good. Also, I haven't noticed an increase in traffic from G or any other bot, but then they show up often enough to keep on top of things anyway so that's not really a concern. What I really wanted was for all those pages I have to be indexed, or at least more of them. Now I've gone from well over 600 to well under 600 on a site with over 2000.
thats weird... on a site with a 3k alexa rank it takes less then 3 mins to run and generate/prioritize 700K+ urls. I run it nightly and have had a lot of success with it. I highly recomend you give it another go. Make sure you go through the process of verifying and other stuff.
Ok, now I'm impressed. I haven't submitted a new sitemap yet, but the results fromt that last one have changed. It had gone down to under 600 pages indexed, but I just checked and now it's up over 9,000! Sweet!
I have a traditional site map (IE html version) on my sites. Should I link the google xml one one the homepage or just leave it in the directory for just google?