Hi, Im looking for a tool that will spider my like (like google) and create me a database of all pages spidered with page name/title/descriptions etc ? Any suggestions.? My site is of some large size so manually trying to index all of my pages is a pain in the a** .... ? Would be great if i dump everything into a database of export the data out as csv or xml ? Thanks
http://tinyurl.com/g88al Google Search link for Google Sitemaps Generator. Google sitemaps is a service that helps Google index your site better if you manually upload a sitemap made through those generators at Google Sitemaps. If you upload the sitemap to your website made by those generators and signup at Google sitemaps with your site and link the program to the sitemap, Google will try and index the pages in that sitemap as fast as possible. IT
google sitemap will just list your urls but inorder to list title and descriptions you need to try some other tool or add it urself by editing the sitemap.. if u knw xml u can create your own tags for title description and urls n then submit to google..
You could consider creating a RSS sitemap. That would contain titles etc. You can also upload that to Google.
i agree with Richie Ni... i have made good use of the free tool he links to on my websites and the files you get are easily uploaded to you website and accepted by Google.
or you can search on forum . you can find many threads like http://forums.digitalpoint.com/showthread.php?t=150528
try my free online sitemap generator, which can make a standard sitemap file for almost all major search engines, including google.