Google Sitemaps

[digitalpoint]

Google is letting people make XML feeds of sitemaps to make crawling faster/easier. :)

https://www.google.com/webmasters/sitemaps/

[sji2671]

I tend to find if I go down on my knees crawling is much faster/easier.
[That was just childish n should be deleted,sorry!]

On any other forum........you would not even read .This,lol

[noppid]

I tend to find if I go down on my knees crawling is much faster/easier.
[That was just childish n should be deleted,sorry!]

On any other forum........you would not even read .This,lol

Are you home alone? :p lmao

I looked at this, I must be in a funk. I can't wrap my head around nuttin'!

[sji2671]

I guess its a different type of site that would need that much updating with regular news etc my site gets google raping it all over the place as is, its like watching something you shouldn't!

However for other sites I can see the good in it, were working towards more streamlined feeds, info is power, 2nd place is no place, and if its just for their benefit then I suppose were all willing :-)

[piniyini]

Seems a tad bit difficult to set up dont ya think? Well not for the digitalpoint man of course

[tflight]

I've got my first sitemap submitted for a simple, static site. Time to start bringing out the heavy tools for my large, dynamic sites.

http://www.timflight.com/2005/06/02/submit-google-sitemaps-xml/

[Blogmaster]

Seems a tad bit difficult to set up dont ya think? Well not for the digitalpoint man of course
It seems that way but since it comes from Google, it should be worth spending some time on it :)

[Dominic]

This is a great idea to save goog money sending it's spiders out, should also decrease the google spider visits to only when content changes.

[tflight]

I submitted my sitemap.xml and the status now shows "OK" with a downloaded time of 2 hours ago. Now to see how my experiments turned out the next time Googlebot comes around. ;)

[Blogmaster]

Sitemaps are very important. I would take this very serious. I have passed it on to our coders right away :)

[ResaleBroker]

Ok, I've submitted my XML feed.

[60 seconds later]

Are we there yet? :p

[Infiniterb]

Crap. I take it I'll need to be able to telnet into my server in order to run the python script. If so, I don't think I have telnet access.

[stephenmunday]

Great heads-up!

All that codie stuff is right over my head, but hopefully someone here will be able to help me out with that....

[ResaleBroker]

If you need help go ahead and post and I'll do my best to help. ;)

[Infiniterb]

NVM my first post. I got it configured right and a site map was created based on my access logs. Very slick IMO and was easy to setup.

[Dziku]

OK, php script for creating sitemap created, sitemap generated and submitted.... Let's wait and see what's gonna happend....

[Infiniterb]

OK, php script for creating sitemap created, sitemap generated and submitted.... Let's wait and see what's gonna happend....

Mind sharing that script? :)

[dyn4mik3]

Going to sleep now, but just wanted to tell everyone I wrote a script to generate a Google Sitemap for Wordpress sites. If anyone wants to take it and improve on it please do. Drop me an email if you do, I'd like a copy :)

Grab it here:
Wordpress script (http://www.socialpatterns.com/search-engine-optimization/google-sitemaps-with-wordpress/)

If you need an overview of the Sitemap protocol, you might want to read this (http://www.socialpatterns.com/search-engine-optimization/breaking-down-google-sitemaps-xml/).

[SEbasic]

Nice.... Thank you. :)

[Blogmaster]

Indeed ... thanks a lot :)

[piniyini]

That was a great read thanks.

How are we to know when the content was last modified?

[ResaleBroker]

Going to sleep now, but just wanted to tell everyone I wrote a script to generate a Google Sitemap for Wordpress sites. If anyone wants to take it and improve on it please do. Drop me an email if you do, I'd like a copy :)

When I followed the link from "Just copy this code into a php file" I received:

Invalid at the top level of the document. Error processing resource 'http://www.socialpatterns.com/wp-content/googlesitemap...

<?php echo '<?xml version="1.0" encoding="UTF-8"?>'; ?>--------------------------------------------------^

[MobileP]

Is this also possible to do with blogs rss?

[Dziku]

Mind sharing that script? :)
NP.

$connection=mysql_connect($baza_adres,$baza_login,$baza_pass);
$db = mysql_select_db($baza_baza);

$plik=fopen("sitemap.xml","w");
flock($plik,2);

fputs($plik,"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n");
fputs($plik,"<urlset xmlns=\"http://www.google.com/schemas/sitemap/0.84\">\n");

$w1=mysql_query("SELECT * FROM artykuly WHERE status='1' ORDER BY 'czas' DESC") or die ("Invalid query 1");
while($r1 = mysql_fetch_array($w1))
{
$skrot=$r1['skrot'];

$linia="<url>\n";
$linia.="<loc>http://www.domain.com/index.php?art=$skrot</loc>\n";
$linia.="<changefreq>daily</changefreq>\n";
$linia.="</url>\n";
fputs($plik,$linia);
}
fputs($plik,"</urlset>\n");
flock($plik,3);
fclose($plik);

mysql_close($connection);

Of course it's just an example...

[BigPhil]

Do you have to install python first? I can't find any documentation in installing and running the script. Some pointers maybe?

[pcdoc]

Yeah, I've been reading the G sitemap docs for the past 45 min, got real excited until realized that my sorry virtual host (godaddy) for this particular site of mine doesn't allow cron jobs, so no python runs.

You ask, why did I chose godaddy for this site?

An incentive from godaddy for all my URL registrations.

Stupid me, and starting day before yesterday, godaddy had a major slowdown on their linux virtual hosting servers... lasted 36 hours.

Here I go, blathering on about godaddy, sorry for trying to hijack this thread.

[boohlick]

I've just submitted my site map.... 30 mins ago

[caroline]

Oh, that's going to come in handy. Thanks!

[kalius]

If the server doesn't have python you could run it in your local computer, just need the url's on the text file. ( I think I'm right on this ones).

I think a good way is to create a script to crawl your site and dump all the urls in the text file, then run the python script to create the sitemap fron the text file.

Is there a crawler that does this already?

[kalius]

I wish I was better at php I just had an idea (maybe a nice php programmer around here will do it and release it):

Modify the create new tread in VB/ PHPBB/ etc, when a new tread is created automatically add it to the .txt file with the url list (posibly add the link at the top).

then run the python script via a cron job so G can easily get new treads...

[BigPhil]

Anyone's site map downloaded yet?
I;m still on "Submitted 2 hours ago"
:)

[ResaleBroker]

Mine was submitted eleven (11) hours ago and downloaded ten (10) hours ago.

[Simon Donkers]

Mine is submitted 54 minutes ago and downloaded 44 minutes ago. So I don't see any problems with google at the moment. What I did see at one point was a status page telling me that my page was submitted half an hour ago and download was pending. And as my current results show that the page was downloaded 10 minutes after submittion that tells me the stat page isn't instant but will be updated every so often. It works for me anyway.
Else there is also the resubmit button which you could try to get Google there attention.
Now on to make one for my other websites.

[pcdoc]

I manually submitted a list or URLs one hour ago, which was just downloaded by google.

Here is how

1. Using a text editor, create a file with your URLs, one URL per line, save it to disk with a name of your choice; e.g,

mysite.urls

contents of mysite.urls will look something like this;

http://www.mysite.com/index.html
http://www.mysite.com/links.html
http://www.mysite.com/contact.html
.
.
.
.

2. ftp this file to the root of your server; e.g.,

http://www.mysite.com/mysite.urls

3) copy / paste the file's path into the address line of your browser to make sure the url is accessible (no robots.txt blockage, permission problems, etc.)

4) add the url of the sitemap by entering it into the textbox at

http://www.google.com/webmasters/sitemaps/showaddsitemap

wait for approval.

[Infiniterb]

I manually submitted a list or URLs one hour ago, which was just downloaded by google.

Here is how

1. Using a text editor, create a file with your URLs, one URL per line, save it to disk with a name of your choice; e.g,

mysite.urls

contents of mysite.urls will look something like this;

http://www.mysite.com/index.html
http://www.mysite.com/links.html
http://www.mysite.com/contact.html
.
.
.
.

2. ftp this file to the root of your server; e.g.,

http://www.mysite.com/mysite.urls

3) copy / paste the file's path into the address line of your browser to make sure the url is accessible (no robots.txt blockage, permission problems, etc.)

4) add the url of the sitemap by entering it into the textbox at

http://www.google.com/webmasters/sitemaps/showaddsitemap

wait for approval.

That's fairly easy to do for small sites, but when you have a few thousand dynamically created urls, that's not too feasable. Nonetheless, thanks for the tip. :)

[pcdoc]

"fairly easy to do for small sites"

true, but it works for now, since I can't run a cron job with my current (godaddy) host.

but, it did serve to upload a goodly number of page URLs.

[jlerner]

Great info pcdoc - thanks. It must be pretty popular... I keep trying to get to that page, but I keep getting a 502 server error. Bet they're getting swamped.

[kalius]

slashdotted google

[Infiniterb]

Oh, they've been slashdotted, digitalpointed, wired, you name it, everyone's going to have a story/discussion about it.

[pcdoc]

Heee, heee, heee, you saw it here (nearly) first.

[john1970]

Any indication that submitting an xml sitemap will result in a faster indexing of a new site?

[TwisterMc]

I manually submitted a list or URLs one hour ago, which was just downloaded by google.

Here is how

1. Using a text editor, create a file with your URLs, one URL per line, save it to disk with a name of your choice; e.g,

mysite.urls

contents of mysite.urls will look something like this;

http://www.mysite.com/index.html
http://www.mysite.com/links.html
http://www.mysite.com/contact.html
.
.
.
.

2. ftp this file to the root of your server; e.g.,

http://www.mysite.com/mysite.urls

3) copy / paste the file's path into the address line of your browser to make sure the url is accessible (no robots.txt blockage, permission problems, etc.)

4) add the url of the sitemap by entering it into the textbox at

http://www.google.com/webmasters/sitemaps/showaddsitemap

wait for approval.

Thanks for that! I missed that simple way of doing things. :D

[Infiniterb]

Any indication that submitting an xml sitemap will result in a faster indexing of a new site?

I can't say with any certainty that submitting my sitemap yielded Googlebot's recent crawl today, but Google has been on my site all morning/afternoon.

[exam]

That's fairly easy to do for small sites, but when you have a few thousand dynamically created urls, that's not too feasable. Nonetheless, thanks for the tip. :)I'd prolly be pretty easy to make your dynamic page generation update your mysite.urls file with all the dynamic urls.....

[Infiniterb]

I'm sure it'd probably would be. I'd love to see one for vbulletin sites. I'm not a programmer, however. :)

[minstrel]

Any indication that submitting an xml sitemap will result in a faster indexing of a new site?
I can't say with any certainty that submitting my sitemap yielded Googlebot's recent crawl today, but Google has been on my site all morning/afternoon.
I was wondering the same thing, John.

Googlebot crawls my forum site several hundred times a week -- only slightly less often for my main site -- do I need really need any more attention than that?

My guess is this might be a way to increase crawling of sites where it isn't already happening, like new sites... where it is, I doubt there's a benefit. Maybe think of it as a "Submit" button for new sites.

[Infiniterb]

I was wondering the same thing, John.

Googlebot crawls my forum site several hundred times a week -- only slightly less often for my main site -- do I need really need any more attention than that?

My guess is this might be a way to increase crawling of sites where it isn't already happening, like new sites... where it is, I doubt there's a benefit. Maybe think of it as a "Submit" button for new sites.

For some reason Google doesn't seem too keen on crawling my site all that often (and it's fairly well established going on 4 years now). I submitted my sitemap, and bam, seemed almost like an instant crawl. Again, though, I can't comment one way or another that this was a direct result, so take that for what it's worth.

[iMacFlats]

I have the ultimate test... I have a site that I put online on May 27th. I submitted it to Google, but no results as of yet. Other than a quick "look and see" with a request for a robots.txt file, Googlebot has not been back.

Let's see if it gets sucked into the ranks or gets ignored.

[NewComputer]

I cannot seem to get the above set of instructions to work. Any idea?

[NewComputer]

Sorry guys, I lied....

I created the file and made the extension .html...

Here is my experience:

As some of you may know, I have recently updated my computer repairs (http://www.newcomputer.ca) website . I actually completed the overhaul yesterday. Obviously, Google was not visiting the new urls because I changed all the extensions and how the domains read (I moved from _ to - and .htm to .html + I added more SE friendly names.)

Anyways, longer story long, since I submitted the new 'sitemap', Google has indexed every single one of my new pages as a result.

This is purely coincidental that they release this tool with my overhaul release, but this is the proof it works...

A google search for "Contact NewComputer.ca using the below information" which is text never used on any of my other pages revealed the following:
Google Search (http://www.google.ca/search?sourceid=mozclient&num=50&scoring=d&ie=utf-8&oe=utf-8&q=Contact+NewComputer%2Eca+using+the+below+information)

[Infiniterb]

Are you saying that you were indexed because of your new site map you submitted using their new site map tool within the last 30 minutes?

[iconv]

Hi all, I just finished putting up an online interface to generate a Google xml sitemap from a simple ls -lR directory listing, you can give it a spin at Create Google xml sitemap from directory listing (http://www.iconv.com/ls2sitemap.htm).

[aspcoder]

oops...i learned some python in last 3 hours with google provided python files to create sitemap

[NewComputer]

Are you saying that you were indexed because of your new site map you submitted using their new site map tool within the last 30 minutes?

It took a total of 5 hours (from submittal to checking each pages cached copy), but yes, that is what I am saying...

Check for yourself:

http://216.239.63.104/search?q=cache:http%3A//www.newcomputer.ca/site-map.html

Look at that date, I just finished that page yesterday... I had an old site map with a different extension and different name.

[iMacFlats]

Okay! Here is the scoop...

I submitted a site that is not even in Google index or anything. I originally submitted it a week or two ago, and although the GoogleBot basically did a "ping" of the main site, it didn't do anything with it. It hasn't been back.

I submitted my XML sitemap last night and google, very quickly, did a "ping" of the sitemap file and that was it. It took 3-5 hours before it showed up that it was "OK". Google has yet to make any attempt to index the site looking at the logs.

So... submitting a sitemap does not mean instant "indexing" of your site. At least, on my test case, it didn't.

[minstrel]

So... submitting a sitemap does not mean instant "indexing" of your site.
Indeed it would be hard to imagine that it would be otherwise. It's not likely to be like a dog whistle that will instantly summon googlebot to do your bidding.

As I think I said earlier, this may be useful for new sites but it's probably best to think of it as a newer better submit button.

[ian_ok]

So if you do the TEXT file version, how do you inform GOOGLE if you change a pages content.

I guess if you add a new page you just add this to the text file.

Ian

[kalius]

I'm sure it'd probably would be. I'd love to see one for vbulletin sites. I'm not a programmer, however. :)

vbsitemap for google (http://forums.digitalpoint.com/showthread.php?t=17798)

Working on it

[NewComputer]

So... submitting a sitemap does not mean instant "indexing" of your site. At least, on my test case, it didn't.

In my case, it was most likely that my site was indexed previously, and as such, the new sitemap just meant Google needed to comeback for new content, which they have...

[minstrel]

I guess, other than the possible "submit" function for new sites, I'm having some difficulty understanding how this is any better than any other site map linked from your home page.

[getsum80]

Could I do this on my amazon store www.lagunasurfshop.com ?

[kalius]

Could I do this on my amazon store www.lagunasurfshop.com ?

probably but I recommend that you first remove session ID's.

Do you do an amazon call for each page or all the data is in a database?

[ziandra]

I did the simple version, just the sitemap.txt with one URL per line.

Google hit it in about 2 hrs. Then with 8 hrs every single page in that file was hit.

I am currently working on a script to automatically generate a sitemap.xml for google and a sitemap.php for everyone else. Seems silly to go to all the hassle of selecting which files you want in the sitemap and which ones you don't for only one purpose.

[iMacFlats]

Well, interesting...

Yahoo! Slurp raped my site today. A site I put online a week or two ago, and I ONLY submitted via normal Google submit site form and 24 hours ago I did the sitemap thingy. Interestingly, GoogleBot did check my sitemap this evening again, about 24 hours after my first submission. No Google rapes yet though.

[lingeriediva]

Interesting possibility for those of us who use Blogger. It looks like the Atom RSS feed can be submitted...Which solves the update issue.

[Roze]

This is great - submitted two simple txt lists, I manually generated all 3,000 forum archive URLs that look like this:

www.site.com/forums/archives/index.php/t-1.html

Using MS EXCEL. I cut and pasted from the spreadsheet into Dreamweaver and used find and replace to quickly put it in the right format! It was very quick and easy...They're downloaded, lets see what happens!

[jlawrence]

For those of us with big sites, I'm looking at a few linkchecker type scripts. The idea is to simply have the script spew a list of all url's in your site into a text file from which you can just create the sitemap.

If anyone knows of a script that can do this 'out of the box' then it could be useful. At the moment, I'm trying to pursuade Gbot to crawl a new 30K page site - it'll be an interesting test for this sitemap stuff. I'd normally expect Gbot to take a good couple of months to crawl the site.

[iconv]

Just finished setting up an online interface that will generate a Google xml sitemap from a ls -lR directory listing, handy to create a sitemap from your local file system, and then upload that to your web server. Create Google xml sitemap from directory listing (http://www.iconv.com/ls2sitemap.htm)

[Boston_JM]

i used that tool but dont see anything besides this

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
</urlset>

[runnerunner]

how soon would you say the pages get indexed after being visited by google?

[seoeffect]

I also had that error on the script :

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
</urlset>

The site I submitted in the end was www.ukmp3.co.uk

This site had 600 pages until May 2005 when more products were added in, making 1,800 pages. There was also a test shopping area added ( affiliate driven ) which left us with over 50,000 pages.

We used Xenu to grab all the hyperlinks and then AddrView to extract the urls in to a .txt file ( 16MB ). This was "fed" to Google.

The file was loaded in to Google after about 2 hours and processed fine.

Looking at the pages present in Google this has already jumped from 800 to 9,100 so all seems to be working.

I was wondering what was stopping someone taking a sitemap, filling it full of 404 pages ( which Google wont like ) and submitting it !


There are many scripts on Hotscripts.com in php for site mapping. best one so far is : http://www.hotscripts.com/Detailed/32948.html and its free. If there was a way of making the site map and creating the xml that would be great.

Maybe Google should make one if they want webmaster to go this way .....
:eek:

[passat]

hmm...how will it look like? I don't want to manually build the xml file.

[boohlick]

I've submitted my site map ... but hey errors occurred..

:confused: PARSING ERRORS!!!!

Ive read the errors.. and followed some of it that cause the error..

But nothing happen

[NewComputer]

what is your site map page?

[boohlick]

http://www.proadultoutsourcing.com/outsourcing-solutions/site-map.html

[tresman]

boohlick ,

You have to send and xml file that has to be placed on the root of the domain. You failed on both: its a html sitemap and is not in the root. Also, the name has to be "sitemap", not "site-map".

Your sitemap has look similar to as it follows:


<?xml version=“1.0” encoding=“UTF-8”?>
<urlset xmlns=“http://www.google.com/schemas/sitemap/0.84”>

<url>
<loc>WWW.PUT-YOUR-MAIN-URL-HERE.COM</loc>
<priority>1.0</priority>
</url>

<!-- delete this comment: repeat the 3 following lines but changing the url for every url of your site -->
<url>
<loc>ANOTHER URL </loc>
</url>

</urlset>



Now do manuallly or code a script that does the sitemap for you, but this is how it has to be.

For the techies reading this: yes, you can specify many other things, change the priority, etc... but this would ok.

[Cheater]

Actually those rules are only formalities.
You can have http://www.domain.com/your/site/map/here.xml.gz if you wanted.
(It does have to be XML)

Google will get it anyway. :)

[BigPhil]

The google sitemaps really work. Got 236 more pages indexed by google :) :)

[boohlick]

boohlick ,

You have to send and xml file that has to be placed on the root of the domain. You failed on both: its a html sitemap and is not in the root. Also, the name has to be "sitemap", not "site-map".

Your sitemap has look similar to as it follows:


<?xml version=“1.0” encoding=“UTF-8”?>
<urlset xmlns=“http://www.google.com/schemas/sitemap/0.84”>

<url>
<loc>WWW.PUT-YOUR-MAIN-URL-HERE.COM</loc>
<priority>1.0</priority>
</url>

<!-- delete this comment: repeat the 3 following lines but changing the url for every url of your site -->
<url>
<loc>ANOTHER URL </loc>
</url>

</urlset>



Now do manuallly or code a script that does the sitemap for you, but this is how it has to be.

For the techies reading this: yes, you can specify many other things, change the priority, etc... but this would ok.


Thanks bro!! :)

[Liminal]

hi all,

sorry if this has been covered before. what benefit does submitting my sitemap xml to google have if i know for a fact taht all of my pages are crawled and are being indexed by google after they are posted?

[NewComputer]

Dev,

It is fully covered in this thread. Some of us have seen an immediate update of new pages added. Some have redesigned sites and pages and had them indexed right away. Some have seen nothing at all.

[Liminal]

newcomp, i see thanks. (there are 9 pages in this thread so i did not have time to read it all)

[ian_ok]

These are the number of pages G is showing:

www.sanlucar-de-barrameda
Was 13 now up to 39
All mainly cached BEFORE I did the G sitemap and has only just shown up on G.

www.costaysierra.com
Was 0 still 0
Can't say anything!!!

www.costaandsierra.com
Was 10-15 now at 42 pages cached
The extra pages seem to be result of the G sitemap

www.casaelbosque.com
All about the same.

Ian

[TwisterMc]

This is good news to me as I just converted my blogger blog to WordPress. Now i'll just tell Google all the new URLs and they'll be happy. Now why doesn't Yahoo do this?

[minstrel]

sorry if this has been covered before. what benefit does submitting my sitemap xml to google have if i know for a fact taht all of my pages are crawled and are being indexed by google after they are posted?
Probably none, AFAICT

[boohlick]

Is there something wrong with my code
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns=http://www.google.com/schemas/sitemap/0.84>

<url>
<loc>http://www.proadultoutsourcing.com/</loc>
<priority>1.0</priority>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/partners.html </loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/solutions.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/contact.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/webmaster.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/tgp2.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/banners2.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/banners2.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/photoeditor.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/photos.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/webdesigner.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/paysites.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/tgp1.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/mgp.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/tgpsites.html</loc>
</url><url>
<loc>http://www.proadultoutsourcing.com/portfolio/avs.html</loc>
</url><url>
<loc>http://www.proadultoutsourcing.com/portfolio/fhg.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/portfolio/freesites.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/videoeditor.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/webdeveloper.html</loc>
</url><url>
<loc>http://www.proadultoutsourcing.com/seo.html </loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/condition.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/Policy.html</loc>
</url>
<url>
<loc>http://www.proadultoutsourcing.com/outsourcing-solutions/site-map.html</loc>
</url>





</urlset>
this the error
An invalid character was found in text content. Error processing resource 'http://www.proadultoutsourcing.com/sitemap.xml'....

<urlset xmlns=



It just a simple site map made in xml..
Any suggestion can do

this the URL :http://www.proadultoutsourcing.com/sitemap.xml

[ikeys]

maybe it is the spaces behind .html
partners.html </loc>
</url>

[boohlick]

maybe it is the spaces behind .html
partners.html </loc>
</url>

Is it? are you sure? nway I'll try it..
Thanks anyway

[kalius]

<?xml version="1.0" encoding="UTF-8" ?>
- <urlset xmlns="http://www.google.com/schemas/sitemap/0.84" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
- <url>
<loc>http://www.aeforums.net</loc>
<lastmod>2005-06-08T09:30:34Z</lastmod>
<changefreq>changefreq=hourly</changefreq>
<priority>priority=1.0</priority>
</url>


your mising some lines from the header, maybe thats it

[boohlick]

<?xml version="1.0" encoding="UTF-8" ?>
- <urlset xmlns="http://www.google.com/schemas/sitemap/0.84" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
- <url>
<loc>http://www.aeforums.net</loc>
<lastmod>2005-06-08T09:30:34Z</lastmod>
<changefreq>changefreq=hourly</changefreq>
<priority>priority=1.0</priority>
</url>


your mising some lines from the header, maybe thats it

Thanks for the Info ... ;)

[NewComputer]

Just finished setting up an online interface that will generate a Google xml sitemap from a ls -lR directory listing, handy to create a sitemap from your local file system, and then upload that to your web server. Create Google xml sitemap from directory listing (http://www.iconv.com/ls2sitemap.htm)

I just cannot figure this thing out. Does it use the folder on my local machine?

[iconv]

Yes, you should run ls -lR > myfile.txt in your local web directory folder, and it will recursively list all your files from there into the myfile.txt file. Then you upload this file, and get a sitemap in return.

[NewComputer]

I have no idea what ls -lR > myfile.txt is... call me a noob...

PS: What about forum and blog posts that are stored server side?

[kepa]

Sitemaps works in getting your pages crawled incredibly faster. I've been waiting for months for my site to get crawled heavily like my other site, but no dice. 5000 products and only about 10-15 pages would get crawled. I set up a cron job to submit my sitemap automatically and bam, the next day I'm getting crawled 1300 pages, now it's an everyday occurence.....sweeeeet.

[microtony]

After 8 hours Google downloaded my sitemap :)

[kalius]

I have no idea what ls -lR > myfile.txt is... call me a noob...

PS: What about forum and blog posts that are stored server side?

It only works for listing your actual files, not for database driven sites.

If your forum is in VBuletin you can use my script to generate a basic sitemap.

for blogs I know of a wordpress plug-in and another for serendipity.

[rossriley]

Just a bit of further information. The pages on our site are tricky to reach via linking, they are normally arrived at via user searches, which obviously google cannot do. For the past few months the number of pages indexed has been stuck around the 1,000 mark. Anyway after submitting a sitemap, google proceeded to visit thousands of the pages, and the number of indexed pages has suddenly increased to 4,500 with some of the datacenters showing 11,500.

Not a bad increase within a few days!!!

Obviously this will not help you if you have a flat site but it seems that the sitemap enables google to go deeper than it would normally.

[NewComputer]

If anyone was having difficulties generating a decent sitemap, I came across this today and took it for a test drive. Worked very well.

http://nodemap.com/software.php

Just download the .exe version and then it will take care of the rest...

[mizt]

Anyone want to help out with a phpbb one? Willing to barter some links in return for it. They already have site map mods so I'm assuming it would just take some editting to make it a xml file.

[Googles76]

are there any alternatives if python is not enabled on my server?

[NewComputer]

are there any alternatives if python is not enabled on my server?
You can use the method I suggested above. It is about 10 seconds worth of work after setup.

[minstrel]

If anyone was having difficulties generating a decent sitemap, I came across this today and took it for a test drive. Worked very well.

http://nodemap.com/software.php

Just download the .exe version and then it will take care of the rest...
Great find, NC...

Someone on another forum was just asking if there was anything like this available... :D

[Googles76]

Why does it take so long for Google to download the sitemap, is it because it is a manual process?

[Infiniterb]

Just an FYI. For those of you using Blogger, you can submit your atom feed as a site map.

[mizt]

http://www.phpbb.com/phpBB/viewtopic.php?t=296051

There you go for your phpbb people. I'm still stuck trying to figure out a solution for this that works with my mod rewrite.

[ServerUnion]

Googles76 - DL of the XML file must be a batch process.

[Googles76]

Looks like Google doesn't update the sitemap status very frequently. I just checked and it says submitted: 4 hours ago, and downloaded: 4 hours ago, yet when I checked it about 30min ago it still said status is pending. Now will see how fast the goole bot comes :D , I need it very much because of complete overhaul of my site.

[Stin]

I setup the sitemap script to generate a new sitemap at 4am every nite and submit it. So far google seems to download it every day at 1pm. They crawled 1/3 of my site the first day and about a third every day afterwards. I haven't seen any more pages indexed but that will probably take a couple days. Im pretty excited.

[microtony]

I submitted 4 sitemaps to Google and it downloads them three times a day :).

[Ian]

I submitted 4 sitemaps to Google and it downloads them three times a day :).

It may download them, but they still haven't crawled one of my sites and added the new pages to the index yet. They've been downloading my sitemap repeatedly since I set up the sitemap last Sunday, yet I haven't seen any of the new pages that I added now show up in the index in almost a week. Anyone else run into this?

Ian

[davedx]

Hmm this is cool. Main advantage I can see is for sites whose freshness > their pagerank->crawl frequency. I.e. you have a news site or blog with about 14 posts/week, but is only being crawled 3 times a week.

It's basically like technorati's ping.

I don't think a cron job is the solution... what you should try to do is setup a manual ping in the admin section of your site that generates a new sitemap and submits it to Google, that you initiate whenever you add or update significant content.

[pachecus]

yes.......sitemaps are very important

[minstrel]

yes.......sitemaps are very important
Thanks for sharing, pachecus.

However, this thread is actually about Google Sitemaps Beta, not about sitemaps in general...

[Stin]

just checked my site: on google, and they just indexed 250 more pages

[chiefshuddle]

It doesn't appear to me that it has helped me much after 4 days after submitting but maybe it takes longer.

[minstrel]

I'm beginning to wonder if this is anything more the Superstition 101... :eek:

[pachecus]

If anyone was having difficulties generating a decent sitemap, I came across this today and took it for a test drive. Worked very well.

http://nodemap.com/software.php

Just download the .exe version and then it will take care of the rest...



Thank you NewComputer

[Googles76]

To sum everything up, if anyone wants to go at it manually, here is how the sitemap layout looks like:


<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
<url>
<loc>http://www.yourbaseurl.com</loc>
<priority>1.0</priority>
</url>

<url>
<loc>http://www.yourbaseurl.com/index.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=1.0</priority>
</url>

<url>
<loc>http://www.yourbaseurl.com/page.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=0.1</priority>
</url>

</urlset>


than just keep repeating:


<url>
<loc>http://www.yourbaseurl.com/page.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=0.1</priority>
</url>


for all your pages you want to map. There are also other options you can put in such as lastupdated but I didn't find that too usefull. The priority is a value from 1.0 to 0.1 with 1.0 carrying the most importance and 0.1 carrying the least.

[City2]

I love the sitemaps, did anyone have success with them

[Redleg]

I love the sitemaps, did anyone have success with them

It looks like it's working for me.. :)

I submitted a couple of sitemaps 1 to 2 days ago, and googlebot is all over the site now.
Been almost zero googlebot activity this month until now..

[Josh]

Hmm, this seems very interesting.. nice way to get those hard to find pages indexed.

Does anybody have any data that shows if submitting a site thats already in the index will get a higher ranking after doing a sitemap for it? I.e. would submitting a site thats already fully indexed be helped at all.


Josh

[minstrel]

Does anybody have any data that shows if submitting a site thats already in the index will get a higher ranking after doing a sitemap for it? I.e. would submitting a site thats already fully indexed be helped at all.
I can't see how it would make any significant difference at all, Josh... once googlebot has found you, devours all your pages, and returns daily for more (often in groups), what more can you expect a sitemap to do?

[noppid]

I can't see how it would make any significant difference at all, Josh... once googlebot has found you, devours all your pages, and returns daily for more (often in groups), what more can you expect a sitemap to do?

I suspect you may be telling the bot only where to go. :eek:

[minstrel]

And when it has already spidered, indexed, and reindexed all your pages through several updates, then what?

I do realize that for sites like this forum or for a storefront, there are always new pages being created -- it may make a difference there when the number of pages reaches a threshold. For an average information site or ecommerce site, I don't see it making a lot of difference.

[Stin]

Yeah my site has several pages added daily, and a lot of them update as well. Im still a PR0 missed the last update, so Googlebot wasnt paying me much attention. After putting the sitemap up I think it has indexed everypage on there, and cached almost all of them!

[Opie]

For anyone that needs it, I wrote a small tutorial on how to create a Google sitemap for a large site using Xenu LinkSleuth and MS Excel. Check it out:

http://www.ethangiffin.com/archives/2005/06/07/08/45/59/

Regards,
Opie

[NewComputer]

I can't see how it would make any significant difference at all, Josh... once googlebot has found you, devours all your pages, and returns daily for more (often in groups), what more can you expect a sitemap to do?

Minstrel,

The one advantage I see of this is if you have a homepage and a few other pages that have not been updated in quite some time, and Google has stopped visiting your site (this has happened to me) but you have added additional internal pages, you just add them to a new sitemap and submit them. Google has now crawled and cached those new pages and they have returned and updated the cache on the homepage as well.

[minstrel]

Okay... so as I suggested a while back, it's basically a fast-track "Submit" button?

[NewComputer]

Okay... so as I suggested a while back, it's basically a fast-track "Submit" button?

Yes and no, with the priority and the rest, it is a little more... remember Min, this is beta, I am sure these guys are not going to tip their hat at the start as to the true reason behind this. If we know the guys at G (and we don't) this will probably turn out to be something pretty cool and useful... it already has been for me...

[minstrel]

The thing is, as I said on another forum, Google's track record doesn't suggest that they generally go out of their way to make it easy for webmasters, given that from the viewpoint of relevant search results webmasters might often be considered to be the enemy. I think it was noppid who wondered publicly what the catch was and that bears some thiking about...

I look at the features (including priority and last-modified-date) and I'm just waiting for the first bright light bulb to start experimenting with scamming this -- for excample, has anyone listed a page yet as anything less than priority 1.0? How useful is that really going to be for Google?

If Google is looking at this as a way to find pages it doesn't know about, I can see why it might be seen as helpful to them. But I think there is little doubt that webmasters are going to at least try to exploit this one way or another and I wonder how Google is going to deal with that.

I have always been impressed by the intelligence in the Google group. I'm waiting with bated breath to see how this one turns out.

[nddb]

Maybe a bit offtopic, but if people are having trouble with the XML, the faq says you can submit just a list of links, divided by newlines (\n in most languages).

That's a very easy way to do it. It's hard to tell if it has improved it's crawl coverage, since it already had most of me indexed, but i believe it has added more because of the sitemap, but I cannot be certain, so could just be wishful thinking.

[NewComputer]

Maybe a bit offtopic, but if people are having trouble with the XML, the faq says you can submit just a list of links, divided by newlines (\n in most languages).

That's a very easy way to do it. It's hard to tell if it has improved it's crawl coverage, since it already had most of me indexed, but i believe it has added more because of the sitemap, but I cannot be certain, so could just be wishful thinking.
If you were using a notepad and just links, do you add the \n after say .html and a space?

[nddb]

no space, just :
http://www.site.com/page.html\nhttp://www.site.com/page2.html\nhttp://www.site.com/page3.html\n

Which, when viewed through a browser as say, sitemap.txt looks like :
http://www.site.com/page.html
http://www.site.com/page2.html
http://www.site.com/page3.html

[Stin]

further update, google is all over my site, has cached basically every page. Google now downloads my sitemap about 2 hours after it is posted every night at 4am. I'm a fan.

[NewComputer]

no space, just :
http://www.site.com/page.html\nhttp://www.site.com/page2.html\nhttp://www.site.com/page3.html\n (http://www.site.com/page.html%5Cnhttp://www.site.com/page2.html%5Cnhttp://www.site.com/page3.html%5Cn)

Which, when viewed through a browser as say, sitemap.txt looks like :
http://www.site.com/page.html
http://www.site.com/page2.html
http://www.site.com/page3.html

What about if you were using an html extenion on the same page, any difference? Is the \n still recognized?

[iskandar]

Google downloaded my sitemap and is basically doing nothing much. Crawls like what it has been doing for the past weeks .. crawling like the robot is dying.

Is it because my site is new (~ 3 weeks old) that the bot has been reluctant to be aggressive eventhough i have submitted my sitemap? Is it the sandbox effect again? Duh! Why can't they just get rid of this sandbox for sites with significant no of pages ( like 50 and above) ?

Anyway, for ppl who use Linux/Unix hosting..you should just try the sitemap generator recommended by google. It's a snap to create a sitemap using it. The beauty is also in the fact that you do not need to recreate it and send it to google when the pages change. Use the cron job to auto-magically do it .. I love this tool

For ppl using windows server..hmm..either do the xml pages by hand (bad idea for big sites) or use third party tools mentioned here. The problem is you need to do it again (create, submit) everytime you add pages. A hassle ...

Oh! Linux - hard to learn but easy to use...thank god i use linux server

[microtony]

Google downloaded my sitemap and is basically doing nothing much. Crawls like what it has been doing for the past weeks .. crawling like the robot is dying.

Is it because my site is new (~ 3 weeks old) that the bot has been reluctant to be aggressive eventhough i have submitted my sitemap? Is it the sandbox effect again? Duh! Why can't they just get rid of this sandbox for sites with significant no of pages ( like 50 and above) ?

Anyway, for ppl who use Linux/Unix hosting..you should just try the sitemap generator recommended by google. It's a snap to create a sitemap using it. The beauty is also in the fact that you do not need to recreate it and send it to google when the pages change. Use the cron job to auto-magically do it .. I love this tool

For ppl using windows server..hmm..either do the xml pages by hand (bad idea for big sites) or use third party tools mentioned here. The problem is you need to do it again (create, submit) everytime you add pages. A hassle ...

Oh! Linux - hard to learn but easy to use...thank god i use linux server

I think it does help very much!
I got all my pages of my sites indexed after Google downloaded my Sitemaps :D :D :D

[nddb]

Iskandar,

Didn't try that, but I think so. googlebot should just see it as a bunch of urls separated by newlines. It looks different in the browser when it has a .php extension, I tried that. But I think it should work as well. It looks like a bunch of urls with a space in between them. But that's just how the browser views \n in .php (and probably .html).

At any rate, if it doesn't work, google will tell you. =)

[iskandar]

That's what ppl have been saying. I think the sitemap is valuable for sites that have been "there" for quite sometime.

I think the sandbox effect still applies - even after you sent the sitemap. Anyway, i think it's better to send the sitemap than not doing anything ..

For those who have succeded .. congrats!

[crazyhorse]

Is there anyone that can confirm that after adding the sitemap he/she saw a increase in traffic?

[NewComputer]

Yes, I saw traffic to pages that were not indexed the day before. Not only that, but when I search for some unique information that is on those pages, Google is showing those pages as being found in the db and are now being found by the consumer... so, yes.

[Perrow]

I've seen a good increase in traffic from G after submitting my sitemap, but then again the site's about getting out of the sandbox age so it might just have coincided.

What I did see however was a nice visit from the G-spider that equals some of the first spiderings of the site, it's been hard to get G to spider more than 100-150 pages each week for oh so long. Now they've spidered 1700 in just a few days (well over 150 a day for 6 days straight).

[nddb]

NewComputer,

If you were using a notepad and just links, do you add the \n after say .html and a space?

Sorry, I didn't answer that, in notepad, you'd just hit enter. \n is more for perl or php

like echo "blahblah\nblahblah"; Would put a return where the \n is.

[NewComputer]

Fair enough, that is what I have done and it seems to be working to date.

[Blogmaster]

Hmm, this seems very interesting.. nice way to get those hard to find pages indexed.

Does anybody have any data that shows if submitting a site thats already in the index will get a higher ranking after doing a sitemap for it? I.e. would submitting a site thats already fully indexed be helped at all.


Josh
Site Maps do help a lot (at this time). You help the robots spider you better and it does increase your rankings. Everytime I have added site maps to a client's site, I have seen increases in rankings very fast without making any other changes or adding links. It doesn't make a major difference when analyzing each keyword as an individual term but since you have all or most of your pages on it, the overall traffic increases substantially thru pulling you up for pretty much all of them.

[isaiasd2003]

I'm just learning how to work with xml, lol. Though the main problem I have with google right now is the fact that I have DIRECT LINKS to my site, though google doesn't recognize it. Anyone know how to get em to recognize?

[ServerUnion]

just takes time. you can always check on MSN as they seem to show up faster there. Just becuase they aren't listed doesn't mean google doesn't see them.

[mizt]

Anyone get an invalid date error?

http://carcommons.com/sitemap.php

Should I copy that information and a make a file with the extension .xml or will good read the php alright?

[Stin]

I dont think that date/time is a valid format. I think u need to replace that T with a space, but I could be wrong.

[Dio]

Anyone found that Post Nuke sites are problematic? My site Viral Auctions has the flat page option ticked so pages have nice easy URLs like this:
http://viralauctions.com/Article15.phtml

I've made a site map (last week in fact), Google has accepted it and downloaded it several times - however, only 4 pages are indexing:
site:viralauctions.com (http://www.google.co.uk/search?q=site%3Aviralauctions.com&sourceid=mozilla-search&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official)

:(

None of the content pages are indexing, does anyone have any idea why? One of them has (since adding the sitemap) but the rest are noticable in their non-showing.

[egdcltd]

Okay, as I'm reading it, you give Google a bunch of URLs on your site to crawl, and tell them when you update with new content. However, when I add new content, it usually winds up on a new URL. How does this affect the sitemap, or am I missing something?

[ServerUnion]

you would have to add the new URLs in. Easiest way is to generate from your database if they are dynamic

[egdcltd]

you would have to add the new URLs in. Easiest way is to generate from your database if they are dynamic

That could be a serious pain. Most pages generated are static, and I generate hundreds of new pages a month. How about if I just put in the URLs of the pages that typically link to new pages in the site?

[Stin]

uhm the google sitemaker can just crawl your directories if your pages are static. You can have it run every night to update and submit the sitemap.

[Imran]

submitted xml sitemap to google as a test for couple of my sites the test resulted in 50-50, one site's pages increased the other sites pages reduced from what google had indexed earlier! srange!

[jlawrence]

Ok, it's now 9 days since I uploaded my sitemap. For a brand new subdomain with 20K pages. G has just indexed and cache nearly 800 pages. /me quite pleased with that, I'm used to waiting about a month for a proper crawl to start.

[crazyhorse]

Wondering whether there are some people that see significant changes since they added the sitemaps? I mean like a increase of number of pages like 30-40 % .

I found a nice sitemap for phpbb users. All credit goes to pentapenguin (http://www.phpbbstyles.com/profile.php?mode=viewprofile&u=1387) for creating Google phpbb sitemap (http://www.phpbbstyles.com/viewtopic.php?t=6232) .


<?php
/***************************************************************************
* googlesitemapgenerator.php
* -------------------
* Copyright/Support http://www.pentapenguin.com
* Last Modified: 06/05/05
*
***************************************************************************/

/***************************************************************************
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
***************************************************************************/

define('IN_PHPBB', true);
$phpbb_root_path = './';
include($phpbb_root_path . 'extension.inc');
include($phpbb_root_path . "common.$phpEx");

// Begin Configuration Section
$included_forum_ids = array();
$excluded_forum_ids = array();
// End Configuration Section

if ( count($included_forum_ids) > 0 )
{
$included_forum_ids_sql = 'forum_id IN (' . implode(', ', $included_forum_ids) . ')';
}

if ( count($excluded_forum_ids) > 0 )
{
$excluded_forum_ids_sql = 'forum_id NOT IN (' . implode(', ', $excluded_forum_ids) . ')';
}

if ( ( count($included_forum_ids) > 0 ) && ( count($excluded_forum_ids) > 0 ) )
{
$and = 'AND';
}

if ( ( count($included_forum_ids) > 0 ) || ( count($excluded_forum_ids) > 0 ) )
{
$where = 'WHERE';
}

$sql = "SELECT topic_id, forum_id, topic_time, topic_type FROM " . TOPICS_TABLE . " $where $included_forum_ids_sql $and $excluded_forum_ids_sql ORDER BY topic_time DESC";

if ( !$result = $db->sql_query($sql) )
{
message_die(GENERAL_ERROR, 'Error: could not retrive topic IDs', '', __LINE__, __FILE__, $sql);
}

$protocol = ( $board_config['cookie_secure'] == 0 ) ? 'http://' : 'https://';
$servername = $board_config['server_name'];
$port = ( $board_config['server_port'] == '80' ) ? '' : ':' . $board_config['server_port'];
$path = $board_config['script_path'];

$output = '<' . '?xml version="1.0" encoding="UTF-8"?' . '>' . "\n";
$output .= '<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">' . "\n";

while ( $row = $db->sql_fetchrow($result) )
{
$topic_id = $row['topic_id'];
$forum_id = $row['forum_id'];
$lastmodified = date('Y-m-d\TG:i:s+00:00', $row['topic_time']);
$viewtopic = "viewtopic.$phpEx?" . POST_TOPIC_URL . "=$topic_id";
$priority = ( $row['topic_type'] == POST_STICKY || $row['topic_type'] == POST_ANNOUNCE ) ? '1.0' : '0.5';

$output .= "<url>\n";
$output .= "\t<loc>$protocol$servername$port$path$viewtopic" . "</loc>\n";
$output .= "\t<lastmod>$lastmodified</lastmod>\n";
$output .= "\t<changefreq>daily</changefreq>\n";
$output .= "\t<priority>$priority</priority>\n";
$output .= "</url>\n\n";
}
$output .= "</urlset>\n";

header('Content-type: application/xml');
echo $output;
?>


For further info please check the phpbbstyles website.

[minstrel]

There's another one at the main phpBB.com site:

http://www.phpbb.com/phpBB/viewtopic.php?t=296051&start=0&postdays=0&postorder=asc&highlight=

[Davilac]

And I in one website only have We couldn't find the Sitemap at the location you provided. Please make sure the Sitemap URL is correct and resubmit your Sitemap.

[eduardomaio]

That's because your .xml or .gz file isn't there. Manually run the script and check if the file is created or not.

[dvduval]

I am hoping that I will get my new site indexed too. I used the script on phpbb.com about 3 days ago. About 100 of 20000 pages have been indexed. Hopefully, more will be added soon.

[WyattEA]

That could be a serious pain. Most pages generated are static, and I generate hundreds of new pages a month. How about if I just put in the URLs of the pages that typically link to new pages in the site?

Sorry for being a little off topic, but when you say you generate hundreds of pages a month, do you mean your site is dynamic but it creates static URL's for the SE's? i've been trying to find information on doing this for my site - anyone have any info on doing this?

WyattEA

[egdcltd]

Sorry for being a little off topic, but when you say you generate hundreds of pages a month, do you mean your site is dynamic but it creates static URL's for the SE's? i've been trying to find information on doing this for my site - anyone have any info on doing this?

WyattEA
In general, no. Most content generated is either html or shtml pages. There are ways of doing this, basically, I think, by mod_rewriting dynamic URLs. My ScriptsGold subdomain generates dynamic URLs, but I had the script modified to make them static.

[jlawrence]

Thought about starting a new thread on this but decided not to.

Are people seeing new sites getting indexed quicker when using the sitemap - or is it still somply a case of pointing enough links into the site to get it indexed ?

[mcdar]

I have a fairly new site that had been indexed by Google. I decided to change the page titles to most of the pages and thought that this would be a good time to try out the new sitemap feature in Google.

I submitted the sitemap and it was uploaded and accepted more that a week ago now.

Google has yet to update to reflect ANY of the new page titles. So apparently, Google has not bothered to spider the site even once as a result of the submission.

So far, I am NOT impressed!

Caryl

[Googles76]

I have a fairly new site that had been indexed by Google. I decided to change the page titles to most of the pages and thought that this would be a good time to try out the new sitemap feature in Google.

I submitted the sitemap and it was uploaded and accepted more that a week ago now.

Google has yet to update to reflect ANY of the new page titles. So apparently, Google has not bothered to spider the site even once as a result of the submission.

So far, I am NOT impressed!

Caryl

I am not impressed either. I've submitted a sitemap more than a week ago and Google has yet to clean up it's index with my site. For some reason it still has pages indexed that were removed weeks ago. It is just one big mess right now.

[Infiniterb]

I have a fairly new site that had been indexed by Google. I decided to change the page titles to most of the pages and thought that this would be a good time to try out the new sitemap feature in Google.

I submitted the sitemap and it was uploaded and accepted more that a week ago now.

Google has yet to update to reflect ANY of the new page titles. So apparently, Google has not bothered to spider the site even once as a result of the submission.

So far, I am NOT impressed!

Caryl

For what it's worth, I've 159,000 pages indexed on one domain since adding my sitemap. The domain is two years old and has a PR5 domain, however.

Sounds to me like the sitemap tool won't help new sites and Google is still applying a sandbox of sorts.

[Jade456]

I wonder how much it will really help. Has anyone sent them a txt file version?

[nddb]

I sent in a text file, just a list of urls separated by newlines. It seems to have worked pretty well, but google was already starting to crawl heavily, so it's not like it went from 0 to 1000 a day. But I think it helps. So does putting random deep links on the front page, that's what seemed to really spur google into action.

[Skeleton]

Check my website about Google Sitemaps. You can find the link in my signature.

[minstrel]

Check my website about Google Sitemaps. You can find the link in my signature.
Actually, no, we can't. You need a certain number of posts before your signature shows up here.

But even if it were there, most people probably wouldn't bother clicking. If you have an opinion or comment, spill a bit here... then if people are at all intrigued, they might go to your site to read the rest of the "article" or whatever it is. As it stands now, my guess is most people will read your post as an attempt to leech traffic to your site, and just move on to the next post.

[NewComputer]

You've been Minstrelized...

[minstrel]

:D just trying to acquaint the new member with how forums work... :o

[Foxy]

Good stuff....hehehe :D

[WebDesignPro]

I am hoping that I will get my new site indexed too. I used the script on phpbb.com about 3 days ago. About 100 of 20000 pages have been indexed. Hopefully, more will be added soon.

From our own observations; It looks like as the workload of sitemap.xml files builds up, the indexing rate is slowing down. As more people submit sitemaps, the time dedicated to indexing your site is decreasing.

Anyone else noticing that revisits are taking longer?
Our record is 19,000 pages in 4 days. That was for a file submitted 5 days ago.

[NewComputer]

Yup, I concur... Which is to be expected. My first site was crawled and indexed within the hour (and those were brand new domains) and the last one I submitted was 6 days ago and looks like G was here today...

[T0PS3O]

So you guys are implying they had dedicated spiders/crawlers available for this.

I doubt that's the case. They just get added to the same URL queue and spidered by the same bots (is my guess - would make most sense from efficiency point of view).

[ian_ok]

I wonder how much it will really help. Has anyone sent them a txt file version?

I added 4 text files.

One of my sites has gad G all over it more importantly G had not touched any of my database pages until submitting this file! Now have over 400 pages up from 20 - 50.

One site still in the sandbox!

One site indexed all forum pages and then 2 days later say's P*ss *ff and now none of the forum pages are done!

Last site happy with slightly better cralwing

Ian

[Imran]

Is there any SiteMap tool for PostNuke? I tried SnakeSEO but it's links are not working Modified the .htaccess according to what snakeSEO says but the links are still broken hence my sitemap's arnt working when I tried other php sitemap tool's they generate use less pages and other stuff in the sitemap instead of including the articles of the site it includes images/registered users list ! I hope there is one for postnuke which works!
If any one know or can help me please?

[minstrel]

Have you tried adapting one of these?

http://www.phpbbstyles.com/viewtopic.php?t=6232&postdays=0&postorder=asc&start=0

http://www.phpbb.com/phpBB/viewtopic.php?t=296051

[Imran]

I actually need for PostNuke not PhpBB.
Thanks!

[minstrel]

Ah, sorry... was thinking of phpNuke and thought you could adapt them...

But you should be able to do that anyway: download the mods at look at the code.

[Imran]

Yeh, There is a strange error I get in this apache server. On my other sites Mod_rewrite works great which does not require any sitemap or I can create my own sitemap manually, but for a site with 12000 articles and other jokes and other category its just a pain to sit down and do manually. I tried sanke seo, you can see, www.bollyheat.com/sitemap.php this works fine but the links dont! when some one clicks on it it says mod not found. Maybe I have to relook at my .htaccess file!
Thanks!

[honey]

I did a sitemap for a test project, and out of 4000 pages, they seem to have read 500 pages, although none of them are in the index yet.

[Stin]

google just picked up the entire site again, this is about a week after the first full site crawl.

[Imran]

Working well, after I submitted by sitemaps [manually created] google has started indexing my pages again!
one thing I learnt for dynamic pages You need not have full site pages in the sitemap, only a link to these pages will do in sitemap!
I have only 8 entries in my sitemap and in one day 3906 pages indexed!

[Spacen]

I was able to successfully include every page from my invision power board. I wrote my own php page to generate the xml file so I could include static content. I then setup a curl cron job to call the page (with querystring input required so that simply accessing the page won't generate the file). The results have been very promising very early.

[crazyhorse]

I was able to successfully include every page from my invision power board. I wrote my own php page to generate the xml file so I could include static content. I then setup a curl cron job to call the page (with querystring input required so that simply accessing the page won't generate the file). The results have been very promising very early.
Would u mind care to explain how you did that? The querystring thing, cuase when you enter mine it almost freezes my sitemap.;)

[Stin]

im not quite sure what you guys are talking about but you might want to put some delays into your program, php has a seconds delay and a milliseconds delay function so check that out

[Stin]

btw of 400 pages in my sitemap 386 are now cached, and I suspect the straglers will soon be cached as well. This from about 30 pages cached a couple weeks ago.

[t2dman]

Hi Minstrel

I have been considering the worth of the Google site map

With no use of the Google site map - just added heaps of new pages to a forum and very soon after Google found the main forum pages, yet to find all the threads
Google often spiders based on the Google PR of the site - the higher PR, the more often, earlier and deeper it seems to spider.
So if you are a low PR site, you can wait quite a few weeks for new content to be found by Google.
And even if you are a higher PR site, you can still wait quite a number of days for Google to find all the new pages that have been added.
So Google site maps can get new pages found faster and it is less dependant on the Google PR of your site as to how fast those pages are found and spidered (IMO).

Where Google ranks a page is a lot about Google PR and link text for that page. So once Google has found all those new pages - if those pages have a lot of link text/anchor text for certain important pages on your site, then those important pages will rank higher earlier.
You need to make sure that you have some good links from existing high PR pages to those new pages, and good html site maps, so Google will calculate in good PR for those pages you have created.
It is always one thing for a page to be cached by Google. It is quite something else for it to then be ranked high by Google. So how about it Minstrel - is that good reason for why Google site maps are a good thing?

[minstrel]

If it's true, t2dman, then I'd say yes... it's reason enough to use Google sitemaps.

[dvduval]

I was wondering if it is okay to submit more than one sitemap per site?
ex. one for forum, and another for the directory.

[Spacen]

Would u mind care to explain how you did that? The querystring thing, cuase when you enter mine it almost freezes my sitemap.;) Sure. I have my php page, let's call it a.php. The code in a.php generates b.xml which is my google xml sitemap. I have setup a cron job to call a.php once every day so a new b.xml is generated everyday. In case anyone accidentally found a.php, I programmed in an if statement that requires a querystring variable in the URL so that if you try to view a.php, nothing happens. If you access a.php?action=generate then the code is executed and a new xml file is generated.

[iconv]

Using google sitemaps has been very successful for me,especially on large sites. One site went from 286 pages indexed to 130'000 indexed, another from 80'000 to 540'000. Took about 4-5 days after submitting the sitemaps, and googlebot started visiting, adding about 30'000 entries per day to the index.

[vectorgraphx]

my indexed pages went up too, still waiting to see the positive effects.

[t2dman]

Following on from the excellant work of Kalius, I have put together a vBulletin Google Site map that has pagination for the forums, threads, archive and archive thread version. All items accessable via the config variables on the file.

Yet to be added is support for multi site maps for large vB sites.

Google site map for vBulletin (http://forums.digitalpoint.com/showpost.php?p=221022&postcount=47)

The copy of it on my personal site:
vBulletin Google site map (http://forum.time2dine.co.nz/showthread.php?t=3976)

[vinturella]

If I read the posts in this forum correctly, those of us hosted by GoDaddy are left out of sitemaps.

Somebody tell me it ain't so and how I get my host to read a $ command, and run python.

Thanks. My direct email is jbv@jbv.com

[t2dman]

If I read the posts in this forum correctly, those of us hosted by GoDaddy are left out of sitemaps.

Somebody tell me it ain't so and how I get my host to read a $ command, and run python.


There are many ways to skin a cat.

You do not have to use the Google version. You can use php and create based on interrogating a database for url's (ie like has been done with vBulletin Google site maps). Or you can run a normal site map spider over your site and manually use php or even excel to create the xml for the site map.

Always lots of different ways.

[tillies]

Submitted my first sitemap to google for our site that has dynamically created pages. I am planning to submit it everytime I add more pages or make content changes.
Not quite sure what difference it will make. Anybody else with experience?

[ServerUnion]

should only have to submit it once, they come by and download it frequently

[vinturella]

Your answer is very provocative, but I am clueless as to how to execute. I use FrontPage, and its site mapper requires that I put each page in a category and is quite unwieldy. I know Excel fairly well.

Am I in the ballpark? Thanks.

John, jbv@jbv.com

There are many ways to skin a cat.

You do not have to use the Google version. You can use php and create based on interrogating a database for url's (ie like has been done with vBulletin Google site maps). Or you can run a normal site map spider over your site and manually use php or even excel to create the xml for the site map.

Always lots of different ways.

[Imran]

Yes You have to submit it only one's but It will download the same sitemap.xml unless you modify it daily!

[ServerUnion]

That is the beauty of scripting a dynamic site map.

[ziandra]

should only have to submit it once, they come by and download it frequently


115 times in the past 7 hours. G is having a ball on one of mine.

[WebDesignPro]

115 times in the past 7 hours. G is having a ball on one of mine.

What log analyzer software do you use?
I wonder if that is just one continuous visit aa it attempts to index your site.
A series of reads of the sitemap.xml file spread over time.
If you look at Google sitemap stats requiring your gmail account, does it show as being downloaded 7 hours ago or 0 hours ago?

[t2dman]

Your answer is very provocative, but I am clueless as to how to execute. I use FrontPage, and its site mapper requires that I put each page in a category and is quite unwieldy. I know Excel fairly well.

Am I in the ballpark? Thanks.

John, jbv@jbv.com

How many pages do you have? Can you copy paste the url's from your front page site map into excel and reformat into the xml format?

Via Freefind (http://www.freefind.com/create-sitemap.html) they will spider your site, and then show you the url's in their control panel - copy and paste into excel and reformat.

Or Google to find a spider program that can spider your site, give you the url's and then excel them.

[ikeys]

maybe start with the basics , you don't need to make a complicated sitemap ,
just make a list of all urls from your site.

http://www.example.com/page1.html
http://www.example.com/page2.html
http://www.example.com/page3.html
...

save the file as
http://www.example.com/sitemap.txt
submit it in the
sitemap service
https://www.google.com/webmasters/sitemaps

more info on the simplest sitemap
https://www.google.com/webmasters/sitemaps/docs/en/faq.html#s9

if that works you can try to make it into an xml sitemap

[ziandra]

What log analyzer software do you use?
I wonder if that is just one continuous visit aa it attempts to index your site.
A series of reads of the sitemap.xml file spread over time.
If you look at Google sitemap stats requiring your gmail account, does it show as being downloaded 7 hours ago or 0 hours ago?

grep sitemap.xml.gz /..../access_log | wc -l

Currently 247 hits
First was: 66.249.65.244 - - [30/Jun/2005:12:46:52 -0700] "HEAD /sitemap.xml.gz when I submitted it
last was: 66.249.65.101 - - [01/Jul/2005:15:00:26 -0700] "GET /sitemap.xml.gz HTTP/1.1"

Google Sitemaps - My Sitemaps says : submitted 1 day ago, downloaded 6 hours ago.
Clock on the computer says: Fri Jul 1 15:27:54 PDT 2005

Google hasn't visited any of the pages in the past day, only hammered the sitemap.

[nddb]

Wow... try it without gzipping perhaps? That is nuts.

[loki]

damn! i got all excited about the Google Sitemap for Wordpress until i realised it is *or Wordpress*.

once i learn to read properly, can anyone point me to a simple solution for producing a gg sitemap for a large dynamically driven site?

ps i haven't learnt to code properly either.

[boohlick]

hope this can help

http://www.google.com/webmasters/sitemaps/docs/en/protocol.html

[mhamdi]

google sitemap just worked great for me, all 150 pages of a new website (less than a month) got indexed a day after submitting the sitemap. (I don't have that much of incoming links, around 30 acc. to msn)

[siraxi]

Is this Google SiteMaps a quicker way to get a new site through the Sanbox waiting time... ?

[mhamdi]

I guess. But it may be the related links too. because the internal pages got crawled more than 10 days before submitting the sitemap but they were indexed the very next day after the sitemap. who knows !!

[siraxi]

This is MOST helpful ! Thank you very much ! It works :)

To sum everything up, if anyone wants to go at it manually, here is how the sitemap layout looks like:


<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
<url>
<loc>http://www.yourbaseurl.com</loc>
<priority>1.0</priority>
</url>

<url>
<loc>http://www.yourbaseurl.com/index.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=1.0</priority>
</url>

<url>
<loc>http://www.yourbaseurl.com/page.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=0.1</priority>
</url>

</urlset>


than just keep repeating:


<url>
<loc>http://www.yourbaseurl.com/page.htm</loc>
<changefreq>changefreq=monthly</changefreq>
<priority>priority=0.1</priority>
</url>


for all your pages you want to map. There are also other options you can put in such as lastupdated but I didn't find that too usefull. The priority is a value from 1.0 to 0.1 with 1.0 carrying the most importance and 0.1 carrying the least.

[KingSEO]

It's only helpful if you're incapable of reading documentation.

[siraxi]

Actually I was ... :) when I started to read about complicated stuff like python and cronjobs, I gave up. I'm too simple minded for that.

[newquestions]

I have a huge website, and so I decided to create a sitemap.
I submitted it about ten days ago, and since then, my traffic
has exploded. A few points to note ...

1. My website is about a year old.
2. I never did create a decent sitemap structure previously.
3. Google had bounced pages in and out of its listing for ages.

But, there is one thing I have noticed. My pages don't necessarily
show up, when I look for saturation. In other words, when I ask
Google how many pages are listed, it gives me a pretty crappy
figure - BUT, I am getting visitors to these pages. To many of
my pages, that are not technically listed.

So, I think this is really helping Google. Basically, if someone is
looking for something really specific, and Google cannot find a
nice website to show them, then perhaps it reaches into its
understanding of XML sitemaps, and retrieves something?

My traffic is around ten times higher than last month, and a lot
of visitors are hitting pages that have never had traffic. Only
difference was that I told Google about my new sitemap.

It works. So, if you are too simple, or too lazy to get it to work,
then pay someone to do it for you. It really is worthwhile ...

[isaiasd2003]

In my opinion, sitemaps aren't as bad as some may think. For example, I submitted my sitemap.xmls to G last month, and just about all the changes I made to my site were recognized faster than the times without. So it came in handy, the sites back in the search keywords it use to be in. I can't say much for my other sites, but I haven't put as much effort into them. So its a small win from the last month when G gave me the smackdown, and shoved my site off to another keyword no one would ever imagine typing but me lmao. Want proof? http://www.google.com/search?hl=en&q=ultimate+spy+software I'm number 8 . ultimatespysoftware.com or you can just go to the big G and search keywords ultimate spy software. It use to be ultimate-spy-software, which is beyond annoying, made a few changes, and the next day I was under my old keyword phrase!:cool: Now if I can only get a lil higher (like rank 1 or 2) it might be worth it! lol:D
PS. Dunno if it'll show up under uk, or other nations, cause I set the metas to only be searchable under specific states and didn't want to get carried away and lose my place.:(

[kool43]

I noticed that alot of people seem to be submitting .php versions of sitemaps to Google sitemaps. I have used a Mod Rewrite to make .xml pages run like a php page so that you can rename your .php page to .xml instead. here it is for those of you want it:

Just add this line to your .htaccess file in the directory of your sitemap

AddType application/x-httpd-php .htm .xml .rss

Hope this helps everyone, as I have seen a significant difference between submitting a .php and now an .xml extension instead with no changes to the code.

If anyone wants to have a look, let me know...

[ServerUnion]

dont forget to change the contenttype you are sending out to make it XML

[Foxy]

I noticed that alot of people seem to be submitting .php versions of sitemaps to Google sitemaps. I have used a Mod Rewrite to make .xml pages run like a php page so that you can rename your .php page to .xml instead. here it is for those of you want it:

Just add this line to your .htaccess file in the directory of your sitemap

AddType application/x-httpd-php .htm .xml .rss

Hope this helps everyone, as I have seen a significant difference between submitting a .php and now an .xml extension instead with no changes to the code.

If anyone wants to have a look, let me know...

Thanks for the heads up
:)

[Up2U - YourWorld]

WordPress has a plugin which auto creates and pings after every update to a post/comment, so that helps your WordPress blogs get noticed.

Got to be a good thing if Google indexes you more - more indexed = more visitors...

[kool43]

We set our xml/via .php page to generate the most current info.

So when Google comes by, it will automatically regenerate the xml feed via the php script before it feeds Google this info. Seems to work great, and Google is eating it up.

[Dusty]

The Google sitemap works. I have a site I just built and by using the sitemap I had all of my pages indexed within a few days.

[m1ke]

It's not doing much for me at the moment. I think I submitted it about a week ago.

[mizt]

Ok I have a .php xml file sumbitted to google right now. What does the mod rewrite look for changing it to xml. Its in my main directory, and the above didn't seem to work.

http://mydomain.com/sitemap.php needs to be http://mydomain.com/sitemap.xml

Sorry. I've been stuck with the amount of pages indexed seeing if this does the trick.

[senexom]

Hey guys...
I see questions like I submitted my sitemap yesterday and Google still hasn't crawled the site. First since I started using Google’s sitemaps especially the first few weeks it has been somewhat un reliable finding errors in one run and then no problems next time it downloads it, once in a while it still does that, no worries thought they're working on it. I had a problem where my sitemap stopped being downloaded for no reason, just stopped for a bout a week, I wrote them an email and within few hours it was back to normal. In the email to Google I specified everything I did and they symptoms I was experiencing. so they are still working out the kinks.

Also depending on how popular your site is they may not want to crawl it every day, it uses up their resources and if the past few crawls your site wasn't updated all that much the crawler knows it.

I found that my PR3 site got re-index 2 days after the sitemap was downloaded, I did it as a test and changed the titles of my pages slightly and they appeared in Google only in a couple of days, so perhaps it works like a queue.

[toocoolforschool]

Fantastic way to get Googlebot to visit your page. Because I have three sites that are wordpress blogs, I downloaded a plug-in that automatically pings Google whenever I update/change something on my blogs. This is probably the easiest and most effective way for those running Wordpress blogs.

[adigitaldreamer]

Seems like a few of my supporting pages are getting spidered, but we shall see. It's really hard to tell if it was just going to happen any ways, or if it has to do with the site map.

Brian

[jugger]

Can anyone tell me if the command "Site:nameofsite" shows all links indexed on google. I am trying to determine the status of our sitemap links that were taken by google, but the status area of the sitemaps is vague. So I thought doing a "site:mywebsite" would let me know. Is this very accurate way of knowing how many links I have indexed.

Thanks.

[North Carolina SEO]

As I understand things, yes. The largest question is whether the page was indexed with the www or without (e.g. site:www.nameofsite vs. site:nameofsite). I have seen some differences in the past between the result sets.

[toocoolforschool]

Hmm, yes, "site:" lists all pages Google has indexed of your site so far.

[rbucich]

Can anyone tell me if the command "Site:nameofsite" shows all links indexed on google. I am trying to determine the status of our sitemap links that were taken by google, but the status area of the sitemaps is vague. So I thought doing a "site:mywebsite" would let me know. Is this very accurate way of knowing how many links I have indexed.

Thanks.

Not sure if this helps but...
Just did a redesign on a site including renaming many pages without redirect. Once we were live, I submitted a sitemap which was downloaded successfully by Google around 2 weeks ago.

I did not include the old pages (they are still live) in the sitemap but fixed all the internal links so they point toward the new pages.

As of today site:www.mysite.com shows only the old pages, none of the new ones from the sitemap.

Yet

[godders]

I've just redesigned my website and it's due for release next Wednesday. My current design is very old and doing a search for