My service has over 500,000 pages. Can't get a good sitemap. Help!

We (me and my wife) started a free classifieds service: www.qwikad.com and while it's still growing we have already encountered a problem. We can't get a full sitemap. After about 20,000 pages sitemap generator freezes. Is there something we are doing wrong? Are there good sitemap generators out there? Any help will be appreciated.

 

you might need something coded by a developer to work server side.

 

I would signup for Attracta.com and the site map tool they provider should be able to help. I also suggest recoding that site so new pages are just mod rewrites which would lower the pages big time and the only thing that would show up is the cities page and the state page.

 

Do you mean I can somehow get a sitemap from the server? If it can be done, I could try to call the web hosting provider to find out if they can help.

 

is this so important to put all of your 500000 pages on the sitemap? If you want to do so, keep in mind that your xml sitemap will have to be divided in pieces of 50000 urls with no more than 10mb each.

 

So WHAT can be done? My classifieds keep growing, will Google still crawl it whether I submit a full sitemap or not?

 

I had a look at your site and a lot of it is empty, it's well and good to have a lot of pages - but Google suggests that you trickle pages into your site - instead of rushing them in at once as it looks fake and like junk - looks automated. Google wont even consider adding your pages if there's no content on them...

http://qwikad.com/-54/posts/1-community/1-activities/ - "there are no ads in this category yet"
http://qwikad.com/269/posts/4-dating/ - "there are no ads in this category yet"
http://qwikad.com/269/posts/5-for-sale/ - "there are no ads in this category yet"

etc. etc.

Then pages like this... there's to many links and not enough content - normal text.

http://qwikad.com/269/posts/3-jobs/

You will have major issues - unless you fill in all of the blanks and get some content into the site, all of the pages with - "there are no ads in this category yet" will be considered junk and duplicate pages because they all look and contain the same content.

 

Thank you for the info, I know I still have quite a few empty pages there, but still it is growing pretty fast. The content of any classifieds is in the ads that come from the visitors.

But you know what? I think I am beginning to realize something. I don't think it really matters if any of the pages (except the index page) are indexed by google in my case. Most visitors/posters I have now didn't find me through google. And once it gets popular google will rank it higher and higher just because.

 

While the attitude is good to let things occur naturally, the things mentioned above still hold true, you still need to be indexed to get "higher and higher" in search rankings, you need to deliver traffic to those who are in your directory - that's your service to them... you still need to target some key word search terms... so even if you cover some basics - like get higher rankings for some of your categories that are popular search terms... Basically you have to be indexed to be an effective directory for several reasons; A) To benefit the listings with a link back to their site (it doesn't count as a link if your page isn't indexed), B) To deliver them traffic... if you're not getting much traffic then you can't deliver much either.

 

you should try
auditmypc.com/

this is a good site

 

You can create sitemap with php and mysql, don't need any sitemaps generator

 

And nobody here knew enough about sitemaps to tell the guy there is a 50.000 URL limit for a sitemap??

sitemaps.org/protocol.html

For 500.000 pages, you'd need to create a sitemap.xml which would serve as an index, listing for example sitemap1.xml, sitemap2.xml...that's the only way to create a sitemap with that many pages. BTW, as biggerlol said, you should create a sitemap with PHP, preferably as a cron job so it updates itself automatically.