Need to find OUR URLs

Hello experts.

Google indicates 20,000 pages from our site, but display only the first 1,000.
That is a known issue.

However wwc crawler has successfully crawled 5,000 pages. That's way more than Google actually gives us, but much less than the 20,000 pages.

Can you suugest a solution?
I'm very open to bids !!

 

Whats the url?

 

PM sent, thanks.

 

I can see 17,600 pages in google.

Are you trying to find out why google has listed so many pages when you think your site has fewer pages?

If so it will be hard as google only shows the first 1000 results. You need to examine the site architecture and make sure all the old pages are 301 redirected to the new pages or to a 404 error page.

 

Try Xenu Link sleuth, it's free, I'd guess it could cope with 20,000 pages though may take some time.
http://home.snafu.de/tilman/xenulink.html

 

Instead of Google, I used WinWebCrawler to crawl the site, and found `6k pages. This bypasses the the 1K limit of Displayed pages by Google.

The site changed hands, and in order to 301 Redirect the pages I want to crawl any Lost page; i.e 12K lost URLs need to be found.
Will be happy to any solution possible

Thanks

 

Instead of Google, I used WinWebCrawler to crawl the site, and found `6k pages. This bypasses the the 1K limit of Displayed pages by Google.

The site changed hands, and in order to 301 Redirect the pages I want to crawl any Lost page; i.e 12K lost URLs need to be found.
Will be happy to any solution possible

Thanks

 

Cool tool, I'll try it out.
Thanks!

 

Can you not just look at your server logs for the error messages? Find all the 404 and 500 errors and solve them one by won.

 

It appears that if I won't be able to relove the lost ones, that's what I'll be doing. Cheers