I downloaded and installed the GSiteCrawler from http://johannesmueller.com/gs/ , but am having some issues with it. When it starts to crawl MuchGames.com, it only grabs the links that are on the main page. It stores all the link into the URL list for crawling, but DOESN'T crawl within those links! Out of around 1000+ links, it returns about 100 links, which it grabbed from the main page. Does anyone know how to make this thingy work?
If its a Google site crawler then it probably works like the real spider. It will index your homepage and you just need to wait a few months for a deep crawl.
It's not the Google crawler. It's a crawler that you set loose on your site to produce a Google sitemap. On the page you mention, there is a link to a Google group for the program. Why don't you ask this question there?
I did But I thought I might get more information here. I posted a question on google groups and am now waiting for a response. Its weird why the crawler won't go deeper. Isn't that the whole point of a crawler.. I mean it's not hard to make an application that retrieves all the links in a site.
Just look through the settings, there should probably be something where you can ajust how deep it crawls.