Try our Search engine Spider Simulator (beta) and tell us your idea. In a few days our simulator will also include all internal and external links from your page, that can be indexed by search engines.
i like it. simple and easy to use and gives you a good breakdown of the information spiders pick up on. i'll definatly be trying it when you get the internal and external links simulator running as well. good work!
Thanks for your positive replies. The internal and external links that can be spidered by search engines are added to our Spider Simulator.
Nice one. Can you add some more features to it? flag any external links that have rel="nofollow" on them. check the robots.txt file on the server to see if this page can be crawled. I was plannng to build my own, but I simply can't find the time... Cheers, Cryo.
just tried it with the links added on as well. very well done i say. i'll certainly be using that from now on. excellent!!!
Very nice Had an initial problm with a trailing whitespace in the URL which led to a pretty ugly error. Maybe need to trim() the URL? Nice tool. Will use.