I started to work on the crawler only 3 weeks ago and so far it is only playing around with different ideas.
General crawler or specifically for myspace? I would be very interested in what youre doing, perhaps from a commercial viewpoint. Perhaps time for a PM
http://www.myspace.com/bigleft2005 - 4 http://www.myspace.com/conformistrecords - 4 http://www.myspace.com/jonfrederikband - 4 http://www.myspace.com/lystraband - 4 http://www.myspace.com/releasepraiseband - 4 http://www.myspace.com/cultclash - 4 http://www.myspace.com/murdermurderyesindeed - 4 http://www.myspace.com/ssharknife - 4 http://www.myspace.com/sheltondavis - 4 http://www.myspace.com/discardthedead - 4 http://www.myspace.com/whackofgoogle - 4 http://www.myspace.com/ztod - 4 http://www.myspace.com/singalongnow - 4 http://www.myspace.com/randyrhoadesmusic - 4
yes, I'm curious too. Are you working on general or specifics crawler? Hmm... I lost some of my blogspot domains, wonder if ppl like you who "steal" those domains
The crawler is for http://www.directoryglobe.com - i want to make a real se/directory out of this dmoz clone. So it wasn't made for the purpose to steal domains or urls but to check if a site is gone already, thats how I came accross these urls.
That is the last one: http://www.myspace.com/shannonsorens - 4 http://www.myspace.com/uglyfrenchduck - 4 http://www.myspace.com/functionalblackouts - 4 http://www.myspace.com/samesexmarriage - 4 http://www.myspace.com/jonathanrfoster - 4 http://www.myspace.com/deadly_altoids_is_bac - 4 Please do not send me any PM for more
Great. Now I am curious how long G* needs to find the links in the profile... @lukaslo: just submitted my three directories to your directory...
@katalugo: I will make links to profiles from other sites and I think the sites should be indexed fast Thanks for adding your directories