Free Add URL Service

Thanks for the URLs and the compliment cashwealth!
I wish I was ready for an IPO...

One submission is enough, because the spider checks if the site has already been crawled before and included in the index; and revisits submitted pages on a regular basis. In fact, as hundreds of URLs are being submitted every day, I'm noticing some spamming already. Some users send the same URL(s) several times per day. This makes me think that perhaps I should start implementing some spam filter for the Add URL script...

 

I decided to go ahead and checc up on this to see if my sites were added and they where. And very nicely at that. Thanks

Repped.

 

Thanks for the link, the complement and the rep as well!

 

I will take a look at your site later. I will consider submitting all my sites to your search engine.

 

Please do and thank you for the support!

 

add mine pls.

 

i added
www.awesome-videos.com/
www.goodcunts.com/

thanks a ton
R+

 

I'm going to add my sites now.
Thanks

 

I have just added my site.

 

Dear kevinzhang,

Please use the Free Add URL form to submit your URLs.

Thanks,
Kris

 

So you are only going to index them? Are you indexing the internal pages as well?

How does it work. just curious?
You store an URL and tag it with keywords? Like you create a column for a keyword and then store all the urls that is relevant to that keyword in that Link?

 

thank for added my blog well. I hope it will be next yahoo!
http://dresswomen.blogspot.com

regard
emha

 

hey the service looks good, i just added a few of my domains, hope to see it get approved

 

Thanks for the question Blogspotter!

The crawler I designed is a simple server side script which goes through the list of submitted URLs. Visits every single one of them unless it is already in the index or if it thinks a URL could be buggy. Then it checks the header if it returns a 200 response and if the content type is HTML or Text. (I am working on a PDF, DOC and XLS filter also which could be a nice addition to this site)

The visited page then gets filtered. It removes all HTML tags and scripts, only keeping the clean text content of the page. It also saves the title between the TITLE tags.

Then using the unfiltered version of the script, it collects all the links from the source including URLs from anchor tags, frames, iframes, redirects etc. These new links will then get cued in for the next crawl.

If the content of the page is sufficient then most of it will get stored the way it is in a MyISAM database running on MySQL.

The crawler waits about a minute and continues crawling the URLs it found in the previous crawl. I did this to ensure that the crawler won't visit pages on a site too heavily to avoid slowing down the site and/or getting my IPs banned on the web server.

It also saves the anchor texts from other sites as incoming links for each site to help determine the ranking of a page later when the query algo goes through the index...

That's it! It's a really simple code I wrote in ASP. The only problem is that it's half automatic so I have to start it and sometimes watch the results to make sure it works fine and to help me find ways of tweaking it. But I'm thinking about automating it fully because -thanks to this forum- new URLs are coming in like crazy (about 200 unique per day which results in about 1000 new pages in the index by the time the crawler is done following all/most the links).

 

hey! Added my URLs as well.

 

Thank you for all the URLs readers of this thread submitted in the past few days!
I really appreciate it!
But I don't understand what happened...
Everything just slowed down.
Not so long ago I was barely able to keep up with feeding your URLs to the spider and I really loved doing that.
A few days ago my spider got over a hundred URLs submitted every single day, but for some reason the number of submitted links dropped to below fifty by yesterday.
I'm curious: What happened?
No more wembasters are reading this thread or they just don't have any more URLs to submit?
What should I do to market this free Add URL service?

My spider is starving!

Seriously, I was so desperate this morning that I went through all the pages of this thread, copied out all the links from it and fed them to my spider!
It only took me a second using firefox, but it was really embarrassing.
I guess that every thread has a lifespan and this one just decided to die.
I'm not being ungrateful. I really do appreciate all the links you all fed my spider, but if this thread is really dying then I need to find a new direction for my mission of gathering fresh URLs for my little baby spider.

As always, I'm open to ideas!
Please let me know if you have any.

Thanks,
Kris

 

added my website.. good service
thanks

 

i added http://www.c4d-renders.com

 

i've added ma site there, but how other ppl will came to know abt ma site...
& how will i know, dat pppl r visiting ma site....

plz dont mind im a newby

 

Thank you for your URL and the question roxn!
Your site was crawled by my spider (a robot/software visiting websites) and included in the UpperSites web index. When a web user comes to UpperSites and submits a query, a program on my search engine does a quick search in the index to find every web sites matching the query. If your website or one of the pages on your site is about the subject of the query, your page will be included on the search engine result page. By default, UpperSites only shows the top ten results, so your site may not be displayed right on the first page. The better you optimize your site meaning that the better and more original content your site has talking about a specific subject on the submitted page, the more chances you are that your site will be included on the first search results page when a websurfer submits a related query...

I hope this helps. Should you have more questions regarding optimization for UpperSites, please post it here and I will do my best to answer it for you as soon as possible.

Best Regards,
Kris Tarr