so im running gsite crawler, so far its found 38,000 links/pages etc. now i know for sure my page doesnt have 38,000 links on it. so what the hell is going on? my stats are below, and its not even done yet, its been going all night long wtf? Statistics for ly2.com Date: 11/28/2005 08:33 Main URL: http://www.ly2.com/ (Case sensitive) Number of URLs listed total: 38730 Number of URLs listed to be included: 38730 Number of URLs listed to be crawled: 38670 Number of URLs still waiting in the crawler: 336054 (may include some already listed) Number of URLs aborted in the crawler: 4 http://www.ly2.com/forgot.htm linked from http://ly2.com/french.html http://www.ly2.com/index.htm linked from http://www.ly2.com/login.htm http://www.ly2.com/myred/home.html linked from http://www.ly2.com/myred/contact.php http://www.ly2.com/order.htm linked from http://www.ly2.com/signUp.htm Top 20 oldest included URLs: http://ly2.com/signUp.htm (49 days) http://www.ly2.com/signUp.htm (49 days) http://ly2.com/terms.htm (49 days) http://www.ly2.com/terms.htm (49 days) http://ly2.com/images/login.jpg (49 days) http://www.ly2.com/images/login.jpg (49 days) http://ly2.com/images/line.jpg (49 days) http://www.ly2.com/images/line.jpg (49 days) http://ly2.com/images/choose.jpg (49 days) http://www.ly2.com/images/choose.jpg (49 days) http://www.ly2.com/main_files/top.jpg (49 days) http://ly2.com/images/top.jpg (49 days) http://ly2.com/images/top2.jpg (49 days) http://www.ly2.com/images/top.jpg (49 days) http://www.ly2.com/images/top2.jpg (49 days) http://ly2.com/login.htm (32 days) http://www.ly2.com/login.htm (32 days) http://www.ly2.com/myred/images/bot_2.jpg (31 days) http://www.ly2.com/myred/images/point_1.jpg (31 days) Ban URLs with these texts in them from being crawled: Drop these parts of URLs being crawled: