I have setup a Business Webmaster Forum BizWebForum.com and I can not figure out what these spiders are doing... today I found over 50 spiders on the forum, then about after an hour they vanish...then a little while ago, I had 20 spiders that showed up, and vanished just after a few minutes....Its getting to the point now that yahoo spider is on the forum on and off all day long looking at much older posts, and can not figure out why these spiders are not jumping on the new content, but rather looking at posts that maybe months old... Then I run across a webmaster forum that has like 2000 spiders online, although his forum has been up like 2 years longer then my forum...so the longer your forum is online the more spiders you attract to your site? any experts out there? Maybe its just me, but I would rather have 50 live members online posting rather then 2000 spiders doing who knows what, spamming my bw, i dunno... I just do not get it... Am I getting any benefit at all ?? What do you guys think?
Hello can you identify with 100% certainty these are SE spiders? because even if so.The increase in the amount of them would suggest ,or at least pinpoint to the "someone is scrapping content from your website" issue. Constant on , off -s , increasing number? The other person gains users based on your content.
A spider, also known as a robot or a crawler, is actually just a program that follows, or "crawls", links throughout the Internet, grabbing content from sites and adding it to search engine indexes.