I have been told that crawlers and spiders are a good way to help your forum, can someone give me some advice how to increase the activity of crawlers etc to help my site, i have created a robots.txt from a website and installed it but its not made much of a difference.
crawlers has their own crawling scheduler, if it has found a lot of new pages on a particular site (or a pattern of adding new pages quickly), it is likely to increase the crawling frequency on that site. If your site is new with a lot of pages waiting to be crawled, you can help the crawler by creating sitemap of your site, this can ease the burden of the crawler discovering new pages in your site.
my site is nearly 2 years old yet it gets next to no activity. its got over 4600 posts but only 85 members. I have a site map which updates every hour, i found i had more spiders etc when i switched from SMF to phpbb where i ended up with 65 spiders etc crawling about but i just cant seem to get much spider activity anymore even though there are regular posts. I have submitted it to nearly every search engine optimizing thingy, and i just dont know what im doing wrong, yet when i first created my tiny music website on a free type site it got hundreds of traffic.
On your forum, get 4 pr5-pr9 back links, get around 200 threads 200 posts and you'll be set google or that will just crew it, hope that helped.
If you are running on vBulletin I can tell you from experience that vBseo helps quite a bit. I think similar products are available for phpBB and SMF if you are running on one of those.
thanks will see if i can get some more back links and also look into VB - been considering VB for a while now but never thought it could improve traffic