I have been searching google and decided to see what my competing forums are doing. I noticed that one forum that is fairly larger than me has just slightly more idexed pages than me (which I expected to be more, but I am very close), then I noticed a site that is smaller than me has 300,000 more indexed pages than me (WOW!). We all use the same forum software (Vbulletin). What is he doing to get that many indexed pages? I am not saying he is ranking higher in the search engines under specific keywords, as a matter of fact he is PR3 and I am a PR4, but to have that many pages? This is the way I look at it. You can throw a handful of stones or just a few, but if you don't hit your intended target, what good was it? Feedback would be appreciated. Thanks!
A few factors can come into the equation here. forum setup - some forums are set up where all threads show up since the beginning, or all threads. Others have theirs set up to just show threads from the last month. How you have this set up is exactly how the search engine spiders are going to see your forum. If you have it set for the last 30 days, that is as far as they go and old threads wont remain indexed in the search engine. sitemap - you can set these up for google as well as submit a .txt version to yahoo. Letting them know what you have can only help your cause in getting pages indexed. robots.txt - Believe it or not, many forums are very popular and linked from several sites. They get all the traffic they want already and therefore limit or completely restrict search engine spiders from their forums. There are others as well, I am sure, but these are the ones I put a high importance on. Hope this helps, and maybe there are some others here that can provide a little more input into this for you.
Your encouragement of "deep links" prompts a question, Shawn (or others). Some directories restrict your submissions to the home page only; others allow an inner page to be submitted. Let's say you already have a good number of links to your home page. Is it wiser then to begin submitting to directories the links to important inner pages, rather than to the home page?
Jim, if the directory in question allows inner pages to be submitted, then you can submit may more pages thus getting many more links, am I right?
That would be ideal, and I'll look into that. I had been presuming (maybe wrongly) that the majority of directories only allow one entry per website.
Yeah Shawn, I agree with that. More links, means your more important. The more important you are, the more pages are indexed by Google.