Hi... I'm just starting an Australian share market discussion forum. When I just submitted my sitemap, there were a few pages indexed. Then I decided to use URL rewrite and added a robots.txt file. Today, Googlebot crawled my site again, but nothing is indexed now. Can anyone see what I did wrong? The URL is: http://www.rookiesharetrader.com The content of my robots.txt is: User-Agent: * Disallow: /forum/login.php Disallow: /forum/privmsg.php Disallow: /phpBB2/login.php Disallow: /phpBB2/privmsg.php Code (markup): (I initially named my directory phpBB2/, then changed it to forum/). The pages with URL rewrite that I expect to get indexed (but are not) are: http://www.rookiesharetrader.com/forum/shares-discussions http://www.rookiesharetrader.com/forum/shares-articles http://www.rookiesharetrader.com/forum/shares-questions-and-answers http://www.rookiesharetrader.com/forum/shares-tipping They are on the first page, are included in the sitempap and are not dead links. This really confuses me. Any help / clues / suggestions will be very much appreciated. Thank you very much.
Have more patience and get more links. Google is fickle and doesn't always index as quickly as we would like.
Your site is already in google index. Submit google sitemap and try to get quality backlinks for your site.
Thanks everyone. It seems that the indexes are old, probably because Google hasn't updated it yet(?) As one of my recent changes (done before last Googlebot's visit), I've removed all "sid=xxxx" part of the URLs, and in the results returned from the link above, they're still there. Also, I checked my web server log and found in its last visit Googlebot only requested for HTTP header for http://www.rookiesharetrader.com and http://www.rookiesharetrader.com/forum/, presumably to check for last modified time, and nothing else. Could it be because it sees that the top level file (index.php) was not modified then it doesn't crawl further down? About sitemap, I'd already submitted it, but in txt format (http://www.rookiesharetrader.com/sitemap.txt). I assume it doesn't make it less "attractive" to Googlebot compared to the XML version? Thanks once again.
just do one thing, create nice internal linking structure which will help u to index whole site quickly.
If you submitt a sitemap to Google that would help. Also, creating back-links will help. Generally, however, it helps to keep in mind that Google also takes some time to index the site.