I have a forum that Google was indexing almost all the new topics that were posted on the forum everyday. It was indexing close to 500 pages of the forum each day. Then we moved our server and in the process lost robots.txt file with some minor modifications. So had to built a new robots.txt again with almost the same things as the previous robots.txt I am not sure if robots.txt is the reason, but after moving server, Google does not index any new pages of the forum. What do you think is wrong and how can I fix it? Thanks.
in a "normal" robots.txt you allow ALL to be indexed and exclude only the absolute MINIMUM folders such as for example cgi-bin or admin folders robots.txt unless you made an error and EXCLUDED Google from major content sections should have zero impact on site indexing look into proper link structure - CHECK entire site for validity of links validate HTML and see if any fatal error is produced check IP to see if your new IP is on any blacklist !! also make sure on your new server Google IP is in no iptables in case your pic-hosting site is the one you have problems: also make sure ALL your hosted images are LEGAL and copyright OWNED by uploading site users/members if the site offering image hosting is the one you have problems MOST sites hosting images have stolen images thus sooner or later those site owners loose G adsense and a few millions or more in $ lawsuits for copyright infringements and sites with stolen pics may get banner by G as well
Well nothing has changed, the links are still the same, links load fine, everything is still the same. I dont think its blacklisted. I still rank #2 for my main keyword, and I get alot of traffic from Google for the pages still indexed in Google. Also my site is Google Authority site. But it is not indexing new pages/topics on the forum.
Are you using Googles Webmaster Tools? That will give you a very good indication of what they are actually seeing. Sometimes it's pretty easy for use to overlook problems because we are not seeing them from fresh eyes. Essentially you changed your address and Google has not caught up. I would expect if nothing really has changed except a new IP then within a couple of days Google will be back on track and things will return to normal.
Even the IP has stayed the same. So all I could guess is the robots.txt is alittle different from what we had before. Everything else is same. Yes, I use webmaster tools, but I dont see anything wrong in it....should I be looking at a specific place?
you may receive more useful help if you actually post the domain here. without site name all else is but a guess and thus waste of time.
How do you check if its ban? Well I just looked at Google and it indexed 1 page today. So I guess that means its not banned?
I am not sure if your pages an with unique content, 500 pages/day seems to be a large number (unless you own a forum bigger than DP). Are most of these posts short (less than ten words), I have a website with the same issue but it seems to be duplicate content refining more than anything else. Feel free to PM me your URL if you want me to have a look
I think you should have to use sitemap from that you can easily indexed your pages by submitting that url
Use Google webmaster tool that indicate you which pages will be index by Google. You will be able to see which page are disallow or allow by robots.txt.