Hey there, I am currently developing a freelance programming website. During the development, I do not want google to crawl my website. I was wondering.. since PR is only about incoming links to a site, does PR still get calculated for a website if the site is disabled by the robots.txt file? The reason I ask is I can still do some SEO during the development. That way when I release the website, everything will not seem sudden. I can increase the backlinks gradually. Any advice will be appreciated. Sincerely, Travis Walters
obviously, if google will not index your site - no PR. Google gives PR only to Google indexed sites. hope i am not wrong,.
Hey there, Thanks for the prompt responses. Well, I do know google is keeping track of my links in the google webmaster tools. My website is currently indexed. I changed the robots.txt file a few days ago so my site would not be crawled. Because a site can not be crawled, will my site be taken out of the google index? When I bought my domain, somebody had it before me so it is over 2 and a half years old, which is decent. The old owner must not have done much with it because it is PR0, and had no backlinks when I started. Let me ask you this. If my site is taken out of the google index temporarily while it is being redeveloped, what will happen? I would expect that when I submit a sitemap to google down the road, it will still have my backlink records, or will find them again. So if I do SEO while it is being developed, my PR should jump up then right? Let me know if I am right.. I am hoping so Sincerely, Travis Walters
Why would you bother putting a robots.txt to block G bots? Although the domain is old, it has 0 PR and 0 backlinks. Even though it's in G's index, G will never show the site when a visitor searched the term. Well, except visitors type the exact keywords in the search box or the browser. I think it's just a waste of time worrying about it. Try check your hosting stat, beside yourself and SE bots, do you have any human visitors? If no, what's the problem of just leaving it as it is. It's in the index already. When you finished the construction the reindexing is faster. People are having problem indexing new domain fast and you want to take your domain out? Just let G index your site with the "site in construction" text.
Hey there, In my experience, I find google indexes websites really fast when you submit a sitemap. However, I find google to be slow reupdating the information. I think if I enabled robots.txt, I would be playing catch up for awhile. Sincerely, Travis Walters
That's exactly what I'm saying. You may think that you are playing for a while, but what if G thinks it's longer that that? You may never be able to predict or "tell" G what to do. If it's in index, it's fine. Use a resubmit of the sitemap to speed up the re-crawl. There are some new sites that having long time of indexing even with the sitemap at webmaster central. If I were you I would never "play" with G. I will never know when the big G will think that this is not a play anymore. If it's not broken, I won't change it. Well, that's me. It's up to you though.