Ongoing test: How long until Google index's an unblocked URL in robots.txt?

I recently had someone tell me that if you have a URL blocked in robots.txt and then unblock it, Google takes 180 days to re-index that URL. That didn't seem right to me (I think they were confusing a manual exclusion request) ... so I'm running a test right now.

Comments/suggestions appreciated on the methodology used ... and read how some of the intermediate steps have happened (i.e. robots.txt has been re-spidered), but after six days, I haven't seen any results in the search engine yet (I'm also testing MSN and Yahoo).

Anyone want to make a guess how long this will take to show back up?

 

So your saying if a site gets banned in google search... it takes 180 days to come back in??? Keep me updated also with the manual exclusion request.

 

No. What I'm saying is if you have a URL blocked in robots.txt (intentionally) ... but then decide that you DO want that URL index'able, how long does it take (after you remove it), for it to show up in the SERP's.

Up to 7 days now, and hasn't shown up yet in Google, MSN, or Yahoo yet - my guess is we'll see some action in the next week though.

 

OH ok... I got your point... keep us updated on your experiment.

 

Does your site point to the url?
are ther any external links to the url?
How often does googlebot visit your site.

i would say at a guess if you have links pointing to the url you want indexed it should show up in google within 2 weeks, if there are very little or no links pointing to it it may take 2 months + .

Kathryn

 

See details on the page including data from web logs and Webmaster Console view

In answer to your questions, there was ONE PR5 link to the previously blocked URL (now there is two, since that URL above has a link to it). 99.9% sure no external links to the test URL, and Googlebot seems to come by fairly frequently to the site. BTW, page has been updated (at T+8 days) to reflect that all three search engines have now spidered the previously blocked URL ... so now it just need to show up in the index.

 

Update: Google is now showing the previousely blocked URL in the the SERP's - took 11 days. No results for MSN or Yahoo ... yet.

 

What I think is search engines check robots.txt every time they crawl your website, and take the recently unblocked URLs as new .. So, how much time it take in getting recently unblocked URLs into SERPs depends on frequency of your website's crawl. If its crawled every 15 days then possibly your pages will get indexed in 15-20 days.

IMO

 

My web site is crawled/spidered quite a bit every day ... so perhaps it was a bit slow based on the theory above (?)

BTW, Yahoo is now indexing the previously blocked URL - this was at T+13 days. Google was at T+11 days ... and so far, hasn't shown up in MSN/LiveSearch.

 

Those two are a bit slower than big G but it will get in there. You can speed it up with Yahoo by submitting a sitemap to them as well. MSN is just slow...period.

 

Just to wrap up this thread, I'm now see results in Live Search. So in answer to the question "Unblocking URL's in robots.txt - how long does it take Google, LiveSearch, and Yahoo to notice?"

Answer: Google indexed the URL after 11 days, Yahoo at 13 days, and LiveSearch at 17 days.

Complete writeup of methodology and other info.

 

Well Google say if you remove a url from their index manually it will be excluded from the index for 6 months no matter if you unblock it or if the Googlebot spider comes across it ... be interesting to see how they react to a robots.txt exclusion my feeling is that it will be the full 6 months also.

 

Must have posted that just before I posted mine ... interesting

 

Yep - I actually mention this in my writeup and here is the Google reference.

My test was what happens if you do NOT do an manual exclude.

 

I'm so glad you posted this and I found some reference to this in google SERPs! I recently moved a site from a hosted wix account to a self hosted wordpress account, and while I was stuck in the process of refining the theme, layout, and organising permalinks, i decided to block robot access. UNFORTUNATELY the whole site became de-indexed from google as a result and in a panic, I resubmitted the site to google, unblocked the google bots, submitted a sitemap, got PR5 links pointing to it, and I'm on day three and still nothing
I really hope it wont take 6 months.
I noticed this information is old, and even the page you reference for google is gone, I know that they have recently debunked as they usually do a number of sites, and quite frankly I'm panicked as there is nothing recent to point to any information on this.
Someone please give me reassurance

 

Well, did you resubmit your site to Google?

 

Well, yes I did resubmit it - I did the whole 'resubmit for reconsideration' thing, and added the sitemap... google has shown that it crawled the site every day for the last 3 days and no indexing yet...