1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Ongoing test: How long until Google index's an unblocked URL in robots.txt?

Discussion in 'Google' started by hulkster, Feb 24, 2007.

  1. #1
    I recently had someone tell me that if you have a URL blocked in robots.txt and then unblock it, Google takes 180 days to re-index that URL. That didn't seem right to me (I think they were confusing a manual exclusion request) ... so I'm running a test right now.

    Comments/suggestions appreciated on the methodology used ... and read how some of the intermediate steps have happened (i.e. robots.txt has been re-spidered), but after six days, I haven't seen any results in the search engine yet (I'm also testing MSN and Yahoo).
    SEMrush
    Anyone want to make a guess how long this will take to show back up?
     
    hulkster, Feb 24, 2007 IP
    Jean-Luc likes this.
    SEMrush
  2. KingofKings

    KingofKings Banned

    Messages:
    5,975
    Likes Received:
    143
    Best Answers:
    0
    Trophy Points:
    0
    #2
    So your saying if a site gets banned in google search... it takes 180 days to come back in??? Keep me updated also with the manual exclusion request.
     
    KingofKings, Feb 25, 2007 IP
  3. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #3
    No. What I'm saying is if you have a URL blocked in robots.txt (intentionally) ... but then decide that you DO want that URL index'able, how long does it take (after you remove it), for it to show up in the SERP's.

    Up to 7 days now, and hasn't shown up yet in Google, MSN, or Yahoo yet - my guess is we'll see some action in the next week though.
     
    hulkster, Feb 25, 2007 IP
  4. KingofKings

    KingofKings Banned

    Messages:
    5,975
    Likes Received:
    143
    Best Answers:
    0
    Trophy Points:
    0
    #4
    OH ok... I got your point... keep us updated on your experiment. :)
     
    KingofKings, Feb 26, 2007 IP
  5. www.mazoo.co.uk

    www.mazoo.co.uk Peon

    Messages:
    225
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Does your site point to the url?
    are ther any external links to the url?
    How often does googlebot visit your site.

    i would say at a guess if you have links pointing to the url you want indexed it should show up in google within 2 weeks, if there are very little or no links pointing to it it may take 2 months + .

    Kathryn
     
    www.mazoo.co.uk, Feb 26, 2007 IP
  6. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #6
    See details on the page including data from web logs and Webmaster Console view

    In answer to your questions, there was ONE PR5 link to the previously blocked URL (now there is two, since that URL above has a link to it). 99.9% sure no external links to the test URL, and Googlebot seems to come by fairly frequently to the site. BTW, page has been updated (at T+8 days) to reflect that all three search engines have now spidered the previously blocked URL ... so now it just need to show up in the index.
     
    hulkster, Feb 26, 2007 IP
  7. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Update: Google is now showing the previousely blocked URL in the the SERP's - took 11 days. No results for MSN or Yahoo ... yet.
     
    hulkster, Mar 2, 2007 IP
  8. Idiot Inside

    Idiot Inside Well-Known Member

    Messages:
    1,300
    Likes Received:
    65
    Best Answers:
    0
    Trophy Points:
    155
    #8
    What I think is search engines check robots.txt every time they crawl your website, and take the recently unblocked URLs as new .. So, how much time it take in getting recently unblocked URLs into SERPs depends on frequency of your website's crawl. If its crawled every 15 days then possibly your pages will get indexed in 15-20 days.

    IMO
     
    Idiot Inside, Mar 3, 2007 IP
  9. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #9
    My web site is crawled/spidered quite a bit every day ... so perhaps it was a bit slow based on the theory above (?)

    BTW, Yahoo is now indexing the previously blocked URL - this was at T+13 days. Google was at T+11 days ... and so far, hasn't shown up in MSN/LiveSearch.
     
    hulkster, Mar 3, 2007 IP
  10. WebGeek182

    WebGeek182 Active Member

    Messages:
    510
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    95
    #10
    Those two are a bit slower than big G but it will get in there. You can speed it up with Yahoo by submitting a sitemap to them as well. MSN is just slow...period.
     
    WebGeek182, Mar 3, 2007 IP
  11. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Just to wrap up this thread, I'm now see results in Live Search. So in answer to the question "Unblocking URL's in robots.txt - how long does it take Google, LiveSearch, and Yahoo to notice?"

    Answer: Google indexed the URL after 11 days, Yahoo at 13 days, and LiveSearch at 17 days.

    Complete writeup of methodology and other info.
     
    hulkster, Mar 13, 2007 IP
  12. Mr Skyline

    Mr Skyline Peon

    Messages:
    55
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Well Google say if you remove a url from their index manually it will be excluded from the index for 6 months no matter if you unblock it or if the Googlebot spider comes across it ... be interesting to see how they react to a robots.txt exclusion my feeling is that it will be the full 6 months also.
     
    Mr Skyline, Mar 13, 2007 IP
  13. Mr Skyline

    Mr Skyline Peon

    Messages:
    55
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Must have posted that just before I posted mine ... interesting :)
     
    Mr Skyline, Mar 13, 2007 IP
  14. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Yep - I actually mention this in my writeup and here is the Google reference.

    My test was what happens if you do NOT do an manual exclude.
     
    hulkster, Mar 13, 2007 IP
  15. julianareed

    julianareed Member

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #15
    I'm so glad you posted this and I found some reference to this in google SERPs! I recently moved a site from a hosted wix account to a self hosted wordpress account, and while I was stuck in the process of refining the theme, layout, and organising permalinks, i decided to block robot access. UNFORTUNATELY the whole site became de-indexed from google as a result and in a panic, I resubmitted the site to google, unblocked the google bots, submitted a sitemap, got PR5 links pointing to it, and I'm on day three and still nothing :(
    I really hope it wont take 6 months.
    I noticed this information is old, and even the page you reference for google is gone, I know that they have recently debunked as they usually do a number of sites, and quite frankly I'm panicked as there is nothing recent to point to any information on this.
    Someone please give me reassurance :(
     
    julianareed, Mar 11, 2011 IP
  16. ForgottenCreature

    ForgottenCreature Notable Member

    Messages:
    7,433
    Likes Received:
    164
    Best Answers:
    0
    Trophy Points:
    260
    #16
    Well, did you resubmit your site to Google?
     
    ForgottenCreature, Mar 11, 2011 IP
  17. julianareed

    julianareed Member

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #17
    Well, yes I did resubmit it - I did the whole 'resubmit for reconsideration' thing, and added the sitemap... google has shown that it crawled the site every day for the last 3 days and no indexing yet...
     
    julianareed, Mar 11, 2011 IP