1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Accidentally barred directory sites with robots.txt

Discussion in 'Google' started by PaddyL, Jan 27, 2009.

  1. #1
    I did something amazingly dumb.

    I changed hosts recently, as my old host has been giving great problems.

    In the process, I accidentally changed my robots.txt to

    User-agent: *
    Disallow: *

    Doh!

    Now, Google has (obviously) started to remove my listings!

    I've fixed robots.txt as it should be.

    How can I get Google to re-read robots.txt quickly?

    The website is
    www.landau.ws

    Thank you!
     
    PaddyL, Jan 27, 2009 IP
  2. malcolm1

    malcolm1 Prominent Member

    Messages:
    7,148
    Likes Received:
    758
    Best Answers:
    0
    Trophy Points:
    310
    #2
    If you buy some traffic (yip its worthless for visitors) but i assure your site will be super indexed by all SEs. ;) "Traffic section DP"
    I did it with a domain that was barely getting indexed and within 2 weeks the site had over 1000 indexed pages.

    You might also want to get a few backlinks to help your site as building backlinks never hurts but only helps.

    thx
    malcolm
     
    malcolm1, Jan 27, 2009 IP
    PaddyL likes this.
  3. PaddyL

    PaddyL Active Member

    Messages:
    69
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    50
    #3
    Thanks, Malcolm, some good advice there.
     
    PaddyL, Jan 27, 2009 IP
  4. hubel

    hubel Grunt

    Messages:
    382
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    168
    #4
    I fail to see how buying traffic is going to make any search engine index your pages quicker. You would be better off submitting an up to date sitemap to Google and if you have already submitted one, then re-submit and hopefully that will get Google to download your new robots.txt file
     
    hubel, Jan 27, 2009 IP
  5. malcolm1

    malcolm1 Prominent Member

    Messages:
    7,148
    Likes Received:
    758
    Best Answers:
    0
    Trophy Points:
    310
    #5
    Hello...

    All i can tell you is that the site wasnt getting indexed very well and after i did
    the traffic thing it started getting spiders coming from everywhere which helped
    the indexing! (from 200 indexed to 800 within 2 weeks)

    I have nothing to gain from saying this as i dont own any "traffic site" but can say
    that the results where well worth the $20 bucks i spent. ;)

    Maybe the traffic was mostly bots? dont know dont care it worked :)

    As for submitting a sitemap.... thats a given/must

    thx
    malcolm
     
    malcolm1, Jan 27, 2009 IP
  6. PaddyL

    PaddyL Active Member

    Messages:
    69
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    50
    #6
    Unfortunately, Google says that it can't download my sitemap because of the robots.txt file.

    The problem is that Google hasn't looked at my restored robots.txt file -- that's what I really needed it to do as the very first step!
     
    PaddyL, Jan 27, 2009 IP
  7. rainborick

    rainborick Well-Known Member

    Messages:
    424
    Likes Received:
    33
    Best Answers:
    0
    Trophy Points:
    120
    #7
    Google ordinarily scans the robots.txt file before each crawl session, so its probably found your changes already. Keep in mind that the reports in the Webmaster Tools console are often 1-2 weeks out of date. I'm sure if you request a new crawl of your sitemap through the console, it will update properly within a few hours.
     
    rainborick, Jan 27, 2009 IP
  8. PaddyL

    PaddyL Active Member

    Messages:
    69
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    50
    #8
    Thanks for the reassurance.

    I've just checked, and you're right... Google has indeed re-indexed robots.txt, and yes, the Webmaster Tools area says that it hasn't!

    I'm curious as to how long Google will take to catch up.
     
    PaddyL, Jan 28, 2009 IP