Yahoo Can't Crawl My Server!!!

Discussion in 'Site & Server Administration' started by twalters84, Feb 22, 2009.

0
  1. #1
    Greetings,

    I have a severe problem with my server. I noticed that Yahoo, Google, and other search engines are not indexing my website.

    Green Watch

    I wrote to Yahoo support about my problem and got the following message:

    Why would my server be behaving like this? I am on a dedicated server so nothing should be timing out.

    Please assist me with fixing this. Thanks for any support and information that could be used to help my situation.

    Sincerely,
    Travis Walters
     
    twalters84, Feb 22, 2009 IP
  2. gate2vn

    gate2vn Peon

    Messages:
    809
    Likes Received:
    33
    Best Answers:
    1
    Trophy Points:
    0
    #2
    Do you have any robots.txt file to prevent SE robots?
     
    gate2vn, Feb 23, 2009 IP
  3. twalters84

    twalters84 Peon

    Messages:
    514
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yeah there is a robots.txt file but its set to crawl the site:

    User-agent: *
    Disallow:
     
    twalters84, Feb 23, 2009 IP
  4. kailash

    kailash Well-Known Member

    Messages:
    1,248
    Likes Received:
    42
    Best Answers:
    0
    Trophy Points:
    190
    #4
    Do you have any firewall installed on your server? If yes, then make sure yahoo bots IP addresses are not blocked. You will get IP list from http://www.iplists.com/. Also, they have mentioned that they got 500 server error from your site. You should check your site logs for this.

    Kailash
     
    kailash, Mar 1, 2009 IP