Is robots.txt available for all engines?

Discussion in 'robots.txt' started by z3nyt, Jan 19, 2009.

0
  1. #1
    Hi,

    I want to know if the robots.txt is available for all search engines?

    thx
     
    z3nyt, Jan 19, 2009 IP
  2. pageloadtime

    pageloadtime Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Yes, but some evil robots choose to ignore it.
     
    pageloadtime, Jan 19, 2009 IP
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    It would be better to use htaccess file to exclude those evil bots..:)
     
    manish.chauhan, Jan 20, 2009 IP
  4. udayns

    udayns Peon

    Messages:
    237
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    if you want to allow all pages index by all search engines than just upload a blank robots.txt file in root.
     
    udayns, Jan 28, 2009 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    OK....:cool: I have neither uploaded a blank robots file nor filled one, still my website's pages are indexed in google..!!
     
    manish.chauhan, Jan 28, 2009 IP
  6. SEOVICE

    SEOVICE Peon

    Messages:
    449
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I know that this is standard for Google,Yahoo and MSN. I expect that all others follow.
     
    SEOVICE, Feb 3, 2009 IP
  7. shailendra

    shailendra Peon

    Messages:
    1,225
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Only the robots of Google, Yahoo! and MSN follow it. Robots can ignore robots.txt file especially malware robots that scan the web for security vulnerabilities. Also, email harvesters used by spammers will pay no attention to the robots.txt file. If a bad robot is operating from single IP address, then its access can be blocked via server config or a network firewall. If copies of the robot operate from different IP address, then advanced firewall rules configuration can be used that automatically block access to IP addresses that make many connections; but it can hit good robots also
     
    shailendra, Feb 4, 2009 IP
  8. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #8
    Best way to block spammy robots is blocking their ips using htaccess. You can find the ips of bad robots from your website admin panel's traffic log..:)
     
    manish.chauhan, Feb 4, 2009 IP