1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Error pages on my site - robots.txt ?

Discussion in 'robots.txt' started by bad_bob00, Sep 21, 2007.

  1. #1
    Hi there,
    SEMrush
    Just been looking at some of the 'Error pages' stats on my website and in a week there was 96 page views of my robots.txt file - and this is shown in the 'error pages' section.

    Does this mean there is something wrong?


    Thanks for any help.
    Matt
     
    bad_bob00, Sep 21, 2007 IP
    SEMrush
  2. bad_bob00

    bad_bob00 Active Member

    Messages:
    3,472
    Likes Received:
    56
    Best Answers:
    0
    Trophy Points:
    90
    #2
    Don't suppose anyone knows why this is happening? :(
     
    bad_bob00, Sep 22, 2007 IP
  3. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Does google say your robots.txt is fine in google webmasters?
     
    evera, Sep 26, 2007 IP
  4. bad_bob00

    bad_bob00 Active Member

    Messages:
    3,472
    Likes Received:
    56
    Best Answers:
    0
    Trophy Points:
    90
    #4
    Sorry but how can I check - i've got google webmasters set up etc, but how do I check my robots.txt file is okay?

    Thanks
     
    bad_bob00, Sep 29, 2007 IP
  5. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #5
    evera, Sep 29, 2007 IP
    bad_bob00 likes this.
  6. bad_bob00

    bad_bob00 Active Member

    Messages:
    3,472
    Likes Received:
    56
    Best Answers:
    0
    Trophy Points:
    90
    #6
    bad_bob00, Oct 1, 2007 IP
  7. mac

    mac Peon

    Messages:
    76
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #7
    If you don't have a robots.txt file that is OK. it is a file that search engines and other web bots look for when they visit a site and it tells them what they can and cannot look at URL wise (such as disallowing admin pages). If you are seeing your 'error pages' as the robots.txt being a 404 error, that means it is being found.

    You can create a blank robots file if you like to avoid seeing those 404 errors.
     
    mac, Oct 8, 2007 IP
    bad_bob00 likes this.
  8. Nucca_The_Warrior

    Nucca_The_Warrior Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Yes, just basically create the robots.txt file in your web directory folder. You can make it blank or start blocking some search engines and/or web bots in the file. If you do not know how to do that, i'm sure you will find some way in the forums here.
     
    Nucca_The_Warrior, Oct 22, 2007 IP