1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to check robots.txt on site?

Discussion in 'robots.txt' started by seoworld1234, Apr 21, 2008.

  1. #1
    If I would like to check whether a website has robots.txt, how would I check it?
     
    seoworld1234, Apr 21, 2008 IP
  2. pj1s

    pj1s Active Member

    Messages:
    337
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    60
    #2
    Just type domain.tld/robots.txt and tadaa...
     
    pj1s, Apr 21, 2008 IP
  3. ServerUnion

    ServerUnion Peon

    Messages:
    3,611
    Likes Received:
    296
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ServerUnion, Apr 21, 2008 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    Just check for the www.domain.com/robots.txt...
    If there is any robots.txt in the website..it can be accessed then...:)
     
    manish.chauhan, Apr 21, 2008 IP
  5. Marcy J.

    Marcy J. Member

    Messages:
    67
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #5
    I didn't know it's so simply
     
    Marcy J., Apr 27, 2008 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    Now you get it..so just check your robots.txt..:)
     
    manish.chauhan, Apr 27, 2008 IP
  7. raymond222

    raymond222 Active Member

    Messages:
    287
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    60
    #7
    I like the Firefox SEO plugin. It allows to check seo and robots.txt for any site you want. Pretty handy to me...
     
    raymond222, May 2, 2008 IP
  8. nopolicestategirl

    nopolicestategirl Peon

    Messages:
    76
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    nopolicestategirl, May 2, 2008 IP
  9. bbrez1

    bbrez1 Banned

    Messages:
    208
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

    It works likes this: a robot wants to visit a Web site URL, say domain.com/welcome.html. Before it does so, it firsts checks for domain.com.com/robots.txt.

    By default you don't have that file in there, you have to create it.

    Example:

    User-agent: *
    Disallow: /myfolder/

    Hope this helps, Cheers
     
    bbrez1, May 2, 2008 IP
  10. mrboomboom

    mrboomboom Peon

    Messages:
    27
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    whats the firefox seo plugin that can check for robot txt? having trouble finding it
     
    mrboomboom, May 7, 2009 IP
  11. compwizards

    compwizards Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    you can just goto submitexpress.com and click on meta tags analyzer. You can type in your domain name and it will check all your meta tags plus also check to see if you have a robots.txt file on your site. If you have a robot.txt file, it will also check to see if it has any errors on that file.
     
    compwizards, May 24, 2009 IP
  12. Sudoku-Master

    Sudoku-Master Peon

    Messages:
    54
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I can find on both domains the robots.txt... I couldn't post a direct link to it, because my account is to young...
     
    Sudoku-Master, May 29, 2009 IP
  13. cyber_demon12

    cyber_demon12 Active Member

    Messages:
    89
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    93
    #13
    It is directly after your domain. So it would be yourdomain.com/robots.txt

    And, it should show you what is in it. If it shows up blank, then you have nothing there. You can change it by editing it under public_html.

    I hope I helped.
     
    cyber_demon12, May 29, 2009 IP
  14. fahadrai

    fahadrai Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    what realy matters when robot.txt is not there?
    and if it is created,, what will be the advantage?
     
    fahadrai, Jun 3, 2009 IP
  15. Jhumon

    Jhumon Peon

    Messages:
    51
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Its Simple, yourdomin.com/robots.txt
     
    Jhumon, Jun 5, 2009 IP
  16. linkdealer

    linkdealer Active Member

    Messages:
    138
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    90
    #16
    Exactly...this is the simplest way to check
     
    linkdealer, Jun 8, 2009 IP
  17. Jectillplum

    Jectillplum Peon

    Messages:
    587
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #17
    agree with him do it like that.
     
    Jectillplum, Jul 20, 2009 IP
  18. mindi9

    mindi9 Member

    Messages:
    72
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #18
    mindi9, Jun 1, 2011 IP
  19. maverick.jones

    maverick.jones Greenhorn

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #19
    Hi

    Just check it out by domainname/robots.txt
     
    maverick.jones, Jun 6, 2011 IP
  20. shweta24

    shweta24 Peon

    Messages:
    41
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #20
    just type your domainname.com/robots.txt and enter, and u can track it on webmaster tools also..
     
    shweta24, Jun 6, 2011 IP