1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Have robot.txt. but always 404

Discussion in 'robots.txt' started by j3r0m3, Mar 23, 2006.

  1. Sem-Advance

    Sem-Advance Notable Member

    Messages:
    6,179
    Likes Received:
    296
    Best Answers:
    0
    Trophy Points:
    230
    #21
    Ok Ryan

    Same home work

    check logs how many spider your site??

    Why don't more??
     
    Sem-Advance, Apr 2, 2006 IP
  2. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #22
    Try studying them, then Sem-Advance.

    A quote from robotstxt.org:

     
    ryan_uk, Apr 2, 2006 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #23
    Sem-Advance, what is the point of posting all these links to references on how to construct a good robots.txt file? None of them back up your claim that if you don't have one spiders will leave.
     
    minstrel, Apr 2, 2006 IP
  4. j3r0m3

    j3r0m3 Peon

    Messages:
    161
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #24
    erm, gents
    we seem to have 2 schools of thought over here.
    why not we leave it at that and not try to convince one another of the benefits , because it will continue til the cows come home.
     
    j3r0m3, Apr 2, 2006 IP
  5. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #25
    Because this isn't a matter of opinion, j3r0m3. This is a matter of Sem-Advance just plain being wrong.
     
    minstrel, Apr 2, 2006 IP
  6. mcfox

    mcfox Wind Maker

    Messages:
    7,526
    Likes Received:
    716
    Best Answers:
    0
    Trophy Points:
    360
    #26
    Have to agree. Having or not having a robots.txt file does not make a difference as to whether your site gets indexed or not and saying the robot will leave the site if you do not have one is 100% wrong.

    If you stipulate parts of your site you do not want robots to index then that's about as good as you can hope for and only some of them will obey.
     
    mcfox, Apr 2, 2006 IP
    ryan_uk likes this.
  7. shauner

    shauner Well-Known Member

    Messages:
    342
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    123
    #27
    I get several hits to robots.txt daily, which doesn't exist on any of my sites. So it shows a 404 hit on my stats page, oh well.

    But I still have Google, MSN and Yahoo crawl HUNDREDS of pages on each site daily. I don't see any point in spending the time creating a robots.txt file when I already get crawled thoroughly.
     
    shauner, Apr 3, 2006 IP
  8. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #28
    The point is, you don't need one to get crawled. It's only if you don't want some pages and/or directories indexed. Either in general or by a particular SE.

    For example, some people exclude Google's Image Bot as it's often unbeneficial (people look at the images and not the pages) and a waste of bandwidth.

    On the other hand, sitemaps might help in getting indexed by google, assuming the sitemap is submitted to google sitemaps and is compatible with google. Maybe this is what Sem-Advance is confused about. However, a sitemap is by no means essential.
     
    ryan_uk, Apr 3, 2006 IP