Do you think Robots.txt is necessary for Google?

Discussion in 'Google' started by Woodcs82, Jun 12, 2010.

0
  1. #1
    Do you think Robots.txt is necessary for Google?

    What is your take on this?

    -C

    :cool:
     
    Woodcs82, Jun 12, 2010 IP
  2. SkullTraill

    SkullTraill Well-Known Member

    Messages:
    270
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    125
    Digital Goods:
    1
    #2
    Yes, I believe it will be advantageous to have robots.txt
     
    SkullTraill, Jun 13, 2010 IP
  3. Shirley1874

    Shirley1874 Peon

    Messages:
    558
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yes, I think so. If robots.txt is missed or wrong, Google might not index your website.
     
    Shirley1874, Jun 13, 2010 IP
  4. smsinhindi

    smsinhindi Peon

    Messages:
    561
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I think it is necessary for website owner to prevent crawler from specific location to index.. I don't think it is a requirement for google.
     
    smsinhindi, Jun 13, 2010 IP
  5. Sxperm

    Sxperm Notable Member

    Messages:
    4,386
    Likes Received:
    142
    Best Answers:
    0
    Trophy Points:
    225
    #5
    It is not except you want to tell specific action to do with search engine bots such as Googlebots such as exclusion specific folder or file from being crawled by them. Search engine bots usually do their jobs by crawling websites both with and without robots.txt contained.
     
    Sxperm, Jun 13, 2010 IP
  6. KISS

    KISS Active Member

    Messages:
    135
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #6
    Why it should be necessary? If you want to block something then use it, if not, then do not.
     
    KISS, Jun 13, 2010 IP
  7. Woodcs82

    Woodcs82 Well-Known Member

    Messages:
    2,588
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    150
    #7
    Thanks for the clarification guys!

    -C

     
    Woodcs82, Jun 13, 2010 IP