Use or not to use robots.txt

Discussion in 'Search Engine Optimization' started by ronijs, Jan 17, 2008.

0
  1. #1
    If I want robots to index all my pages, I need to put robots.txt in main directory of website? is there some differece to rank site higher, when I have robots.txt file or not?
     
    ronijs, Jan 17, 2008 IP
  2. Web Gazelle

    Web Gazelle Well-Known Member

    Messages:
    3,590
    Likes Received:
    259
    Best Answers:
    0
    Trophy Points:
    155
    #2
    I include the robots.txt even if I am not going to disallow anything because search engine spiders will be looking for the file.
     
    Web Gazelle, Jan 17, 2008 IP
  3. pedring

    pedring Banned

    Messages:
    376
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    There is no problem with that... as long as you don't put any dis allow command...

    Allow all spiders to index everything:

    User-agent: *
    Disallow:
     
    pedring, Jan 17, 2008 IP
  4. Allinpoker.cc

    Allinpoker.cc Peon

    Messages:
    252
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Definetly use it. It will never hurt, and will sometimes help.
     
    Allinpoker.cc, Jan 17, 2008 IP
  5. Ace0

    Ace0 Active Member

    Messages:
    526
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    58
    #5
    I am using it and blocking some of my content from being indexed in order to avoid duplicate content...it depends on each site.
     
    Ace0, Jan 17, 2008 IP
  6. budhanes

    budhanes Peon

    Messages:
    245
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I use one and dissallow nothing (except a bad spider that found my sites) but otherwise, liked having one just to avoid the 404's. If you dont really want to get too involved with a robots.txt, just do as pedring said and move on. Good luck with your sites.
     
    budhanes, Jan 17, 2008 IP