how important is robots.txt file

Discussion in 'Search Engine Optimization' started by clouting, Feb 5, 2008.

0
  1. #1
    hi people,

    i have a games arcade site and have noticed that many pages are not being found (404) when google crawls my site. apparently i can set up a robots.txt file which will stop google crawling these particular pages.

    how beneficial would this be??

    and

    how easy is this file to create?

    thanks in advance
    clouting
     
    clouting, Feb 5, 2008 IP
  2. jayeshpaun

    jayeshpaun Peon

    Messages:
    83
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hi Dear

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

    Its a very Simple , First You need to create txt file Like ,
    http://www.mydomain.com/robots.txt

    And read Below link for next steps, it will Guide you properly Which You are trying to do .

    webconfs.com/what-is-robots-txt-article-12.php


    I Hope, this will Help You.

    Thanks
    Jay
     
    jayeshpaun, Feb 5, 2008 IP
  3. clouting

    clouting Guest

    Messages:
    120
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ok, cool.

    thanks a lot!!
     
    clouting, Feb 6, 2008 IP
  4. muiomuio

    muiomuio Active Member

    Messages:
    225
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    83
    #4
    Robots.txt helps you control access to your domain.
    It's important in many ways for controling search engine robots.

    Search information about them in order to create an effective file
     
    muiomuio, Feb 6, 2008 IP
  5. JWRmedia

    JWRmedia Banned

    Messages:
    499
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    0
    #5
    You can also create a custom 404 page which with links to the most popular areas of your site. The crawlers can follow these links.
     
    JWRmedia, Feb 6, 2008 IP