Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to add robots.txt file in my site?

Discussion in 'Search Engine Optimization' started by Jagdisg, Sep 13, 2013.

0
  1. #1
    Please suggest me the way of adding robots.txt file in my site.
    SEMrush
     
    Jagdisg, Sep 13, 2013 IP
    SEMrush
  2. alex06295

    alex06295 Active Member

    Messages:
    933
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    80
    #2
    For example, your robots.txt file would cointain:



    # robots.txt generated at http://www.mydomain.com
    User-agent: *
    Disallow:
    Disallow: /cgi-bin/


    __________
    it is a text file located in your root directory
     
    alex06295, Sep 13, 2013 IP
  3. proson

    proson Well-Known Member

    Messages:
    573
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    130
    #3
    create a .txt file on your computer and then upload to the server...
     
    proson, Sep 13, 2013 IP
  4. webdev007

    webdev007 Active Member

    Messages:
    1,037
    Likes Received:
    13
    Best Answers:
    3
    Trophy Points:
    88
    #4
    Put the file named 'robots.txt' on the root of your server. It will be automatically detected by the search engines.
     
    webdev007, Sep 13, 2013 IP
  5. goodstudy

    goodstudy Banned

    Messages:
    94
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    38
    #5
    try to copy some other website robots.txt
     
    goodstudy, Sep 13, 2013 IP
  6. hockeystick

    hockeystick Active Member

    Messages:
    95
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    58
    #6
    Best place to get robots.txt specifications is directly from Google itself. Easy to read, easy to use, great resource.

    Robots.txt Specifications - https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
     
    hockeystick, Sep 13, 2013 IP
  7. fare

    fare Active Member

    Messages:
    89
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    95
    #7
    You can create the file using notepad by copying the template from other website's robots.txt and edit it according to your own preference then save "robots.txt" as the file name.
    You can also use online tools to generate the robots.txt file. After you got the file, upload it to public_html directory.
     
    fare, Sep 13, 2013 IP
  8. hockeystick

    hockeystick Active Member

    Messages:
    95
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    58
    #8
    Copying someone else's robots.txt only works if your sites are similar and you know what you're doing. Use the above guideline and it will help you out immensely. You're probably as good doing a Google search or better than taking the risk of copying someone else's incorrectly created robots.
     
    hockeystick, Sep 13, 2013 IP
    xanth likes this.
  9. hockeystick

    hockeystick Active Member

    Messages:
    95
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    58
    #9
    It should have a sitemap in there as well.
     
    hockeystick, Sep 14, 2013 IP