1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robot.txt

Discussion in 'robots.txt' started by Abdullah adil, Feb 11, 2017.

  1. #1
    how to create robot.txt file and how can i install it in my website
    SEMrush
     
    Abdullah adil, Feb 11, 2017 IP
    SEMrush
  2. VictorWelfred

    VictorWelfred Greenhorn

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #2
    Robots.txt file is used to guide the search engines automatically to the page that you want it to search for and then index the page. Most sites also have the directories and files do not need to search engine robots visit. So create robots.txt file can help you in SEO.
    How to create a robots.txt file is very simple, you just open the notepad or notepad ++ insert the code and save the file as ".txt"

    For example my robots.txt file has the following format:
    User-agent: *

    Allow: /media/

    Disallow: /wp-admin/

    Disallow: /wp-includes/

    Sitemap: domain/sitemap.xml
    User-agent: used to identify bot of search engines.
    Allow: allows the bot to the directory, site.
    Disallow: blocking the bot folder, pages.
     
    VictorWelfred, Feb 12, 2017 IP
  3. Ankurgupta87

    Ankurgupta87 Greenhorn

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    A very simple way to do create a file is yoast SEO plugin. Since it is the best SEO plugin you must be having it on your website.
    Goto yoast --> Advanced tool--> robots.txt editor-> Click create a file. Simple :)
     
    Ankurgupta87, Feb 12, 2017 IP
  4. Allen James

    Allen James Active Member

    Messages:
    120
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    53
    #4
    robots.txt ;)
     
    Allen James, Feb 13, 2017 IP
  5. Real Estate Lawyer

    Real Estate Lawyer Greenhorn

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #5
    Search on google, or YouTube. There are so many tutorials on this topic..
     
    Real Estate Lawyer, Feb 14, 2017 IP
  6. saurabh1985

    saurabh1985 Active Member

    Messages:
    41
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #6
    The robots.txt file is to tell the search bots which files should and which should not be indexed by them. Most often it is used to specify the files which should not be indexed by search engines.

    In order to allow search bots to crawl and index the entire content of your website, you can add the following lines in your robots.txt file:

    User-agent: *
    Disallow:

    On the other hand, if you wish to disallow your website from being indexed entirely, you can use the lines below:

    User-agent: *
    Disallow: /

    For more advanced results you will need to understand the sections in the robots.txt file. The "User-agent:" line specifies for which bots the settings should be valid. You can use "*" as a value to create the rule for all search bots or the name of the bot you wish to make specific rules for.

    The "Disallow:" part defines the files and folders that should not be indexed by search engines. Each folder or file must be defined on a new line. For example, the lines below will tell all the search bots not to index the "private" and "security" folders in your public_html folder:

    User-agent: *
    Disallow: /private
    Disallow: /security
     
    saurabh1985, Feb 16, 2017 IP
  7. sarahk

    sarahk iTamer Staff

    Messages:
    24,129
    Likes Received:
    3,252
    Best Answers:
    92
    Trophy Points:
    615
    #7
    That does assume that the OP is using WordPress
     
    sarahk, Feb 16, 2017 IP