1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

importance of robots.txt

Discussion in 'robots.txt' started by kabali, Sep 27, 2016.

  1. #1
    i want to know the importance of robots.txt for a website
    SEMrush
     
    kabali, Sep 27, 2016 IP
    SEMrush
  2. theworldwinner

    theworldwinner Active Member

    Messages:
    903
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    58
    #2
    If you want to restrict some of the pages of your website from appearing in Google search then write them down in the robots.txt and Google won't be able to index those.
     
    theworldwinner, Sep 27, 2016 IP
  3. BITNETHOST

    BITNETHOST Member

    Messages:
    43
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #3
    Got an admin area / or other secured area in your website and wish not to list it in Search engines like Google, Yahoo, MSN ? Just make a robot.txt and place the code to disallow .
     
    BITNETHOST, Feb 3, 2017 IP
  4. highphiverr

    highphiverr Greenhorn

    Messages:
    47
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    6
    #4
    robots.txt can block indexability to certain pages of your site so you should know what you're doing before you mess with it. also a good idea to check it if one of your plugins is blocking indexing of your pages.
     
    highphiverr, Feb 4, 2017 IP