1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

importance of robots.txt

Discussion in 'robots.txt' started by kabali, Sep 27, 2016.

0
  1. #1
    i want to know the importance of robots.txt for a website
     
    kabali, Sep 27, 2016 IP
  2. theworldwinner

    theworldwinner Well-Known Member

    Messages:
    903
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    108
    #2
    If you want to restrict some of the pages of your website from appearing in Google search then write them down in the robots.txt and Google won't be able to index those.
     
    theworldwinner, Sep 27, 2016 IP
  3. BITNETHOST

    BITNETHOST Active Member

    Messages:
    53
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    98
    #3
    Got an admin area / or other secured area in your website and wish not to list it in Search engines like Google, Yahoo, MSN ? Just make a robot.txt and place the code to disallow .
     
    BITNETHOST, Feb 3, 2017 IP
  4. highphiverr

    highphiverr Greenhorn

    Messages:
    46
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    6
    #4
    robots.txt can block indexability to certain pages of your site so you should know what you're doing before you mess with it. also a good idea to check it if one of your plugins is blocking indexing of your pages.
     
    highphiverr, Feb 4, 2017 IP
  5. Raquib

    Raquib Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #5
    robots.text instructs the crawler.
     
    Raquib, Nov 19, 2020 IP
  6. MayurKawale

    MayurKawale Well-Known Member

    Messages:
    221
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    170
    #6
    Importance are follows,

    It gives you more control over search engines movement on your website. While a single accidental disallow instruction can cause Googlebot from crawling your entire site, there are some common cases where it can really be handy.

    • Prevents server overload.
    • Prevents sensitive information from getting exposed.
    • Prevents crawl budget from getting wasted.
    • Prevents crawling of duplicate content
    • Prevents indexing of unnecessary files on your website (e.g. images, video, PDFs).
    • Helps to keep sections of your website private (e.g. staging site).
    • Prevents crawling for internal search results pages.
    cheers !!! :p:D
     
    MayurKawale, Nov 24, 2020 IP