1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Why is Google indexing our robots.txt file and showing it in search results?

Discussion in 'Search Engine Optimization' started by Knocks, Oct 6, 2019.

  1. #1
    For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below.

    Our robots.txt file is not linked from anywhere on the site and contains just the following:

    User-agent: *
    Crawl-delay: 5

    This only happens for some sites. Why is this happening and how do we stop it?

    [​IMG]
    SEMrush
    Screenshot 1: Google Search console

    [​IMG]

    Screenshot 2: Google search results
     
    Knocks, Oct 6, 2019 IP
    SEMrush
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    5,938
    Likes Received:
    1,099
    Best Answers:
    22
    Trophy Points:
    400
    #2
    qwikad.com, Oct 6, 2019 IP
    Knocks likes this.