Why is Google indexing our robots.txt file and showing it in search results?

Discussion in 'Search Engine Optimization' started by Knocks, Oct 6, 2019.

  1. #1
    For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below.

    Our robots.txt file is not linked from anywhere on the site and contains just the following:

    User-agent: *
    Crawl-delay: 5

    This only happens for some sites. Why is this happening and how do we stop it?

    [​IMG]

    Screenshot 1: Google Search console

    [​IMG]

    Screenshot 2: Google search results
     
    Knocks, Oct 6, 2019 IP
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    7,291
    Likes Received:
    1,698
    Best Answers:
    31
    Trophy Points:
    475
    #2
    qwikad.com, Oct 6, 2019 IP
    Knocks likes this.