How to Block All Robots to Crawl/Index All html Pages within My Website

Discussion in 'robots.txt' started by vema123, Jul 9, 2019.

0
  1. #1
    How to block all robots from indexing/crawling all the html pages in my website. I can't block the folders because all those html pages are distributed in so many folders. So I only want all robots NOT to index/crawl all html pages within my website.
     
    vema123, Jul 9, 2019 IP
  2. vema123

    vema123 Active Member

    Messages:
    43
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #2
    Is this correct (to block all robots from crawling all html pages):

    User-agent: *
    Disallow: /.html

    Allow: /wp-admin/admin-ajax.php

    Sitemap: mydomain/sitemap.xml
     
    vema123, Jul 9, 2019 IP
  3. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    818
    Best Answers:
    7
    Trophy Points:
    320
    #3
    There are only three ways to block all bots. 1) password protect your pages, 2) block the bots on your server, or 3) do not put your pages on the public internet. Anything else that 'works' will block honorable robots but will not stop rogue robots. Since you specifically stated ALL, you will need to do #1 #2, or #3 above.
     
    mmerlinn, Jul 9, 2019 IP