Robot.txt or remove the page

Discussion in 'robots.txt' started by Mubashir Ahmad, Mar 21, 2017.

0
  1. #1
    A simple question. Why we need robot.txt while we can simply remove the page from the website?
     
    Mubashir Ahmad, Mar 21, 2017 IP
  2. zibolin

    zibolin Greenhorn

    Messages:
    53
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    8
    #2
    But if you remove page normal users (not bots) will not see it.
     
    zibolin, Mar 21, 2017 IP
  3. rajkumar mudaliyar

    rajkumar mudaliyar Member

    Messages:
    188
    Likes Received:
    7
    Best Answers:
    2
    Trophy Points:
    43
    #3
    rajkumar mudaliyar, Mar 22, 2017 IP
  4. Allen James

    Allen James Active Member

    Messages:
    120
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    53
    #4
    It's robots.txt :D
     
    Allen James, Mar 23, 2017 IP
  5. Omi Mazumder

    Omi Mazumder Member

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #5
    By using Robot.txt a website owners can control their website from web robot. Website owners can give instruction whose are allow and whose are disallow to visit any page or site.
     
    Omi Mazumder, Mar 29, 2017 IP
  6. Scriptechweb

    Scriptechweb Member

    Messages:
    53
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    33
    #6
    Its the best tool to disallow private directories/pages listing in Google. You can use it for privacy of your website. But make sure to hide those directories that you need to make private.
     
    Scriptechweb, Apr 5, 2017 IP
  7. Tutorials Feed

    Tutorials Feed Greenhorn

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #7
    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be crawled. Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results. So, simply use the Allow query with website sitemap link and remove the link of the page from sitemap you don't want to show.
     
    Tutorials Feed, Apr 19, 2017 IP