1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemap, Links and Bots

Discussion in 'Google Sitemaps' started by jrbiz, Feb 26, 2017.

  1. #1
    If I have a website and I put up a new page which is not listed in the sitemap and no other page on the website (or anywhere on the web) has a link to it, can a Google bot or any other bot still find and index it? I would like this to be a private webpage that can only be found by inputting the specific URL and not via a search engine.
    SEMrush
     
    Solved! View solution.
    jrbiz, Feb 26, 2017 IP
    SEMrush
  2. #2
    Yes, it can still be found by robots. It can also still be found by robots even if you ban them via robots.txt, mess around with .htaccess, and so on. The only sane thing to do, to keep a page mostly hidden, is implementing a login-system, or at least only show the information if the request is coming from white-listed sources.
     
    PoPSiCLe, Feb 26, 2017 IP
  3. David Jones42

    David Jones42 Member

    Messages:
    192
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    35
    #3
    yes they found easily without submitting in index. but if you want keep private your page you can hiding by adding dis allow code in the robot.txt
     
    David Jones42, Feb 26, 2017 IP
  4. jrbiz

    jrbiz Illustrious Member

    Messages:
    3,832
    Likes Received:
    1,471
    Best Answers:
    1
    Trophy Points:
    420
    #4
    Thank the feedback. I mostly just don't want to be listed on Google, Yahoo, and Bing search engines so that the page does not show up in search results. Another bot stumbling into it is not a problem. Do the Google/Yahoo/Bing bots do a good job of paying attention to robot.txt?
     
    jrbiz, Feb 27, 2017 IP
  5. PoPSiCLe

    PoPSiCLe Illustrious Member

    Messages:
    4,622
    Likes Received:
    717
    Best Answers:
    151
    Trophy Points:
    420
    #5
    Yeah, they do
     
    PoPSiCLe, Feb 27, 2017 IP
  6. jrbiz

    jrbiz Illustrious Member

    Messages:
    3,832
    Likes Received:
    1,471
    Best Answers:
    1
    Trophy Points:
    420
    #6
    Thanks!
     
    jrbiz, Feb 27, 2017 IP
  7. billzo

    billzo Well-Known Member

    Messages:
    957
    Likes Received:
    276
    Best Answers:
    15
    Trophy Points:
    113
    #7
    Yes, Google can find it if you use any third-party Google service that it uses as a tracking vector including Google Analytics, Google Fonts, and any Javascript or other libraries hosted on Google services.

    If you don't want anybody or any bot to find it, put it in a password protected directory.
     
    billzo, Feb 27, 2017 IP
  8. jrbiz

    jrbiz Illustrious Member

    Messages:
    3,832
    Likes Received:
    1,471
    Best Answers:
    1
    Trophy Points:
    420
    #8
    Thanks!
     
    jrbiz, Feb 28, 2017 IP