1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is there a query to search for websites that are"no index"?

Discussion in 'Google' started by Forsh, Nov 18, 2016.

  1. #1
    I am looking for a way to search websites with "no index" in their html, as those would be pages that Google doesn't crawl, right?
    SEMrush
     
    Forsh, Nov 18, 2016 IP
    SEMrush
  2. Miroslav Chodak

    Miroslav Chodak Member

    Messages:
    98
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    45
    #2
    If they are not in Google index, there is no way to search for them in Google. I'm not familiar with any tool that would crawl sites independently and then compare with Google index.
     
    Miroslav Chodak, Nov 23, 2016 IP
  3. ajparker

    ajparker Member

    Messages:
    90
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    48
    #3
    That would defeat the idea of noindex. That's a directive to keep Gogle from indexing it to begin with, Google can't search what it doesn't index.
     
    ajparker, Nov 23, 2016 IP
    sarahk likes this.
  4. Forsh

    Forsh Well-Known Member

    Messages:
    511
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    158
    #4
    Yes, but Google still shows Facebook pages and Facebook is nofollow, though this may just be the outer pages. Google sees this information already. It runs into "noindex" & "nofollow" tags, so it's safe to assume that there would be a way to only show pages that it hits and can't crawl.

    For instance, many forums are "noindex" and there are search engines that only crawl "nofollow" forums. I think Google would crawl "noindex" websites, but just don't show the pages in search results. Besides, not all bots and crawlers obey meta infomation, htaccess, and robots.txt. I have had 40+ Bingbots on a site before, because bots that spoof can be hard to track-down
     
    Forsh, Nov 24, 2016 IP
  5. BrijeshB

    BrijeshB Member

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #5
    I want to make things clear, there is not any meta tag which stops crawler from crawling the website. So even a page has meta nofollow and noindex tag, Google will crawl it. But as crawler sees nofollow, it stops crawling further pages which are linked from that page. Again which means, the pages which exclusively linked through that page won't be crawled. Otherwise it they are linked from other page, it will be crawled and indexed. And noindex directive suggest crawler to not index that particular page. So if you want to search it in Google, you won't find it for sure. For facebook, there is some mistake on your side. Facebook profile pages are not set to noindex. They are indexed and can be found on Google.
     
    BrijeshB, Dec 16, 2016 IP
  6. sweetlouise

    sweetlouise Well-Known Member

    Messages:
    1,858
    Likes Received:
    38
    Best Answers:
    0
    Trophy Points:
    165
    #6
    Yes, you can got to http://publicwww.com/ and search the term "noindex" it comes up with hundreds of thousands of results.
     
    sweetlouise, Dec 16, 2016 IP
  7. Mithchen

    Mithchen Member

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #7
    Is it possible that Google is indexing something that is already on a nofollow? As far as I believe that those pages were indexed only by the times the no follow tags were posted. Afterward Google crawler would've stopped crawling things further.
     
    Mithchen, Jan 12, 2017 IP