Removing all links to known blacklisted domains

Discussion in 'Link Development' started by bgillingham, Sep 23, 2006.

  1. #1
    I have started to go through my set of nearly 18,000 distinct domain names that I link to from my site to remove the ones that are blacklisted according to whois information. I would be more interested in determining exactly what makes Google decide to not index a page for linking to SPAM or blacklisted sites. :confused:

    1) Should this even be done (with respect to being closer to 100% indexed by Google)?

    2) When a site is blacklisted according to whois lookup, is that essentially the same thing as a SPAM site - or are there cases where Google doesn't care when linking to a site blacklisted for a different reason?

    3) Is there a good way to look up a domain thru Google (their API perhaps) to determine whether or not the domain is blacklisted according to Google and shouldn't be linked to?
     
    bgillingham, Sep 23, 2006 IP
  2. Talker

    Talker Notable Member

    Messages:
    2,795
    Likes Received:
    108
    Best Answers:
    0
    Trophy Points:
    210
    #2
    Im looking for the same thing, i need to know what domains are or have been banned/ blacklisted.

    Sahel
     
    Talker, Sep 23, 2006 IP
  3. phealey

    phealey Well-Known Member

    Messages:
    186
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    138
    #3
    That would be a great tool if it could check all the links on your site for banned sites..

    Come on techies invent one!!!
     
    phealey, Sep 23, 2006 IP
  4. max pain

    max pain Notable Member

    Messages:
    2,179
    Likes Received:
    521
    Best Answers:
    0
    Trophy Points:
    260
    #4
    max pain, Sep 23, 2006 IP
  5. bgillingham

    bgillingham Peon

    Messages:
    109
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    The "Banned Website Checking Tool" that max pain refers to answers a few of my questions.... I think that that program would choke on my website... way too many links - it would get bogged down (my web has over 1,000,000 pages).

    I am more interested in checking a specific list of links - not ALL of my links.

    I don't want to remove more links than I have to. I like my content too much to throw a lot of it away - kinda like a pack rat in that sense.

    I am working on a program that takes my list of domains - and creates from that list a set of blacklisted domains. The program also will let you construct SQL commands so that you can effect data that refers to the blacklisted set of domains. I am still looking into other options for the program (integrating a XML API that returns whois info per domain - rather than web scraping).

    If anybody is interested in my program, PM me - it is still in BETA, and I could use a couple testers.

    I am still a bit uncertain what rule Google goes by... if a site is blacklisted, you shouldn't link to it - regardless of the blacklist status? Blacklisted is blacklisted is blacklisted.
     
    bgillingham, Sep 23, 2006 IP
  6. classifieds

    classifieds Sopchoppy Flash

    Messages:
    825
    Likes Received:
    51
    Best Answers:
    0
    Trophy Points:
    150
    #6
    The whois Db doesn't contain any information that can be used to determine if a site is "blacklisted." If you're not sure about the site you should add the rel="nofollow" attrubute to the anchor tag.
     
    classifieds, Sep 23, 2006 IP
  7. bgillingham

    bgillingham Peon

    Messages:
    109
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Under the Server Data section, for most pages - I can see a line like the following for domains that are blacklisted:
    Blacklist Status: Currently Listed

    For an example, http://whois.domaintools.com/duplibase.com

    Also, I would think that if my site contains any links that are blacklisted - whether or not there is the rel="nofollow" attribute, the page that contains that link will rank low - or not be indexed at all. Is my logic bad here, or does the "nofollow" avoid all problems with linking to blacklisted URLs?
     
    bgillingham, Sep 23, 2006 IP
  8. bgillingham

    bgillingham Peon

    Messages:
    109
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #8
    I just want to confirm that after a bit more research on the subject, the addition of rel="nofollow" to any link should avoid any contribution to PR or any effect.

    I guess that my question now is - whether or not any page of mine that I know displays a blacklisted link - should I just keep that page from getting indexed at all, or will the page get indexed and treated the same as a similar page on my web that has no blacklisted domains listed.
     
    bgillingham, Sep 28, 2006 IP