No-Follow Robots.txt - FRIENDLY sitemap generator

Discussion in 'Google Sitemaps' started by SEO_WatchDog, Apr 22, 2008.

0
  1. #1
    Hello, does anybody know here of a sitemap generator that would NOT harvest no-follow and robots.txt blocked links? Would be much appreciated!
     
    SEO_WatchDog, Apr 22, 2008 IP
  2. CommonDavid

    CommonDavid Peon

    Messages:
    440
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #2
    gsitecrawler you can chose to import it or not, but why wouldn't you want to block the robots.txt in your sitemap?
     
    CommonDavid, Apr 22, 2008 IP
  3. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #3
    The A1 Sitemap Generator program has "crawler filter" options for:

    download robots.txt
    obey robots.txt
    obey <meta> "robots" noindex
    obey <meta> "robots" nofollow
    obey <a> "rel" nofollow
     
    websitetools, Apr 22, 2008 IP