1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemap tools that work with include files

Discussion in 'Google Sitemaps' started by pher, Jun 12, 2006.

  1. #1
    I've run into a small problem. I've been trying out numerous sitemap tools (ones recommended by Google's page), and they all seem to choke at the same place. My directory structure is as such:

    / - root files (default.aspx, page1.aspx, page2.aspx, etc)
    /includes/ - top and bottom include files that contain all my repetitive code (top.aspx, bottom.aspx, calendar.aspx, etc)

    The utilities run fine on the root files. They seem to locate the include files just fine. However, all of them want to try and scan the includes directory seperately. I can't seem to exclude that directory. The problem is that my top include contains all my namespace declarations, style sheet references, and any javascript code that is used on all of my pages. When the sitemap utility hits calendar.aspx (one example), it bombs because it doesn't use those references. I cannot put those references on the site because calendar.aspx is an include file that is used on various pages (based upon user's preferences).

    I hope this was clear enough for you to understand. If anyone can suggest a utility, with the ability to filter our specific files/directories, it would be greatly appreciated. Ideally, I'd like one that is generated on-the-fly whenever the sitemap is requested. I believe my host supports php, though I have limited experience with that language. I'd prefer .NET if at all possible.

    Thanks in advance.
    SEMrush
     
    pher, Jun 12, 2006 IP
    SEMrush
  2. Pahrump Mike

    Pahrump Mike Life Is Good

    Messages:
    190
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    123
    #2
    I used the third party tool sitemap_gen.asp. It took me just a little while to get it to work, and I was able to exclude my includes and images directory.
     
    Pahrump Mike, Jun 12, 2006 IP
  3. pher

    pher Well-Known Member

    Messages:
    403
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    125
    #3
    Thanks. Working great for me with a few modifications. Added it to Google sitemaps a few mins ago, so we'll see if it works out for me.
     
    pher, Jun 13, 2006 IP
  4. MaxPowers

    MaxPowers Peon

    Messages:
    261
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #4
    AutoMapIt has directory exclusion and a manual override if that doesn't work. A 'normal' spider ought to catch on that your include directory isn't directly linked to unless they are using a regex-driven spider. True HTML parsing should be able to distinguish between 'links' and 'include' files.

    If all else fails, it sounds like your architecture is perfect to setup an actual server-level block on the includes directory. This could be used to stop all access to the includes directory unless the request originates from your root directory.
     
    MaxPowers, Jun 13, 2006 IP