1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is there a way to restrict certain urls for sitemap ?

Discussion in 'Google Sitemaps' started by QubeSys Technologies, Mar 27, 2012.

  1. #1
    Hi, the automatic Sitemap generator is taking all the urls from our site, while we dont need a lot of it.

    How do we restrict it from getting added to the sitemap.xml ?


    And we are using http://www.freesitemapgenerator.com/ to generate the sitemap. So, it doesnt get auto refreshed, right ?

    Does that mean, in every 2-3 months, I need to regenerate the sitemap for our site and upload again ?
    SEMrush
     
    QubeSys Technologies, Mar 27, 2012 IP
    SEMrush
  2. Hiiiiitch

    Hiiiiitch Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You can't restrict it as it's an automated tool, but you can use Htaccess files to restrict Google from sending crawlers to them :)
     
    Hiiiiitch, Mar 27, 2012 IP
  3. QubeSys Technologies

    QubeSys Technologies Active Member

    Messages:
    204
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #3
    Okay, example please.

    And do you know a better sitemap generator, which would refresh automatically the sitemap.xml placed on the server ?
     
    QubeSys Technologies, Mar 27, 2012 IP
  4. ErikdVP

    ErikdVP Member

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #4
    The best one is: php sitemap ng from enarion.net
     
    ErikdVP, Mar 28, 2012 IP
  5. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #5
    Hi,

    Some sitemap tools allow you to:
    * Use some kind of exclude filters (e.g. both crawler "analysis filters" and sitemap "output filters")
    * Automate it through command line to e.g. scan + upload + ping e.g. each weekend automatically.


    You can always ask the developer of the sitemap tool you are using, maybe you missed an option or something.
     
    websitetools, Mar 28, 2012 IP
  6. vitamintsus

    vitamintsus Guest

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Htaccess files may certainly be helpful for you, yo u just need to upload your sitemep by using this commend RewriteRule .? http://www.yourdomain.com%{REQUEST_URI} [R=301,L]. this is htaccess file upload it by ftp.
     
    vitamintsus, Mar 29, 2012 IP
  7. gauthamvit2003

    gauthamvit2003 Peon

    Messages:
    151
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    There are many sitemaps both paid and free version. Paid version has the option to include/exclude certain pages, refresh rate etc whereas most of the free version adds all url in your site and very few have the refresh option. If you need to restrict SE bots use htaccess file and disallow particular page or else you can use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on the page/url you want to block.
     
    gauthamvit2003, Mar 30, 2012 IP
  8. QubeSys Technologies

    QubeSys Technologies Active Member

    Messages:
    204
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #8
    Is there a way to restrict direct download links from sitemap , .doc .pdf , etc ?
     
    QubeSys Technologies, Apr 2, 2012 IP
  9. monalikedare

    monalikedare Greenhorn

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #9
    If you go for paid version sitemap generator then you will get this option to restrict certain urls.
     
    monalikedare, Apr 5, 2012 IP
  10. biggerlol

    biggerlol Peon

    Messages:
    74
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    You can use :
    in robost.txt
     
    biggerlol, Jul 19, 2012 IP