Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

URL restricted by robots.txt

Discussion in 'Google Sitemaps' started by fade, Feb 8, 2007.

0
  1. #1
    URL Detail Last Calculated
    http://www.pokeraffiliated.com/blog/ URL restricted by robots.txt [?] Feb 3, 2007
    http://www.pokeraffiliated.com/blog/2007/01 URL restricted by robots.txt [?] Feb 4, 2007
    http://www.pokeraffiliated.com/blog/ni...g/putting-together-a-marketing-strategy URL restricted by robots.txt [?] Feb 6, 2007
    http://www.pokeraffiliated.com/blog/ni.../putting-together-a-marketing-strategy/ URL restricted by robots.txt [?] Feb 4, 2007
    http://www.pokeraffiliated.com/blog/po...-to-choose-the-right-affiliate-program/ URL restricted by robots.txt [?] Feb 5, 2007
    http://www.pokeraffiliated.com/blog/poker-hysterics/dutch-boyd-is-on-crack/ URL restricted by robots.txt [?] Feb 3, 2007
    http://www.pokeraffiliated.com/blog/po...ish-takes-a-bad-beat-this-is-hilarious/ URL restricted by robots.txt [?] Jan 26, 2007
    http://www.pokeraffiliated.com/blog/po...2-this-guy-is-the-biggest-donkey-alive/ URL restricted by robots.txt [?] Jan 27, 2007
    http://www.pokeraffiliated.com/blog/seo/backlinks URL restricted by robots.txt [?] Feb 6, 2007
    http://www.pokeraffiliated.com/blog/seo/backlinks/ URL restricted by robots.txt [?] Feb 3, 2007
    http://www.pokeraffiliated.com/blog/seo/why-content-matters-to-poker-affiliates/ URL restricted by robots.txt [?] Feb 5, 2007
    http://www.pokeraffiliated.com/blog/vi...inkbaiting-and-link-baiting-strategies/ URL restricted by robots.txt [?] Feb 4, 2007


    now is it because my site is under http://www.pokeraffiliated.com/blog and not using the main domain, or is it something wrong with my sitemap, i used a wordpress sitemap generator plugin, and my robots.txt is the basic
    User-agent: *
    Disallow:

    Im not sure why these errors are coming up, if anyone can help it would be appreciated.
    SEMrush
     
    fade, Feb 8, 2007 IP
    SEMrush
  2. hhheng

    hhheng Banned

    Messages:
    2,633
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I think you shall contact the sitemap generator provider, but I think this is funny.
     
    hhheng, Feb 9, 2007 IP
  3. earhost

    earhost Peon

    Messages:
    36
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    i have the same experience with this, but not all my page are restricted, usually only two pages, but next crawl, succeeded
     
    earhost, Feb 9, 2007 IP
  4. Corey Bryant

    Corey Bryant Texan at Heart

    Messages:
    1,127
    Likes Received:
    51
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You have 
    User-agent: *
Disallow:
    Code (markup):
    in your robots.txt file. From what I read once on Google's site add 
    User-agent: Mediapartners-Google* 
Disallow:
    Code (markup):
    to your robots.txt file to make them happy and that might fix the problem hopefully. You might check out How do I use the robots.txt file analysis tool for a bit more information
     
    Corey Bryant, Feb 10, 2007 IP
  5. Peace4all

    Peace4all Active Member

    Messages:
    180
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #5
    I made a through search on google and found solution that you may create a robots.txt file with the help of google webmaster tool like this one:

    User-agent: *
    Allow: /

    User-agent: Mediapartners-Google
    Allow: /

    User-agent: Googlebot
    Allow: /

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Image
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    and after downloading that file you required to upload that robots.txt file in your top level of the domain.
    That's fine.
     
    Peace4all, Dec 28, 2009 IP