1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

URL restricted by robots.txt

Discussion in 'robots.txt' started by junrey24, Mar 28, 2010.

0
  1. #1
    sorry if I'm in the wrong thread. Just want to ask if it is all about program? or..

    why google robots restricted this URL?

    jun-luxury-watches.blogspot.com/search/label/The%20Luxury%20Watch%20Today

    please help me explain this.

    Thanks.
     
    junrey24, Mar 28, 2010 IP
  2. AirForce1

    AirForce1 Peon

    Messages:
    1,325
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I don't really understand your question but I think the link in our post is normally accessible.

    BTW, Google crawler would not restrict any urls, IMO. Of course, it might don't want to pick up some certain urls. :)

    Have a nice day,
     
    AirForce1, Mar 28, 2010 IP
  3. ing

    ing Well-Known Member

    Messages:
    500
    Likes Received:
    38
    Best Answers:
    0
    Trophy Points:
    195
    #3
    I think blogger automatically adds the /search or /label directories to the robots.txt file to avoid duplicate content.
    If you want these kinds of searches or results pages to also be indexed you should probably go with your own domain where you have more control.
    :)
     
    ing, Mar 28, 2010 IP
  4. Sxperm

    Sxperm Notable Member

    Messages:
    4,386
    Likes Received:
    142
    Best Answers:
    0
    Trophy Points:
    225
    #4
    Did you mean the report in Google Webmaster Tools? If so, most of the time you see this does mean that url has been marked as "nofollow" or "noindex" via robots.txt.
     
    Sxperm, Mar 28, 2010 IP
  5. Bompa

    Bompa Active Member

    Messages:
    461
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    58
    #5
    Bompa, Mar 28, 2010 IP
  6. JeddahBikers

    JeddahBikers Guest

    Messages:
    305
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Yes you can place the robot.txt in your root and allow the disallowed pages :)
     
    JeddahBikers, Mar 28, 2010 IP
  7. snaeem

    snaeem Member

    Messages:
    486
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #7
    you can use google webmaster to make ur robot.txt :) some ppl have really advanced robots.txt
     
    snaeem, Mar 28, 2010 IP
  8. desperateboy

    desperateboy Peon

    Messages:
    610
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    URL you mentioned may not be search engine friendly.
     
    desperateboy, Mar 28, 2010 IP
  9. junrey24

    junrey24 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    This one I mean bro...

    [​IMG]
     
    junrey24, Mar 29, 2010 IP
  10. junrey24

    junrey24 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    why? please tell me, what do I need to become an SE friendly?

    can you teach me how to become SE friendly?
     
    junrey24, Mar 29, 2010 IP
  11. junrey24

    junrey24 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    sorry I do not know how to do this..

    please teach me...thanks..
     
    junrey24, Mar 29, 2010 IP
  12. junrey24

    junrey24 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    junrey24, Mar 29, 2010 IP
  13. elmostreet

    elmostreet Peon

    Messages:
    80
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    I think it can never be unless you have to own your domain.
     
    elmostreet, Mar 29, 2010 IP
  14. magda

    magda Notable Member

    Messages:
    5,197
    Likes Received:
    315
    Best Answers:
    0
    Trophy Points:
    280
    #14
    Log-in to yout blogspot dashboard. Go to Settings > Basic
    Scroll down to Let search engines find your blog? and change the setting to 'Yes'
     
    magda, Mar 29, 2010 IP
  15. vagrant

    vagrant Peon

    Messages:
    2,284
    Likes Received:
    181
    Best Answers:
    0
    Trophy Points:
    0
    #15
    no wonder you are confused... with so many different answers from people who have answered but not read (or understood) the thread *sigh*

    blogspot ALWAYS sets robots txt not to index internal search pages within the blog, otherwise it would be never ending as there are millions of words you could search for within your blog.
    Your robots txt is NORMAL for a blog spot blog. there is nothing you need to change.

    Your posts will be index as normal. the only thing not indexed is the results of any internal search page made from the search box on your blog.

    In you case, most of those pages are from your "tag cloud" and blogspot does not enable those pages to be index by bots ... just the way blogspot works.
     
    vagrant, Mar 29, 2010 IP
  16. LookAtHerBeautifulFace

    LookAtHerBeautifulFace Peon

    Messages:
    171
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    Don't worry Pal, it's something usual for Blogger. The answer is simple: Blogger is not a Self-Hosted.

    It also happened to my Blog, even more than 1 URL restricted by robot.txt, I got 20 URL's restrictions. And everything is back to normal within 4 or 5 days and No Crawl Errors anymore. Next day, it will appears again and back to normal again. It's an Effect of Default robot.txt from Blogger, i.e:

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search


    You cannot change it. But if You want to make a test, Use following robot txt:

    User-agent: *
    Allow: /

    for your URL:
    jun-luxury-watches.blogspot.com/search/label/The%20Luxury%20Watch%20Today
     
    LookAtHerBeautifulFace, Mar 29, 2010 IP