Robots.txt help

Discussion in 'Search Engine Optimization' started by anandsapple, Aug 21, 2013.

  1. #1
    Block Google on all pages except homepage via robots.txt

    ....
    User-agent: googlebot
    Disallow: /*

    Will above mentioned code work?
     
    anandsapple, Aug 21, 2013 IP
  2. Helge Sverre

    Helge Sverre Prominent Member Affiliate Manager

    Messages:
    840
    Likes Received:
    99
    Best Answers:
    2
    Trophy Points:
    305
    Digital Goods:
    2
    #2
    Helge Sverre, Aug 22, 2013 IP
  3. anandsapple

    anandsapple Greenhorn

    Messages:
    64
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #3
    That will never work because i want to block only Google not all the robots.
     
    anandsapple, Aug 22, 2013 IP
  4. Helge Sverre

    Helge Sverre Prominent Member Affiliate Manager

    Messages:
    840
    Likes Received:
    99
    Best Answers:
    2
    Trophy Points:
    305
    Digital Goods:
    2
    #4
    Do
    User-Agent: googlebot
    Allow: /$
    Disallow: /
    Code (markup):
    then...
     
    Helge Sverre, Aug 22, 2013 IP
  5. anandsapple

    anandsapple Greenhorn

    Messages:
    64
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #5
    Allow: /$

    what does middle line denotes? I think $ denotes dynamic pages ...
     
    anandsapple, Aug 22, 2013 IP
  6. Helge Sverre

    Helge Sverre Prominent Member Affiliate Manager

    Messages:
    840
    Likes Received:
    99
    Best Answers:
    2
    Trophy Points:
    305
    Digital Goods:
    2
    #6
    Helge Sverre, Aug 22, 2013 IP
  7. Roger Pelt

    Roger Pelt Active Member

    Messages:
    121
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    53
    #7
    Yes, I do agree with Helge,

    $ designates the end of the URL so when you follow below code then you may block your entire website for Google bot except your home page.

    User-agent: googlebot
    Allow: /$
    Disallow: /*
    
    Code (markup):
    If you have configured your website in Google webmaster then follow this step to test your robots.txt file live

    • Sign in your Google Webmaster account
    • Click on Crawl from the left side
    • Select Blocked URLs
    • Here you can check your robots.txt file live
    I have tasted above code from Google webmaster tool and it's work
     
    Roger Pelt, Aug 22, 2013 IP
    Helge Sverre likes this.
  8. ManiQuadraincorp

    ManiQuadraincorp Active Member

    Messages:
    119
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    56
    #8
    User-agent: * Disallow: /
    *allow :/ method only follow this one
     
    ManiQuadraincorp, Sep 16, 2013 IP