Robots.txt

Discussion in 'robots.txt' started by Angelabailey, Nov 15, 2013.

  1. #1
    I have problems when i try to fetch my site with google webmaster tools.
    I created a website on webs.com.
    Need some help..
     
    Angelabailey, Nov 15, 2013 IP
  2. danasurvey

    danasurvey Well-Known Member

    Messages:
    1,975
    Likes Received:
    95
    Best Answers:
    0
    Trophy Points:
    138
    #2
    Has your site been indexed by Google yet? What is the specific issue or error you are running into?
     
    danasurvey, Nov 15, 2013 IP
  3. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #3
    ryan_uk, Nov 15, 2013 IP
  4. Angelabailey

    Angelabailey Greenhorn

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #4

    I don't know when i try to fetch, it say blocking by robots.txt .

    Thanks ''Ryan_uk'' but allready viewed that topic, and it's not helpful -_-.
     
    Last edited by a moderator: Jul 29, 2016
    Angelabailey, Nov 15, 2013 IP
  5. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #5
    I don't think you did or you would have done what the official rep mentions - to contact them if robots.txt Disallow: / is still there after seven days.
     
    ryan_uk, Nov 15, 2013 IP
  6. dtsystems

    dtsystems Greenhorn

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    did your code is verify by webmasters.
     
    dtsystems, Nov 20, 2013 IP
  7. greenoro0

    greenoro0 Active Member

    Messages:
    435
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    70
    #7
    I think you used this one Disallow: /

    give me your robots.txt url. So, i could fix it....
     
    Last edited by a moderator: Jul 29, 2016
    greenoro0, Nov 20, 2013 IP
  8. xegen

    xegen Member

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #8
    beauty can have a desire to help, I am an expert in this matter
     
    xegen, Nov 20, 2013 IP
  9. RobinInTexas

    RobinInTexas Active Member

    Messages:
    217
    Likes Received:
    14
    Best Answers:
    2
    Trophy Points:
    65
    #9
    Just create a robots.txt file that says
    User-Agent: Googlebot
    Allow: /
    User-Agent: *
    Disallow: /
    User-Agent: Googlebot
    Disallow: /privatstuff
    Code (markup):
    Add anything you don't want indexed to the fourth line or add more pairs of lines as Shown above
     
    RobinInTexas, Nov 20, 2013 IP
  10. evgenktulu

    evgenktulu Active Member

    Messages:
    85
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    50
    #10
    Why would you open a site to Google only?
     
    evgenktulu, Dec 6, 2013 IP
  11. RobinInTexas

    RobinInTexas Active Member

    Messages:
    217
    Likes Received:
    14
    Best Answers:
    2
    Trophy Points:
    65
    #11
    Looking back, I can't think of a good reason. For the OP's problem all needed is
    User-Agent: Googlebot
    Allow: /
    Code (markup):
     
    RobinInTexas, Dec 6, 2013 IP
  12. Ehteshamshaikh

    Ehteshamshaikh Active Member

    Messages:
    59
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    56
    #12
    Check out your robots.txt file and try to remove that link from robots.txt file and then fetch your websites link in Google Webmaster tools & I believe it will work.
     
    Ehteshamshaikh, Dec 25, 2013 IP
  13. greenoro0

    greenoro0 Active Member

    Messages:
    435
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    70
    #13
    I really appreciate your answer..........
     
    greenoro0, Feb 3, 2014 IP
  14. danish001

    danish001 Well-Known Member

    Messages:
    635
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    103
    #14
    What is different between allow and disallow in robots.txt and can i make two different file in robots.txt one for allow and other one is for disallow?
    Also please give me example so that i can apply for my website.
     
    danish001, Oct 28, 2014 IP
  15. Ashwini Kumar

    Ashwini Kumar Greenhorn

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #15
    Check your robots.txt file. read about it first.
     
    Ashwini Kumar, Mar 11, 2016 IP
  16. Panda123

    Panda123 Active Member

    Messages:
    43
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    63
    #16
    User-agent: *
    Disallow: /example/
    Disallow: /unknown/

    This is the syntax I use on mine :)
     
    Panda123, Mar 27, 2016 IP