1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt

Discussion in 'robots.txt' started by Angelabailey, Nov 15, 2013.

  1. #1
    I have problems when i try to fetch my site with google webmaster tools.
    I created a website on webs.com.
    Need some help..
    SEMrush
     
    Angelabailey, Nov 15, 2013 IP
    SEMrush
  2. danasurvey

    danasurvey Well-Known Member

    Messages:
    1,930
    Likes Received:
    79
    Best Answers:
    0
    Trophy Points:
    138
    #2
    Has your site been indexed by Google yet? What is the specific issue or error you are running into?
     
    danasurvey, Nov 15, 2013 IP
  3. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,985
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #3
    ryan_uk, Nov 15, 2013 IP
  4. Angelabailey

    Angelabailey Greenhorn

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #4

    I don't know when i try to fetch, it say blocking by robots.txt .

    Thanks ''Ryan_uk'' but allready viewed that topic, and it's not helpful -_-.
     
    Last edited by a moderator: Jul 29, 2016
    Angelabailey, Nov 15, 2013 IP
  5. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,985
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #5
    I don't think you did or you would have done what the official rep mentions - to contact them if robots.txt Disallow: / is still there after seven days.
     
    ryan_uk, Nov 15, 2013 IP
  6. dtsystems

    dtsystems Greenhorn

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    did your code is verify by webmasters.
     
    dtsystems, Nov 20, 2013 IP
  7. greenoro0

    greenoro0 Active Member

    Messages:
    435
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    70
    #7
    I think you used this one Disallow: /

    give me your robots.txt url. So, i could fix it....
     
    Last edited by a moderator: Jul 29, 2016
    greenoro0, Nov 20, 2013 IP
  8. xegen

    xegen Member

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #8
    beauty can have a desire to help, I am an expert in this matter
     
    xegen, Nov 20, 2013 IP
  9. RobinInTexas

    RobinInTexas Active Member

    Messages:
    217
    Likes Received:
    14
    Best Answers:
    2
    Trophy Points:
    65
    #9
    Just create a robots.txt file that says
    User-Agent: Googlebot
    Allow: /
    User-Agent: *
    Disallow: /
    User-Agent: Googlebot
    Disallow: /privatstuff
    Code (markup):
    Add anything you don't want indexed to the fourth line or add more pairs of lines as Shown above
     
    RobinInTexas, Nov 20, 2013 IP
  10. evgenktulu

    evgenktulu Active Member

    Messages:
    85
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    50
    #10
    Why would you open a site to Google only?
     
    evgenktulu, Dec 6, 2013 IP
  11. RobinInTexas

    RobinInTexas Active Member

    Messages:
    217
    Likes Received:
    14
    Best Answers:
    2
    Trophy Points:
    65
    #11
    Looking back, I can't think of a good reason. For the OP's problem all needed is
    User-Agent: Googlebot
    Allow: /
    Code (markup):
     
    RobinInTexas, Dec 6, 2013 IP
  12. Ehteshamshaikh

    Ehteshamshaikh Active Member

    Messages:
    59
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    56
    #12
    Check out your robots.txt file and try to remove that link from robots.txt file and then fetch your websites link in Google Webmaster tools & I believe it will work.
     
    Ehteshamshaikh, Dec 25, 2013 IP
  13. greenoro0

    greenoro0 Active Member

    Messages:
    435
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    70
    #13
    I really appreciate your answer..........
     
    greenoro0, Feb 3, 2014 IP
  14. danish001

    danish001 Member

    Messages:
    593
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #14
    What is different between allow and disallow in robots.txt and can i make two different file in robots.txt one for allow and other one is for disallow?
    Also please give me example so that i can apply for my website.
     
    danish001, Oct 28, 2014 IP
  15. Ashwini Kumar

    Ashwini Kumar Greenhorn

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #15
    Check your robots.txt file. read about it first.
     
    Ashwini Kumar, Mar 11, 2016 IP
  16. Panda123

    Panda123 Active Member

    Messages:
    43
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    63
    #16
    User-agent: *
    Disallow: /example/
    Disallow: /unknown/

    This is the syntax I use on mine :)
     
    Panda123, Mar 27, 2016 IP