Blocked URL checking

Discussion in 'robots.txt' started by ntrdtr, Jun 11, 2013.

0
  1. #1
    Hi everybody please help me, How can I check which URL block by robots.text?
     
    ntrdtr, Jun 11, 2013 IP
  2. palsman

    palsman Active Member

    Messages:
    156
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    95
    #2
    give me the file to see what is containing?
     
    palsman, Jun 18, 2013 IP
  3. lovetowork

    lovetowork Active Member

    Messages:
    103
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    65
    #3
    check on Robot.txt file, which file path are mentioned as disallowed
     
    lovetowork, Jun 18, 2013 IP
  4. naveenjosh

    naveenjosh Active Member

    Messages:
    189
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    73
    #4
    You can check it in your Google webmaster tools account as well. It will tell you that how many of your web page have been blocked by robots.txt

    Hope it helps
    My best
    N
     
    naveenjosh, Jun 19, 2013 IP
  5. ntrdtr

    ntrdtr Member

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #5
    Which file do you want?
     
    ntrdtr, Jun 22, 2013 IP
  6. anuragswebstreet

    anuragswebstreet Member

    Messages:
    83
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #6
    Check your robots.txt file as www.yoursite.com/robots.txt and see what pages or folders are blocked.
    Or else go to your website's GWT interface and check it there.
     
    anuragswebstreet, Jun 24, 2013 IP
  7. palsman

    palsman Active Member

    Messages:
    156
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    95
    #7
    give me your domain and i will check the robots.txt
     
    palsman, Jun 26, 2013 IP
  8. ntrdtr

    ntrdtr Member

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #8
    Hi, I gave the robots.txt file, please see it and give me a advice.


    Sitemap: http://paydayloansonlinebusiness.com/sitemap.xml

    User-agent: Mediapartners-Google
    Allow: /

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Image
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /


    # global
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /trackback/
    Disallow: /feed/
    Disallow: /comments/
    Disallow: /category/*/*
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: */comments/
    Disallow: /*?
    Disallow: /date/
    Disallow: /author/
    Disallow: /archives/
    Allow: /wp-content/uploads/
     
    ntrdtr, Jul 10, 2013 IP
  9. ntrdtr

    ntrdtr Member

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #9
    Hi, please see the robots.txt file.
    Sitemap: http://paydayloansonlinebusiness.com/sitemap.xml

    User-agent: Mediapartners-Google
    Allow: /

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Image
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /


    # global
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /trackback/
    Disallow: /feed/
    Disallow: /comments/
    Disallow: /category/*/*
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: */comments/
    Disallow: /*?
    Disallow: /date/
    Disallow: /author/
    Disallow: /archives/
    Allow: /wp-content/uploads/
     
    ntrdtr, Jul 10, 2013 IP
  10. Kyle Stester

    Kyle Stester Greenhorn

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #10
    Thanks for sharing your ideas, now i can checked blocked URL as well.
     
    Kyle Stester, Jul 11, 2013 IP