I have 11 000 url blocked by Robots in Google Webmaster

Discussion in 'vBulletin' started by goy, May 30, 2011.

0
  1. #1
    I have 11 000 url blocked by Robots in Google Webmaster for my VB forum is that normal? What kind of URL should you blog in VB?
     
    goy, May 30, 2011 IP
  2. -Aw-

    -Aw- Active Member

    Messages:
    319
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    80
    #2
    What does your robots.txt file look like? Maybe you blocked a few things here and there.
     
    -Aw-, May 30, 2011 IP
  3. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #3
    It being normal or not just depends on what you are blocking with robots.txt...
     
    digitalpoint, May 30, 2011 IP
  4. goy

    goy Well-Known Member

    Messages:
    1,010
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    130
    #4
    I do not have a robot txt in my root folder? Is there is a setting in VB that blocks?

    I only have this 

    User-Agent: *
Disallow: /cgi-bin/
Disallow: /expert-advisor/admin/
Disallow: /thailand-directory/admin/
Disallow: /thailand-pictures/admin/
Disallow: /wordpress/wp-admin
Disallow: /wordpress/wp-includes
Disallow: /wordpress/wp-content/plugins
Disallow: /wordpress/wp-content/cache
Disallow: /wordpress/wp-content/themes
Disallow: /wordpress/trackback
Disallow: /wordpress/feed
Disallow: /wordpress/comments
Disallow: /trackback
Disallow: /generator
Disallow: /generator_new
Disallow: /feed
Disallow: /comments
#Disallow: /category/*/*
#Disallow: */trackback
#Disallow: */feed
#Disallow: */comments
#Disallow: /*?*
#Disallow: /*?
#Allow: /wordpress/wp-content/uploads
Allow: /threads/
Allow: /forums/
Allow: /archive/
Allow: *.*

# Google Image
User-agent: Googlebot-Image
#Disallow:
#Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
#Disallow:
#Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
#Disallow: /
    PHP:

    Blocked websites are like this by robots

    threads/4728-The-National-Rice-and-Farmers-2011?p=5098&viewfull=1

tags.php?tag=crack+down
    PHP:
     
    Last edited: May 30, 2011
    goy, May 30, 2011 IP
  5. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #5
    No... vB isn't going to block anything. If Google says they can't spider stuff due to robots.txt file, Google thinks you have one for some reason.
     
    digitalpoint, May 30, 2011 IP
  6. goy

    goy Well-Known Member

    Messages:
    1,010
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    130
    #6
    This is the only robot file i have in the root where the i have the VB i think. Should i modify this robot text to allow the last URL i show you? How should a VB robot txt look like?
     
    goy, May 30, 2011 IP