robots.txt

Discussion in 'Search Engine Optimization' started by saqibkhan2311, Jun 11, 2010.

0
  1. #1
    i hav my blog site redirectedon blogger and in blogger webmaster it shows restricted by robots.txt 10 and i have posted only 10 posts what does it means?
    do please help me
     
    saqibkhan2311, Jun 11, 2010 IP
  2. ninja_va

    ninja_va Peon

    Messages:
    35
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    hun ????

    can you rephrase the question pls?
     
    ninja_va, Jun 11, 2010 IP
  3. jvfconsulting

    jvfconsulting Active Member

    Messages:
    1,089
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    90
    #3
    have an example for us? Your question is a bit confusing.
     
    jvfconsulting, Jun 11, 2010 IP
  4. jamescar

    jamescar Guest

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Can you please describe your problem with example, so i can help you.
     
    jamescar, Jun 11, 2010 IP
  5. saqibkhan2311

    saqibkhan2311 Peon

    Messages:
    28
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    i have my blog on blogger and using the googles webmasters tools. when i checked the crawl errors in the webmaster tools, it shows in the crawl errors that you have 10 pages restricted by robots.txt and i have posted only 10 pages what is robots.txt what should i do
     
    saqibkhan2311, Jun 11, 2010 IP
  6. oz198

    oz198 Member

    Messages:
    61
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #6
    go to webmaster tools. click on site configuration. then click on crawler access and see what it says in the little box where it says "text of www.yourdomain.com/robots.txt. See if it says something like this:

    User-Agent:*
    Disallow: /

    That would mean that you have a robots.txt file that is restricting access from googlebot and all other spiders. You probably have no need for a robots.txt file unless you have some parts of you site that you do not want google to crawl, such as the admin side of you site. If you want to block google from that you need to go to the robots.txt file that is on the root directory of your server and change the file to say this:

    User-Agent:*
    Disallow: /admin

    Or whatever you have you backend file saved as.

    Hope this helps. Once you change the file or delete it, the next time google crawls you site it will look for a robots.txt file and abide by whatever rules you have in place. If you have non, it will simply crawl everything it wants to.

    OZ
     
    oz198, Jun 11, 2010 IP