Looking for WordPress robot.txt Guru

Discussion in 'robots.txt' started by squeaky, Jan 2, 2009.

0
  1. #1
    I am looking for a robots.txt Guru who know their way around WordPress.

    I want to make sure that I am not excluding things which will prevent getting inner Pagerank to my post pages, and that I am avoiding dupicate content.

    The site I need help with is at, http://www.madmouseblog.com

    Any suggestions would be helpful.

    Thanks,
     
    squeaky, Jan 2, 2009 IP
  2. justinlorder

    justinlorder Peon

    Messages:
    4,160
    Likes Received:
    61
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Are you looking for robot.txt Guru ?
    It should be robots.txt .
    It is quite easy to write and understand .
    I may help you . what is the problem ?
     
    justinlorder, Jan 2, 2009 IP
  3. sampathsl

    sampathsl Guest

    Messages:
    861
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Please post your problem with the robot.txt file.
     
    sampathsl, Jan 2, 2009 IP
  4. squeaky

    squeaky Well-Known Member

    Messages:
    371
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    108
    #4
    I have updated my post to reflect what my actual problem/question is.
     
    squeaky, Jan 2, 2009 IP
  5. squeaky

    squeaky Well-Known Member

    Messages:
    371
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    108
    #5
    Post updated, with type of problem.
     
    squeaky, Jan 2, 2009 IP
  6. FightRice

    FightRice Peon

    Messages:
    1,082
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    0
    #6 
    
User-agent:  Googlebot
 
# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
 
# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
 
# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: */trackback/
Disallow: */feed/
Disallow: /category/*
    Code (markup):
     
    FightRice, Jan 3, 2009 IP
  7. squeaky

    squeaky Well-Known Member

    Messages:
    371
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    108
    #7
    Thanks for the information, but your example won't validate. My robots.txt is a little more advanced, so it doesn't help me much. What I currently have may be a little over kill, but I would like to sort it out a bit.
     
    squeaky, Jan 9, 2009 IP
  8. badboys

    badboys Active Member

    Messages:
    246
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    58
    #8
    User-agent: *
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /cgi-bin/

    User-agent: Mediapartners-Google*
    Disallow:

    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://www.yoursite.com/sitemap.xml.gz
    # END XML-SITEMAP-PLUGIN

    This is my Valid and simple robot txt for my wordpress blog
     
    badboys, Jan 11, 2009 IP
  9. squeaky

    squeaky Well-Known Member

    Messages:
    371
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    108
    #9
    Thanks for your input. This is a very simple one as well, while it works I was hoping for something more.

    I am looking for a Guru, and I figured that he would go to my site and view my robots.txt file and just analyze it, telling me if there were any problems with my current one.


    Moderators - Please close thread.
     
    squeaky, Jan 11, 2009 IP
  10. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #10
    Pm me your website details, so that I can analyze your robots.txt file..:)
     
    manish.chauhan, Jan 20, 2009 IP