1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt Wordpress help

Discussion in 'robots.txt' started by tonyfloyd, Apr 11, 2008.

  1. #1
    Hi....

    This is what i have in my robots.txt file.....it was created with a plugin i am using....is this sufficient?...do i now need to add anything to this file?...confused....lost...etc...

    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://www.greekhomerecipes.com/sitemap.xml.gz
    # END XML-SITEMAP-PLUGIN

    thanks
    tony
     
    tonyfloyd, Apr 11, 2008 IP
  2. tonyfloyd

    tonyfloyd Well-Known Member

    Messages:
    1,230
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    140
    #2
    any help with this?

    is my robots.tst file ok as is?
     
    tonyfloyd, Apr 11, 2008 IP
  3. dewald

    dewald Peon

    Messages:
    13
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Add the following to your robots.txt. It will prevent spiders from spidering the WP code folders, and it will keep your feeds from being indexed, which helps with avoiding duplicate content.

    User-agent: *
    Disallow: /wp-
    User-agent: *
    Disallow: /search
    User-agent: *
    Disallow: /feed
    User-agent: *
    Disallow: /comments/feed
    User-agent: *
    Disallow: /feed/$
    User-agent: *
    Disallow: /*/feed/$
    User-agent: *
    Disallow: /*/feed/rss/$
    User-agent: *
    Disallow: /*/trackback/$
    User-agent: *
    Disallow: /*/*/feed/$
    User-agent: *
    Disallow: /*/*/feed/rss/$
    User-agent: *
    Disallow: /*/*/trackback/$
    User-agent: *
    Disallow: /*/*/*/feed/$
    User-agent: *
    Disallow: /*/*/*/feed/rss/$
    User-agent: *
    Disallow: /*/*/*/trackback/$
    Code (markup):
    Best regards,

    Dewald
     
    dewald, Apr 11, 2008 IP
  4. tonyfloyd

    tonyfloyd Well-Known Member

    Messages:
    1,230
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    140
    #4
    ok...thanks dewald.....

    this shows in my header currently.....<meta name="robots" content="follow, all" />.....do i need to delete this if i am creating a robots.txt file?

    alos...where do i put the above in this file....# BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://www.greekhomerecipes.com/sitemap.xml.gz
    # END XML-SITEMAP-PLUGIN

    in between the # Begin and # end?....or after all of this?
     
    tonyfloyd, Apr 11, 2008 IP
  5. tonyfloyd

    tonyfloyd Well-Known Member

    Messages:
    1,230
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    140
    #5
    any help on what to do with meta tag in header?....do i delete it if i add a robots.txt file?
     
    tonyfloyd, Apr 11, 2008 IP
  6. tonyfloyd

    tonyfloyd Well-Known Member

    Messages:
    1,230
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    140
    #6
    any help?

    dewald.....do i really need all those in my robots.txt file?
     
    tonyfloyd, Apr 13, 2008 IP
  7. justlikeit

    justlikeit Active Member

    Messages:
    200
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #7
    This is the robots.txtx from Johnchow.com :D
    And google's http://googleblog.blogspot.com/robots.txt

     
    justlikeit, Apr 25, 2008 IP
  8. masoud376

    masoud376 Peon

    Messages:
    4
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    the file robot.txt where bi chang ?
     
    masoud376, Apr 29, 2008 IP
  9. masoud376

    masoud376 Peon

    Messages:
    4
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    the cod :

    User-agent: *
    Disallow:

    to be all search anginier crawl and index the site file and imag
     
    masoud376, Apr 29, 2008 IP
  10. lateuk

    lateuk Active Member

    Messages:
    317
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    58
  11. Sara Tancredi

    Sara Tancredi Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0