1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt Please Check mine

Discussion in 'robots.txt' started by articleshop4all, Oct 30, 2010.

  1. #1
    This is my robots.txt

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content
    Disallow: /tag
    Disallow: /category
    Disallow: /author
    Disallow: /page/
    Allow: /

    But Google still crawling and indexing the tag. Please help me. I'm using all in one seo pack for my wordpress blog and i set noindex for tag and categories from the very begining. But don't know what's wrong with my blog.

    Above all i don't want to let google index my attatchment pages. Please suggest me a good robots.txt.

    Thanks
     
    articleshop4all, Oct 30, 2010 IP
  2. SolidServ

    SolidServ Peon

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Your robots file looks fine, have you considered blocking the googlebot with .htaccess?
     
    SolidServ, Oct 30, 2010 IP
  3. articleshop4all

    articleshop4all Member

    Messages:
    496
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    30
    #3
    no i didn't block google bot via .htaccess
     
    articleshop4all, Oct 30, 2010 IP
  4. AtomicPages

    AtomicPages Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Try this:

    
    User-agent: *
    Allow: /
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /tag/
    Disallow: /category/
    Disallow: /author/
    Disallow: /page/
    
    sitemap: http://www.example.com/sitemap.xml
    
    Code (markup):
    Note: the sitemap uri must be the absolute link to your sitemap and you also need to have a space between the sitemap and the last entry. You also should close all of the directories you wish to block as well with a "/".

    Oops, I forgot to mention that you can request that google take down a specific url that they have indexed. I believe you can do this through google webmaster
     
    AtomicPages, Oct 31, 2010 IP
  5. articleshop4all

    articleshop4all Member

    Messages:
    496
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    30
    #5
    Ya i know that and i already requested them to remove those URLs. Thanks for this Tips :) I've updated my Robots.txt Hope it will solve my problem :)
     
    articleshop4all, Oct 31, 2010 IP
  6. thenetspiders

    thenetspiders Active Member

    Messages:
    341
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #6
    try plugin for robots meta tag this will solve your problem
     
    thenetspiders, Nov 29, 2010 IP
  7. pbonweb

    pbonweb Peon

    Messages:
    172
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    sir a also disallow to my admin files but that also crawled by google is that take time to remove urls from google or it will do as google crawl
     
    pbonweb, Dec 9, 2010 IP
  8. patamski9v

    patamski9v Peon

    Messages:
    49
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    use no-index on your tag.
     
    patamski9v, Dec 20, 2010 IP
  9. ayushi infotech

    ayushi infotech Peon

    Messages:
    1,814
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #9
    where to use no-index. in which tag i have to use this. pls be explicative
     
    ayushi infotech, Dec 28, 2010 IP
  10. reihaan

    reihaan Member

    Messages:
    153
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #10
    You can change your blog links with rel="nofollow attribute then google will not crawl them..
    for example <a href="http://yourdomain.com/page" rel="nofollow" >Link</a>
    Hope this will work for you..
     
    reihaan, Dec 30, 2010 IP