1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is robots noindex nofollow is the reason my site has not been crawled by google?

Discussion in 'Google' started by expressonline, May 26, 2011.

  1. #1
    Due to some server issues, I have applied <meta name="robots" content="noindex, nofollow"> on my site's homepage and other internal pages. but now i want only google to crawl my site and rest crawlers should not.
    Do i need to remove this code? or Should I put another <meta name="googlebot" content="index, follow"> right below the code. I also know googlebot have the default behaviour to index and follow but does <meta name="robots" content="noindex, nofollow"> effects googlebot too?
    I am little confused. Your answer will definitely be appreciable.
    SEMrush
     
    expressonline, May 26, 2011 IP
    SEMrush
  2. socialmediamarketing01

    socialmediamarketing01 Peon

    Messages:
    427
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Please remove the code & write the tag as index & follow.
    In your Robot.txt file you can write the code as you are allowing the GoogleBot, not the other.
     
    socialmediamarketing01, May 26, 2011 IP
  3. imuncutno1

    imuncutno1 Well-Known Member

    Messages:
    599
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    170
    #3
    Hi there,
    Yes, I agree to remove " No index. No followed" to your homepage and change as mentioned above,
    By common sense, "No index, no followed " should be more applicable apply to Non-targeted landing page as like Private policy, terms of use, contact, disclaimer, the only exceptional is sitemap.
    I had experienced Google indexed these on my site when I used "Index, follow", this also because I have been unending learning new things for doing a better IM.

    However, I will come back to any of my sites if I have time for correcting towards this :)
    Have a great day
     
    imuncutno1, May 26, 2011 IP
  4. expressonline

    expressonline Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Thanks for your answer
    So, Should I inject <meta name="googlebot" content="index, follow"> instead of <meta name="robots" content="noindex, nofollow">. But, if I remove that code from home page then other crawlers will start making load on server. I just want to allow googlebot, no one else.
    Also, I have read elsewhere that googlebot have default behavior to index and follow. So is this necessary to put <meta name="googlebot" content="index, follow">.
     
    expressonline, May 26, 2011 IP
  5. F--K

    F--K Well-Known Member

    Messages:
    2,116
    Likes Received:
    51
    Best Answers:
    0
    Trophy Points:
    148
    #5
    Yes that was the reason - also add your website in webmaster tool that would boost up the whole process
     
    F--K, May 26, 2011 IP
  6. expressonline

    expressonline Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks for the responses.
    I have removed <meta name="robots" content="noindex, nofollow"> from the homepage and allowed googlebot to crawl by robots.txt. But by homepage did not crawled yet. :(
     
    expressonline, May 26, 2011 IP
  7. DomainerHelper

    DomainerHelper Well-Known Member

    Messages:
    445
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    100
    #7
    It takes time; it doesn't come when we want it to, it comes after finishing it's coffee and newspaper.
     
    DomainerHelper, May 26, 2011 IP
  8. saqib889

    saqib889 Peon

    Messages:
    1,291
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #8
    You should mate it follow so that google is allowed to crawl it
     
    saqib889, May 26, 2011 IP
  9. expressonline

    expressonline Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Many thanks to all!!!
    My home page gets crawled. :)
    Yes I know, even done that. Thank You :)

    NOW ... other bad bots are still crawling my site. I DONT want them to crawl. I know they do not follow robots.txt commands. (coz i hv done that). So, major issues are:
    # Load on server due to bad bots.
    # Some unnecessary bots like web18bot, Mozilla/5.0 are not following robots.txt.
    # Will blocking individual bots through htaccess be helpful? if yes, then please share the code for it?
     
    expressonline, May 27, 2011 IP
  10. virtuat

    virtuat Peon

    Messages:
    109
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    include your sitemap in robot.txt so all of your pages can be indexed by search engines.
     
    virtuat, May 27, 2011 IP
  11. bbrij871

    bbrij871 Peon

    Messages:
    37
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    You need to do index, follow for google bot and rest for others nofollow, no index.
     
    bbrij871, May 28, 2011 IP
  12. expressonline

    expressonline Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Now ... I do not have problem with Google. My major issue is to STOP bad bots, those are responsible of making unnecessary load on server.
    Thank you... But bad bots do not follow these instructions. :( So ... I am helpless :(
     
    expressonline, May 29, 2011 IP
  13. jyotikhatabi

    jyotikhatabi Member

    Messages:
    126
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    46
    #13
    Hii Friends

    You remove the tag on "No Index" No Follow" and use the <meta name="robots" content="index, follow" /> tag bcoz this tag help u index ur site page. They allow to Spider n Crawler to crawl the site.
     
    jyotikhatabi, May 29, 2011 IP
  14. websitetools

    websitetools Well-Known Member

    Messages:
    1,510
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #14
    If you want to restrict other crawlers... And they don't obey robots.txt you will have to *block* them with force in .htaccess. Either based on useragent string matching *or* IP address
     
    websitetools, May 30, 2011 IP
  15. john81

    john81 Member

    Messages:
    153
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #15
    You should be managing these things via robots.txt file
     
    john81, May 30, 2011 IP
  16. clive2

    clive2 Peon

    Messages:
    343
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    Yes, You should apply <meta name="googlebot" content="index, follow"> and remove <meta name="robots" content="noindex, nofollow" />. The Google works as per code. To block the other search engines, disallow that search engines through the robots.txt.
     
    clive2, May 31, 2011 IP
  17. DomainerHelper

    DomainerHelper Well-Known Member

    Messages:
    445
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    100
    #17
    Many many bots ignore robots meta tags and robots.txt. If you want not spidered, you had better pword protect your site or limit it by ip address(es).
     
    DomainerHelper, May 31, 2011 IP
  18. expressonline

    expressonline Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Hi Friends,
    Thanks for your valuable comments :) ... Now, I have removed the <meta name="robots" content="noindex, nofollow"> from homepage... coz server load issue has been resolved upto some instinct ... but bad bots still coming on server although intensity is low... Google is crawling and others are NOT (not exactly :( but still...).
    This was the best answer I was waiting for :) Thanks ... can I know how long these changes get effected?
     
    expressonline, Jun 7, 2011 IP
  19. emily

    emily Well-Known Member

    Messages:
    1,253
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    175
    #19
    Changes to .htaccess are picked up immediately...
     
    emily, Jun 7, 2011 IP