I want some page of my website is not crawled by google bot. Can i use nofollow tag for the link that directed to that page? That page i don't want to becrawled is only connected from one page (with nofollow tag). or i have to set disallow on the robots.txt? thank you EDIT: sorry, i have found the answer from http://forums.digitalpoint.com/showthread.php?t=1841012&highlight=crawl+nofollow Moderator, you can erase this thread.
prefer not to edit the robots.txt . No follow tag will not pass the PR so its important. Using noindex meta tag in the header of that page will tell all search engines not to index that page. so best way is using nofollow tag as well as noindex metatag. if you did anything wrong in robots.txt then it may create trouble for your entire site.
The google bot will follow nofollow links, but not count them in the link popularity/rank of the target page. In order for you to avoid having one of your pages crawled/ indexed at all by google or any of the other SE's, you need to specify as much in your robots.txt file. Hope that helps!
If your links have some confidential thing and for the security reason you could use robot.text that prevents your website's link indexing in search engine and google crawler skips that link (s). ------------------------------------------
Google bot will crawl nofollow links but it will not consider into ranking calculations. for example , google webmaster tools shows nofollow links from twitter profile page.
Google WILL NOT crawl a no follow page.. BUT! - If someone was to "say..." link to that page, and google followed that link, then it would index that page. Regards, M1chael
Just add the noindex tag in the page header that you don't want to have indexed. It's not rocket science people