Assume i have a site page .......abc.html and there are lots of inner links (inner page) can i disallow robot to index these url or keyword, ( for avoid keyword stuff) i m asking abt that because i dont want SE crawl or index my extra keyword. is there any method to hide these keyword frm SE crawlers to index them , can i add <rel="noindex"> in my anchor tag of particular url
another way is robots.txt using robots.txt, disallow the link which you don't want to index and crawler. but from this instruction google never go in that link. but if you want to give the permission to google to follow the link but not index then you are right? use <rel="noindex">
Only Google will comply with the 'nofollow' command. If you are concerned with this issue on other search engines, you will have to take further steps such as blocking the page with robots.txt
yes each SE has their own method and rules or commands which is used in robot.txt, so i wanted to SE robo do wnt to crawl my some content because if i add more content or keywords then it wil be mess(page) , so i dont know if i was wrong or right to ask this question but if u have any idea abt how to control robo or SE bot to avoid crawling on some content or tags. explain it. i think things like this is not existed in web market
You can stop SE bots from crawling a particular page of your site or a particular section (a whole folder) by using robots.txt The above robotx.txt will stop search engine bots from crawling any content under "images" folder. The next line will also stop them from crawling abc.html Most of the top SE bots(including MSN, Google & yahoo) will obey robots.txt But in rare cases if you got incoming links pointing from external sites to abc.html it might still get indexed. In such a case use google webmaster tools to let google know that you do not want abc.html to be indexed. -Neo
thanks u all to trying to help me out, i m aware abt this tactics, but my initial question is, can i stop any SE to crawl my <div> or any tags like this,. reply if interested
It is so simple, you can use any script language to bring specific content from server side. As you knw that Google can't read any scripting... Or you can use SeoGrr's suggestion about using iFrame. Both ways work. You can also encrypt your HTML coding and by using this encrypted content you can stop Google to get your content.