Good facts about robot.txt file. I want to know more about robot file. where will i get to learn the codes..?
Robot.txt is the very beautiful concept . Advantages of robot.txt is using this we Prevent to consider page in our web site by google LIke admin page. If admin page consider by google and index in that .Hacker easily find our control panel and hack our site . This is the main advantage of Robot.txt.
In simple words robots.txt file is used to tell search engines ( or bots, or robots) what they can access on the website and limiting the access to our private data.
Exactly, you don't want certain parts of your website to be indexed into search engines for the whole world to see so you restrict them by using robots.txt
Private and personal part of site can be restricated so that google crawler cannot access those pages of the site.
Robots.txt file controls the crawlers to view contents of your website...so that if you want to hide some of the contents from and don't want it to be indexed it is done by the use of robots.txt file...