Should we use robots.txt?

According to the Guidelines-If I do not use robot.txt in my site that Search engine will not crawl my site web pages. Some expert says that search engine crawl web pages automatically. Can anybody tell me the truth of robot.txt?

 

robot.txt helps search engine bots to look for folders or directory and whether to scan it or not.

 

I agree. Robots.txt restrict access to the site for Google crawlers.

 

If you do not use the robot.txt then it's like your all secure folders and files are crawl able by search engines and might be listed there too. So, we use robots.txt in order to let the search engines to crawl at specific areas instead of the whole directories of the site. Therefore, it also helps bot to crawl and list your site easily and faster because they are knowing what they have to crawl for and making listing through robots.txt.

 

Thanks for giving the knowledge of Robot.txt