How do I block a particular file name via robots.txt file no matter where it appears ? So, I would like to block a file gallery.htm and I have this particular file in various folders in my website. For example, example.com/abc/gallery.htm example.com/xyz/gallery.htm It will take a lot of time, if I have to manually identify and block this file. I have tried the below code – Disallow: */gallery.htm However, it doesn’t work. Any suggestion regarding this would be great.
You can find the answer here http://www.robotstxt.org/robotstxt.html. You will have the complete understanding of how it works and how to allow and disallow the file or folder. Please note here, even if you block the file from robots.txt. It will take some time to disappear from the net as it has already been indexed by google, bing and all other bots. So i would suggest, wait for some time (could be weeks ). Alternative to that is you can block the url from being indexed in the webmaster console.