I have used a .htaccess file to rewrite my urls without .html extension. Does this mean that when I define which file I wish to exclude from the bots in my robots.txt I also write them without url? The reason I ask is I notice some robots.txt files I've looked at refer to directories like this: /folder and some like this /folder/ in my case /folder would refer to an actual file written without the extension and not a folder... Hop this is clear - my first post by the way - so Hi.
It is simpler for you to place your files which need to be excluded in one directory and mark the directory in your robots.txt like so: (assuming that the directory name is private) User-agent:* Disallow: /private
But if I have one file, say example.html, do I refer to this in my robots.txt as example or as example.html (In the browser address bar the user sees the file name without the extension due to the .htaccess rewrite.)