my website is running on Wordpress platform and i want to block a page and all its child pages from my robots.txt file the format of the url is www.domain.com/sdr i want to remove the sdr and all its child pages, what is the syntax of the robots.txt file to disallow this page ?
User-Agent: Googlebot Disallow: /sdr/ Add the above lines to your robots.txt. This will block all the content and page inside the folder sdr.
robots.txt only disallows a section of your site from getting indexed by search engines. This is respected by search engines bots and they do not index those sections or pages. Usually, you specify bot name and then section. You can use * for all bots, for example : user-agent : * Disallow : /sdr/ But this does not block those pages from being viewed. Anybody can type in manually the address and access those pages. To block the pages or section of a site, you can use htaccess method or just block them from webhost interface (like cPanel).
Hi there, I hope I can help. Within your robot.txt form ad user-agent : * Disallow : /sdr.html (or what every extension you use) etc. This will dis-allow this particular page from being indexed and searched. Note: its best to have the files you want to hide from search engines within folders, then all you have to do is use: user-agent : * Disallow : /sdr/ And this will block everything within this folder. I hoped this has helped.