Hello I have this question. Let's say I have a domain called: www.domainx.com Let's say that: - I have an htaccess rewrite (hidden redirect) in order to display the website stored under the folder /websitex whenever an http request gets www.domainx.com That is to say, if you go to www.domainx.com the url will not change. However, the content displayed in the browser will be really the one stored under the folder called "websitex". In other words, the browser displays the same content as if you had typed www.domainx.com/websitex - I do not want search engines to index the contents under the root folder of www.domainx.com. I just want them to index www.domainx.com and the contents under www.domainx.com/websitex Is there any way to achieve this through the use of a robots file? I would try something like this but I am not sure: User-Agent: * Disallow: / Allow: /websitex My concern is that I am not sure whether this would prevent www.domainx.com from being listed by search engines or not. Actually I would like www.domainx.com to be listed by search engines. Any idea? Thanks
I would like you to be more clear in your query. I assume that you want to index your main website domainx.com and want to block the folder domainx.com/websitex. If this is what your query is about then you can simply do it using following code: User-agent: * Allow: / Disallow: /websitex Otherwise, Use following code User-agent: * Allow: /websitex Disallow: /