Hey there, i've searched the internet and asked on bunch of forums about this for a while and nobody seems to respond. This is the last place so i hope there will be generous people and answer me this question. Why when i type the robots.txt on this website (http://imustdateher.com/) i can't see the file, it redirects me to the home page but in the URL section stays what i've typed? Here it is you can see for your self: http://imustdateher.com/robots.txt Thanks in advance for this Cheers
Its a standard wordpress site so the file doesn't actually exist. Go into the admin settings and look at the bottom of the reading settings that box should be unchecked. Why you don't see the normal file though is unclear
Hey sarahk thanks for your reply. I know that that box is if you want to be indexed or not. The website i provided above is not mine i just wanted to know why and how is that done. The SEOquake firefox plug in says that there is a robots.txt file. Thanks
Aaaah, I don't know seoquake but they might be doing something funky like hiding it from human visitors and showing it to the bots. Try doing a test with curl to see what you get
Yep they might, but how do you mean to test it with curl, sorry but i am not so advanced in those stuff..
Well i can't see what the advantage is but definitely there is something.. + the same website has a pretty nice structure so google ranks the same very high. I see you are a web developer, can i have you on skype?
Your http://imustdateher.com/robots.txt page serves 200 server response, that's why all plugins say that the file exists. If it would return 404, the plugin would work correctly.