Problem with Robots.txt

Hi all,

I manage The Movie Analyst, the url is www.themovieanalyst.com - a new blog on crisp movie reviews and looks like i've a serious problem at hand. There is currently no robots.txt file uploaded in the folder but when one goes to the following URL - http://www.themovieanalyst.com/robots.txt/, one sees the following text disallowing all bots.

User-agent: *
Disallow: /

Screenshot of the same is attached. I'm using the Ultimate Baseball Wordpress theme on my blog.

Can someone tell me what's happening here? How is it that even when there is no robots.txt file uploaded, something of this sort happens??

 

Maybe your host put that up, upload a blank one and see, a blank one would allow SE's

 

the attachment image got error so i cant check it
all they way there is some kind of plugins might do that u just have to upload robots.txt and it will be the default and by the way http://www.themovieanalyst.com/robots.txt/ thats where u got that robots but when u go to http://www.themovieanalyst.com/robots.txt u will find nothing there so just upload a robots.txt file and everything should be ok

 

i jus uploaded a robots.txt file and it now when i go to themovieanalyst.com/robots.txt, i see the one that i've uploaded and when i go to themovieanalyst.com/robots.txt/, there seems to be another one which disallows all the bots.. how is that possible?

 

am not sure how that happen as i told u might be a plugins u installed cuz that for u but google and other search engine read the /robots.txt file and dosnt read /robots.txt/ so u could check that by adding ur site to google webmaster and then go to robots.txt analyze and u will see witch robots.txt google read from ur site

 

just checked in Google Webmaster tools and as feared, Google Bot is being blocked

 

It's a problem with WordPress - chances are you've got your privacy settings turned on (to block Technorati and so forth). What you need to do is manually create a robots.txt file and upload it to your root directory.

(I know this for a fact because it happened to me with a site that hasn't even launched yet.)

 

Hi Dan,

I've checked on this and the privacy settings are turned off (it allows Technorati and the other bots).. In addition to all this, i've also tried disabling all the plugins, changing the theme etc but i continue to have the same problem.

 

Create new fresh robots.txt and upload that.
Google webmaster tool take some time to re-consider your new one. so i think you should try again..

 

Which is what I said earlier. Create a robots.txt file and put it in your root directory. It'll "over-write" the false positive that's blocking your site from the Googlebot.

It's unfortunately the only way to fix it.

 

Hi Prabhu,

I uploaded a different robots.txt more than a week back.. but the prob is there seems to be 2 robots.txt while i can see only one (the one that i have uploaded)

if u check - www.themovieanalyst.com/robots.txt gives u the one that i have uploaded and the same is reflected in Google Webmaster Tools.. but there's another robots.txt at www.themovieanalyst.com/robots.txt/.. this seems to have an adverse effect as i'm unable to see any linkbacks even though there are (there's one from www.techspikes.com and blog.shankarganesh.com as far as i know - there may be more).. i'm also not getting any hits from search engines - the blog is more than a month old and has close to 50 posts.. havent got even a hit till date..

 

Dan - there's already a robots.txt file in the root directory and unfortunately for me, there seem to be 2 of them simultaneously.. n hence causing havoc

 

You're not using the KB-Robots plugin by any chance are you?

 

nope.. initially i had uploaded a robots.txt that i manually created.. the one that is there currently was created by the XML Sitemap Generator plugin..

 

Can you link me to the download page for the plugin? I'd like to be able to take a look at it.

 

Here's the link for the sitemap that i've created

http://www.themovieanalyst.com/robots.txt

and here's the one that's appeared from nowhere..

http://www.themovieanalyst.com/robots.txt/

 

No no no, I meant the page where you downloaded the plugin you are using. In other words, where did you get it from?

 

It's a know fact that some host block bots this helps reduce bandwith! and saves them money.

It was mentioned on another forum

 

There are a couple of other blogs that I host with the same host and they are working fine! Its only this one that's causing me trouble