Google Ignoring robots.txt and Sitemap?

Hi guys,

I run www.garyog.com which is a pretty popular website for a singer, anyway:
I have a sitemap file (www.garyog.com/sitemap.xml) which lists the page I want indexed. Basically I don't want the message boards to crop up, so I also have www.garyog.com/robots.txt which should block Google from everything in /boards/ although preferable I want the actual /boards/ to be listed.

Anyway, if you do a search for "site:garyog.com" you get results like:

Index of /boards/Themes/GaryOg/images
http://www.garyog.com/boards/Themes/GaryOg/images/

Index of /boards/Themes/default/images
http://www.garyog.com/boards/Themes/default/images/

It's also ignoring my Priority settings, for example my homepage is no longer #1 in the results, instead it has /boards/ (with the title: 'Untitled' wtf?)

Also I was wondering how long it takes for a site/page to be given it's PR (i've a good few links from 3-4 rated pages so I'm sure it shouldn't still be 0).

Thanks guys

 

does anyone know anything about this?

ta

 

It can take a while for changes to take effect, like months.

Your robots.txt will currently exclude '/boards/' as well as everything within it. You'd be better off disallowing '/boards/index.php' and '/boards/Themes/' instead. Use the functionality within Google Sitemaps to test your robots.txt file.

Priority doesn't have any influence on the page's positions in the SERPs, only how often it is spidered (amongst other factors).

PageRank is only updated every 3 or 4 months. Next update is expected later this month.

Cryo.

 

Google is pretty good with adhering to the rules laid out in the robot.txt file. Is your robot.txt file correct? I copped an example from http://www.robotstxt.org/orig.html#code:

# robots.txt for http://www.garyog.com/

User-agent: *
Disallow: /boards/ # don't spider this directory

M

 

It seems that the problem is with his robots.txt. There is a 500 http error when I tried to see your file TOSCS.

 

Replace /boards/* with /boards/index.php?* and see what happens.

Do you want the forum posts to be blocked as well, or just the index?

 

This thread is 18 months old...

 

Huh, I didn't even bother to look at the date of the thread. I just saw what appeared to be an "open support ticket" and replied to it (you'd be surprised at how many threads I skip on a given day).