Refinance - Bleach 149 . Bleach 150 - Xecuter 3 Mod Chip - Debt Help - Cheap Flights

PDA

View Full Version : Meta noindex OR robots.txt OR rel=nofollow


cellphone_store
May 5th 2005, 8:08 pm
I have a website and I noticed that some pages that I "disallow"ed in the robots.txt are showing up in Google, but without the title and cache.

What code would make Google or any SE not even list those pages in their SERP?

Robots.txt? (i found out this didn't work)
Meta noindex ?
rel="nofollow" ?

City2
May 5th 2005, 10:18 pm
meta no index works fine. Not sure about nofollow

dcristo
May 6th 2005, 2:44 am
Did you implement the robots.txt file previous to the pages getting indexed?

J.D.
May 6th 2005, 8:12 am
I have a website and I noticed that some pages that I "disallow"ed in the robots.txt are showing up in Google, but without the title and cache.

What code would make Google or any SE not even list those pages in their SERP?

Robots.txt? (i found out this didn't work)
Meta noindex ?
rel="nofollow" ?
Post your robots.txt file - it may be misconfigured (e.g. you have to use absolute paths, etc). If you believe that is not, you should contact Google and let them know about this problem.

J.D.

cellphone_store
May 6th 2005, 9:08 am
User-agent: *

Disallow: /account.php
Disallow: /conditions.php
Disallow: /contact_us.php
Disallow: /create_account.php
Disallow: /login.php
Disallow: /logoff.php
Disallow: /password_forgotten.php
Disallow: /privacy.php
Disallow: /shopping_cart.php

------------------
the robots.txt was implemented before google even found my website.

ServerUnion
May 6th 2005, 11:00 am
I do not believe you have to use the "/" before filenames, only directories...

J.D.
May 6th 2005, 12:33 pm
User-agent: *

Disallow: /account.php
I see nothing wrong with this file. I would contact Google and let them know about this. It might be useful for them if you actually include a few log lines that show how Google got robots.txt first and then ignored its instructions and requested the disallowed pages. Also, I believe that Googlebot retrieves robots.txt only once in a day (may be once in a few hours), so the request for this file may be quite far in the log file from the page request.

J.D.

J.D.
May 6th 2005, 12:35 pm
I do not believe you have to use the "/" before filenames, only directories...You have to use absolute paths in robots.txt. Relative paths (i.e. those that don't have a leading slash) are not allowed. There's no distinction between files and directories - bots simply match each line in robots.txt to the beginning of the actual path.

J.D.

ServerUnion
May 6th 2005, 1:26 pm
even after you ad the needed text to the file or robots.txt, it will take a while before the pages are removed from the listing....