Sorry if this is a dupe. thread I did not see my first post appear on the board. Ok I have done a search for this and have not been able to find anything but was wondering if someone knows anything about this. I have sites which run on PHP and use includes for certian parts of the site. I am going to add a Cool Links section with the links being placed in a seperate file and will be displayed on the site using the include function. These links will not be exchanges just sites that I think are cool and want to offer links to for my visitors. But I don't want Google or any other search engine to see them. So my question is if I use my robots.txt file to dissallow that one file what will be the result....from my guessing their are 3 possible outcomes 1. Googlebot will not follow those links but will index and follow the rest of the page. 2. Googlebot will not index those links or any page that has that file included on it. 3. Googlebot will index the page down to the point where the included file appears in the source code but will not spider any futher down the page. I know I could use the rel nofollow tag but I dont want the added weight on the page as I am very careful about my text to code ratio. Any thoughts....
If you add the filename to your robots.txt file, the Google will not even attempt to download it. This has to be the actual file on the address bar, not the file that is included, otherwise it will not work. Alternatively, you can add <meta name="robots" content="noindex, nofollow"> to the <head> of your code. This will tell Google (and other SEs) not to index the page and not to follow any links. More info here: www.robotstxt.org Cryo.
What I don't understand is why you want to use nofollow on links you think are cool. To "nofollow" links in blog comments and the like, well, why not, but for sites you appreciate and recommand ? I mean, if everybody do the same, this is end of the web... Sorry for the off topics, but I couldn't help me on this... Back to you question : Cryogenius is right : put your resulting php page in the robots.txt file and all bots respecting it will not see the page
I can see your point about the rel nofollow and while I generally have no problem giving a cool site a link I have to think of it from an SEO standpoint and I have reached the maximum number of outgoing link I want to give from my site......
Yes, I'm sure it's can be legitimate, but I can't help thinking that sometimes we do (myself included) think to much in terms of SEO resulting for instance in not passing value to sites we trust. Bad SEO, bad