I recently picked up a client who's distraught with the fact that their site isn't getting any search traffic. I wondered why, because the business is actually fairly large and well known within the community. Anyway, the first thing I check, of course, is the robots.txt and meta tags to see if the site is being prevented from being indexed. Sure enough, I found the noindex, nofollow, noarchive tag on every single page. Horrible! So, now I'm trying to put together a strategy to get these pages indexed, but I'm wondering how I should go about it. I know that they have some pages that may end up in the supplementals, so I'd like to use a slow "trickle" process at getting them all spidered. That way, I can check each page for uniqueness before I just let the flood gates open. What do you think?
Right, but they do still exist don't they? It's just harder for us to tell because Google no longer tags them as such.
Now that you've fixed the robots.txt, I think you should just wait patiently and do slow link building. Just submitting the site to a few directories and few social bookmarks(especially digg n stumbleupon) can get it indexed quite soon. Cheers..
Hi Henry, The robots.txt file was actually ok to begin with. It's the individual pages that have the the meta tags which prevent indexing. Every page has one of these <meta name="robots" content="noindex,nofollow,noarchive">. So, basically my strategy is to go through every one of those pages and add unique descriptions, unique titles, and unique content before I take the nofollow tags out. It's going to be a ton of work, but I'm just wondering if my strategy could be better or more efficient in a different way. Any thoughts?
Well, what i really think is along with adding unique desc, title and content you should also take the nofollow tags out. IMHO this would be more effective. I knw its goin to be a tough job but after all our job is to make it look easy to the clients.. Cheers...
Google is pretty good about figuring out which pages are most important and should make it into their index first based on your internal linking structure as well as external links. Drawing out the process is only going to prolong the process of getting their site indexed with little or no gains from an SEO perspective. If this is a transactional site or lead generation site, they are likely losing money every day they are not indexed. If their entire site has been blocked all this time, their crawl frequency is going to be crap unless they have a lot of inbound links. I would simply remove all of the <meta name="robots"> elements and let the chips fall where they may. I would start adding more and more content and building inbound links to up their crawl frequency as well.