1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Very slow indexing/caching of fresh content. Thoughts?

Discussion in 'Search Engine Optimization' started by spycraft, Mar 2, 2013.

  1. #1
    Hello all,

    I wanted to start this thread and see if you can help me figure this one out.. I've been doing SEO for several years very successfully, but SEO is one of those industries that somehow manages to make even the "experts" wonder. So, here is the issue.

    I have managed 100s of blogs/sites. Through posting good (and unique) content and running SEO campaigns, all of them would get their new posts indexed in Google within minutes of publishing them. That's pretty standard for a well set up and linked blog.

    Now, I am managing a huge business directory type of website that also has a very active blog. The tens of thousands of directory listings are ALL indexed, and are rarely updated. The blog is updated every day with unique and very high quality content. While all the posts eventually get indexed, it takes almost a week for Google to index them.

    The blog pings several services (including Google) every time a new post is published, and I have an auto-updated sitemap submitted as well.

    Now, keep in mind that I am not asking how I can get those posts indexed - a simple answer would be to do some bookmarking when publishing a new post - but the question here is WHY does a small young blog with some links get indexed instantly, while a MEGA site with age + a powerful and ongoing linkbuilding campaign + a blog that gets updated regularly takes 4 to 10 days for its new posts to get indexed?

    Any thougths? Please share.

    Thanks
    spycraft, Mar 2, 2013 IP
  2. MattsBackpack

    MattsBackpack Greenhorn

    Messages:
    29
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    23
    #2
    Hey guys!

    This seems like a bit of an interesting one!

    My first thought was that it could just be down to the nature of the site in general. If it's quite a large site (which it sounds like this is) and the majority of the pages are static and are never really updated, then Google may have the site logged as one which doesn't get updated or changed that often. Although the blog is updated very regularly, what percentage of the sites pages does the blog make up? From reading over what you've said in the OP, it sounds like the blog would only make up less than 1% of the total pages of the site. This means that the vast majority of the site is static and never updated. Compare that to a small, everyday blog, where most of the pages are updated regularly.

    If G thinks that the site is largely static and not updated, they may be less inclined to crawl it regularly.

    Just my opinion anyway, I'd be interested to know what the rest of you guys think? Has anyone else experienced a similar situation?
    MattsBackpack, Mar 2, 2013 IP
  3. spycraft

    spycraft Member

    Messages:
    795
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    35
    #3
    Very interesting point. Thank you.

    Then again, isn't that what sitemaps and priorities are there for? I have clearly marked all the business listings with very low priority/frequency, the categories with slightly higher, the homepage which is the hub of it all with max, and the blog that's updated so regularly with max as well.

    Also, I only ping Google when a new post has been published with the URL of that post. I am telling Google about that new URL that has been created. I am not waiting for them to recrawl the site in order to find it (in which case, the delay would make sense). So, in essence, it seems as if Google is actually neglecting my pings for some reason.

    To answer your question though, yes, the blog barely makes up for 0.01% of the total pages.
    spycraft, Mar 2, 2013 IP
  4. MattsBackpack

    MattsBackpack Greenhorn

    Messages:
    29
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    23
    #4
    Hmm...yeah that's an interesting point regarding the priorities. Perhaps if G consider the site as a whole to be subject to relatively small or rare updates in this way, then they mark the site as a whole with a lower crawl priority than a site where most of the pages are updated regularly. This would mean that your pings are not prioritised as highly as that of a small scale niche blog site.
    MattsBackpack, Mar 2, 2013 IP
  5. spycraft

    spycraft Member

    Messages:
    795
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    35
    #5
    You could be right.. If so, then a point is to be made that for situations like this, maybe we should set up subdomains - many seem to believe that Google treats subdomains as separate sites...

    But still, I would like to hear from someone who has had this happen to them as well in order to confirm that this is the case.

    Also, I wonder is there is a workaround to it.
    spycraft, Mar 2, 2013 IP
  6. coloma21

    coloma21 Active Member

    Messages:
    300
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    80
    Digital Goods:
    1
    #6
    Well, sometimes the smallest details like shouldn't stress you out. After all, we can't predict everything about Google, nor should we try.

    The only thing that matters is that your website is getting indexed lol. But my only guess is that Google will index newer blogs quickly so that it can keep adding newer sites to its collection. It's very mysterious haha
    coloma21, Mar 2, 2013 IP
  7. Serp Creation

    Serp Creation Member

    Messages:
    240
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    43
    #7
    For fast indexing, what you can do is put a list of pinging sites in your WP dashboard, there is an option there, it would help in fast indexing, without any doubt.
    Serp Creation, Mar 3, 2013 IP
  8. MattsBackpack

    MattsBackpack Greenhorn

    Messages:
    29
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    23
    #8
    That doesn't really answer the question though does it? OP clearly states:
    "Now, keep in mind that I am not asking how I can get those posts indexed - a simple answer would be to do some bookmarking when publishing a new post - but the question here is WHY does a small young blog with some links get indexed instantly, while a MEGA site with age + a powerful and ongoing linkbuilding campaign + a blog that gets updated regularly takes 4 to 10 days for its new posts to get indexed?"
    He's not asking how to get a page crawled, he's asking for opinions on WHY the site is in the situation that it is.
    MattsBackpack, Mar 4, 2013 IP
  9. DIA_8497

    DIA_8497 Active Member

    Messages:
    302
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #9
    What i think is google always look for something new... keeps it's database upto date and also it's visitors upto date.
    Subdomains are definitely treated as different sites for eg: oberoihotels, it has sub domains for it's hotel in each state or country, they are ranked on their specific keywords.
    DIA_8497, Mar 4, 2013 IP
  10. alex06295

    alex06295 Active Member

    Messages:
    879
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    55
    #10
    If any site has good indexing rate then it has qualified for this, there are many factors in Google SEO, and it is very difficult to understand,

    What we can try to do is to know the ways how to get our pages indexed faster.. if any newly created site is indexed fast then there would be something, there may be unique content, domain name, hosting server, length of domain registration and so on..
    alex06295, Mar 4, 2013 IP
  11. SeoKungFu

    SeoKungFu Member

    Messages:
    206
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    30
    #11
    Guess what, I am coming here, because I have a similar problem, although with not a big site.
    The cache on the home page sits somewhere in the quite distant - in internet terms - time. Three weeks back at the moment, while there have been new content coming in. Just cannot see why...
    SeoKungFu, Mar 20, 2013 IP