Big problem with crawling, indexing and SERPS

Discussion in 'Search Engine Optimization' started by ringwraith, Jul 2, 2009.

  1. #1
    Hello.

    I have a very big site (8 years old, hundreds of thousands of links) that has the following problem.

    1. I publish an article
    2. Google comes and crawls the article very fast, in 30 minutes tops.
    3. The page is crawled and indexed but it doesn't get included in SERPS in the first 6-8 days after the first crawl. In thsi period of time, the page is recrawled and rechaced, but still doesn't usually get ranked.
    4. After this time, the page appears in SERPs, usually in pretty good positions.

    Here's an example (content is not in English):

    Article from 27.06: http://www.apropo.ro/cotidian/social/alcoolul-ucide-unul-din-25-de-oameni-la-nivel-global-4600786

    Indexed: http://www.google.ro/search?hl=ro&q...el-global-4600786&btnG=Căutare&meta=&aq=f&oq=

    But not in SERPS: http://www.google.ro/search?hl=ro&q...+la+nivel+global"&btnG=Căutare&meta=&aq=f&oq=

    Here's an older article that has no problem ranking: http://www.apropo.ro/cotidian/util/codul-galben-extins-pentru-aproape-toata-tara-4532691

    The things described before happen to every article on this page. Actually, they happen to every new thing published on any of the subdomains of the site and also for a couple of other sites hosted on the same IP.

    Like i said, this site and the others hosted on that IP are old and big. I know that's the way Google does stuff (indexes and then ranks), i'm not a SEO newbie by any means. But, that should not be the case for such big sites, when the page should appear in SERPS from the first moment it gets ranked.

    I have a couple of explanations about why does this happen.

    1. First of all, this site has a category that treats subjects like "love, sex and all" and ranks pretty well for a couple of explicit terms. Also, there's another site on the server with a similar case. These kind of subjects are like 5% of the content on the site, not more. Is is possible that Google would apply some kind of filter on this site because of this that would affect all the other sites on the same IP?

    2. The site doesn't have a very good link distribution. I mean, the main page has many many links, but deep pages have quite a few.

    3. This site used to not have very much unique content (with a proportion of around 30% unique - 70% cited and rewritten) until a year or so. Things have changed now, but could this problem be an effect from those days?

    4. Finally, the site hasn't got the best link structure. it's mod-rewritten but it's usually 2-4 directories deep and deep content is not so well linked from the home-page. Still, it's pretty OK, but i'm trying to follow every aspect here.

    In the end, i must say the pages have no problem ranking after these first 8-10 days. The site has tens of thousands on visitors daily from SE, so it's not banned or penalized or something.

    So, i'm asking you guys what do you think about this problem and what would you advise me to do in this case. I'm interested to make the pages appear in SERPS asap, cause the site has quite a big news part, and it's loosing tons of traffic by not ranking when those news are hot.

    Sry for the long post and thanks in advance for your answer. Pls keep them on topic though.
     
    ringwraith, Jul 2, 2009 IP
  2. lindamood1

    lindamood1 Active Member

    Messages:
    1,705
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    78
    #2
    your each page and post are interlink with webpages, have u check xml sitemap in multiple section, have u got high pr backlink for every pages.
     
    lindamood1, Jul 2, 2009 IP
  3. submitinme

    submitinme Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Do you have "Metakeyword" It will help the crawlers to identify your site fast.Reciprocal link building will help to get good SERP
     
    submitinme, Jul 2, 2009 IP
  4. ringwraith

    ringwraith Peon

    Messages:
    436
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #4
    come on guys, you can do better than this. I really need good advices here :)
     
    ringwraith, Jul 6, 2009 IP
  5. ~kev~

    ~kev~ Well-Known Member

    Messages:
    2,866
    Likes Received:
    194
    Best Answers:
    0
    Trophy Points:
    110
    #5
    When your site has tens or hundreds of thousands of pages, its normal for a few to be missing from googles index. The larger the site, the more pages that will be missing.

    Take a site like digitalpoint. Its been online for years, has 1,340,643 threads, 11,282,120 post but only has 713,000 pages listed with google. Even if just look at the number of threads. That means that google is missing about 600,000 digitalpoint threads.

    My own personal forum is missing about 600,000 - 650,000 urls from googles index.

    The behavior of your site looks normal. Google indexes fast, but may lose the page later - its a hit an miss kind of thing.

    Even with submitting a sitemap, there is no promise that "every" URL will be indexed. In my google webmaster tools, it says that google has indexed 180,000 urls. But that is out of 800,000 urls in my forum. So where are the other 620,000 urls? I dont know.

    The one thing that I do know, is that google is not perfect. Its a program written by people, and its not going to be perfect.

    The one thing that I do recommend is to use a site map that "pings" the search engines when you update your page. Instead of waiting for the search engines to find your content, they are notified of changes as soon as their made.
     
    ~kev~, Jul 6, 2009 IP
  6. sampath1

    sampath1 Well-Known Member

    Messages:
    151
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #6
    In my experience googe drop PR and some back links too in last few weeks so in this matter I hope same thing happening to you too.
     
    sampath1, Jul 6, 2009 IP