1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Google is not indexing website anymore (check image)

Discussion in 'Google Sitemaps' started by rkempo, May 16, 2015.

  1. #1
    At first, i will thank you for this great community + support. This is my first issue i'd like to discuss with you.
    SEMrush
    2 months ago, google was indexing the community threads after 1/2 hours. 18500 urls in sitemap, 18400 indexed. So lets say, no problems.

    But the last month google was not indexing the new posted threads anymore, not even after a week. So i decided to rebuild my sitemap, and resubmit it on webmaster tools.

    After a week of resubmit, google indexed just 30% of the 18400 urls. And i can see in search results that Google is deleting a lot of posts. And when we create a new thread, it doesnt appear anymore on Google. I dont understand it, nothing changed, there is still the same cms and url structure

    website: http://forum.rkempo.nl/

    [​IMG]
     
    rkempo, May 16, 2015 IP
    SEMrush
  2. braulio

    braulio Active Member

    Messages:
    70
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    95
    #2
    Have you confirmed that you have 0 sitemap errors and 0 crawl errors ? These errors can keep your site from being crawled and eventually indexed.
     
    braulio, May 16, 2015 IP
  3. rkempo

    rkempo Greenhorn

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    Thank you for your reply Braulio. I have 300 crawl errors, 403 access errors (caused by rules in robots.txt). But i always have these errors. and 0 sitemap errors
     
    rkempo, May 16, 2015 IP
  4. braulio

    braulio Active Member

    Messages:
    70
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    95
    #4
    You need to drill down to the root cause of the errors and eliminate the root cause. Please post some of the most common errors so we may help you.
     
    braulio, May 16, 2015 IP
  5. rkempo

    rkempo Greenhorn

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #5
    Ok thank you.
    Right now, i have 2 kinds of crawl errors.
    [​IMG]

    1) Lot of Access denied errors (response code 403) for all profile urls like:
    http://forum.rkempo.nl/members/samir.17/ - I dont have these profile errors in sitemap & I blocked view access for guests (from xenforo cpanel).

    2) not found attachments errors like:
    http://forum.rkempo.nl/attachments/1755/

    this is my robots.txt file
    User-agent: *
    Disallow: /find-new/
    Disallow: /account/
    Disallow: /attachments/
    Disallow: /goto/
    Disallow: /posts/
    Disallow: /login/
    Disallow: /admin.php
    Allow: /
    
    Sitemap: http://forum.rkempo.nl/sitemap.php
    Code (markup):
    I have this forum in a sub directory, and this robots.txt is inside this map. Do i have to move it to the root?

    Today i received a message from Google:
    Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.
     
    rkempo, May 17, 2015 IP
  6. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #6
    Generally speaking, blocking Google from URLs you do not want indexed is a good idea, but will of course result in fewer URLs indexed. You may also look into if you have duplicate URLs and then use a mix of canonical and robots.txt to solve that as well. That way, Google will only spend time crawling your real content URLs meaning you may also get more fresh content crawled faster.
     
    websitetools, May 18, 2015 IP