1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Blocked by robots in index status

Discussion in 'Google' started by RON6, Oct 16, 2013.

  1. #1
    When I go to google index-> index status, in webmasters tool, and click advanced, this is what I see:

    For all the last "fetches" I made, the numbers are not looking good.
    The last one was 3 days ago -
    "Added to index: 281
    Blocked by robots: 3395"
    !
    This might be the reason for blog not to appear higher in Google search, or to appear at all...
    Many of my posts were on page 1, but after the September 4th update I lost 90% of it.
    Total destruction.
    What can I do with the info above in order to get out of this situation?
    Can someone explain what this actually means? That 3395 blocked by robots? Do you think I have the means to fix it?
     
    RON6, Oct 16, 2013 IP
  2. admee

    admee Well-Known Member

    Messages:
    401
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    170
    #2
    I had same issue - top be honest have you checked you robots.txt file?
     
    admee, Oct 16, 2013 IP
  3. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #3
    So how did you overcome this?

    And yeah my robots.txt is fine, here -
    "
    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search
    Allow: /

    "
     
    RON6, Oct 16, 2013 IP
  4. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #4
    Firstly, the allow directive is unnecessary. I assume you meant the October 4 update. Penguin has nothing to do with your robots.txt. You're blocking /search and depending on how your site works that is potentially thousands of URLs. Audit your backlinks and read up on Penguin in order to understand your problem.
     
    ryan_uk, Oct 16, 2013 IP
  5. admee

    admee Well-Known Member

    Messages:
    401
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    170
    #5
    Yes remove "Disallow /search" thats a Huge issue - and then you may see increase in indexing and your site may not even be hit by Googles Zzzzzzzzzzzzzz boring update.

    Also I would use a bulkpinging service to give you an extra chance of your URLs being indexed by Google.
     
    admee, Oct 16, 2013 IP
  6. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #6
    Well that allow directive isn't really the problem right?
    And your assumption is incorrect. My site, like many others, got hit on September 4th. My page rank was completely destroyed that day.
    In the last few days there was another drop, which I tend to relate to the October 4th update, but that was of a much smaller scale than the September 4th one.
    September 4th- 6th, Total daily Impressions dropped from 18000 to 2000, Since October 4th it dropped again, to 1000 (so 15000 dropped to 2000 in 3 days and this month dropped to 1000).

    The thing is, I got a penalty at the end of May this year, and worked every post on my blog to remove anything "spammy".
    A month and a half later, the beginning of July, the penalty was lifted (Again, in like 2 days).
    At around August 20th my site sky rocketed and reached to an high of 22K daily impressions.

    The current Penalty was imposed at the 4th of September (again, not October) but this time, there are no signs of recovery what so ever.

    So what is the problem now? I didn't change anything. My back links are the same as always... Are YouTube videos back links forbidden?
    I don't think so. Maybe it's yahoo? Prior to September 4th I begun seeing more and more yahoo Q&A that lead to my blog, Might Google be punishing for that?


    The Disallow /search should prevent duplicates originating in my own site search, I don't think I should remove it. Did you try removing it in the past?

    And how would bulk pinging help?
     
    RON6, Oct 16, 2013 IP
  7. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #7
    You seem to understand robots.txt isn't the problem, so why did you mention it?

    I'm surprised you got hit so badly by Hummingbird. Take a look at your content - Hummingbird is designed to provide answers to searchers. Are your titles and content structured in the right way (as to pose the question and answer), or more focused on keyword ranking? Is your site niche-specific or quite broad?

    Have a read through this and it should give you a good starting point for finding any issues and fixing them.

    Keep in mind semantic markup and structured data. Write your content for humans, but try to help Google understand it "under-the-hood". ("Hello, on-page SEO.")

    Hummingbird is a bit different - for Panda and Penguin you need to wait for the next update in order to recover, whereas with Hummingbird you can recover at any time as long as you identify and fix the issues.
     
    ryan_uk, Oct 16, 2013 IP
  8. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #8
    Ok, so the September 4th drop is hummingbird.
    I've read a bit about it,and in accordance with it, I see nothing wrong with my content structure.
    I'll just have to try some stuff, but I don't think I'm close to nailing it.

    Thanks anyway.
     
    RON6, Oct 16, 2013 IP
  9. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #9
    Being objective of one's own site can be the most difficult thing. Ask a few friends/family members who know nothing of SEO and web design to give you feedback.
     
    ryan_uk, Oct 16, 2013 IP
  10. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #10
    What about some posts that are still ranked 1-5, while other posts, completely identical (structure wise), literally disappeared from the search? And basically I have to recover from both hummingbird and Penguin 2.1 (which dropped my site from "zero" exposure to less than zero).

    Well I'm gonna try and change the structure of my titles, I think there is something there.
     
    Last edited: Oct 16, 2013
    RON6, Oct 16, 2013 IP
  11. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #11
    Google has started to get more specific, so some pages might be impacted while others won't be. So, for example, you could have bad links just to three pages, so just those three pages dropped. Or only three of your pages have poorly written titles (or something about the main content), for example.

    So, that's a Good Thing (tm) if it's not a site-wide hit.

    If your major drop was due to Hummingbird, then don't think about Penguin too much right now, but rather on-site issues. Sometimes a site can just be hit by knock-on effect rather than an update directly. You'll have a better understanding of Penguin's impact after you've fixed the other issues.
     
    ryan_uk, Oct 16, 2013 IP
    Arick unirow likes this.
  12. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #12
    Well I have an idea regarding title structure, I'm about to change the titles of all 260 posts.
    I'll update here if there is any change in the coming weeks.

    Thanks Ryan.
     
    RON6, Oct 16, 2013 IP
    ryan_uk likes this.
  13. Arick unirow

    Arick unirow Acclaimed Member

    Messages:
    719
    Likes Received:
    298
    Best Answers:
    30
    Trophy Points:
    500
    #13
    I have an old site and it has 'higher rank' and trust from Google but rarely shown up in Search Engine for specific keywords.
    After changing the title, revising the content and adding more detailed information, the site got more than 2 times in term of traffic. It is now ranked well in many keywords. Not in first position but it is always in the first page.
    I do learn simple thing from this experience, It seems Google love my site but decided not to give the traffic because the site is not suitable for any keywords. Once I designed the site for specific and common keywords, the traffic keep flowing in. To design the site for specific or common keywords, jut make detailed research about the niche and make sure the content is good enough to satisfy both Search Engine and visitors. Sometimes, modifying the template/theme is necessary as we could get well structured content easily just by using well coded template.
    I am not good enough in term of SEO so my opinion could be wrong.
     
    Arick unirow, Oct 17, 2013 IP
    ryan_uk likes this.
  14. RON6

    RON6 Peon

    Messages:
    11
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    3
    #14
    Yeah, I've been there. I used to have amazing results, topping the 3 top places (!) for many keywords. 4 month of heaven until May 2013.
    Even that didn't stop me, I worked on every post and on July 6th got back on the horse. August was amazing (1700$ with some simple, not so annoying ads).
    And on September 4th I was struck again. So since May I had only two "painless" month...

    The main thing I'm doing this time is simply analyzing my competitors, those who are now topping the Google search, I did this for a whole day and for many keywords in different variations and I think I got an idea on how to handle this hummingbird issue.
    I already edited 110 posts out of 260.
     
    RON6, Oct 17, 2013 IP