How much Bandwidth are Search Engines taking from your site?

Discussion in 'All Other Search Engines' started by Gifts From Scotland, Aug 14, 2012.

?

In 3 months how much data is used by Spiders?

  1. Under 10GB (in 3months)

    50.0%
  2. 10GB to 20GB (in 3months)

    0 vote(s)
    0.0%
  3. 20GB to 30GB (in 3months)

    0 vote(s)
    0.0%
  4. 30GB to 40GB (in 3months)

    0 vote(s)
    0.0%
  5. 40GB to 50GB (in 3months)

    0 vote(s)
    0.0%
  6. 50GB to 99GB (in 3months)

    25.0%
  7. 100GB to 300GB (in 3months)

    25.0%
  8. Over 300GB (in 3months)

    0 vote(s)
    0.0%
  1. #1
    I am very curious to know from other webmasters or server/service providers what the numbers look like.

    My main focus is on splitting up the search engines to see which are most greedy and for what type of return.

    I'v done some simple log analysis so far but have yet to split the data by unique spider.

    Please leave a comment or two about your data usage with the big players and little guys alike.
    Also, please include if the bandwidth usage includes media like pictures sounds or videos! As this will massively affect the data.

    My attached data is for html/text content only ( taken from apache2 access.log using http://www.weblogexpert.com )

    Top_Spiders.png
    Chart from WebLogExpert for domain iluvscotland.co.uk over the space of 90 days from May to July 2012
    A truly massive amount of hits from Baidu Spider (even all throughout this year), for a search engine that doesn't index my site they sure do love to spider constantly.
    We can see Googlebot is the only bot really interested in fresh content.
    Where is the Yahoo bot? does the BingBot now replace yahoo's own bot?

    Lets see how these amounts of robot and spider access turns into actual visitors...

    Top_Search_Engines_i_luv_scotland.png
    Chart from WebLogExpert for domain iluvscotland.co.uk over the space of 90 days from May to July 2012 Showing nothing here we all wouldn't already guess, google dominates completely providing 33,000+ visitors while the others manage a few hundred or in the low thousands over the same amount of time.

    other handy info from the WebLogExpert is the summery page including Bandwidth uses by day week month etc.

    [TABLE="width: 100%"]
    [TR="class: TableSolidRow, bgcolor: #C8D6E5"]
    [TD="class: TableCell, colspan: 2"]Bandwidth[/TD]
    [/TR]
    [TR="class: TableRow1, bgcolor: #FFFFFF"]
    [TD="class: TableCell"]Total Bandwidth[/TD]
    [TD="class: TableCell"]141.13 GB[/TD]
    [/TR]
    [TR="class: TableRow2, bgcolor: #EEEEEE"]
    [TD="class: TableCell"]Visitor Bandwidth[/TD]
    [TD="class: TableCell"]65.69 GB[/TD]
    [/TR]
    [TR="class: TableRow1, bgcolor: #FFFFFF"]
    [TD="class: TableCell"]Spider Bandwidth[/TD]
    [TD="class: TableCell"]75.44 GB[/TD]
    [/TR]
    [TR="class: TableRow2, bgcolor: #EEEEEE"]
    [/TR]
    [/TABLE]


    While looking for programs to help me analyse my apache logs to get a clear view of what was going on i also came across http://www.apacheviewer.com/ though i preferred the program i used above in the end.

    Lets get your stats in here so we can see the bigger picture !
     
    Last edited: Aug 22, 2012
    Gifts From Scotland, Aug 14, 2012 IP
  2. OSSEO

    OSSEO Active Member

    Messages:
    1,430
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    53
    #2
    I did not know that how much Google search engine getting bandwidth from our website. useful post :)
     
    OSSEO, Aug 16, 2012 IP