1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Working with large server log files

Discussion in 'Products & Tools' started by Bernard, Mar 14, 2005.

0
  1. #1
    Recently, I had cause to dig into my server logs for a detailed analysis of Googlebot activity. Urchin stats doesn't let me see exactly when it "comes", what pages it "visits" and what result occurred. So, I had to download the server log for the month of February to see the activity during the last few days (it was a 16MB file uncompressed).

    Working with such a large file in Wordpad is awkward because it does not load the full file all at once. You have to keep scrolling down and scrolling down as it pages through the file. Excel didn't help either because it has a limit of ~64,000 records/lines that it will load.

    So, I built a quick, no frills utility to chop up large text files into smaller, more manageable text files. I figure it might be useful to some of you too, so I've packaged it up as freeware.

    http://www.seo-help.com/html-tools/file-splitter.html
     
    Bernard, Mar 14, 2005 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Might not help you, but for me, this is what I do... If your logs are on a *nix serer (and you have shell access), or you are running Mac OS X (BSD at it's core), you can use grep to quickly do it. I can spin through gigs of logs in seconds.

    For example:

    grep Googlebot [your log file name]
    Code (markup):
    Will spit out just lines containing "Googlebot".
     
    digitalpoint, Mar 14, 2005 IP
  3. Bernard

    Bernard Well-Known Member

    Messages:
    1,608
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #3
    Yea, I just have a lowly hosting account. In my recent, specific case, the logs for February were already archived and I had to uncompress it before I could dig into it. It's just as well that I did it manually - I noticed a few unrelated things that I needed to address.
     
    Bernard, Mar 14, 2005 IP
  4. Chrissicom

    Chrissicom Guest

    Messages:
    261
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Chrissicom, Mar 15, 2005 IP