1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to slice and tag array data correctly

Discussion in 'JavaScript' started by ketting00, May 10, 2015.

  1. #1
    Hi guys,
    I'm working on a huge array. It's webGL and I store data as ArrayBuffer (typed array) for speedy reason (the internet says that, I have no idea), but it works just like a normal array. So my simple question is: I've an array of 142000 bytes length. I want to chop it off in four parts and reassemble them with other function. The webGL code is:
    
    function chopping_factory( meat ) {
    
        var meatball = new ArrayBuffer(meat.byteLength/4 + 1),
            tag = new Int8Array(meatball, 0, 1),
            roll = new Uint8Array(meatball, 1 * Int8Array.BYTES_PER_ELEMENT);
       
        var package_1 = meat.subarray(0, meat.byteLength/4),
            package_2 = meat.subarray(meat.byteLength/4, (meat.byteLength/4) * 2),
            package_3 = meat.subarray((meat.byteLength/4) * 2, (meat.byteLength/4) * 3),
            package_4 = meat.subarray((meat.byteLength/4) * 3, meat.byteLength);
       
        tag.set(tagging_id);
        roll.set(package_1);
        roll.set(package_2);
        roll.set(package_3);
        roll.set(package_4);
       
        var transporting = new reassemble_factory( meatball );
    }
    
    Code (markup):
    which I roughly translate it as (may be wrong):
    
        var meatball = new Array(meat.length/4 + 1),
            tag = meatball[0],
            roll = meatball.shift();
           
        var package_1 = meat.slice(0, meat.length/4),
            package_2 = meat.slice(meat.length/4, (meat.length/4) * 2),
            package_3 = meat.slice((meat.length/4) * 2, (meat.length/4) * 3),
            package_4 = meat.slice((meat.length/4) * 3, meat.length);
       
        tag.push(tagging_id);
        roll.push(package_1);
        roll.push(package_2);
        roll.push(package_3);
        roll.push(package_4);
       
        var transporting = new reassemble_factory(meatball); // transport in roll + tag package
    
    Code (markup):
    At the reassemble factory I reassemble the meat like this (translated):
    
    function reassemble_factory( package ) {
       
        var tag = package[0],
            roll = package.shift();
       
        if(!this.counter) { // initialize the variables
            this.data = new Array(roll.length * 4);
            this.counter = 0;
        }
         
        // append this input to the rest     
        this.data.push(roll);
        this.data.push.apply(roll, roll.length);
        this.data.push.apply(roll, roll.length);
        this.data.push.apply(roll, roll.length);
    
        this.counter++;
       
        if(this.counter == 4) {
            // do something with the output
            var output = this.data;
         
            // clear the data to repeat the cycle
            this.counter = 0;
        }
       
        var sell_this = new delivery( output );
    }
    
    Code (markup):
    However, when the salesman, woman, transgender, whatever, presents the product to customers. It looks like this:
    
    2   |  1
    ____|____
    1   |  2
    
    Code (markup):
    How do I make it look like this:
    
    1   |  2
    ____|____
    3   |  4
    
    Code (markup):
    Thanks for reading this long story and thanks for your help in advance;
    (you can change meat to chocolate ball if you like) :)
     
    ketting00, May 10, 2015 IP
  2. deathshadow

    deathshadow Acclaimed Member

    Messages:
    9,732
    Likes Received:
    1,998
    Best Answers:
    253
    Trophy Points:
    515
    #2
    My question would be WHY are you chopping it apart, WHY are you using so many variables for NOTHING, why are you re-assembling it as a regular array and not also arrayBuffer... I'd probably be using slice instead of subarray... and I'm not sure what that .apply method is but it looks uneccessary; but push only pushes one element not the element's contents so I'd assume that those are all wrong, the correct command for a typed array would be set.

    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/slice
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/set

    Note that roll.set is an OFFSET, so you are declaring none so the offset is 0 over and over again, so the array is overwriting itself...

    I'd also make damned sure to null all those array copies when you are done given the amount of memory doing that wastes... which again would be why I'm questioning even slicing them apart in the first place. Not quite sure what you are trying to accomplish with all that apart from making it painfully slow and chewing on memory for no good reason.
     
    deathshadow, May 13, 2015 IP
    ketting00 likes this.
  3. ketting00

    ketting00 Well-Known Member

    Messages:
    772
    Likes Received:
    27
    Best Answers:
    3
    Trophy Points:
    128
    #3
    Hi @deathshadow,
    It seems you've helped me save my hair again. Thanks you so much.

    Actually, I reassemble chunking arrays as ArrayBuffer. The problem was when I talk about this with people they do not understand it. I just make sure that I speak English not Italian when I talk about that (sound wired, isn't it). So I translate it to what they used to.

    I think that is it.... The problem.

    There goes another problem I guess.

    The reason why I slice array is it seems there's limitation in sending data over TCP/UDP. Sending large data seems to take something like four light-years to reach the destination.

    I found that slicing huge data and reassemble them take a mere ms. You don't even feel a thing when you get chopped. What you do is waiting for data to come in full, strip off tag and reassemble that. That's simple process.

    As for memory gulp, I'm not worry about that. My customer have an eight data-core server with 8GB memory, powerful enough to broadcast data to thousands of subscribers. :)
     
    Last edited: May 13, 2015
    ketting00, May 13, 2015 IP
  4. deathshadow

    deathshadow Acclaimed Member

    Messages:
    9,732
    Likes Received:
    1,998
    Best Answers:
    253
    Trophy Points:
    515
    #4
    Wait, is this server side (something like node.js) AND client side JS? That would explain a good deal.

    Usually sending as separate files should take LONGER due to handshakes, unless you've found a way to parallelize the requests, and for some users that could have an artificial result where it's fast on clean/open connections, but you go down to Panera Bread or McDonalds and leech their free wireless you'll be less than impressed. Actually used to be part of my testing cycle, and something I think more Dev's should do instead of trusting your magical home FIOS/better or bazillion dollar a month phone data plan! Get up, walk downtown to various local restaurants with free wireless and see how it performs there during lunch rush!

    Is that both client and server side code? (Not sure why you'd both break it apart and combine it again client side, which is likely a big chunk of any confusion explaining this to people) -- if so how are you transmitting it? Static includes? AJAX? Mix? Is it static content or generated?
     
    deathshadow, May 13, 2015 IP
  5. ketting00

    ketting00 Well-Known Member

    Messages:
    772
    Likes Received:
    27
    Best Answers:
    3
    Trophy Points:
    128
    #5
    It's client side. But yes, I do it at server as node.js enables me to do this. This actually is a Java app. The client want to go mobile and across devices. So it's where I come in.

    I've convinced this clients to move from Java to JavaScript in term of data transportation. (I was once coding Java I know how it communicates with JavaScript.) Since the data was hand-picked and operated by few staffs I think there will be no problem in mid term and near term.

    Everything is working fine with text data. I use TCP socket to communicate with Java app on a user's device and Websocket to communicate with browser.

    Browser is a marvelous technology. Chrome Beta for Android even receives data in sleep mode. It works pretty much like a native app, at lower power consumption rate.

    The problem was, the client want to go beyond text. He want to send image and graphic. Websocket seems to take dataframe not larger than 64k in data transmission.

    That was the problem and why I cut data in pieces. I search and searches. So far, information suggested that I'm the very first person try to doing this.

    Very good idea :)

    I'll see if I can make it happened soon enough.

    Cheer!
     
    Last edited: May 13, 2015
    ketting00, May 13, 2015 IP