Lazy ArrayBuffer concatenation

The bl module in Node.js-land is handy because it provides a way to handle sequences of buffers as if they were a single buffer. It would be helpful to have this at the language level, and be able to use sequences of ArrayBuffers wherever ArrayBuffers are currently usable. This would prevent not-necessarily-needed copy operations, hopefully speeding up some classes of programs that do a lot of data processing.

I've sketched this out a bit here, introducing a class called ArrayBufferList, usable wherever ArrayBuffers are usable today. I also mentioned some alternative approaches.

What do folks here think?

Have you considered ArrayBuffer.join([...buffers])?

Also, anything that doesn't resolve it to a single memory chunk is going to be inefficient to iterate - you can't just look up the value from memory.

You're conflating iteration with random access. Efficient iteration of a chain of iterables is pretty common.

Besides, my hunch is that the target use case is more like a queue. You append incoming data to the end of the list, while consuming from the head of the list. Joining incoming chunks for that sounds like unnecessary overhead.

1 Like

Since the size of an ArrayBuffer is fixed at creation, might skip lists provide efficient-ish random access?

2 Likes

You're conflating iteration with random access. Efficient iteration of a chain of iterables is pretty common.

Iteration's fine, and if that's all you're supporting, that's fine. I thought you were going further than that with the "use wherever ArrayBuffers are needed". Allowing things like new Uint8Array(arrayBufferList) would slow down even current code as you'd need to do a type check just to amortize the added cost of the new feature (which would require a C array iteration for every random access). But if you created a new type that only allows iteration, that could work.

I've got an alternate approach that would make it a little more strictly a queue: Suggestion: make this a binary queue rather than a pure append-only list ยท Issue #1 ยท bengl/proposal-arraybufferlist ยท GitHub