Trying one more time because this is pretty important and I have a lot more practical experience with this idea and so more evidence than I have had previously.
The proposal I wish to champion (or find a champion for) is to create a for of loop over iterator steps, but which only awaits iterator steps and only when it is necessary because the value of step is a promise.
for await?(let item of iterator);
The existence of this iteration syntax directly implies the existence of a new core iteration protocol, though protocols are essentially duck-typed in JS. The new protocol would be denoted Symbol.streamIterator. To implement the protocol the iterator’s next() method would return either a { done, value } object or a Promise which would then be expected to resolve to { done, value }.
This solves several seemingly-unrelated problems at the same time:
- It provides a primitive suitable for concurrent processing as it is a complete embedding space for data – neutral towards all possible values passed as data. Async iterators as designed are netural to almost all data values: except promises.
- It obviates the needs for WebStreams by providing a primitive fast enough to abstract over a single data source in which the individual items become available in chunks.
- It permits stream transformers to be defined once while being usable for both sync and async data. You might define a transformer like
toUpperCaseand now you could do bothtoUpperCase("str")andtoUpperCase(readFile('./file.txt')). Things that are sync stay sync (keeping fast things fast), and awaits happen when they are needed.
It's not terribly hard to polyfill this behavior in current-day JS. In a what-color-is-your-function sense you might call a generator function of this kind a "purple generator" as it can (potentially) be used natively in both red and blue calling functions.
You can polyfill purple generators into Javascript fairly easily. Most of the difficulty is in doing iteration without the help of a for loop.
let toUpperCase = (source) => new Wrapper(toUpperCase__(source));
function *toUpperCase__(source) {
let iter = source[Symbol.streamIterator]();
let step;
try {
while(true) {
step = iter.next();
if (step instanceOf Promise) {
// the purple magic happens here!
// wrapper ensures the yielded promise doesn't end up in the data stream
step = yield step;
}
}
if (step.done) break;
let chr = step.value;
yield chr.toUpperCase();
} finally {
iter?.return();
}
}
class Wrapper() {
constructor(generator) {
this.generator = generator;
}
next(value) {
let step = this.generator.next(value);
if (step.done) {
return { value: undefined, done: true };
} else if (step.value instanceof Promise) {
// resolve the promise for the wrapped generator
// rolls the promise into `step` for the consumer
return step.value.then((value) => {
return this.next(value);
});
} else {
let { value } = step;
return { value, done: false };
}
}
}
This proposal is to shrink all that boilerplate down to this:
async? function *toUpperCase(source) {
for await?(let chr of source) {
yield chr.toUpperCase();
}
}
If you were to try to visualize how data is moving around it would be useful to think of a purple generator function a bit like a bucket brigate moving water. As long as there's buckets ready to go from the water source regularly, each worker in the line will be in a steady rhythm: take from the left, pass to the right, take from the left, pass to the right. Both sync-ness and async-ness propagate through this system. When buckets are ready to go, the system is in a synchronous state with high throughput. When buckets are no longer ready though, perhaps because the tub of water at the source of the chain has run dry, then each worker will look to their left and see a "promise of future bucket" and will understand that they should wait until synchronous operation may resume. If buckets are only available very slowly, each worker may return to the idling state after every bucket, mimicking how async iteration works currently, yet there is no reason to force each worker into an idling state as a defensive measure, as async iteration does.
I'm just putting together the final details to be able to release a fully-featured stream parser framework implemented as purple generator functions, so there should soon be a lively and rapidly-growing ecosystem of people relying on this age-old technique for moving large numbers of small items quickly