Hi Kevin,
Thanks for the answers, and yes, it makes sense, even though that I sense that it is a stretched patchwork to add an extra temporary environment for handling the iterator results. I'll metabolize the thing and get back to you! Indeed, actually having a linear code(pseudo-code, meta-code etc..) interpretation, and no distinction between the evaluation of the user code and the intervention of an invisible hand that somehow updates the state makes things easier! Or at least, I have this feeling for now.
I'm not sure such a distinction exists, but even if it does, the steps in GeneratorYield are executed as part of GeneratorStart step 4.a.i or 4.b.ii, which is explicitly the code evaluation state of genContext.
Ah yes, I missed that. Thank you for pointing it out.
Our mistake in our formalization was to encode the control flow of abstract steps structurally: calling an abstract algorithm corresponds to a recursive call in our formalization, and control must return there at the end of the abstract algorithm. Unfortunately it does not seem to be the case in the spec, and Iām following Editorial: Notes that tell you where a `Return` step returns to Ā· Issue #2400 Ā· tc39/ecma262 Ā· GitHub closely.
I tried following this thread and am unclear both what the issue is and the resolution. Given that I have been coding in Javascript for many years I must assume that I am not the only one who is confused. For the mere mortals that follow these threads like yours truly, could someone write an explanation of this issue that (for example) does not include words like "metabolize" in its explanation?
This issue is about understanding how the specification itself describes the mechanics of generators (and related things), and how to map that to an implementation of the language. It's not really relevant to people who are using the language.
If you're interested anyway, the short summary is that the specification uses a mechanism it refers to as the "execution context stack", but it isn't very precise about how this mechanism is supposed to work, and in at least one place it had a bug where it failed to update the execution context stack correctly.
Iāve been looking at https://github.com/tc39/ecma262/pull/2045 in details, and I think before it was applied there was a clean distinction between iterators and generators. Iterators store their current state in an object, whereas generators explicitly manipulate the execution context. It seems that before that change, the code associated to generators could only be a parse node (the body of a function *, if I understand the single reference to https://262.ecma-international.org/#sec-generatorstart).
There still is: a generator is a particular mechanism for producing an iterator, but iterators are anything which obeys a particular contract.
Previously there were a number of places in the spec where iterators were implemented by explicitly tracking all of their state. Now they are mostly implemented by using a spec-internal generator to produce an iterator. But the observable semantics are the same.
Yes, that's true.
A naive/literal implementation of the iterators of ES2020 are objects that are updated when fetching the next value. Generators, on the other hand, need to manipulate the continuation of the code of generatorBody to be able to suspend and resume them. I can see how to literally implement this when generatorBody is a parse node, but not when it is specification steps.
I trust you when you say the observable semantics is the same, but this is far from obvious to me. I think we will go back to formalizing ES 2020 until I can find a way to add continuation manipulation to our simple formalization language (interestingly, the pull request in engine262 that implements this change relies on generators in the host language, see Implement āYieldā and āCreate(Async)IteratorFromClosureā by ExE-Boss Ā· Pull Request #142 Ā· engine262/engine262 Ā· GitHub).
Hm. I'm not sure it really makes sense to talk about "the continuation of the code of generatorBody" - from the perspective of the specification, JavaScript code is data in a very literal sense. It does not execute. Rather, the spec executes, and sometimes part of its execution state is data which happens to be JavaScript code.
To take a concrete example, consider EvaluateCall, which is used in the evaluation of CallExpressions. Evaluating a call entails evaluating all of its arguments, so this takes an |Arguments| parse node, in addition to several language values. It then evaluates the arguments. Within a generator or async function, that evaluation can call Yield or Await, which involves suspending the running execution context. Doing so is not just a matter of noting where in the JavaScript you are: you have to be tracking enough state so that when the generator or async function is resumed you are able to continue evaluation from within the arguments list - meaning you have to be able to pick up in the middle of ArgumentListEvalution, with all of its state intact, and knowing to return to the middle of the earlier EvaluateCall call with all of its state intact, and then to the middle of the Evaluation steps for CallExpression, and then to the middle of the Evaluation steps for whichever expression or statement contained the call, and so on.
That is to say, implementing parse-node-based generators requires suspending specification steps in exactly the same sense that implementing abstract-operation-based generators does. In the former case it happens to always be true that some of the state is a parse node, but that's really just a detail: you aren't suspending the parse node, just keeping it around alongside all the other data you're keeping around when you suspend the current evaluation.
interestingly, the pull request in engine262 that implements this change relies on generators in the host language
Yeah. It also uses them for JS generators and async functions, in exactly the same way.
That was the idea behind the change in ES2021: since conforming hosts are already required to deal with generators, the spec can reuse that machinery for producing iterators rather than explicitly tracking all of the state in internal slots on the iterator object.
Anyway, going back to ES2020 for now seems reasonable. If you've successfully implemented JS-language generators and async functions in your formalism, though, I'd love to understand what makes them fundamentally easier for you to implement than AO-based generators; I don't currently understand the distinction from your perspective.
Thank you for your detailed answers, itās starting to make more sense now.
Weāre clearly not that far, we are trying to implement vanilla function calls at the momentā¦ which require implementing generators.