Feature Request: Quoting Expressions

Could you share a concrete example were an AST would be useful for debugging?

Sure. If we take a closer look at the example:

contract.requires(x != null, "Developer has to type some explanatory string...");
contract.requires(quote(x != null));

we can consider that requires(), in the second example, receives an Expression or a Node object rather than a Boolean and a string. Accordingly, it could process the AST to detect that the requirement is or involves a nullity check. If x were null, then an Error object could provide information about x being null. So, again in theory, an Error object could stringify the AST or otherwise process it to automatically provide developers with more information. This would, apparently, require evaluate() on Expression or Node to obtain a resultant Boolean...

One can already use Function.prototype.toString to get the sourcecode of a function object and then parse that with a parser of your choice.

Function.prototype.toString(), as implemented by Node.js (V8), has not met all of my needs. For example, as I remember, when a function contains nested anonymous functions or arrow functions, the provided text representation of the function elides or only refers to its nested functions.

Here again I wonder what concrete usecases you'd have in mind, which can't already be solved through other means.

I didn't mean to make a claim about any of the above use cases not being solveable through other means or having other, potentially inelegant, workarounds. To the above example, for instance, one could reply, "but the developer could just type the error description string."

Some of the use cases with iterables and observables can be considered in response to your question. One use case that comes to mind resembles "LINQ to SQL" where AST's could be processed, potentially optimized, translated into a DSL, e.g., SQL, and transmitted to a server.

With the aforementioned iterables and observables, one can envision subscribing to streams transmitted from server machines. In this case, having the AST's of functions used to filter() such streams would be useful for optimization scenarios, e.g., to transmit the semantics of the filtering functions to the server, reducing transmission cost. So having access to the AST's involved in formulating queries to servers can facilitate a number of optimizations.

An interesting software to consider, on these topics, is RethinkDB (source) which provides a cool JS API which includes support for something called changefeeds. With databases like RethinkDB, developers can subscribe to receive notifications when tables, rows, or, more interestingly, query results change or update.

I think this is the least interesting, maybe even bad, use of what you're proposing.
The source code of the failing condition is rarely helpful on its own. The most important pieces of information in this case are source location, and natural language explanation of what the expectation was. The condition itself is something I'll only need to see when I go there to fix it.

1 Like

@lightmare, ok. Let us consider, instead, then, the other, more interesting uses of the proposal.

Seems like if you're going to do all the work of creating accessible ASTs, you might as well go full lisp and have macros.

It appears that with this feature macros can be built on top of decorators.

I get Greenspun vibes from the way this discussion's going...

I guess this is what you meant:

Greenspun 's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.
😂😂

Ok. Time to be a wet blanket. What happens with this?

console.log(quote(class SomeClass {
   #pvt = 42;
   print() { console.log(`SomeClass::pvt = ${this.#pvt}`); }
}));

Here's the problem I see. Sure, you're getting the AST of the expression passed to the quote . However, to my eyes, this reads weird. From everything in this chain, this should dump the AST for this class, but just a glance says it should dump the AST for the default constructor function as the class keyword has to be evaluated first due to the parenthesis surrounding it. The same goes for all other quote examples that have been given.

It get's a little worse though. The most useful use case for quote wouldn't be in dumping the AST of immediately fed syntax as has been presented in all posts above, but in dumping the syntax of either already processed logic, or of a string containing yet-to-be-evaluated ES code. For the latter case, that's easy to understand how one would get the string. On the other hand, how would one take a result and retrieve the syntax that generated it? That would be an incredibly large expense to bind all products to the syntax that generated it just for the sake of such a tool.

Maybe I missed something here.

Regarding your first paragraph, you shouldn't be parsing the examples as actual syntax. It's obviously pseudo-code, a placeholder to showcase a feature without introducing new syntax. quote(...) is a function call and so it cannot be changed to mean something different. How the actual syntax would look like is not important at this point, could be %%( ... )%% or whatever. As I understand it, it would not evaluate the class.

As for the second paragraph, you're venturing into eval territory. If you trust the input, you can do

ast = eval(`quote(${input})`);
1 Like

This is kind of a strawman. No such thing is being proposed here. You'd also need to define what "the syntax that generated it" means.

let { a, ...rest } = { a: 1, b: 2 };

What syntax generated the value of rest in this program? No slice of the source code did, you'd have to fabricate source code that constructs { b: 2}.

Honestly, even while ignoring the undecided syntax, I still have to take in to consideration potential syntaxes. If it is based on functional syntax, then what I said in paragraph 1 still applies. If it's based on keyword or operator syntax, then it becomes nearly useless as it would need to be the intent of the writer of the code being analyzed that it be rendered as AST.

That's because "eval territory" seems to be the best place for such a feature. Unless I'm mistaken, this feature is trying to dump the AST of code (i.e. source), not process (i.e. compiled result).

And how did you come about that { b: 2 } ? You "sliced" away all parts of the let statement that didn't contribute to the destructured value of rest, right? That makes that "fabricated source code" a slice of the original program imho. Let's not quibble over that.

The point of that admittedly straw argument is that such a feature's primary practical use is to analyze and return the AST of code that it is not itself directly part of. That being the case, some of the suggested use cases don't make much sense. For instance:

Even as pseudo-code, I can't see this being implemented any other way than as a function. That being said, while contract.requires() receives an AST, there's not much it can do with it as there's yet to be developed an exposed means in ES to look up a scope binding on the calling function by name. Even merely being able to access the calling function seems to be frowned on by TC39 given their push towards defacto use strict, let alone the never-been-externally-accessible scope of that function.

Put differently, using the generated AST to decide what to do with program elements mentioned in the AST is just not viable without being able to create/modify AST or source code and evaluate it in the desired execution context. From within ES currently, the only way to do this viably is if the generated AST is of code yet to be compiled.

There are similar limits on each of the proposed usages in this thread. Using such a feature to process code before it is compiled and run is viable. However, without several other features, there's no real way this feature is viable for analyzing and affecting any code already running.

Yes, quote() was intended as example pseudo-code. To explore and to visualize syntactic options beyond quote(), and based on the feedback, here is an example @{...} syntax which uses a symbol, in this case @, to modify the scopal {} brackets.

So, the code contracts example could resemble:

contract.requires(@{ x != null });

and the lambda expressions examples could resemble:

foo(@{ (x:number) => x + 1 });
foo(@{ function (x: number) { return x + 1; } });

Hopefully this example pseudo-code is a bit more readable.

@AdamSobieski - @rdking does also bring up a good point about retrieving local variables from the current scope. How are you proposing the "x" variable's value is accessed by assert() in the following code snippet?

let x = f()
assert(@{ x === 2 })

Here's another thought.

If such a feature were to go in, I wouldn't want the end-user to be able to transform the AST tree and execute the transformed tree, I'd rather they received a frozen, immutable tree. If this kind of AST-transforming thing starts happening, then the stuff inside the @{} isn't really Javascript anymore, it's Javascript with modified behaviors, and it's probably best to think of it that way. It would be better to just use a template tag for that kind of feature, For example:

// Instead of allowing support for this kind of thing:
let varFromOuterScope = 1
flipTernaryOperands(@{true ? varFromOuterScope : 2}) // outputs 2

// we just allow people to do this instead
flipTernaryOperands`true ? ${varFromOuterScope} : 2`

That second version is possible today. The string can be parsed into an AST tree using a library like Acorn. The template string version allows for more flexibility than we could ever provide with "@{}" syntax, like adding new operators, etc. Template strings, to me, feel like the best way to embed sub-languages into Javascript.

@rdking, @theScottyJam, in my rough-draft sketches, I use an AST model which includes a CaptureExpression type for specifically representing things captured from containing scopes.

In the following example, obj_y is captured:

let obj_y = { value: 123 };
let lambda = @{ function (x) { return x + obj_y.value; } };
let compiled = lambda.compile();

The following visualization shows constructing an AST manually:

let obj_y = { value: 123 };

let x = Expression.parameter('x');
let y = Expression.capture(obj_y);
let y_value = Expression.member(y, 'value');
let add = Expression.addition(x, y_value);
let ret = Expression.return(add);
let lambda = Expression.lambda([x], Expression.block([ret]));

let compiled = lambda.compile();

So, then, we could:

console.log(compiled(0)); // outputs 123
console.log(compiled(1)); // outputs 124
console.log(compiled(2)); // outputs 125

and, next, change the value of obj_y.value,

obj_y.value = 126;

the result of which would be such that:

console.log(compiled(0)); // outputs 126
console.log(compiled(1)); // outputs 127
console.log(compiled(2)); // outputs 128
1 Like

Ok. With this, I can see that you're using composable AST almost like a modifiable intermediate language. That makes Expression.compile() the AST equivalent of eval for ES code in a string. What I don't see is where in the notation example you marked obj_y as a captured value. I also don't see it as necessary. Capturing the value of obj_y would create a snapshot of that value in the lambda expression, essentially making it immutable. However, you showed that changing the value of obj_y.value changes the output of the compiled result. This means obj_y was simply referenced as one would expect for something in local scope. Maybe I misunderstood something again. I'm not fully versed on AST terminology, but that doesn't strike me as a captured value.

That being said, at least now the direction of conversation is headed towards actual use cases for such a feature. The only problem I can see now is why this would be something worth integrating into the language when it serves quite well as an external library (Acorn) as is. What would be gained by making this a language feature? If this feature were implemented using the engine's core translation routines, then there wouldn't be any guarantee that the same AST would be generated across different engines without first standardizing that AST. If it wouldn't be implemented that way, then there's little advantage over a library.

I would think that it's just "captured by reference", not "captured by value", so you would be able to mutate it externally and see it reflected within the lambda.

It doesn't need to be marked. If the variable isn't found within the AST itself, it can know that it needs to grab a reference to it from outside scopes.

As I understand it, this capturing is really governed by the same principles as functions. When you declare an inner function, it captures variables from its outer scope by reference. If you mutate the variables in the outer scope, you'll see that reflected in the captured value. It knows which variables it needs to capture simply because it knows what variables aren't defined locally.

Function example

let x = 2
let y = { data: 2 }
function f(z) {
  function g() { // This function "captures" x, y, and z from outer scopes.
    let w = 2
    return { w, x, y, z }
  }
  return g
}

f(2) // { w: 2, x: 2, y: { data: 2 }, z: 2 }
x++
y.data++
f(2) // { w: 2, x: 3, y: { data: 3 }, z: 2 }

Quote example:

let x = 2
let y = { data: 2 }
function f(z) {
  let quotedExpr = @{ // This "captures" x, y, and z from outer scopes.
    (() => {
      let w = 2
      return { w, x, y, z }
    })()
  }
  return quotedExpr
}

f(2).exec() // { w: 2, x: 2, y: { data: 2 }, z: 2 }
x++
y.data++
f(2).exec() // { w: 2, x: 3, y: { data: 3 }, z: 2 }

let quotedExpr = f(2)
quotedExpr.invoke.function.body.statements[1].expr.objLiteral.entries.get('x') // 3

This does mean these quoted expressions aren't "true" AST trees - they've been augmented with additional runtime information, which is fine.

I can understand that way of thinking about it, though it causes certain inconsistencies in how I think about things. For instance, to me a function doesn't do any capturing of any kind. It has a scope "object" with a prototype chain composed of prior scope "objects". In this way, a function behaves in an an independent and isolated fashion.

Applying that way of thinking to this AST generator, the AST itself would either need to be able notationally reference the scope it is to be compiled in, or it has the same limitations as eval and the evaluated code inherits the scope it's compiled in. I'd like to see a way to reference other scopes, but I'm sure that TC39 would call that idea DOA for security reasons.

That's the way I currently understand it - it would only be able to capture the variables from the location it was compiled in.


On another note - I don't know if people agree or not with my previous thoughts of limiting the generated AST to be immutable only, but I want to keep following that thread for a bit, and revisit the use cases we're trying to target.

The primary use case seems to be for debugging purposes - we want to provide nicer error messages when something goes wrong. (Are there any other use cases that don't involve mutating the AST tree? I can't remember any from the previous conversation, but fill me in if anyone can think of anything else). So now, I want to see if we can still provide this use case, without having to have a feature request as big and difficult to use as this one.

Asserts got mentioned previously as a concrete scenario where this would be helpful. What if, we simply provided a debugger.assert() pseudo-function, that had the capabilities of examining the parameters and determining what went wrong. e.g. let x = 2; debugger.assert(x === 3) could throw a nice error saying "Assertion Error: Assertion '2 === 3' failed". We could even let it have an second optional string parameter that allows the assert user to provide more context to the error, and that context will just be concatenated with the auto-generated assertion message.,

This has the disadvantage(/advantage) that we're simply building-in this feature, and not giving userland the tools to build it themselves. But, once a good, helpful assert function is in place, a lot of potential use cases for this "quote" feature could be done instead by just using the assert function. Plus, this assert() function will always be up to date, to provide good debugging information with the coolest new operators (you don't have to keep manually updating assert() each time a new language feature comes out, the browser can handle that for you).

@theScottyJam, enhancing the debugging experience isn't intended to be the primary use case. That particular use case arose from brainstorming uses of quoting content other than lambda expressions. I was thinking about use cases for quoting simple portions of code or simple expressions. Also, beyond quoting functions into lambda expressions, @rdking indicated a scenario that I hadn't considered where entire class definitions could be quoted.

There appear to be at least three categories for use cases:

  1. Quoting small portions of code or simple expressions
  2. Quoting functions, anonymous functions, and arrow functions into lambda expressions
  3. Quoting larger portions of code, e.g., class definitions

With respect to quoting functions into lambda expressions, scenarios to consider include the processing of iterables and observables.

I can put together a rough-draft set of use cases soon and post it here.