Yes, please do, I think that would help a lot.
I'm not entirely sure yet what the value is in quoting a lambda expression over just creating a lambda function - I don't think any of the examples presented yet showed the value in that.
Most of the time that I'm aware of, when people want to use an AST, it is for analysis and modification. Even just providing the AST for analysis requires that it be standardized. So I can understand why you might want to dispense with the modification side of things. That being said, reducing it all the way down to a simple assert() pseudo function completely guts the original intent of the proposal idea. Maybe that should be something all its own, especially given that such assert statements could be resolved at compile time while the source is being processed.
I'm curious to know whether the quote function / syntax construct / whatever; can only take an expression and not a statement?
// this is possible
quote(function() { ... })
// but are these valid?
quote(function name(args) { ... })
// or
quote(for(let x of arr) { ... })
// or
quote(let {x, y} = obj)
If it does will there be any ambiguities?
Consider this:
class Foo { ... }
quote(Foo)
will it produce the AST of an identifier named Foo or the class Foo?
Introduction
Quoting would be a powerful language feature which, along with an accompanying expression tree / abstract syntax tree model, could support a wide range of metaprogramming scenarios.
Types of quoting scenarios and some specific uses are indicated below.
A syntax for quoting remains to be determined and an example syntax, @{...}, is utilized here.
Types of quoting scenarios
I. Variables
Developers could quote in-scope variables, e.g.,
@{ x }
II. Simple expressions
Developers could quote simple expressions, e.g.,
@{ x.value + 1 }
III. Arbitrary expressions and statements
Developers could quote arbitrary expressions and statements, e.g.,
@{ for(let x of arr) { ... } }
IV. Functions, anonymous functions, and arrow functions
Developers could quote named functions, anonymous functions, and arrow functions, including generators and async varieties thereof.
let obj_y = { value: 123 };
let lambda = @{ function (x) { return x + obj_y.value; } };
Such lambda expressions could be compiled
let compiled = lambda.compile();
V. Class definitions
Entire class definitions could be quoted.
let classdeclaration = @{ class foo { ... } };
Some use cases and scenarios
1. Enhanced debugging and code contracts
Debugging experiences and design by contract scenarios could be supported with quoting and an accompanying expression tree / abstract syntax tree model.
2. Aspect-oriented programming
Some aspect-oriented programming scenarios could be facilitated by processing the abstract syntax trees of functions as input and outputting trees.
3. Data structures
Data structures (such as and-or trees and and-or-not trees) could utilize expression tree and abstract syntax tree models.
4. Inspectable mathematical sets
Mathematical sets could be utilized by developers which include transparent definitions, lambda expressions for functions which process candidate elements to return Boolean values (also known as intensional definitions). These mathematical sets could be intersected, unioned, complemented, and so forth.
5. Stream processing, relational algebra, and querying
Iterables and observables could be processed utilizing composable operators which could receive lambda expressions as arguments. See also: LINQ.
objects.filter(@{ x => x.property == 'value' });
6. Decorators
Decorators and related scenarios could be supported with quoting and an accompanying expression tree / abstract syntax tree model.
7. Hygienic macros
Macros, specifically hygienic macros, could be facilitated with quoting and an accompanying expression tree / abstract syntax tree model.
8. Runtime generation of functions
A runtime abstract syntax tree model could be a means for developers to create and compile functions for runtime use.
9. Runtime generation of classes
A runtime abstract syntax tree model which includes class definitions could be a means for developers to create and compile classes for runtime use.
10. Parsing and generating source code
With a runtime abstract syntax tree model, developers could parse source code into it and produce source code from it.
1 Like
Thanks so much - that helps.
And just to make sure I understand your listed use cases, let me try and give concrete examples to all of them, and let me know if I'm getting the gist of what you were thinking.
- Enhanced debugging and code contracts
I assume this is mostly just talking about the existing points we've already discussed? i.e. providing more useful errors when a function receives an invalid parameter, like, function f(x) { tool.assert(quote(x === 2)); ... }; f(3) would give "Error! Assertion '3 === 2' failed!"
- Aspect-oriented programming
So, this would allow userland code to, for example, write a function that logs out the result of each line that gets executed. e.g.
tools.logEachLine(${
{ // start of block, so I can execute multiple statements
let x = 2
let y = 3
x + y
}
})
/* Logs:
2
3
5
*/
- Data structures
Are you thinking of something like this?
const result = tools.interpretAsDataStructure(@{
1 + 1 && 3 || 4 && 5
})
console.log(result)
/*
{
type: 'OR',
left: {
type: 'AND',
left: 2,
right: 3
},
right: {
left: 4,
right: 5
}
}
*/
- Inspectable mathematical sets
I'm not sure I really follow you on this one - could you provide a concrete example?
- Stream processing, relational algebra, and querying
How would your example snippet behave differently from this?
objects.filter(x => x.property == 'value');
Why is it valuable for an iterable or observable to inspect the AST of the predicate?
- Decorators
Are you just saying that the quote feature can understand decorators? e.g. it would correctly turn the following into an AST tree?: @decorate function() {}
- Hygienic macros
I do know what macros are, and can understand how quoting would enable macros.
8/9/10. Runtime generation of classes/functions / Parsing and generating source code
I can see how this would be possible as well.
Thanks for helping me slowly understand what you're going for.
Thank you.
With respect to #4, inspectable mathematical sets, one can consider developing some interfaces for mathematical sets. When so doing, one might define something like:
interface Set
{
contains(element: any): boolean;
}
Next, towards inspectable sets, one could add a property, expression, which returns the lambda expression of that contains() function.
interface InspectableSet extends Set
{
get expression(): LambdaExpression<(element: any) => boolean>;
// ...
}
Effectively, expression is the abstract syntax tree representation of the contains() function. That is, compiling that lambda expression would result in the same function which is invoked when contains() is called.
The universal set would have a contains() function which returns true for any input and the empty set would have a contains() function which returns false for any input.
One could also add methods like complement(), intersect(), and union() to Set or InspectableSet. To implement these for inspectable sets, one could, then, process the lambda expressions to produce resultant lambda expression for resultant inspectable sets. For two sets, set1 and set2, the result of a set intersection would be an inspectable set with a lambda expression which optimizes the semantics of set1.contains(x) & set2.contains(x), utilizing the bodies of the lambda expressions.
Optimizations are possible when logically combining lambda expressions, resembling an optimizing runtime compiler, resembling a rule-based or knowledge-based system, resembling a computer algebra system (see also: Mathematica's FullSimplify).
To the question on #5, we can look to IQueryable and IQbservable in .NET . There is a pattern where a provider is utilized to produce resultant instances when composeable operations are utilized on instances. For IQueryable (iterable-based), this is IQueryProvider and, for IQbservable (observable-based), this is IQbservableProvider.
So, when calling where (or filter) on an IQueryable, it routes to the instance's query provider (IQueryProvider) which processes an expression tree / abstract syntax tree to provide a resultant IQueryable. See also: CreateQuery().
The pattern allows for customizations and optimizations.
Hopefully, clarifying some of the differences between potential uses of:
objects.filter(x => x.property == 'value');
and
objects.filter(@{ x => x.property == 'value' });
In the second case, the implementation could utilize the provided lambda expression and route to a provider object which then processes it or a derived expression to produce a resultant instance capable of evaluating a composed query upon one or more iterables or observables.
The way I understand the difference between these two, is that if objects is e.g. a proxy for a remote dataset, the example with plain arrow filters on the client (i.e. pulls the whole dataset, then calls the arrow function for each received record), whereas the second can translate the quoted arrow function into SQL and filter on the server.
2 Likes
Alright - now that I have a better picture of what's going on, I'm going to ramble out my thoughts on it all.
Firstly, I see that this quoting feature is extremely powerful. A quoted expression can be processed in all sorts of unimaginable ways. This, I think, is also the danger of it. Take these two code snippets for example:
addListener(data => {
const value = data.value
console.log(value + 1)
})
addListener(@{data => {
const value = data.value
console.log(value + 1)
}})
What does the first code snippet do? Well, you have no idea. But, you at least have a rough idea of what will happen when that callback executes - it's going to take the parameter, pull out the "value" property, and log its value plus one. That's all that callback can do, because that callback is just javascript.
OK, what about the second code snippet. What happens when the quoted callback executes? You've lost all guarantees that Javascript provides inside a quoted function. const declarations might not be constant anymore. A class's private state could be publicly exposed. Object.freeze() may not behave as expected. Two plus two might equal five. There's nothing you can tell me about that callback with certainty - and this is because of how much power this quoting feature has. You're relying completely on the documentation for that function, and hoping it's been implemented in a good and predictable way.
Because of this issue mentioned above, I think anyone who's trying to maintain a readable codebase must use this quoting feature as little as possible, and when it gets used, limit how much code gets quoted as much as possible (don't go quoting entire modules). Alternatives should be used whenever it's possible and practical. But, there are times when alternatives just don't do a great job, and maybe a quoting feature like this would be the best solution.
With that in mind, I'm going to go over some of these use cases, and try and see if we can provide alternative solutions that don't require the usage of such a powerful tool.
- Enhanced debugging and code contracts
As mentioned previously, we can provide a debugger.assert() function to provide this kind of functionality. this unfortunately doesn't provide the same power to provide user-friendly error messages that quoting can provide, but overall it'll make for a much more readable codebase, and IMO this is an appropriate tradeoff.
// Does this expression work as expected?
// Or does this thirdPartyLibrary try and do
// any extra magic with that expression?
thirdPartyLibrary.assert(@{getAge(user) > 18})
- Aspect-oriented programming
This sounds really cool up-front, but in actuality, it might lead to really bad code. Aspect-oriented programming relies on the ability to do sweeping changes over lots of code with little effort. This means:
- You have to quote larger chunks of code for aspect-oriented programming to provide any value
- You're applying a lot of unexpected magic to the whole chunk of code you're quoting
These are my two biggest no-nos I laid out at the beginning. If aspect-oriented programming is something desired in Javascript, perhaps we kind find ways to introduce it using a more confined, less powerful feature, so as to keep the overal program more predictable and readable.
- Data structures
From how I understood this idea, we're basically repurposing Javascript syntax to do something else than what you would expect. I think a better, more understandable solution to this problem would be to provide developers with the power to define their own custom operators, that they can then use to construct a data structure.
The operator overloading proposal does mention this idea briefly (here), but they don't currently plan on going that direction. But, operator overloading by itself can still get you part of the way there. Operator overloading is designed to be a feature that you opt-into at the place you want to use overloaded operators, so you shouldn't ever be surprised by the fact that particular operators are behaving differently in a particular block. The one limitation is the fact that the overloaded operators must act on non-primitive.
- Hygienic macros
Macros are a powerful feature that provide a whole lot of extra power to framework creators. However, using this quoting feature for macros suffers from the same issue that we saw with the data structure point, namely, you have to repurpose existing Javascript features to mean something they normally don't mean. I can't think of any situation where I would rather go with a macro over just using whatever code the macro would expand into, precisely because the expanded code has more constraints applied to it, that makes it easier to read.
8/9/10. Runtime generation of classes/functions / Parsing and generating source code
This isn't a common use case, and the acorn library does a good job of providing this feature to those who need it.
There are two use cases I haven't discussed yet (aside from decorator support, which is more of a feature than a use case): "Inspectable mathematical sets" and "Stream processing, relational algebra, and querying". I can sympathize with this class of use cases. I'm reminded of a time when I needed to speed up a program by moving logic into the GPU. I used gpu.js which I found to be a really well-designed library. Here's an example code snippet from their website:
const gpu = new GPU();
const multiplyMatrix = gpu.createKernel(function(a, b) {
let sum = 0;
for (let i = 0; i < 512; i++) {
sum += a[this.thread.y][i] * b[i][this.thread.x];
}
return sum;
}).setOutput([512, 512])
In this example, the function being passed into gpu.createKernal() never actually gets executed. Instead, they stringify it, turn it into an AST tree, and compile the instructions into a format that the GPU can understand, then execute it directly on the GPU. Basically, they've created a compiler that supports a sub-set of Javascript, and ran its output runs on the GPU.
The unfortunate thing is, there's a number of gotchas you have to be aware of when using this library, e.g., the function passed into createKernal() can never access variables from its enclosing scope, which makes sense, there's no way for createKernal() to access those values when it compiles this function, but it's not something you would normally expect.
This is an example scenario where a whole lot of magic is being done within that block of code, and there's really no other good way for them to support this kind of feature (unlike the previous bullet points I went over, where there are often acceptable alternatives). I think a scenario like this would be a great use case for this feature-quoting syntax being proposed. Instead of passing in a function, a quoted function could be passed in - this makes it immediately obvious to code readers that there's going to be weird stuff going on inside that function, and you should expect it to behave a little differently from a traditional function.
The querying idea also falls into this category - Just like with the GPU.js example, you're trying to provide a subset of Javascript functionality that runs in an exotic location. This time, that exotic location is on some remote machine, instead of on the GPU. I think "Inspectable mathematical sets" fall in here as well - you're likewise only going to support a subset of Javascript features in this scenario, as the underlying framework would need a deep understanding of how each operator works, so that it can combine or shuffle them around (and some, less-pure operations just wouldn't work, so they shouldn't be supported).
The thing to notice from the use cases that I (personally) find valuable, none of them really require the ability to compile the AST tree back into Javascript, or to execute the AST tree. Instead, they're trying to recreate a strict subset of Javascript, for different purposes. Because of this, I'm going to continue to propose that we keep the AST trees immutable and un-executable - it'll help keep them from getting used in bad ways, and help limit their power, so that when they do get used, you'll have a better idea of what they do.
Another thought - we could potentially follow the idea proposed previously of allowing the AST-accepting function to specify which features it expects in the AST tree. e.g. Perhaps you're making an implementation of the inspectable mathematical sets idea, and you don't want to support x++, because you only want pure logic in there. If an x++ does get found within the quoted block, a syntax error could be thrown.
Anyways, I'm done rambling now about my opinions on this. I'm sure people will disagree with a lot of what I said, but that's all my current point of view on the matter.
I have a doubt about its functionality;
let abc = "foo"
let quotted
quotted = quote(abc)
// or
quotted = @{ abc }
console.log(quotted)
what will be the output?
is it something like (rough idea):
{
type: "Identifier",
name: "abc"
}
i.e if abc is parsed but not evaluated within the quote construct.
or
{
type: "StringLiteral",
value: "foo"
}
i.e if abc is parsed and evaluated within the quote construct.
or
{
type: "Identifier",
name: "abc",
value: {
type: "StringLiteral",
value: "foo"
}
}
i.e if abc is parsed and evaluated but never executed within the quote construct.
How will it behave when presented with a situation similar to:
@{
for(let x of abc) {
// ...
}
}
will it produce:
{
type: "ForOfStatement",
head: {
type: "OfExpression",
head: {
type: "LetDeclaration",
body: [
{
type: "Identifier",
name: "x",
value: null
}
],
}
body: [
{
type: "Identifier",
name: "abc"
value: {
// ...
}
}
]
},
body: [
// ...
]
}
@jithujoshyjy, in my current conceptual model, a capture expression would be created for captured variables. As envisioned, a capture expression node wouldn't snapshot, clone, or serialize captured objects.
Is the following snippet useful?
{
type: "Capture",
value: [Object]
}
A rough-draft sketch:
class CaptureExpression extends Expression
{
constructor(value: object)
{
super('capture');
this.#value = value;
}
#value: object;
get value(): object
{
return this.#value;
}
override accept<R, C>(visitor: Visitor<R, C>, context?: C): R
{
if (visitor == null) throw new Error("Visitor is null.");
return visitor.visitCapture(this, context);
}
}