JSSugar: a concrete implementation with BABLR

[conartist6]

I woud like to put forward BABLR as the API layer for a JSSugar/JS0 divide. I've spent years polishing this proposal to standards-quality, and now I believe it is there. What I have created is a natural successor to Babel, and a mechanism that allows JS syntax to be extended in practice.

This proposal is designed to resolve or supercede several other outstanding proposals, including JSSugar, JSX/ESX, as well as the Type Annotations proposal.

This proposal won't look much like other proposals though. Its APIs are defined not in terms of spec text but in terms of a reference implementation. That reference implementation is bootstrapped, meaning that it creates new formalisms and defines them formally in terms of... its own formalisms. These formalisms also are also then used to define Javascript. We've created the first fully extensible parser since Acorn, and we show off this parser extension ability by creating separate and fully independent definitions of es3, es5, es6, and the more modern variants each of which extends from the previous definition of the language.

Here is an example of an es3 parse of the expression foo.bar = baz

accessible version of above image content

The CLI is expressing its output in a language called CSTML, which (like HTML) lets us talk about the content of trees without talking about exactly how they are stored in memory. CSTML is the Concrete Syntax Tree Markup Language, and like HTML it is a means of embedding structural metadata into a stream of plain text. It's just that instead of doing this by hand, we let a parser embed the metadata, most ofbviously open and close tags to show the boundaries of particular parse tree nodes. The text stream embedded inside the metadata is highlighted in green.

You'll also notice the most distinctive feature about CSTML compared to HTML or XML: it has a special kind of tag called a "shift tag" which looks like ^^^ that ensures we can still have streaming output from a parser even in languages like JS with infix notations. The shift tag means "take the node above, and put it into the gap below", where a gap is written as <//>.

The ability to stream input into a worker and stream parsed syntax out is what in practice allows us to spin up a worker and use it to do on-the-fly transformation of syntax as a preprocessing step.

In the long run this system also creates a clear path to implementing optimizations which have heretofore been impossible, most importantly the ability to efficiently load trees of ES modules. Trees of individual ESModules face a major perf penalty (in terms of wall clock time spent) because the current parser must wait to receive the complete binary content of a document before it can emit any parse results, which include the parse results which indicate the next batch of imports.

A BABLR parser can and does emit parsed imports before the rest of the content of the module is received at all, meaning that it can eliminate the dead wall clock time making it much more pleasant to work with applications where the scripts are not pre-bundled. The same streaming ability is also critical for efficient feeding of input and output through a worker-isolated syntax transformer.

A complete formal description of CSTML can be found in the CSTML grammar.

I'm happy if not eager to answer any questions!

[ljharb]

To be clear, there is not likely to be any chance of consensus on the implementation side of the JSSugar presentation; the only consensus was on the shared problem.

[conartist6]

@ljharb Well we have someplace to start then, right? We can start with the shared problem, which is that people want runtimes to be able to run the code they write. They seem to want this quite badly, which I think creates room to get people talking and slowly work in the direction of consensus.

The underlying technical problem has a couple major parts:

1. How can the official parser be extended to recognize unofficial syntax?
2. How can unofficial syntax be transformed into official syntax efficiently?
3. How can TC39 ship official syntax extensions in a more 0day fashion?

To even get to the point of having design disagreements, some strawman/bikeshed implementation has to be proposed. As far as I'm aware, this is the first proposed implementation of JSSugar, and therefore hopefully the start of a new conversation.

[ljharb]

For at least two of those questions (1 and 3), before discussing an implementation, one would have to contend with the reality that nonzero delegates don't believe they should be possible.

[conartist6]

How do I get the people who have those concerns to make them known to me? Are they already documented somewhere?

I was hoping that you might be one of the most knowledgable and opinionated delegates on the topics involved thanks to your work on Babel and ESLint. Do you think parser extension should be possible? What are your concerns?

[aclaymore]

It sounds like custom parser extensions would hurt performance of websites.

Is the intention that it would only be used during local development of sites?

[conartist6]

That's a complicated question to answer, not least because what I intend may have little bearing on what happens in the real world.

But I think we can confidently say that yes, there would be some perf cost to this so that the Facebooks and Googles of the world would certainly precompile to avoid paying them. We have the infrastructure in place for precompiling and I don't see any reason to think that it will cease to be maintained and used -- unless the perf benefits should dwindle to a point where the overhead is no longer worth it (which I'd say would be a good thing).

For the small or new sites in the long tail of the web, who knows if they would choose to ship source code written in Gleam or Vue or Rescript or something! I imagine if given the option they'd weight the costs and benefits and decide what was best for them.

In terms of the perf trade-off, there's several facets to consider:

• Source code may be more expressive and so more concise and few bytes over the wire
• Sourcemaps no longer need to be sent over the wire
• Client uses compute resources on transformation and source map generation.
• You might see wall-clock wins even as you use more resources because the server can be sending the data (and the client receiving and parsing that data) in a highly parallel way when you aren't bundling.

This set of tradeoffs does not have a universally clear outcome in my mind. Each cost/savings has a YMMV factor, and then there's a bigger question of, "Will the user of a device like a phone find battery life or data more precious?" In the part of the world I live in we treat data as virtually free so normally developers optimize for battery life, but the value math won't work out the same everywhere.

[conartist6]

What's the best source of information about how JS modules are parsed? I think to accurately describe the possible perf benefits of my solution I need to better understand how module code is parsed in current browsers.

Let's say you have a file like this:

import "https://server/modules/other.js";

/*
  1 MiB of code here
*/

How soon can you start parsing the other.js module? My intuition about how the current parsers work is that you have two main phases of wall clock penalty:

First, you can't really start parsing at all until you've received the full mebibyte of data over the network, which is because parsers expect to be able to make synchronous function calls during the process of parsing.

Second, you can't really access any of the parse results until you've finished producing a full tree of parse results for a given file.

Before I go on to draw conclusions about my approach based on these assumptions, is anyone able to confirm that this information is correct?

[aclaymore]

Chrome, as one example, parses the network stream as it downloads. Blazingly fast parsing, part 1: optimizing the scanner · V8

[conartist6]

In addition to supporting more than one encoding, the separation between scanner and character stream allows V8 to transparently scan as if the entire source is available, even though we may only have received a portion of the data over the network so far.

OK that's definitely interesting, but I'm left wanting a lot more information. v8 is creating tokens on the fly as data becomes available, but it doesn't really say how the parser works when the token stream is not fully available, and they also don't say whether they analyze the token stream for imports.

I'm also aware that adding JSX syntax to the JS parse rules eliminates the possibility of doing tokenization as a pure pre-pass: GitHub - lydell/js-tokens: Tiny JavaScript tokenizer.

[ljharb]

fwiw, I'm one of those people. I don't think it's a good idea to allow an anarchy of ways to express runtime JavaScript. If you want to write a different language, I think it's perfectly reasonable - and ideal - to force you to transpile.

[conartist6]

@ljharb You say it's reasonable and ideal to transpile, and so far I think I agree wholeheartedly.

I've tried to make sure that that's all this proposal really is: to define a syntax extension you're forced to transpile the new syntax away until only standard syntax is left. This stands in sharp contrast to something like proposal-type-annotations which allows arbitrary extension syntax to be used without forcing it to be transpiled.

I guess my point is that I don't see any new layer of risk here that arises from having a plurality of ways that syntax is written because syntax is already written in a plurality of non-standard ways. Not only that, but those nonstandard syntaxes already have made their way inside the browser with the help of sourcemapping.

So I'm arguing that whatever the consequences of syntax plurality are, we're already living with them.

But right now there's a massive wave of demand to be able to run code without a build step. It's the number one issue reported on the annual devX survey, and it's so politically persuasive that engine authors are embracing Typescript as a pseudostandard, which I see as a huge, practical risk for TC39. I definitely do not want communally owned standards give way to a model where Microsoft has direct control, yet I observe this to be where we are going, particularly if TC39 won't help engine authors figure out how to meet their users' needs without blessing Microsoft in an anticompetitive way.

[aclaymore]

@conartist6

How soon can you start parsing the other.js module?

I believe it would not be spec compliant to start loading the imports before finishing parsing the full module - if the module has a syntax error then it's dependencies are not loaded.

If a website does want to get the load started early one technique is to use script preload tags.

[conartist6]

@aclaymore OK, interesting, I didn't know that the spec said that.

Still, it could only really stops you doing things that have side effects, which neither starting a network stream to fetch the related module's data nor parsing that data should... If a module is just doing other imports and defining functions then there shouldn't be any real harm from eagerly evaluating it I'd think, and there could be quite a bit of real perf gain since it's a pretty common pattern to limit effectful code the top modules of a project.

[ljharb]

You can't know if it's harmful or not until you evaluate it, because it could eg modify globals, or console.log, or do any kind of i/o - which is why the spec prohibits it.

[conartist6]

I'm in agreement there, but I still feel pretty good about where that leaves us in terms of potential to have some real perf wins as a result of embracing stream parsing.

If the three main phases are receiving the data, parsing, and evaluating, I'd say a browser is free to receive data and to parse it in a streaming fashion. Those two steps will allow you to see a streaming parse result for an import .. from 'foo' statement, and at that point the browser could safely kick off another thread of work to be able to start receiving and parsing foo's data.

The gain in effective throughput would be most pronounced for unbundled development workflows, which fits with the general theme of this proposal: making web development as accessible as it was in the long-ago days by eliminating the need for a would-be novice website builder to own a bundler or transpiler toolchain just to ship an efficient webapp written in a modern style.

The perf we can win back is time the network is sitting idle. There's still more work for it to do loading further related modules, but right now within a single thread the system will tend to sort of see-saw back and forth between parsing a module and then making network requests for more modules. When it's parsing the network will be idle, and when it's waiting for network data the processor will be idle. So at any given moment, each thread is leaving half its resources idle.

Again this fits with the theme of: technically some overhead is introduced, but you win it back in more responsive user experience: more parallelism and so less wall-clock time spent on the same work.

[conartist6]

I just posted this guide to the basics of CSTML syntax: Intro to CSTML | Docs

It's also a chance to show off a bit:

• The HTML was authored as CSTML: bablr-docs/src/content/docs/guides/cstml.cstml at 4fca1f18cf4fc59615ba5b1dd2d436dd27d5f0c4 · bablr-lang/bablr-docs · GitHub
• The code examples in the page have semantic selection, so as you drag your mouse what you see is the boundaries of nodes in the syntax tree
• We're providing ASTExplorer's experience right in the page for every code example, letting you see the names of the various kinds of syntax tree nodes and their hierarchical relation to each other
• We support localization of parsers as well as localization of text, so we will be able to translate this document and the embedded syntax node names into any of the human languages spoken around the globe
• To avoid UX lockup while the code examples on this page are being parsed we leverage our stream parsing ability. After every 20th character of input text we put an artificial promise deferral into the input stream. This allows other work to be scheduled ahead of parsing so that the UI stays responsive.

Also there's another page I published which is both relevant and interesting: the page about stream iterators: Stream iterators | Docs

I think it's the best version of the argument for this pattern, distilled from earlier arguments I've made here. Stream iterators are a separate but related proposal, one which is foundational to this one. As the post mentions, they have both a proposed implementation and a proposed syntax. I have included the proposed syntax in the post. It looks like this:

async? function* streamParse(chrs) {
  for await? (let chr of chrs) {
    /* ...do parsing stuff */
    yield tag;
  }
}

What's especially interesting though is that in the page which documents this proposed extension syntax, our client is actually sugaring the core Javascript parser with our proposed extension right there in the page, which you can see happening here: bablr-docs/src/content/docs/components/solid/Highlight.js at 87998e1c0fa80bdc2ddb74e2c776a687ccd2a27c · bablr-lang/bablr-docs · GitHub

What really tickles me that there's something so universally accessible about the experience of dragging the mouse around on code and having the computer react in a way that is revealing about the structure of the thing being interacted with. If you took away everything else and just left the semantic selection feature, the feature would still be incredibly revealing to me when exploring code examples in a language I was unfamiliar with. It would let me explore which pieces of syntax are siblings, which are children to what parents, and even just something simple like how deep the hierarchy is, all without ever needing to use any words at all to convey meaning.