Inline Macros

💡 Ideas
1

Disclaimer: No AI was used in the making of this post

What are Macros?

Note: the syntax semantics are only an example for demonstration

Macros are a pre-processing step that allows developers to use templating-like syntax to describe code generation.

For example, an inline macro:

let result = add!(1, 1);

Prior to evaluation, would expand into:

let result = 2;

Why is this needed?

The Current Landscape

Almost every single non-trivial JavaScript and TypeScript application already uses "macros" (or at least the concept of). These macros are hard coded into the tools they use OR are added into custom compilers;

React

React, Preact, Solid, or any library that depends on JSX uses a built-in macro available in their transpiler

import { createElement } from 'react'
export App = () => <div>Hello World</div>

Expands into

import { createElement } from 'react'
export App = () => createElement('div', {}, 'Hello World')

Due to slight differences in implementation, there was a need to add configuration options like jsxFactory.

However, frameworks that want to take a different approach or wish to experiment with alternative approaches must use custom compilers.

Angular, Vue and Svelte

Angular uses a custom templating system powered by a custom compiler and a custom LSP.

Vue and Svelte use their own custom templating systems powered by custom compilers. Both require custom LSPs, and due to requiring a dedicated file extension, require additional tooling to support TypeScript, testing, etc

Styling Libraries

Styling libraries often use template literals or a combination of custom compilers and template literals to create programmatically defined controlled styling directives.

How do Macros Help?

Custom Language Syntax Becomes a Library Concern

Rather than tools like TypeScript's tsc, babel, swc, oxc, etc implementing expansion of jsx syntax and all other custom syntax formats (angular, vue, svelte, styling libs) requiring an additional bundler layer configured with custom compilers/loaders - the transformations for these libraries could be defined and exported by the libraries themselves

For example jsx could be described inside a normal .js file as:

import {jsx} from 'react'
const App = () => jsx!{<div>Hello World</div>}

Other frameworks/libraries could also leverage the same capability within .js files as:

import {vue} from 'vue'

export const MyComponent = function () {
  return {
    template: vue.html!{ 
      <div>{this.data.value}</div> 
    },
    data: { 
      value: 'Hello World' 
    },
  }
}

This would drastically reduce the amount of tooling required to create a dynamic web application and allow the tools to distribute their custom transformations directly.

Custom Compilers and Build Tools have Short Shelf Lives

Many of the tools re-implement solutions to problems solved by other tools. The tools have high maintenance burdens and, unless they have large budgets, often have short lifespans. This means developers have to frequently hop from tool to tool without any material benefit to their workflow.

Running Code in the Browser and/or Runtime Directly

Currently, none of these custom implementations can be run directly in a JavaScript runtime without first requiring the pre-processing step.

Examples of the need for Macros

There are efforts to incorporate these custom transformations using tagged template literals, various eval hacks or baking external tooling into the runtime:

These approaches reimplement existing tools and are either unsafe or not-reusable/non-portable.

Having first class support in the EcmaScript specification for inline macros would consolidate all of these efforts

Compile Time Versus Run Time Evaluation

Macros are typically thought of as a pre-processing step done Ahead-Of-Time by a compiler. EcmaScript is different in that it's an evaluated language.

Having macros defined within the EcmaScript specification allows these pre-processing steps to be evaluated at runtime OR expanded/optimized by compilers in a build step prior to runtime.

This has the advantage of allowing source code to be use symmetrically at both run time and during the development loop.

Examples of Macros used for These Exact Use Cases

Here are some examples where the use of inline macros enabled the rapid experimentation and development of GUI frameworks and utilities without the need for an external transpiler.

  • [1] Moved to comment due to links limit
    • Rust wasm framework, similar to Vue, that adds html support into the language without an external transpiler
  • [2]
    • Rust wasm framework, similar to React, that adds jsx-like support to the language without an external transpiler
  • [3]
    • Rust native desktop framework that has a Flutter-like syntax to define elements/styles without an external transpiler
  • [4]
    • Rust library that adds inline json support to the language without an external transpiler
  • [5]
    • Rust library that adds inline xml support to the language without an external transpiler
  • ...etc

If a developer wanted to experiment with a new/novel framework - they could just write it within the language and would not face the cliff of needing to build, document and maintain an external transpiler, LSP and integration tooling.

This lack of friction empowers developers/tool makers and fosters competition within the ecosystem which is highly valuable.

Macro Syntax

I won't dive into this now as I'd like to discuss the inclusion of macros into EcmaScript before getting into the technicals.

That said, there are a few key points.

  • Macros must be expandable statically AOT by native transpilers (like swc oxc) and therefore should not use ES syntax in their definition
  • Macros must not conflict with existing ES syntax

Below is a example of a macro syntax that meets point 1 (I am aware of the conflict of using !)

This is just an example based on Rust's inline macro templating system. If progressed, something similar would be defined in a more ES idiomatic way.

This is an inline macro that adds two numbers at at expand-time.

macro_rules! add {
    ($a:expr, $b:expr) => {
        $a + $b
    };
}

const result = add!(1, 1)

... That looks complicated

It is, but so is maintaining a transpiler, LSP and multiple bundler plugin/loaders for that custom transpiler - which is what we do today any way.

Ultimately, macros are intended to be used by framework/library maintainers/builders as a means to reduce complexity for consumers of those libraries. Outside of edge cases, it's not something most developers would actually write themselves.

Procedural Macros vs Inline Macros

There would perhaps need to be two proposals to incorporate both of these elements.

Inline Macros

Inline macros are simple "pure" templates that take arguments and are replaced by the result of their macro syntax

For example, I could write a library that does this conversion:

Users write:

import {css} from 'magic-styles';

const className = css!{ color: red; };
document.querySelector('#my-elm').classList.add(className);

Expands into:

import {css} from 'magic-styles'; // Can be stripped by dead-code elimination

const className = (() => {
  const className = "xyz123"
  const elm = document.createElement('style')
  elm.innerHtml = `.${className} { color: red; }` 
  document.head.appendChild(elm)
  return className
})();

document.querySelector('#my-elm').classList.add(className);

Procedural Macros (A Later Discussion)

I don't want to dive too deeply into this in this discussion, but I wanted to share as it's part of the larger vision.

Procedural Macros are similar to decorators, but can be applied to variable declarations, functions, classes, class members, etc, and augment the behavior of the target.

For example; Frameworks like Vue, Angular, Mobx, etc currently use complex combinations of compiler tricks and/or runtime libraries like signals to give users ergonomic access to detect state mutations and automatically trigger rendering calculations.

Procedural macros would grant both tool makers and developers alternative and/or simpler approaches.

For example, a procedural macro that in-place converts class properties into getters/setters;

import {reactive, subscribe} from 'reactive'

#[reactive.class]
class Foo {
  #[reactive.push]
  bar = 'Initial Value'

  constructor() {
    // This is difficult (or impossible?) to replicate with decorators
    setTimeout(() => this.bar = 'Updated', 1000)
  }
}

const foo = new Foo()
subscribe(foo, () => console.log('updated'))

Would expand into something like

import {subscribe} from 'reactive'

class Foo {
  [Symbol.for('reactive-state')] = {
    emitter: new EventTarget(),
    bar: 'Initial Value',
  }

  get bar() { return this[Symbol.for('reactive-state')].bar}
  set bar(update) { 
    this[Symbol.for('reactive-state')].bar = update
    this[Symbol.for('reactive-state')].emitter
      .dispatchEvent(new CustomEvent('reactive:update'))
  }

  constructor() {
    setTimeout(() => this.bar = 'Updated', 1000)
  }
}

const foo = new Foo()
subscribe(foo, () => console.log('updated'))

There are many other use cases, like simplified serialization/deserialization and creating RPC wrappers for instances located in Worker threads or external processes - however the key is that these implementation are done in user-land and can be compiled/optimized AOT by bundlers

Alternatives Considered

  • Using an external macro compiler
    • [6]
    • [7]
  • Adding this into TypeScript
    • [8]
    • [9]

The TypeScript team reject the idea of adding macros unless it's an EcmaScript feature (fair enough, it makes sense given the project goals).

Support for custom TypeScript transformers is going away with ts-go (see tool's have short shelf lives).

An external macro-aware compiler requires a custom LSP as well as documenting and maintaining plugins/loaders for bundlers.

2

Missing links due to link post limit for new users:

3
4

Hi! I'm the macro guy here, I think. I have a very detailed proposal about how to bring macros into JavaScript.

You mention a number of real problems with maintenance burden, and I agree wholeheartedly that maintenance burdens in our ecosystem have, in aggregate, spiralled out of control. It's becoming a problem especially rapidly now with multiple major tooling forks in Rust and Go, meaning that where before JSX was just tool factoring debt that existed in one set of tools, now many sets of tool maintainers have copied the debt and the ecosystem is bearing highly multiplied maintenance costs.

JSX is everywhere already so I think that to be able to clean up the mess we already made we have to find a way to be able to describe the reality of how things are already working. For me that means that i prefer to think of what's needed as language extension instead of macros, because JSX is, in practice, a custom language extension.

The debt that we owe is that tools now consider it pseudo-standard, and implement it as part of their core definition of JavaScript. They do this because there's is no other mechanism for them to use to support that extension. There could be such a mechanism in the future though! That's why I've chosen to focus my work on extensible parsers like this es3 parser which we later extend to es5 and then to es6 and beyond. We already support JSX as a higher-order language!

1 Like
5

JSX is everywhere already so I think that to be able to clean up the mess we already made we have to find a way to be able to describe the reality of how things are already working.

The benefit of my proposed macros approach is that it provides library makers a backwards compatible means to migrate their implementations to macros without breaking support for existing bundler configurations.

Macros can also coexist so you can have React, Preact and Vue in the same project without a transpiler

import {jsx} from 'react/macros'
import {jsx as pjsx} from 'preact/macros'

const ReactApp = jsx!(<div>Hello World</div>)
const PreactApp = pjsx!(<div>Hello World</div>)

Realistically, if a macro system was integrated into ES, I'm certain a JSX macro would be developed within hours. Such a macro would be compatible with React without changes to React.

It's also worth mentioning that JSX debt only affects React and React-like libraries. There are many other libraries (and prototypes that couldn't get off the ground) which would be able to adopt the new capability afforded by macros immediately - as well as non-gui use cases (e.g. a comlink-fork, serialization/deserialization libraries, and any library that uses decorators)

JSX is everywhere already so I think that to be able to clean up the mess we already made we have to find a way to be able to describe the reality of how things are already working

Personally, I'd caution against doubling down on the existing debt with a closed approach that is tailored to one specific GUI library.

i prefer to think of what's needed as language extension instead of macros

I'm assuming you mean there is an API exposed to allow extending the language parser at runtime (otherwise it would just be adding jsx to the language specification)

Something like:

import {JSX} from 'react/parser'
globalThis.engine.extendParser(JSX.parser({ pragma: "createElement" }))

The biggest issue I see with an extensible parser is the complexity and performance impacts associated with running it.

To run in the browser, you'd need to supply the parser extension to the browser. To run in a bundler, you'd need to supply the parser extension to webpack, rspack, parcel, rollup, swc, tsc, oxc - which begs the question of what language the parser extension will be written in.

If a parser extension is written in JavaScript - then native transpilers (ts-go, swc, oxc) will not be able to use it. swc and oxc might be able to embed a v8 runtime to consume the parser extension, but then they would need expensive mapping logic. ts-go on the other hand simply couldn't consume extensions written in JavaScript.

Native transpilers also need to identify the custom parser before using it. Which begs the question of control flow. When do you declare a custom parser? What happens before it's declared? What happens inside of Workers or iframes?

By contrast, static macro syntax can be implemented in v8 as native code - which means it can be highly efficient as it's a simple static language expansion (basically handlebars). Extensions supplied at runtime have to be in JavaScript, which would be slow (important for Nodejs targets that would use them during testing).

Additionally, what happens if you need to use multiple custom parsers? For example, you are writing Vue and need templates and the ability to modify class properties to be getters/setters - but you also use React because you're migrating. Or perhaps you are using React and migrating to Preact.

The benefit of macros are they are a generic method to open the language to any extension which would evolve at the pace of the implementer (rather than TC39). This is evidenced in how Rust, despite being a systems language, has support for a React-like GUI library and a Vue-like GUI library without the use of a transpiler. The design of their macro system opened the language up to creativity of library implementers and amazing solutions were developed without the need to involve their standards committee or use any external tooling.

6

To be clear, I'm not against macros. I'm both a user and one of the maintainers for babel-plugin-macros, so I'm pretty well aware of how nice it is to be able to scope transformations to particular syntax, and to make the macro transformations explicit and discoverable in the process.

I reiterate what I said however: the focus should be on being able to describe the code we have already written. We've probably got hundreds of millions of files out there using non-macro JSX as well as other nonstandard syntactic extensions like Typescript types or CSS-in-JS imports. To me that's a major debt: a lot of code that belongs to our ecosystem that we just don't comprehend well enough to be able to run. People wish they could just run their code!

I suspect we'll eventually support scoped macros like jsx!() and your example is a succinct display of a real purpose they might be put to, but the vast majority of JSX users aren't going to want to write jsx!() around every single JSX tag in their application. They're not going to want to do it because it would cost them something (ease of reading and writing their code) and they wouldn't get anything in return except for what they already had. For most real applications writing jsx!() repeatedly would only really be visual noise making it harder to scan the code to quickly grok its purpose.

You mentioned:

I'm assuming you mean there is an API exposed to allow extending the language parser at runtime (otherwise it would just be adding jsx to the language specification)

You assume correctly. Wouldn't your idea also require a mechanism like this though? After all, you used nonstandard JSX syntax in the source file, so now as far as I can tell you still need a custom parser to be able to parse the source file. There are nearly no limitations on what kind of crazy syntax can go between JSX tags, including arbitrary JS expressions which may be interpolated. If you don't give the jsx macro control of the runtime's parser, you're going to have to make draconian restictions like "the character ) is reserved inside macros" as that would be the only way the parser could tell when the expandable macro is over.

I agree with most of your analysis of the difficulties involved in parser extension, I just think that that work will be absolutely necessary both for macros and to be able to support the code we've already written.

1 Like
7

Thanks for the response! Hope I'm not overly interrogating your ideas, I appreciate the sparring as you've obviously thought deeply about the subject for a lot longer than I have.

the focus should be on being able to describe the code we have already written. We've probably got hundreds of millions of files out there using non-macro JSX.

That's a fair point. I can't back this up but my gut tells me to trust in the adaptability and competency of developers using the language. Plus moving jsx to an inline parameterized macro is pretty easy to automate via codemod - and the benefits are tangible enough to justify the migration.

as well as other nonstandard syntactic extensions like Typescript types or CSS-in-JS imports

That's true. Macros can't solve css-in-js ES imports. It targets use cases like css-in-js (like styled-components).

In Rust, they do have a mechanism for macros to work with files

let foo = css!("./path/to/file.css"); // Internally calls fs.read_to_string(path)
// Or
let bar = css!{ color: blue; };

But that would mean macros need access to fetch (not in the spec) to load the css file as text. Then you need to worry about macros fetching remote files and how transpilers would work with file loading.

For most real applications writing jsx!() repeatedly would only really be visual noise making it harder to scan the code to quickly grok its purpose.

I can see this point, my gut tells me that it's too subjective to quantify. This doesn't really occur in the Rust world as macros overall reduce the verbosity of code.

I can't speak for all but personally I value visual traceability, having import {jsx} from 'react/jsx' in scope is unambiguous - by comparison a .jsx or .tsx file requires me to interrogate the tsconfig and the bundler configuration (which might be out of sync).

That said, macros wouldn't replace bundlers, they'd simply reduce the demands on the pre-processors as much of the transformations can be described within the libraries themselves.

Also, it's not that ugly :joy: Here's jsx in rust:

fn render_greeting(name: &str) -> DOMNode {
  let stylesheet = css!("example.css");

  return rsx! {
    <text style={stylesheet.take(".text")}>
      { greeting_str(name) }
    </text>
  };
}

Wouldn't your idea also require a mechanism like this though?

Hmm.. great point. You're right, they totally would.

While we'd have the ability to use a simple templating syntax for some simple inline macros - non-trivial macros (like css, jsx, html, etc) would need to be expanded programmatically which necessitates using JavaScript to parse and reorganize tokens.

That alleviates the issue of how/where to "register" a parser (as the implementation is just imported at the call site) - but it does present new challenges;

  1. Macros can't have access to globals like localStorage or anything set by the user, etc.
  2. Native transpilers need to identify macros and run them inside of a JavaScript runtime.

So I guess that naturally leads us to requiring the isolation of macros (I believe parsers would share this challenge) to running within their own lightweight restricted contexts, which then requires the addition of some kind of sandboxed ParserWorker/MacroWorker.

So library authors would need to write something like

// react/jsx.js
export const jsx = MacroRegistry.registerCallable('./react_jsx.js')

// app.js
import {jsx} from 'react/jsx' 
const App = () => jsx!(<div>Hello World</div>)

The good news is that, if the isolated environment only features ES syntax, native transpilers can essentially use an unmodified quickjs - which has bindings for Go and Rust.

The bad news it that it's difficult for transpilers to statically identify usage.

Tricky tricky.

Perhaps, similarly to import, it's enough for the ES spec to define how to register a macro and how to call a macro, but not how the macro is evaluated. The runtime can then determine it wants to spawn worker threads, give them fetch and load balance between them

8

I don't quite understand, why are tagged template literals not a sufficient solution for the presented use cases?

9

I can't speak for anyone else but I'm here to advocate for something much richer than just macros, which is a method for parser extension.

A method for parser extension would allow existing JSX and Typescript code to become valid according to the standards again, healing a tremendous wound in the ecosystem caused when a plurality of people started writing code in .ts files which are completely opaque to TC39's standards and so to any standard interpreter.

I don't think it's my job to second-guess that people want to be able to extend the parser. They've demonstrated overwhelmingly that they want to extend it -- that they will extend it.

What I will advocate fiercely for is to create a level playing ground for technologies. I object to the idea that runtimes like Deno or Bun should be baking in specific support for .ts files and JSX. In practice that makes these syntax extensions have the weight and force of standards, while new or competing syntax extensions are nearly impossible to make since they involve adding support to every single parser -- tens if not hundreds. If it's inevitable that people are going to extend the parser, there should be a standard mechanism that ensures that access to the opportunity is equitable and can't be used as leverage to take control away from TC39 and can't be used as a cudgel by entrenched technologies to freeze out upstarts.