`utf8` / `buffer` import type assertions

With import ... from ... assert {type: 'json'}, one can now safely import JSON files into an application. Because this syntax is native to the language, all environments support it, including bundlers, making it entirely portable. As a perks, it also supports dynamic loading through import(...), avoiding authors from having to always include potentially large datasets into their applications.

However, while many applications use JSON in order to transfer data, not all do. Either because the files are originally provided as third-party formats or because the project author crafted a special format to more efficiently store data, it may occur that project authors have to deal with various arbitrary types. Some examples:

  • Text: Markdown, 3D OBJ files, XML, CSV, ...
  • Binary: Wasm, archives, protobuf, images, ...

There is currently no portable way to import any of these files in a JS application today. Bundlers (and some runtimes) support it via proprietary logic (loaders), but they don't share any common syntax. Browser environments would tend to use fetch to load files, and Node-like environments could read the files from the file system using dedicated APIs, but they don't mix (fetch on local files could potentially be supported in Node environments, but it has various issues security-wise, more on that here).

At the same time, supporting loaders in general is a very difficult topic as well (different runtime have wildly different requirements), so this isn't the point of this proposal. Instead, I'd like to focus on the minimal features which would let JS authors portably take the matter into their own hands, by letting them import any file and process them however they'd like.

To this end, I'd like to propose two additional type assertions:

  • utf8 would return a synthetic module whose default export would be a string representing the resource
  • buffer would do the same but the default export would be an ArrayBuffer (or Uint8Array?) instance

Using these two primitives, project authors would be able to load any file type they want, and process them according to their needs.

How would people feel about those?

1 Like

I would definitely find these useful, although it would make more sense if they were loaded using an import reflection as assertions don't change interpretation, they simply validate it.

Yes, interpretation would need to change indeed as things like import(import.meta.url, {type: 'utf8'}) wouldn't work otherwise. However I'm not sure import reflection would work - it doesn't seem to provide a way to make explicit what is the module type, correct? So how would it know that the file to load is a text file, not a JS one?

However I'm not sure import reflection would work - it doesn't seem to provide a way to make explicit what is the module type, correct?

In the import reflection proposal currently only raw module interpretation is proposed, specifically:

import module wasmModule from "./module.wasm";
// Also requires some stuff from module expressions proposal for this
import module jsModule from "./module.js";

console.log(wasmModule); // WebAssembly.Module
console.log(jsModule); // Module (proposed in module expressions proposal)

Dynamic import is similar, although it just puts it an options bag:

const { default: wasmModule } = await import("./module.wasm", { reflect: "module" });
const { default: jsModule } = await import("./module.js", { reflect: "module" });

Note that it is possible to load both:

// Presuming wasm JS integration as proposed https://github.com/WebAssembly/esm-integration
import * as wasmInstance from "./module.wasm";
import module wasmModule from "./module.wasm";

console.log(wasmInstance); // js module wrapper around a WebAssembly.Instance
console.log(wasmModule); // WebAssembly.Module

In this regards adding a new format like text would presumably just be a new keyword, i.e.:

import yaml from "yaml";
import text config from "./config.yaml";

// ...
yaml.parse(config);
// ...

Personally I don't think the fact it needs new syntax for import reflection is great given it means adding text/binary as kinds would need proposals for treating those as keywords in those positions, previously the proposal added a more generic syntax:

import wasmModule from "./module.wasm" as "module";

There is still an open issue about this though.

There are lots of discussions around these kind of use cases happening right now in the context of Module Loading Harmony.

The overarching repo is still GitHub - tc39/proposal-compartments: Compartmentalization of host behavior hooks for JS but that may change to restructure things more as an "Epic" proposal composed of individual proposals like import reflection and module expressions.

The matrix logs are here: https://matrixlogs.bakkot.com/TC39_Loader

The TL;DR right now seems that there may be a new "as" modifier which unlike import assertions would be part of the module cache key, and would allowed a single identifier being loaded differently as separate modules. This modifier would be revealed to importHook, allowing userland to implement support for values not supported by the host.

Module reflection is different because it simply deals with the different loading phases of a given module. Aka the underlying module is the same, you just have a representation of a different stage of its instantiation.