# Streaming JSON **URL:** \ **Category:** đź’ˇ Ideas\ **Created:** [September 18, 2021, 5:22am UTC](https://es.discourse.group/t/streaming-json/986 "2021-09-18T05:22:00Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![claudiameadows](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/claudiameadows/32/126_2.png) [@claudiameadows](https://es.discourse.group/u/claudiameadows)\ **Post date:** [September 18, 2021, 5:22am UTC](https://es.discourse.group/t/streaming-json/986/1 "2021-09-18T05:22:00Z")
`JSON.parse` and `JSON.stringify` are eager (and blazingly fast), but doesn't scale beyond a few megabytes on servers and a few hundred megabytes on clients. It's also not especially useful for resource-constrained IoT in general for similar reasons. I would like to see a streaming variant similar in spirit to Jackson for JVM apps. Here's a rough sketch as a starting point: - `enumerator = JSON.tokenize(iterable)` - Tokenize an iterable/async iterable of strings and/or array buffers into a token enumerator\* - Newline-delimited JSON is accepted by default, allowing it to restart and be usable in such an environment. If you only want a single value, just read that one value and immediately call `enumerator.close()`, and it'll just work. - No reviver is supported as it's...kinda useless in this context. (That data path is already exposed, so it doesn't matter.) - `enumerator.nextToken(typeMask = JSON.ALLOW_ALL, maxScale = BigInt(Number.MAX_SAFE_VALUE))` gets the next token matching a type bit mask\*, returning `JSON.INVALID` on mismatch - `JSON.ALLOW_NULL` - Accept `null` - `JSON.ALLOW_TRUE` - Accept `true` - `JSON.ALLOW_FALSE` - Accept `false` - `JSON.ALLOW_BOOLEAN` - Sugar for `ALLOW_TRUE | ALLOW_FALSE` - `JSON.ALLOW_STRING` - Allow strings - `JSON.ALLOW_NUMBER` - Allow number - `JSON.ALLOW_INTEGER` - Allow integer and distinguish integers from floats by returning integers as bigints - `JSON.ALLOW_OBJECT_START` - Allow that relevant token - `JSON.ALLOW_OBJECT_END` - Allow that relevant token - `JSON.ALLOW_OBJECT_SEEK` - Allow object end and ignore entries leading up to it - `JSON.ALLOW_ARRAY_START` - Allow that relevant token - `JSON.ALLOW_ARRAY_END` - Allow that relevant token - `JSON.ALLOW_ARRAY_SEEK` - Allow array end and ignore entries leading up to it - `JSON.ALLOW_ALL` - Sugar for all the above OR'd together except `ALLOW_INTEGER` - A bit mask is used to reduce GC overhead, as this is a very perf-sensitive operation. - Max scale (applies to bigints and strings) can be passed to optimize for things like dates, Ethereum hashes, and compiled schema enum variants, to reject obviously invalid data early. - `enumerator.ready()` returns a promise that resolves once it either has data to parse or completes - `{writer, output: iterable} = JSON.generate(opts?)` - Generate an async iterator of strings or array buffers (by option) - This generates newline-delimited JSON. `writer.close()` can be used to terminate such a stream, and a single value can be written followed by `writer.close()` to ignore subsequent values. - `writer.ready()` returns a promise that resolves once it is ready to output more - `writer.closed` returns `true` if either a full value has been written or if `iterable.return()` has been called. - `writer.write(token, treatNumberAsNonInteger = false)` - Write a token - `treatNumberAsNonInteger`, if `true`, ensures that all numbers are suffixed with a `.0` if they're integer numbers. - Returns `false` if the iterable isn't ready for more. - Async iterators here can have their `next` method invoked with an optional chunk size in bytes, to help resolve backpressure. - Replacers and revivers are not supported as I don't want to require state beyond the bare minimum necessary to sustain this - it's supposed to be lightweight.\*\* - Tokens are immediate values (if neither objects nor arrays), or the following symbols: - `JSON.INVALID` - can only be returned from `enumerator.nextToken` and is not valid as an argument for `writer.write` or `enumerator.maybeConsume` - `JSON.CLOSED` - `JSON.UNAVAILABLE` - `JSON.OBJECT_START` - `JSON.OBJECT_END` - `JSON.ARRAY_START` - `JSON.ARRAY_END` - Note: objects are simply alternating string + value pairs, to simplify the API. Users are expected to handle these appropriately, even though there's defined behavior if they don't. \* This explicitly eschews the existing reviver/replacer idioms in favor of manual handling of values to avoid the overhead, and it also uses bit masks for the type specification to avoid garbage collection overhead. This is because the code lies on a critical performance path, and necessarily needs to be decently fast to compete with `JSON.stringify` at all. (It's a fairly data-driven API.) \*\* The stack could be implemented as a bit stack, where 0 = array and 1 = object value - an engine could optimize for the common case by simply using a 31-bit integer initially and only upgrading to a heap bit array later. Everything else could be tracked via simple static states, resulting in near zero memory overhead aside from buffered values. Replacers require tracking cycles, and I want to avoid the overhead of that in this API. \*\*\* This obviously pairs well with the `yield.sent` proposal, but neither depends on the other. ---
**Author:** ![conartist6](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/conartist6/32/515_2.png) [@conartist6](https://es.discourse.group/u/conartist6)\ **Post date:** [April 16, 2022, 5:57pm UTC](https://es.discourse.group/t/streaming-json/986/2 "2022-04-16T17:57:37Z")
I'm working on userspace tooling for building streaming parsers at the moment using my [streaming regex engine](https://github.com/iter-tools/regex). It wouldn't have anything like the level of speed that what you're proposing would, but maybe it's of use to someone in a resource-constrained environment anyway: [A human-friendly json parser with parserate · GitHub](https://gist.github.com/conartist6/71fd8e40f34a6a2267d4fd6990cb686c) Note that this approach isn't really actually viable yet because 1) I haven't written the library and 2) it's 10x slower than it should be for reasons (tons of awaits) that I'm still trying to get to the bottom of. ---
**Author:** ![jschoi](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/jschoi/32/917_2.png) [@jschoi](https://es.discourse.group/u/jschoi)\ **Post date:** [May 9, 2022, 3:06pm UTC](https://es.discourse.group/t/streaming-json/986/3 "2022-05-09T15:06:39Z")
I have always found this problem (processing large and asynchronously loading JSON boluses) and this solution (a streaming parser API) to be interesting. However, I do not currently have the mental bandwidth to champion this in the foreseeable future. My apologies. As an aside, I note that we would have to decide whether to go with an async-generator-based API or an HTML5 ReadableStream-based API. This would have ramifications for in which specification would standard JSON streaming live. I do think that it would be appropriate to live in TC39’s Ecma262, rather than WHATWG’s HTML5, given that JSON is such a widespread need independently of host environment. But that would imply that the API would be based on async iterators, which would lack ReadableStream’s cancellation etc. ---
**Author:** ![claudiameadows](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/claudiameadows/32/126_2.png) [@claudiameadows](https://es.discourse.group/u/claudiameadows)\ **Post date:** [May 10, 2022, 3:43pm UTC](https://es.discourse.group/t/streaming-json/986/4 "2022-05-10T15:43:23Z")
I've been excessively busy myself, hence why I've not followed up on this. ---
**Author:** ![Delapouite](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/delapouite/32/340_2.png) [@Delapouite](https://es.discourse.group/u/Delapouite)\ **Post date:** [December 1, 2025, 1:17pm UTC](https://es.discourse.group/t/streaming-json/986/5 "2025-12-01T13:17:04Z")
Deno provides some utilities around this topic with `JsonStringifyStream` in their std lib [@std/json - JSR](https://jsr.io/@std/json) ```js import { JsonStringifyStream } from "@std/json"; import { assertEquals } from "@std/assert"; const stream = ReadableStream.from([{ foo: "bar" }, { baz: 100 }]) .pipeThrough(new JsonStringifyStream()); assertEquals(await Array.fromAsync(stream), [ `{"foo":"bar"}\n`, `{"baz":100}\n` ]); ``` ---
**Author:** ![conartist6](https://yyz2.discourse-cdn.com/free1/user_avatar/es.discourse.group/conartist6/32/515_2.png) [@conartist6](https://es.discourse.group/u/conartist6)\ **Post date:** [December 4, 2025, 2:30pm UTC](https://es.discourse.group/t/streaming-json/986/6 "2025-12-04T14:30:57Z")
As it turns out I've done nothing but work on streaming parsers for JS for the last several years. I can now even do a streaming parse of Javascript from Javascript which allows me to do things like emit import statements for further analysis before I'm even finished receiving the full file over the network pipe.