One of the major criticisms I've encountered so far is of my focus on speed in the abstract without evidence that there's a sufficiently broad need for the perf characteristics I'm talking about -- that perhaps these needs are "niche" because they mostly apply to streaming parsers. I want to very specifically rebut that claim. Code is probably the most common data type most of us here interact with, and also one of the most valuable. Streaming parsers have the potential to have a significant impact on the way we are able to interact with code because they don't just perform a bit faster or slower, they perform differently. Here's how:
Streaming parsers are significantly more memory efficient as they allow you to avoid ever having to load the entire input into memory. This also enables them to run better in memory-constrained environments. It also allows them to run on infinite streams of data. Imagine being able to build a chat that effortlessly streams data in the exact format it stores it: a JSON array of messages.
Even when they are slower than traditional parsers streaming parsers may still be more responsive to humans due to the advantages of concurrency. This was understood by web browsers, which implemented streaming parsing as a core technology.
Streaming parsers allow unnecessary work to be skipped. For example the top of a file could be read and parsed for docblocks and imports without needing to read the body of a file from disk. The building (and garbage collecting) of AST nodes can also be skipped when in is not needed, such as in syntax-aware code searches (!!). The ability to skip code may have perf impact even beyond the cost of code not executed as storage and processor caches can be more effective.
Note that while the creation of streaming parsers is already possible, I'm specifically thinking of imperative parsers, which are some of the easiest to write due to the ease of understanding and debugging their behavior. This is due to the purely top-down control which tends to create the most relevant and succinct call stacks.
To sum up: I think this shift is coming, and not just in the abstract. I'm working full time to build the first generation of these tools, because I'm pretty sure the market will reward me for being able to create a variety of powerful and performant tools that can only be created when you have a comfortable, powerful, and reliable way of expressing streaming parsers with code.
[Sorry for completely rewriting this in edits. It was hard to keep it organized and focused.]