Function call context via `function.meta`

💡 Ideas
1

Edit: update link

Proposal: link

Initial draft, hidden for brevity

Here's the initial code snippet, to explain it briefly:

let rootMeta = function.meta
let fooMeta
let barMeta
let bazMeta

function foo() {
    fooMeta = function.meta
    bar.callWithContext({[three]: 3}, /* `this` value */ void 0, () => {
        console.log(function.meta === fooMeta) // true - bound to `this` context
        console.log(function.meta[three]) // undefined - bound to `this` context
    })
}

function bar(init) {
    barMeta = function.meta
    console.log(barMeta[three]) // `3`
    init()
    baz()
}

function baz(init) {
    bazMeta = function.meta
    console.log(bazMeta[three]) // `3`
}

foo.callWithContext({one: 1, two: 2})

// They don't follow the standard prototype chain, for safety.
console.log(rootMeta) // {[[Prototype]]: null}

// They inherit their parent contexts, to save on memory.
console.log(fooMeta) // {[[Prototype]]: rootMeta, one: 1, two: 2}
console.log(barMeta) // {[[Prototype]]: fooMeta, one: 1, two: 2, [three]: 3}
console.log(bazMeta) // {[[Prototype]]: fooMeta, one: 1, two: 2, [three]: 3}

// They only update whenever explicitly updated, to save on memory.
console.log(rootMeta === fooMeta) // false
console.log(fooMeta === barMeta) // false
console.log(barMeta === bazMeta) // true

// They're frozen, for additional safety and some extra optimizability.
console.log(Object.isFrozen(rootMeta)) // true
console.log(Object.isFrozen(fooMeta)) // true
console.log(Object.isFrozen(barMeta)) // true

// New realms create new root metas
let newRealmMeta = new Realm().eval("function.meta")
console.log(newRealmMeta) // {[[Prototype]]: null}
console.log(Object.isFrozen(newRealmMeta)) // true

// Call contexts don't penetrate realm boundaries
console.log(newRealmMeta === rootMeta) // false
console.log({}.isPrototypeOf.call(rootMeta, newRealmMeta)) // false
1 Like
2

This doesn't explain much to me - can you elaborate on what problem it's solving?

3

Yes! This is something I've been actually wanting recently. I've actually made an npm package in the past to try and achieve this purpose (here), but it's severely limited in that it won't work with async functions, and I couldn't figure out any way around that limitation without native support (or tricky, fragile hacks).

@ljharb - here's how I understand it (@claudiameadows can correct me if I'm off). The idea is to implicitly pass certain parameters around, without having to wire than through each step of the call stack. For example, when working in node, it's common to pass a "req" and "res" parameter all over the place. With a function call context, you can cause these parameters to be available to all of your functions (from a certain stack frame and higher), without explicitly passing them around.

For example:

app.get('/my/route', async (req, res) => {
  const callback = async () => {
    await f()
  }
  await callback.callWithContext({ req, res })
})

async function f() {
  await g()
}

async function g() {
  await forSomething()
  const context = function.meta
  console.log(context.req.params.xyz)
}

In the above example, notice how we can get access to the req parameter, even though we didn't explicitly pass it into the f or g function? When a callWithContext() occurs, any stack frame above the current one will have access to the passed-in context object.

(I'm not sure if @claudiameadows's proposal supports async functions either, so it's technically possible that the above example does not work.)

Some use cases:

  • React has been using a context idea similar to this for some time now, and it's worked very well. It allows parent components to pass things like "theme colors" to all of their child components without having to be explicit. I don't know if React could use a built-in language feature like this or not (it's a little bit different of a setup), but from a usability point of view, it's got a similar concept.
  • You can pass around values such as "req" and "res" without putting those in every function signature
  • You can pass around configuration for your whole app (like, is a "verbose" flag set?), without having to rely on global state (which makes testing easier).
  • It makes refactoring easier. Say, for example, you want to start tying your log output to the current request, so you decide to instantiate a new instance of a Logger each time a request is made, and you give that logger details about the current request. Now imagine you want to start using these logging instances throughout your program. Instead of having to explicitly wire the logger instance through all of your function calls, you can just provide it everywhere, by putting this logger instance into a function call context before calling any of your utility functions. (At work, we have a similar issue, and we just attached the logger instance onto the req object, since req gets passed around everywhere anyways. This is more of a hack solution, and isn't very great).
  • Here's a theoretical API I wanted to make recently until I realized it was not possible. I needed to throw on a quick-and-dirty async locking mechanism for part of our API. My original plan was create an AsyncLock instance that provided a "lockWhile(() => { ... }" method. As long as it's callback was executing, a lock would be set, and no other lockWhile() callback could execute until the first callback finished. The problem was that I wanted to be able to nest lockWhile() callbacks (e.g. maybe getUserProperty() would use lockWhile() during execution, and incrementUserProperty would also use lockWhile(), but would call out to getUserProperty() during execution). The issue was that there was no way of detecting if I was in a nested lock callback, and so such an API would just deadlock as soon as a nested lock was encountered. The current "correct" solution would be to explicitly pass this lock instance around everywhere, but I wasn't about to modify that much of our codebase to do this basic task.
4

A couple of bike-shedding notes:

I feel like a common use case would be to call anonymous functions with a call context, which is currently awkward to do:

(() => {
  // ...
}).callWithContext({ x: 2 })

// or

const callback = () => {
  // ...
}
callback.callWithContext({ x: 2 })

What if we made this a static property on Function instead?

Function.callWithContext({ x: 2 }, () => {
  // ...
})

Next, if everyone shares the same context, there's lots of potential for name collision (unless you're careful enough to use symbols). What if you can make a number of distinct context (similar to React contexts), and the only way to access the value of your particular context is if you have a reference to an object that represents that context. For example:

// We create new contexts.
const configCtx = Function.createContext()
const anotherCtx = Function.createContext()
const aThirdCtx = Function.createContext()

// We're supplying the context value { verbose: true} to configCtx
Function.callWithContext(configCtx, { verbose: true }, () => {
  f() // f() will print a verbose message
})
Function.callWithContext(configCtx, { verbose: false }, () => {
  f() // f() will not print a verbose message
})

function f() {
  // We're retrieving the context value with .get().
  const { verbose } = configCtx.get()
  if (verbose) console.log('f() called')
}

Function.callWithContext(anotherCtx, { x: 2 }, () => {
  Function.callWithContext(aThirdCtx, { y: 3 }, () => {
    console.log(anotherCtx.get().x) // works
    console.log(aThirdCtx.get().y) // works
    console.log(configCtx.get()) // Throws an error, configCtx is not currently being supplied a value
  })
})

This idea is similar to how I implemented it in my npm package, which might explain it better than what I did here.

5

React's context feature that works like this is one they've moved away from - the modern approach is the useContext hook, which has an explicit "get" call in each component.

While I've had lots of use cases for React's "legacy" context, there is a lot of implicit magic about it - slightly mitigated by the fact that a React component has no access to any context it hasn't explicitly requested via .contextTypes.

In other words, a big difference between function.meta and React legacy context is that in function.meta, you have a set of implicit, magic globals, with no static way to figure out which functions are using which data. In other words, the explicit presence of a "context request" seems like a very important thing to have.

1 Like
6

I don't actually know much about React's legacy context API, I was mostly referring to the new one, but looking it up, I guess the legacy context API more closely resembles the current proposal.

Does the second point I brought up in my last "bike-shedding" post fulfill this requirement? There's separate context objects, and if you want a particular one, you have to explicitly call contextInstance.get(). You don't magically get all of the provided context values, just the ones you sepcifically request. This sounds like it more closely resembles the newer React context API.

Edit: I just updated it the previous post to make it more clear how multiple contexts would work together

7

Related proposal (if I've understood correctly):

8

The proposal's definitely not perfect (that callWithContext is deliberately kinda ugly, and function.meta.foo is also obviously cumbersome).

You could imagine an alternate API like this (note: this can't be polyfilled or transpiled at all if you consider generators and async functions):

  • context = new Context(defaultValue?) - Create a context
  • context.get() - Get the context's value for the current call context
  • result = context.run(value, () => result) - Run a callback with the context's value set to that value within that context.

I'm just trying to spur some discussion here on the core concept of call context as a native thing.

Edit: Redid the proposal off this, just with use instead of run (works the same way).

9

Mine is similar to that, but is a bit cleaner and is better defined with non-async functions as well.

10

Edit: Made some edits after I made this post, and now it's a bit outdated.

@theScottyJam

Redid my proposal to better match that. I also went based on React's current context primitives, which I agree are cleaner.

  • context = new Context() ≈ context = React.createContext()
  • context.get() ≈ useContext(Context)
  • context.use(value, init) ≈ <Context.Provider value={value}>...</Context.Provider>

Trying to avoid the temptation to tag it onto Function because while it looks like it deals with functions, it really does things with the execution context, not really the function. And unlike async context as @aclaymore brought up, this doesn't really do dynamic scoping like that.

1 Like
11

That is an interesting and related proposal. It seems with it, you would be able to mimic this feature for async functions without too much trouble (but I don't think you would be able to mimic generator support).

It does look like it's intended to solve a different problem. It also looks more powerful than I would want - I like the restriction that you can only access a context value higher-up the callstack, it makes it easier to track down where it got set. in that proposal, if I understand it correctly, one location high up a call stack could set a value, then another very different location that's still part of the same promise chain could access that value.

12

I'm trying to figure out how this would actually work with async contexts. Take the following example:

myContext = new Context()

const wait = ms => new Promise(resolve => setTimeout(resolve, ms))

context.use({ x: 2 }, async function() {
  setTimeout(function () {
    console.log(myContext.get())
  }, 100)
  setTimeout(function () {
    console.log(myContext.get())
  }, 500)
  await wait(300)
})

What would be printed out by these two console.logs()? Do they both log out the context value? Would just the first one log it out? Or do neither of them log it out?

Also, @claudiameadows, in your updated proposal, you mentioned that arrow functions don't work with context.use(). Could you expound on this and the reasoning behind it?

13

This is an interesting idea, but I think you're going to have a very, very hard time convincing the committee to add anything which looks like dynamic scope to the language. So if you want to pursue this, I'd recommend trying to make it much clearer what problem this is intended to solve before worrying about exactly what it should look like.

2 Likes
14

It's propagated by stack traces, but is stored in the this context.

1 Like
15

Yeah, I need to sit on this a little more and refine it better. I slapped it together in the span of about an hour, without too much thought, even though I knew of a few use cases for it. I'll keep @theScottyJam's initial example in mind when I redo the proposal (again) and put more actual thought into it.

16

Maybe I'll try to verbalize better why I think this proposal is useful (just to express my own point of view on the matter). I listed off a number of use cases already, some of which are more important than others, but I think the primary thing this proposal provides is the proper facilities to deal with "variadic dependencies" (I made that phrase up, but I'll explain what I mean)

In coding, unless you're doing dependency injection, we generally make each module in charge of declaring their own dependencies. If, for example, I'm wishing to import and use an internal logging module, I shouldn't have to worry about what it depends on to write these logs. It might just use node's fs module, or an npm package, or maybe it sends logs over the network, I don't really care how it logs, I just want it to do its job. If at some point this logging module switches to using an npm package, I shouldn't have to change a single line of code outside of the logger. These are how "static dependencies" work.

Sometimes a module/function requires "variadic dependencies". These are dependencies that vary depending on the current context. Currently, we just call these dependencies "parameters" and treat them as such, but doing so can lead to some nasty issues. Let's go back to our logger example. We decide to make a change to how logging works, and wish to link logs to the request the log happens in, That is, each inbound request will be assigned a randomly generated request ID, and each time we log something out, that request ID will get logged with our message. There's currently only one way to achieve this desired effect - we've got to update every single intermediate function between where the request is received and where logging happens, and add the request ID as a parameter. These intermediate functions shouldn't care whether or not logging wants to use a request ID or not, this kind of change shouldn't be something that requires updating half the codebase. With function call contexts, this request ID can be treated as a variadic dependency instead of a parameter, and we only need to update two places to provide this dependency - the logging module, and a file where we can inject a bit of code that runs with each request, which will provide the request with a function call context.

For prior art, there's actually a good number of places where this concept has been put into practice (albeit in different shapes). React contexts are an obvious shining example, but there are other subtle ones too. For example, here's a code snippet showing how to use Python's flask library:

from flask import request

@app.route(...)
def login():
    myParam = request.args.get('myParam')

Notice that the request parameters are not being passed into the login function? Flask is able to automatically detect which request is currently being handled. Many frameworks provide this kind of functionality.

In Javascript, we don't have the option to design an API like that. All variadic dependencies have to be explicitly passed around. In earlier versions of express, it was not uncommon for people to tack values onto the req object, as that req object tended to get passed around everywhere (req and res themselves are variadic dependencies to many functions), and it allowed other values to hitch a free ride to many parts of a project without having to explicitly update tons of function signatures. This was effectively a hack due to the missing ability of providing variadic dependencies to other areas of a project. Later versions of express standardized this hack by adding a blank res.local object - this was the intended location to attach arbitrary values for the whole project to access. The fact that people everywhere are using this res.locals object to provide values to their whole program shows how much of a need the community has to provide a proper context system, and providing a proper function call context system would allow us to give developers a standard, more flexible, and less error prone way to deal with these variadic dependencies.

17

A paper (no paywall) on a similar concept that might be of interest:

Section 3.6. Ambient Programming

2 Likes
18

this is dynamically scoped.

As others pointed out, that's essentially dynamic scoping. Although the term usually refers to how this is scoped in JS, or local vars in Perl or

Bash 
f() { echo "f:var == $var"; }
g() { local var=bar; f; }
var=foo
g # prints f:var == bar
f # prints f:var == foo

In essence, it's still dynamic scoping even if implemented purely in library without extra language support, like contextvars in Python, or merely as a designated request.local scope object.

That's pretty confusing, your example code explicitly says twoCtx.use(2, ()=> ...) and the function ignores that. I'm also not sure what you mean by "stored in the this context". You'd probably need to add a new field to func.env.rec. (or to decl.env.rec. so that it's also available at module level), in which case I see no point giving it the this treatment in arrow functions.

I'm going to refer to this new field as [[DynamicScope]]. It could be a plain object, but only accessible via "ContextVar" (which you called Context) keys to prevent name clashes. Instead of Context.use method that only allows you to set one value for the call, I'd rather call-with-copied-scope, and set new values inside.

const a = new DynamicallyScopedVar('a', 1)
const b = new DynamicallyScopedVar('b', 2)
const aplusb = () => a.get() + b.get()

function bar(anew) {
  a.set(anew)
  return aplusb()
}

assert( aplusb() === 3 )
// similar to how bar.call(that, ...args)
//   calls bar with `env.rec.[[ThisValue]]` set to `that`
// bar.subScope(...args)
//   would call bar with `env.rec.[[DynamicScope]]`
//   set to a copy of the current `[[DynamicScope]]`
assert( bar.subScope( 40 ) === 42 ) // modifies copy, 40+2
assert( aplusb() === 3 )
b.set( 13 )
assert( bar( 8 ) === 21 ) // modifies current dyn.scope, 8+13
assert( aplusb() === 21 )
assert( bar.subScope( 40 ) === 53 ) // modifies copy, 40+13
assert( aplusb() === 21 )
19

Why would you rather this? I certainly don't deny that what's being proposed is very similar to dynamic scoping, but the currently proposed implementation at least has certain restrictions on how the dynamic scope can be used.

The current design prevents someone from modifying a context variable deep down in one callstack and having that effect code far off in another direction. If my context variable is not what I expected it to be, I just have to look down the current callstack and find the most recent location that was setting it. These context values are "constant" in a way, and can only be shadowed, not modified. In your proposed version, any code that's been executed within the current context could have made the unexpected modification to the context variable - there's a lot more stuff to search through in order to debug this issue.

(Also, if a concern is that only one context can be set to a value at a time with the current proposal, that can certainly be updated in different ways. In my own npm package, I achieved this by allowing a key-value mapping of contexts to be set. i.e. something like this: Context.use({ [myCtx1]: 2, [myCtx2]: 3 }, () => return myCtx1.get() + myCtx2.get()) )

20

That's an interesting paper @aclaymore - I really like the idea the Koka language was founded upon (from the paper). It makes a lot of sense to be required to list a function's side-effects in the type signature. Their specific solution for express's req/res problem is interesting - using their effect system, they found a way to implicitly pass around the req/res parameters. This makes it so you don't have to explicitly call these functions with a req/res object anymore, but you do still have to update type-signatures to signify that you use the req/res effect. A variation of this could be used as a halfway solution if contexts are too extreme.

Say I need to refactor my codebase to provide, say, a request ID to the logger, here are my options:

  1. the current proper solution would be to explicitly pass around the request ID everywhere
  2. If I'm already using express and passing around req/res everywhere, I can let the request ID hitch a rid to the rest of my codebase on the res object
  3. We could make it so I have to update function signatures everywhere to indicate that I'm implicitly passing around another item to anyone with this updated function signature. (This would still require updates everywhere, but only in function signatures, not in function calls)
  4. We could implement a context API, and I can just give it to the places that need it.

I certainly would rather go with the fourth option, but the third option could maybe be a fallback if the fourth is rejected.