doc: add more extension internals documentation

src/backend/doc/extensions/pages/core-devs.md

@@ -31,3 +31,105 @@ service.
 Kernel.js loads all core modules/services before any extensions. This allows
 core modules and services to create [runtime modules](./runtime-modules.md)
 which can be imported by services.
+
+### How Extensions are Loaded
+
+Before extensions are loaded, all of Puter's core modules have their `.install()`
+methods called. The core modules are the ones added with `kernel.add_module`,
+for example in [run-selfhosted.js](../../../../../tools/run-selfhosted.js).
+
+Then, `Kernel.install_extern_mods_` is called. This is where a `readdir` is
+performed on each directory listed in the `"mod_directories"` configuration
+parameter, which has a default value of `["{repo}/extensions"]` (the
+placeholder `{repo}` is automatically replaced with the path to the Puter
+repository).
+
+For each item in each mod directory, except for ignored items like `.git`
+directories, a mod is installed. First a directory is created in Puter's
+runtime directory (`volatile/runtime` locally, `/var/puter` on a server).
+If the item is a file then a `package.json` will be created for it after
+`//@extension` directives are processed. If the item is a directory then
+it is copied as is and `//@extension` directives are not supported
+(`puter.json` is used instead). Source files for the mod are copied to
+the mod directory under the runtime directory.
+
+It is at this point the pseudo-globals are added be prepending `cost`
+declarations at the top of `.js` files in the extension. This is not
+a great way to do this, but there is a severe lack of options here.
+See the heading below - "Extension Pseudo-Globals" - for details.
+
+Before the entry file for the extension is `require()`'d a couple of
+objects are created: an `ExtensionModule` and an `Extension`.
+The `ExtensionModule` is a Puter module just like any of the Puter core
+modules, so it has an `.install()` method that installs services before
+Puter's kernel starts the initialization sequence. In this case it will
+install the implied service that an extension creates if it registers
+routes or performs any other action that's typically done inside services
+in core modules.
+
+A RuntimeModule is also created. This could be thought of as analygous
+to node's own `Module` class, but instead of being for imports/exports
+between npm modules it's for imports/exports between Puter extensions
+loaded at runtime. (see [runtime modules](./runtime-modules.md))
+
+### Extension Pseudo-Globals
+
+The `extension` global is a different object per extension, which will
+make it possible to develop "remapping" for imports/exports when
+extension names collide among other functions that need context about
+which extension is calling them. Implementing this per-extension global
+was very tricky and many solutions were considered, including using the
+`node:vm` builtin module to run the extension in a different instance.
+Unfortunately `node:vm` support for EMCAScript Modules is lacking;
+`vm.Module` has a drastically different API from `vm.Script`, requires
+an experimental feature flag to be passed to node, and does not provide
+any alternative to `createRequire` to make a valid linker for the
+dependencies of a package being run in `node:vm`.
+
+The current solution - which sucks - is as follows: prepend `const`
+definitions to the top of every `.js` file in the extension's installation
+directory unless it's under a directory called `node_modules` or `gui`.
+This type of "pseudo-global" has a quirk when compared to real globals,
+which is that they can't be shadowed at the root scope without an error
+being thrown. The naive solution of wrapping the rest of the file's
+contents in a scope limiter (`{ ... }`) would break ES Module support
+because `import` directives must be in the top-level scope, and the naive
+solution to that problem of moving imports to the top of the file after
+adding the scope limiter requires invoking a javascript parser do
+determine the difference between a line starting with `import` because
+it's actually an import and this unholy abomination of a situation:
+```
+console.log(`
+import { me, and, everything, breaks } from 'lackOfLexicalAnalysis';
+`);
+```
+
+Exposing the same instance for `extension` to all extensions with a
+real global and using AsyncLocalStorage to get the necessary information
+about the calling extension on each of `extension`'s methods was another
+idea. This would cause surprising behavior for extension developers when
+calling methods on `extension` in callbacks that lose the async context
+fail because of missing extension information.
+
+Eventually a better compromise will be to have commonjs extensions
+run using `vm.Script` and ESM extensions continue to run using this hack.
+
+### Event Listener Sub-Context
+
+In extensions, event handlers are registered using `extension.on`. These
+handlers, when called, are supplemented with identifying information for
+the extension through AsyncLocalStorage. This means any methods called
+on the object passed from the event (usually just called `event`) will
+be able to access the extension's name.
+
+This is used by CommandService's `create.commands` event. For example
+the following extension code will register the command `utils:say-hello`
+if it is invoked form an extension named `utils`:
+
+```javascript
+extension.on('create.commands', event => {
+    event.createCommand('say-hello', async (args, console) => {
+        console.log('Hello,', ...args);
+    });
+});
+```

src/backend/doc/extensions/pages/runtime-modules.md

@@ -37,3 +37,13 @@ Separating RuntimeModule allows core code that has not yet been migrated
 to extensions to export values as if they came from extensions.
 Since core modules are loaded before extensions, this allows any legacy
 `useapi` definitions be be exported where modules are installed.
+
+For example, in [CoreModule.js](../../../src/CoreModule.js) this snippet
+of code is used to add a runtime module called `core`:
+
+```javascript
+// Extension compatibility
+const runtimeModule = new RuntimeModule({ name: 'core' });
+context.get('runtime-modules').register(runtimeModule);
+runtimeModule.exports = useapi.use('core');
+```