doc: improve docs organization

- Move extension docs back into this repo
- Add index README.md under `doc/`
- Improve documentation of project structure

README.md

@@ -50,7 +50,7 @@ npm start
 
 This will launch Puter at http://puter.localhost:4100 (or the next available port).
 
-If this does not work, see [First Run Issues](./doc/first-run-issues.md) for
+If this does not work, see [First Run Issues](./doc/self-hosters/first-run-issues.md) for
 troubleshooting steps.
 
 <br/>

doc/README.md

@@ -0,0 +1,24 @@
+## User Documentation
+
+- [Hosting Instructions](./self-hosters/instructions.md)
+- [Domain Setup](./self-hosters/domains.md)
+- [Support Levels](./self-hosters/support.md)
+
+## Contributor Documentation
+
+### Where to Start
+
+Start with [Repo Structure and Tooling](./contributors/structure.md).
+
+### Index
+
+- **Conventions**
+  - [Repo Structure and Tooling](./contributors/structure.md)
+    - How directories and files are organized in our GitHub repo
+    - What tools are used to build parts of Puter
+  - [Comment Prefixes](./contributors/comment_prefixes.md)
+    - A convention we use for line comments in code
+
+- [Frontend Documentation](/src/gui/doc)
+- [Backend Documentation](/src/backend/doc)
+- [Extensions](./contributors/extensions/)

doc/contributors/extensions.md

@@ -1,3 +1,38 @@
-## Puter Extensions
+# Puter Extensions
 
-See the [Wiki Page](https://github.com/HeyPuter/puter/wiki/ex_extensions)
+## Quickstart
+
+Create and edit this file: `mods/mods_enabled/hello-puter.js`
+
+```javascript
+const { UserActorType, AppUnderUserActorType } = use.core;
+
+extension.get('/hello-puter', (req, res) => {
+    const actor = req.actor;
+    let who = 'unknown';
+    if ( actor.type instanceof UserActorType ) {
+        who = actor.type.user.username;
+    }
+    if ( actor.type instanceof AppUnderUserActorType ) {
+        who = actor.type.app.name + ' on behalf of ' + actor.type.user.username;
+    }
+    res.send(`Hello, ${who}!`);
+});
+```
+
+## Events
+
+//
+
+This is subject to change as we make efforts to simplify the process.
+
+### Step 1: Configure a Mod Directory
+
+Add this to your config:
+```json
+"mod_directories": [
+    "{source}/../mods/mods_available"
+]
+```
+
+This adds the `mods/mods_available` directory to this 

doc/contributors/extensions/README.md

@@ -0,0 +1,89 @@
+# Puter Extensions
+
+## Quickstart
+
+Create and edit this file: `mods/mods_enabled/hello-puter.js`
+
+```javascript
+// You can get definitions exposed by Puter via `use`
+const { UserActorType, AppUnderUserActorType } = use.core;
+
+// Endpoints can be registered directly on an extension
+extension.get('/hello-puter', (req, res) => {
+    const actor = req.actor;
+    
+
+    // Make a string "who" which says:
+    //   "<username>", or:
+    //   "<app> acting on behalf of <username>"
+    let who = 'unknown';
+    if ( actor.type instanceof UserActorType ) {
+        who = actor.type.user.username;
+    }
+    if ( actor.type instanceof AppUnderUserActorType ) {
+        who = actor.type.app.name
+            + ' on behalf of '
+            + actor.type.user.username;
+    }
+
+    res.send(`Hello, ${who}!`);
+});
+
+// Extensions can listen to events and manipulate Puter's behavior
+extension.on('core.email.validate', event => {
+    if ( event.email.includes('evil') ) {
+        event.allow = false;
+    }
+});
+```
+
+### Scope of `extension` and `use`
+
+It is important to know that the `extension` global is temporary and does not
+exist after your extension is loaded. If you wish to access the extension
+object within a callback you will need to first bind it to a variable in
+your extension's scope.
+
+```javascript
+const ext = extension;
+extension.on('some-event', () => {
+    // This would throw an error
+    // extension.something();
+
+    // This works
+    ext.example();
+})
+```
+
+The same is true for `use`. Calls to `use` should happen at the top of
+the file, just like imports in ES6.
+
+## Database Access
+
+A database access object is provided to the extension via `extension.db`.
+You **must** scope `extension` to another variable (`ext` in this example)
+in order to access `db` from callbacks.
+
+```javascript
+const ext = extension;
+
+extension.get('/user-count', { noauth: true }, (req, res) => {
+    const [count] = await ext.db.read(
+        'SELECT COUNT(*) as c FROM `user`'
+    );
+});
+```
+
+The database access object has the following methods:
+- `read(query, params)` - read from the database using a prepared statement. If read-replicas are enabled, this will use a replica.
+- `write(query, params)` - write to the database using a prepared statement. If read-replicas are enabled, this will write to the primary.
+- `pread(query, params)` - read from the database using a prepared statement. If read-replicas are enabled, this will read from the primary.
+- `requireRead(query, params)` - read from the database using a prepared statement. If read-replicas are enabled, this will try reading from the replica first. If there are no results, a second attempt will be made on the primary.
+
+## Events
+
+See [events.md](./events.md)
+
+## Definitions
+
+See [definitions.md](./definitions.md)

doc/contributors/extensions/definitions.md

@@ -0,0 +1,46 @@
+## Definitions
+
+### `core.config` - Configuration
+
+Puter's configuration object. This includes values from `config.json` or their
+defaults, and computed values like `origin` and `api_origin`.
+
+```javascript
+const config = use('core.config');
+
+extension.get('/get-origin', { noauth: true }, (req, res) => {
+    res.send(config.origin);
+})
+```
+
+### `core.util.*` - Utility Functions
+
+These utilities come from `src/backend/src/util` in Puter's repo.
+Each file in this directory has its exports auto-loaded into this
+namespace. For example, `src/backend/src/util/langutil.js` is available
+via `use('core.util.langutil')` or `use.core.util.langutil`.
+
+#### `core.util.helpers` - Helper Functions
+
+Common utility functions used throughout Puter's backend. Use with caution as
+some of these functions may be deprecated.
+
+> **note:** the following documentation is incomplete
+
+#### `core.util.langutil` - Language Helpers
+
+##### `whatis(thing :any)`
+
+- Returns `"array"` if `thing` is an array.
+- Returns `"null"` if `thing` is `null`.
+- Returns `typeof thing` for any other case.
+
+##### `nou(value :any)`
+
+Simply a "null or undefined" check.
+
+##### `can(value :any, capabilities :Array<string>)`
+
+Checks if something has the specified capabilities. At the time of
+writing the only one supported is `iterate`, which will check if
+`value[Symbol.iterator]` is truthy

doc/contributors/extensions/events.json.js

@@ -0,0 +1,27 @@
+module.exports = [
+    {
+        id: 'core.email.validate',
+        description: `
+            This event is emitted when an email is being validated.
+            The event can be used to block certain emails from being validated.
+        `,
+        properties: {
+            email: {
+                type: 'string',
+                mutability: 'no-effect',
+                summary: 'the email being validated',
+                notes: [
+                    'The email may have already been cleaned.',
+                ]
+            },
+            allow: {
+                type: 'boolean',
+                mutability: 'mutable',
+                summary: 'whether the email is allowed',
+                notes: [
+                    'If set to false, the email will be considered invalid.',
+                ]
+            },
+        },
+    }
+];

doc/contributors/extensions/events.md

@@ -0,0 +1,38 @@
+### `core.email.validate`
+
+This event is emitted when an email is being validated.
+The event can be used to block certain emails from being validated.
+
+#### Property `email`
+
+the email being validated
+- **Type**: string
+- **Mutability**: no-effect
+- **Notes**: undefined
+  - The email may have already been cleaned.
+#### Property `allow`
+
+whether the email is allowed
+- **Type**: boolean
+- **Mutability**: mutable
+- **Notes**: undefined
+  - If set to false, the email will be considered invalid.
+
+### `core.request.measured`
+
+This event is emitted when a requests incoming and outgoing bytes
+have been measured.
+
+```javascript
+extension.on('core.request.measured', data => {
+    const measurements = data.measurements;
+    //    measurements = { sz_incoming: integer, sz_outgoing: integer }
+
+    const actor = data.actor; // instance of Actor
+
+    console.log('\x1B[36;1m === MEASUREMENT ===\x1B[0m\n', {
+        actor: data.actor.uid,
+        measurements: data.measurements
+    });
+});
+```

doc/contributors/extensions/gen.js

@@ -0,0 +1,30 @@
+const dedent = require('dedent');
+const events = require('./events.json.js');
+
+const mdlib = {};
+mdlib.h = (out, n, str) => {
+    out(`${'#'.repeat(n)} ${str}\n\n`);
+}
+
+const N_START = 3;
+
+const out = str => process.stdout.write(str);
+for ( const event of events ) {
+    mdlib.h(out, N_START, `\`${event.id}\``);
+    out(dedent(event.description) + '\n\n');
+    
+    for ( const k in event.properties ) {
+        const prop = event.properties[k];
+        mdlib.h(out, N_START + 1, `Property \`${k}\``);
+        out(prop.summary + '\n');
+        out(`- **Type**: ${prop.type}\n`);
+        out(`- **Mutability**: ${prop.mutability}\n`);
+        if ( prop.notes ) {
+            out(`- **Notes**: ${prop.n}\n`);
+            for ( const note of prop.notes ) {
+                out(`  - ${note}\n`);
+            }
+        }
+    }
+
+}

doc/contributors/index.md

@@ -1,25 +0,0 @@
-# Contributing to Puter
-
-## Essential / General Knowledge
-
-### Repository Dichotomy
-
-- Puter's GUI is at the root; `/src` is the GUI
-- Puter's backend is a workspace npm package;
-  it resides in `packages/backend(/src)`
-
-The above may seem counter-intuitive; backend and frontend are siblings, right?
-Consider this: by a different intuition, the backend is at a "deeper" level
-of function; this directory structure better adheres to soon-to-be contributors
-sifting around through the files to discover "what's what".
-
-The directory `volatile` exists _for your convenience_ to simplify running
-Puter for development. When Puter is run
-run with the backend with this repository as its working directory, it
-will use `volatile/config` and `volatile/runtime` instead of
-`/etc/puter` and `/var/puter`.
-
-## See Next
-
-- [Backend Documentation](../../packages/backend/doc/contributors/index.md)
-<!-- - [Frontend Documentation](./frontend.md) -->

doc/contributors/structure.md

@@ -0,0 +1,65 @@
+# Repository Structure and Tooling
+
+Puter has many of its parts in a single [monorepo](https://en.wikipedia.org/wiki/Monorepo),
+rather than a single repository for each cohesive part.
+We feel this makes it easier for new contributors to develop Puter since you don't
+need to figure out how to tie the parts together or how to work with Git submodules.
+It also makes it easier for us to maintain project-wide conventions and tooling.
+
+Some tools, like [puter-cli](https://github.com/HeyPuter/puter-cli), exist in separate
+repositories. The `puter-cli` tool is used externally and can communicate with Puter's
+API on our production (puter.com) instance or your own instance of Puter, so there's
+not really any advantage to putting it in the monorepo.
+
+## Top-Level directories
+
+### The `doc` directory
+
+The top-level `doc` directory contains the file you're reading right now.
+Its scope is documentation for using and contributing to Puter in general,
+and linking to more specific documentation in other places.
+
+All `doc` directories will have a `README.md` which should be considered as
+the index file for the documentation. All documentation under a `doc`
+directory should be accessible via a path of links starting from `README.md`.
+
+### The `src` directory
+
+Every directory under `/tools` is [an npm "workspaces" module](https://docs.npmjs.com/cli/v8/using-npm/workspaces). Every direct child of this directory (generally) has a `package.json` and a `src` directory.
+
+Some of these modules are core pieces of Puter:
+- **Puter's backend** is [`/src/backend`](/src/backend)
+- **Puter's GUI** is [`/src/gui`](/src/gui)
+
+Some of these modules are apps:
+- **Puter's Terminal**: [`/src/terminal`](/src/terminal)
+- **Puter's Shell**: [`/src/phoenix`](/src/phoenix)
+- **Experimental v86 Integration**: [`/src/emulator`](/src/emulator)
+  - **Note:** development is focused on Puter PDE files instead (docs pending)
+
+Some of these modules are libraries:
+- **common javascript**: [`/src/putility`](/src/putility)
+- **runtime import mechanism**: [`/src/useapi`](/src/useapi)
+- **Puter's "puter.js" browser SDK**: [`/src/puter-js`](/src/puter-js)
+
+### The `volatile` directory
+
+When you're running Puter with development instructions (i.e. `npm start`),
+Puter's configuration directory will be `volatile/config` and Puter's
+runtime directory will be `volatile/runtime`, instead of the standard
+`/etc/puter` and `/var/puter` directories in production installations.
+
+We should probably rename this directory, actually, but it would inconvenience
+a lot of people right now if we did.
+
+### The `tools` directory
+
+Every directory under `/tools` is [an npm "workspaces" module](https://docs.npmjs.com/cli/v8/using-npm/workspaces).
+
+This is where `run-selfhosted.js` is. That's the entrypoint for `npm start`.
+
+These tools are underdocumented and may not behave well if they're not executed
+from the correct working directory (which is different for different tools).
+Consider this a work-in-progress. If you want to use or contribute to anything
+under this directory, for now you should
+[tag @KernelDeimos on the community Discord](https://discord.gg/PQcx7Teh8u).