xoong
Reference Search
Menu · Data flow
DocsArchitectureData flow

Data flow

A screen reads through one road: the OpenAPI document, a generated table, one client, a query cache, a shared entity store, and the scope of <xo-view>. A write travels back along the same road and refreshes what it touches.

The road from spec to screen

Nothing in the browser knows OpenAPI. xoong sync reads the document once, at development time, and writes a small table that the client uses at run time.

The road from the OpenAPI document to the DOM Two stages. At build time, in Node: the OpenAPI document goes into xoong sync, which writes operations.json, api.d.ts and xoong.html-data.json. In the browser: configure() creates the one client from operations.json; the client keeps a query cache with one query per operation and canonical parameters; queries normalise responses into the entity store, which holds one signal per record; an xo-view reads its query and the store to build its scope with the variables $loading, $error and the items; effects write only the DOM nodes that read a changed signal. Build time (Node) OpenAPI documentyour backend's contract xoong syncwrites .xoong/operations.json,api.d.ts and xoong.html-data.json In the browser configure({ api, baseUrl, ... })creates the one client: requests, retries, errors Query cacheone query per operation + canonical params Entity storeone signal per record: User#1, User#2 xo-view scope$loading, $error, items, p-* params DOMonly nodes that read a signal update operations.json query(op, params) normalize(response) denormalize (tracked) effects

One road for every read. The arrows are the direction the data travels; a change in the store travels the last two arrows again, without a request.

The rest of this page follows the road from the top, then the write path, live updates and errors.

Sync: what the generated files hold

File (in .xoong/)What it is for
operations.jsonThe runtime table. Per operation: id, method, path template, the names of its query parameters, whether it has a body, the request schema (bodySchema, used by forms), and a response descriptor that says where entities sit in the answer. Plus the entities: which schemas are records and which property is their id.
api.d.tsTypeScript types for every operation, so createClient<Operations>() checks names, params and bodies. JavaScript users never load it.
xoong.html-data.jsonCompletion and hover for xo-* names and your operation names in editors. See Tooling.
{
  "version": 1,
  "operations": {
    "listUsers": {
      "id": "listUsers", "method": "GET", "path": "/users",
      "query": ["limit", "sort"],
      "response": { "t": "object", "props": { "items": { "t": "array", "items": { "t": "entity", "name": "User", "replace": true } } } }
    },
    "updateUser": {
      "id": "updateUser", "method": "PATCH", "path": "/users/{id}",
      "body": true,
      "bodySchema": { "t": "object", "props": { "name": { "t": "string", "minLength": 2 }, "email": { "t": "string", "format": "email" } }, "required": ["name"] },
      "response": { "t": "entity", "name": "User", "replace": true }
    }
  },
  "entities": { "User": { "id": "id" } }
}
import { configure } from '@xoong/data';
import '@xoong/data/view';
import operations from '/.xoong/operations.json' with { type: 'json' };

configure({ api: operations, baseUrl: '/api', streamUrl: '/api/events' });
npx xoong sync openapi.json --hints xoong.hints.json

An operation is named by its operationId. Without one, sync builds a name from the method and path. A schema is an entity when it has a property named id, or one named by x-xoong-id in the document, or by a hints file (idNames, or schemas.X.id); an id may be several properties (a composite id). A response that uses the entity's own schema is stored by replacing the record (replace: true above); several schemas that share one entity through a hint merge field by field, because projections carry fewer fields. xoong doctor lists what the document leaves unclear, with a fix for each.

The client

configure() makes the one app-wide client (createClient()); getClient() returns it. Everything that talks to the API goes through it. A request is built like this:

A failure is always a XoongError with status (0 for a network failure), title, detail, code, retryAfterS and fields. It is built from application/problem+json, { error: { code, message, fields } }, FastAPI-style { detail: [{ loc, msg }] } or { errors: [{ pointer, detail }] }. A 401 calls onUnauthorized, then every failure calls onError.

OptionDefaultMeaning
apirequiredthe contents of operations.json
baseUrl, headers, credentials, fetchwhere and how requests go out
staleTime30000 mshow long a GET result counts as fresh
gcTime300000 mshow long a query that nothing watches is kept
retry, retryDelay2, 300 msGET retries and the first delay
evictEntitiestruedrop records no cached query reaches when a query is collected
invalidatenoneextra rules: mutation id to the operation ids to refresh
onError, onUnauthorized, streamUrlglobal error handler, 401 handler, the server-sent events endpoint

The query cache

A read is a query. Its key is the operation id plus the canonical params: keys sorted, empty values dropped, every value as its text in the URL. So the same data is the same query, however you ask for it, and a developer never writes a key.

One read Columns, left to right: the screen the person sees (a window with a table of people), the query cache, the entity store, and your services and APIs, drawn as example tiles of which only the People API is called. Above the arrows, between the cache and the store, is the code you write, in a window named people.html, which makes the screen. The code window shows people.html: <xo-view get="listPeople"> <xo-table columns="name,email,role" sortable></xo-table> </xo-view> <!-- loading, errors, cache and types come from your OpenAPI document -->. Steps: 1. The xo-view with get="listPeople" starts. It selects its query, listPeople, and retains it: it asks the query cache to watch that query. 2. The cache has no fresh answer, so the query's status is loading. The view sees $loading as true, and the screen shows loading rows. 3. The cache sends GET /people to your People API. The path and the query names come from your OpenAPI document. 4. Your People API answers 200 with the people as JSON. 5. The cache normalises the answer into the entity store: each person once, as Person#1, Person#2 and so on. 6. The query's status becomes success, so $loading is false and the loading rows go away. 7. The view reads the records from the store, tracked, and its effects write the rows into the xo-table on the screen. When the animation plays, a dot travels along the arrows one after another, top to bottom, while the line of code that causes the step is lit and the screen changes with it: loading rows first, then the table. A hollow dot rides a dashed arrow. Query cache Entity store Your services and APIsany language, one base URL,described by OpenAPI Orders API Files service People APIGET /people 1 query(op, params).retain() 2 $loading = true 3 GET /people 4 200 JSON 5 upsert Person#1, Person#2 6 $loading = false 7 read records (tracked)

A read, top to bottom. Dashed arrows are answers or signal updates. The code is all you write; the people and the names of the services are examples. In motion, a dot travels along the arrows one after another, top to bottom, while the line of code that causes the step is lit and the screen changes with it: loading rows first, then the table. A hollow dot rides a dashed arrow. Use the slider, or press a number, to look at one step.

const a = client.query('listUsers', { limit: 10 });
const b = client.query('GET /users', { limit: '10' });
a === b;   // true: both are the query  listUsers:{"limit":"10"}

A query holds signals: status (idle, loading, success or error), error, the normalised root and data, a computed value that fills the records back in. It also holds the rules of its life:

The entity store

When a response arrives, the response descriptor tells normalize() where the records are. Each record goes into the store once, under Name#id, and the query keeps a reference ({ $ref: "User#1" }) in its place. Every record is one signal. A GET response is only one of four ways a record gets in:

Four ways data enters the entity store Four sources on the left all write into the one entity store in the middle: a GET response fetched by a query, a mutation response, client.seed() with data the server already has, and a stream event (patch, upsert or delete). The store feeds three readers on the right: xo-view, derived() and a pristine form, which follows its record. GET responsea query fetches Mutation responseclient.mutate() client.seed()data the server already has Stream eventpatch, upsert, delete Entity storeone signalper record xo-viewreads through its query derived()computed over all(name) a pristine formfollows its record

Whatever the source, a record is stored once, so every reader of that record updates.

const q = client.query('listUsers', { limit: 10 });
await q.fetch();
q.root.peek();                 // { items: [{ $ref: 'User#1' }, { $ref: 'User#2' }], total: 2 }
client.store.read('User#1');   // { id: 1, name: 'Ada', email: '...' }  (a tracked read)
q.data.peek();                 // the same list with the records filled in

The effect is sharing: a list, a detail page and a dashboard that show User#1 read the same signal, so one change reaches all of them. Reading the data (denormalize) resolves the references; a reference to a deleted record drops out of lists instead of leaving a hole. A union is stored under the right entity when the members can be told apart (a discriminator, a fixed tag value, or a property only one member has); otherwise the value stays inline and xoong doctor reports it (XO160).

derived(({ all, get }) => ...) is a computed over the store: all("Order") is every loaded order, from any query, and the result recomputes when one of them or the set changes. It never fetches; fetching is declared on a view.

One caveat. When a query is collected, records that no remaining cached query reaches (following nested references) are dropped, and a screen still reading them sees them as deleted. So a screen should show its data through a query. Turn it off with evictEntities: false.

The view and the DOM

<xo-view get> picks its query in an effect over its get attribute and its params (the p-* attributes, xo-param, the URL). When either changes, it releases the old query and retains the new one. Its scope has $loading (status is loading), $error (the XoongError), the response fields (items), and the params, so {id} works in an action. Text bindings read the scope inside effects, so only the nodes whose values changed are written. Rendering shows how.

<xo-view get="listUsers" p-limit="10">
  <p xo-if="$loading" aria-busy="true">Loading…</p>
  <p xo-if="$error" role="alert">{$error.code}: {$error.detail}</p>
  <ul>
    <li xo-each="items">
      {name}
      <button xo-click="POST /users/{id}/disable" xo-optimistic="status=disabled">Disable</button>
    </li>
  </ul>
</xo-view>

The write path

A write is xo-click on a button or xo-submit on a form. Both end in client.mutate(op, params, body).

One write Columns, left to right: the screen the person sees (a table of people with a Disable button in each row), client.mutate, the entity store, the query cache, and your services and APIs, drawn as example tiles of which only the People API is called. Above the arrows, between mutate and the cache, is the code you write, in a window named people.html, which makes the screen. A note says that on failure the optimistic patch is undone, a XoongError is thrown and xo-error fires. The code window shows people.html: <xo-view get="listPeople"> <xo-table columns="name,status"> <button xo-click="POST /people/{id}/disable" xo-confirm="Disable {name}?" xo-optimistic="status=disabled" >Disable</button> </xo-table> </xo-view>. Steps: 1. You click Disable on a row. The button's xo-click starts client.mutate, and xo-confirm asks first: "Disable Ada?" You press OK. 2. With xo-optimistic, mutate patches the loaded record in the entity store at once, before the server has answered. 3. The row reads that record, so its effect writes "disabled" on the screen at once. 4. mutate sends POST /people/1/disable to your People API. The {id} comes from the row. 5. Your People API answers 200 with the updated person. 6. mutate puts the answer in the store: Person#1 is now the server's version, so every screen that shows it is up to date. 7. mutate marks every cached query whose path overlaps /people/1/disable as stale. GET /people is one. 8. Something is watching that query (the list on the screen), so the cache sends GET /people again, now. 9. Your People API answers with the list as it is now. 10. mutate resolves, and the xo-done event fires on the view. The list is up to date. When the animation plays, a dot travels along the arrows one after another, top to bottom, while the code that causes the step is lit and the screen changes with it: the question, then the row that says "disabled". A hollow dot rides a dashed arrow. The note about a failure is not played: it is the other road. client.mutate Entity store Query cache Your services and APIsany language, one base URL,described by OpenAPI Orders API Files service People APIGET /peoplePOST /people/{id}/disable 1 mutate(op, params, body) 2 xo-optimistic: provisional patch 3 effects: row says disabled 4 POST /people/1/disable 5 200 + Person 6 upsert Person#1 7 invalidate paths that overlap /people/1/disable 8 refetch GET /people 9 200 list 10 resolves, xo-done On failure: patch undone, xo-error fires

A write, top to bottom. Invalidation is by path; only queries that something is watching refetch at once. The people and the names of the services are examples. In motion, a dot travels along the arrows one after another, top to bottom, while the code that causes the step is lit and the screen changes with it: the question, then the row that says "disabled". A hollow dot rides a dashed arrow. The note about a failure is not played: it is the other road. Use the slider, or press a number, to look at one step.

  1. The handler. It ignores a second click while one call is running (and sets aria-busy). xo-confirm asks first. Path and query params come from the scope: the current item first, then the view's p-* params. The body is the form model's result, the form's fields, xo-body (a record from the scope), plus xo-with fields; xo-selected runs once per ticked row or sends the ticked ids in one request.
  2. Optimistic update. With xo-optimistic="status=disabled" (or the optimistic option), the patch is applied to loaded records at once. If the call fails, only the fields that still hold the provisional value are put back, so a newer change from the server or another screen is never erased.
  3. The response is normalised into the store, so every screen that shows the record updates. The call resolves with the denormalised result.
  4. Invalidation. Every cached query whose path overlaps the mutation's path is marked stale, and refetched now if something is watching it. Two paths overlap when one is a prefix of the other, segment by segment.
  5. Events. The view's host fires xo-done with the result, and xo-notice when the button has a xo-notice message (shown by <xo-toast>).
Cached queryAfter POST /users/1/disable
GET /usersrefreshed (a prefix of the mutation's path)
GET /users/1refreshed, unless the response was that very record: it was just rewritten, so it is marked fresh instead
GET /users/1/postskept (neither path is a prefix of the other)
GET /users/2kept (differs at a segment)

The rule is by path, so it needs no declaration. When it is not enough, name the extra queries once: configure({ invalidate: { shipOrder: ["getDashboard"] } }). Those are refreshed whatever their path.

await client.mutate('updateUser', { id: 1 }, { name: 'Ada L.' }, {
  optimistic: [{ $ref: 'User#1', patch: { name: 'Ada L.' } }],
});

A deleted record is not removed from the store by a DELETE call: the lists that contained it refetch and no longer name it. Only a stream delete event removes a record at once.

Server-known data: seed

When the server already has the answer (it rendered the page), put it in the cache instead of asking again. The server writes the JSON into the page as <script type="application/json"> (not code, so a strict CSP allows it), one module script calls client.seed(op, params, data), and the data is stored like any response, entities included. A view that asks for the same operation and params shows it at once and makes no request until it is stale.

// main.js: the server wrote the answer into the page as <script type="application/json" id="seed-users">
const data = JSON.parse(document.getElementById('seed-users').textContent);
getClient().seed('listUsers', { limit: 10 }, data);
// a view with get="listUsers" p-limit="10" shows it at once and makes no request until it is stale

Live updates

With streamUrl set and live on a view, the client opens one server-sent events connection, shared by all live views and closed when the last one goes away. It is loaded lazily, so an app without live views never pays for it. Each message is one JSON event, applied to the store, not to a view: every screen showing that record updates.

{ "type": "patch",  "entity": "User", "id": "42", "path": "/email", "value": "a@b.c", "version": 17 }
{ "type": "upsert", "entity": "User", "id": "42", "value": { "id": 42, "name": "Ada" }, "version": 18 }
{ "type": "delete", "entity": "User", "id": "42", "version": 19 }
{ "type": "resync" }

When something fails

Where to look in the code