Fragments and Datastar

A page update that does not reload the page, in 198 lines of Lisp and one

vendored script. No build step, no component model, and the route still

works with scripting turned off.

Chapters 4 and 5 describe a UI where every interaction is a link or a form and the answer is a whole page. For forms, lists and a timeline that is the right shape. For a diff with collapsible hunks, a file tree, live search or a theme picker it stops being right: the reader is asking for one piece of the page to change and getting all 36 KB of it back.

mélangej 0.1.0 answers that with fragments, over the Datastar protocol. This chapter is what that is, what it costs, and the four things about it that are worth not rediscovering.

The insight that makes it small

Datastar's server side is not a library, it is a wire format. Two event types, each a few lines of Server-Sent Events:

event: datastar-patch-elements
data: selector #foo
data: mode inner
data: elements <div>Hello</div>

event: datastar-patch-signals
data: signals {"count": 3}

And the part that keeps it to 198 lines: Datastar's @get/@post actions read the response as a stream of events, and a response that sends its events and closes is a complete interaction. There is nothing to keep alive. A fragment endpoint is an ordinary route that returns a string with text/event-stream on it — no long-lived connection, no registry of subscribers, no thread per client.

That is why there is no dependency here. spice-ds is the format, written out:

(patch-elements "<div id=\"skingrid\">…</div>")
(patch-signals  "{\"open\": true}")
(remove-elements "#banner")
(ds-events ctx event-1 event-2)   ; sets the type and the no-buffer headers

ds-events sets Cache-Control: no-cache and X-Accel-Buffering: no. The second one is the one that catches people: nginx will otherwise hold a short event-stream response until it has enough bytes to think forwarding is worth it, and the symptom is an interaction that works locally and hangs behind the proxy.

def-fragment, the third stack

Chapter 4 has two named middleware stacks, def-page and def-post. This is the third, and it exists for a reason stronger than tidiness:

(def-fragment (app "POST" "/skin") (ctx)
  …
  (values events page-thunk))

Language, session and skin are bound exactly as they are for a page. A fragment rendered without those bindings comes back in the default language wearing the first skin — which, on a partial page update, looks like a styling bug rather than a missing binding. Bind them.

The body returns two values, and that pair is the whole degradation story:

• the events — sent when the request carries Datastar-Request: true;
• a thunk that renders or redirects the full page — called when it does not.

A reader with scripting off, a crawler, or somebody who pasted the URL into a terminal gets the second path. Both exist for every fragment route from the first one, because retrofitting that at the twentieth is not something anybody does. If a route is written events-only and a plain browser asks for it, the macro answers 406 with a page that says what happened, rather than returning an event stream for the browser to render as text.

A fragment URL is a URL. Everything in chapter 5 about a hidden button being a hint applies unchanged. def-fragment establishes who is asking; the body still has to establish that they may.

The worked example: the skin picker

Before 0.1.0, choosing a skin was a POST, a Set-Cookie, and a redirect back to the settings page — correct, and a full reload to repaint ten cards. Now the same route returns two events:

(values
 (concatenate 'string
   ;; the token block, which is the whole of what a skin is (chapter 8)
   (spice/ds:patch-elements
    (html (list :style :id "skin" :data-skin (skin-id *skin*)
                (r (skin-style)))))
   ;; and the grid, so the selected card moves
   (spice/ds:patch-elements
    (html (list :div :class "skingrid" :id "skingrid"
                (loop for sk in +skins+ collect (r (skin-card sk)))))))
 (lambda () (redirect ctx "/settings/appearance?m=…")))

Two details in there are the general pattern rather than this route's business. The grid is rendered by the same function that renders it inside the full page, which is what keeps a fragment from drifting away from the page it came out of; and the new skin is bound with skin for the rest of the response, so the fragments come out wearing the skin the reader just chose rather than the one the request arrived in.

The cost, measured on one local instance:

  GET  /                              36 417 bytes
  POST /skin  with Datastar-Request    6 296 bytes  ← two events
  POST /skin  without it               a 302, then the page again

A fragment is 5.8× less to send than the page it updates. It is also less to render — an interaction that used to build a whole document now builds a style element and a grid — though not in the same proportion, since the two events here include the widest block of markup the settings page has. Either way the direction is the point: moving interactions to fragments reduces server cost per interaction rather than trading server cost for interactivity, which is the opposite of how client frameworks are usually paid for.

Four facts about the wire format

None of these appear in an error message. That is why they are in a book.

1. The attribute is data-on:click, with a colon. Datastar v1.0.4 does not recognise data-on-click and ignores it without a word. data-text and data-signals have no colon; the modifier that stops the anchor also following its link appends with two underscores:

<a href="/?skin=carbon" data-on:click__prevent="@post('/skin?skin=carbon')">

2. npm's latest is older than the current release. @starfederation/datastar@latest resolves to 1.0.0-beta.11, which speaks the previous protocol — datastar-merge-fragments, not datastar-patch-elements. Read today's documentation while running npm's default and you get a client that silently ignores every event your server sends. The client is vendored at v1.0.4 from the git tag, in src/dist/static, with a .sha256 beside it.

It is vendored rather than loaded from a CDN for a reason that is this project's premise rather than a performance opinion: a CDN is a third party on every page of a service whose selling point is that nobody gets to watch its readers.

3. A data: line cannot contain a newline, because a newline ends the line. HTML with line breaks in it therefore becomes several data: elements lines, which the client rejoins. spice-ds does the splitting, so a fragment with line breaks in it arrives whole.

4. A live patch does not change the served document. After a patch the data-skin attribute on <html> is whatever the page was served with; the live state is on the patched <style id="skin"> element. The skin suite reads the second and asserts that the first did not move, which is what tells a patched page from a reloaded one.

Reading signals

Datastar keeps client state in signals, and sends them along with the request. Where they arrive depends on the verb:

(let* ((signals (spice/ds:read-signals ctx))
       (q (spice/ds:signal-value signals "query")))
  …)

• a GET carries them in a datastar query parameter;
• everything else carries them in a JSON body, wrapped as {"datastar": {…}} — which is why spice-json grew json-object.
• a signal whose name begins with _ is not sent at all. That is the client's own default filter, and it is the right place to put a value the server should never see.

read-signals hides that difference and returns the alist json-parse returns, or NIL. The header comment on the file says the thing that matters more than the shape: signals are attacker-supplied. A signal is a value the browser was asked to keep, not a value the browser was trusted with, and anything read out of one goes through the same checks a form field does.

What is not done yet

CSRF in a header. A fragment POST has no form, so it has no hidden field, and csrf-ok-p reads a form field today. Datastar's actions take a headers option, so the token has somewhere to go; csrf-ok-p has to learn to read either, and the anonymous case — the skin picker works without an account, and an anonymous caller has no session token to prove anything with — is a decision rather than a line of code. The one converted route changes a display preference and sets a cookie, which is why it was chosen first. No fragment route may mutate repository or account state until that is built, and that sentence is the rule rather than an intention.

Forms stream (0.59.0). That rule is about fragments with no form. A form already carries its token, so every POST form on every page is now sent by Datastar as a form (@post(action, {contentType: 'form'}), which sends the hidden csrf field and the button pressed), to the same def-post route a page load reaches, checked the same way. The route does not change at all: its redirect is followed by the fetch, and page answers a Datastar request with #wrap alone, which Datastar morphs in. Three small rules make that honest. A refusal page is answered 200 for Datastar (Datastar drops anything else, which would leave a button that did nothing). A redirect's #fragment travels as _at=, because a fetch drops it. And #wrap carries its own address and title, which the stream script puts in the address bar and the tab; Back reloads. Forms marked data-plain, and the conversations, which send their own turns, post the old way, as every form does without scripts.

Everything else. The editor's live preview is a fragment of the same kind. The ones that want it next are in the repository rather than in this chapter: diff hunks, the file tree, live search.

What this is not

It is not server-pushed UI state. Chapter 13 says a page stays a function of the request and the only server state stays a session row, and fragments do not change that. The events this chapter describes are a response, and the response is computed from the request exactly as a page is. That is what keeps a second instance a configuration change.

It is not the live feed. Server push already exists and is separate: /api/v0/events/stream is an SSE feed fanned out by EventHub, 64 lines of Java, over the durable events table. Datastar can consume that feed directly, which is the main reason it was chosen over htmx. This chapter is the request/response half, which is the half nearly every interaction needs.

It is not a build step. One <script type="module">, served from the instance. Still no bundler, no node_modules, no lockfile.

The portability note

spice-ds contains zero Java interop — it is the format, and the format is strings. Loaded into SBCL with nothing supplied but a stub package for spice/http, it compiles and emits correct events:

  OK  spice-ds
  (patch-elements "<div id=\"a\">x</div>")
  => "event: datastar-patch-elements
      data: mode outer
      data: elements <div id=\"a\">x</div>
      "

Which is chapter 2's measurement arriving in a new file: the interop stayed where it was, and the feature that got added is portable Common Lisp.