What comes next

Two of the four things this chapter planned are done, one of them

answered a question that was being asked a lot, and a few things are

deliberately not planned. It exists so that the answer to "are you going

to…" is written down rather than reconstructed each time — which means

it also has to record what the answer turned out to be.

What 0.1.0 contains, and what the order was

The database came first and the reasoning is in DATABASE.md. Three of the four items on that list are done, which is why this chapter now reads as a record for the first two sections and a plan after them.

1. A connection pool and transactions. Done. There was one shared JDBC connection serialized with threads:synchronized-on — correct, and survivable on a local SQLite file, but it meant there were no transactions anywhere, because a transaction needs one connection held across several statements and holding it meant holding the lock. HikariCP, pool size io.threads + worker.threads + 2, and a re-entrant with-transaction that rolls back on any non-local exit. Four operations that write more than one table are wrapped in it.

The measurement is worth keeping because half of it is negative: eight concurrent 100 ms Postgres queries take 0.84 s at pool=1 and 0.28 s at pool=8, tracking 1/N as it should — and the landing page measures the same at pool=1 and at pool=6, about 1,370 requests a second either way, because at io=2 worker=2 the renderer is the bottleneck and not the round trip. The pool is a correctness prerequisite and headroom. It is not a throughput win today, and saying it was would be the kind of claim this book is supposed to avoid.

1. A versioned migration runner. Done. Numbered forward-only files, a schema_migrations ledger with checksums, one transaction per migration, pg_advisory_lock held on one leased connection, and --dry-run. 0001 is the previous schema as create table if not exists, so it is a no-op on an existing database and there is no baseline step. Verified against a real pre-0.9.0 database: six missing columns added, every row intact. A Lisp migration's checksum hashes its source text, not the function object, so it is the same in every image.

1. PostgreSQL on every commit. Partly. core/run-tests-all.sh runs the whole gate — build, NixOS module evaluation, the Lisp suites, then the browser suites against SQLite and against a throwaway Postgres cluster. What does not exist is something that runs it without being asked. SQLite is permissive where PostgreSQL is strict, so a suite that runs on one engine tests one engine; until the gate is automatic, a dialect divergence is caught only by whoever runs it.

1. Expand/contract written down before it is needed. Not done. Expand, backfill, contract — in the contributor docs before the first rename, not during it.

Then the parts of spicemine that are specified and unbuilt: the Guardian, webhooks, teams, search, and the issue tracker, project boards and merge queues designed in TRACKERS.md. Those are product rather than framework and live in SPEC.md.

htmx or Datastar: answered, and it is Datastar

This was the most frequently asked question about this framework, and the answer is now a chapter rather than a plan. Chapter 6 is fragments and the Datastar wire format: 191 lines of Lisp, no dependency, a client vendored at v1.0.4, and def-fragment as a third middleware stack beside def-page and def-post.

What is worth keeping here is the choice, because the earlier edition of this book leaned the other way.

  htmx       ~14 KB gzipped   attributes on HTML, server returns HTML
                              largest community, most documentation
                              no client state model - which is the point
  Datastar   ~10 KB gzipped   signals plus SSE, server pushes updates
                              smaller, newer, smaller community
                              fits the existing event stream directly

The lean was htmx, for the documentation and the hiring pool. Datastar won on two things that only matter to this codebase:

• The transport was already here. /api/v0/events/stream is an SSE feed fanned out by EventHub over the durable events table. Datastar's model is SSE, so the live half of the UI and the request/response half speak the same protocol, and a page that wants both takes one client rather than two.
• The server side is a format, not a library. Two event types and a handful of data: lines. That is a thing this project can own completely, which is the same trade it made for Markdown, HTML, JSON and migrations. htmx would have been a dependency on the client, which is fine, but it would not have removed the fragment plumbing from the server; Datastar's did, because there is nothing left to depend on.

Three things from the original plan survived contact and are worth restating, since they were the actual design work rather than the library:

Degradation is a decision, not a default. Every fragment route also answers as a full page, and that is enforced by the macro's two return values rather than by remembering. It was a rule from the first fragment because it is not a thing anybody retrofits at the twentieth.

A fragment URL is a URL. def-fragment establishes who is asking; the body still has to establish that they may. A fragment endpoint is public API with a smaller response, not a private implementation detail of a page.

CSRF beyond forms is still open. A fragment POST has no form, so it has no hidden field, and csrf-ok-p reads a form field today. Datastar's actions take a headers option, so the token has somewhere to go; the anonymous case is a decision rather than a line of code, and until it is made, no fragment route may mutate repository or account state. The one converted route sets a display preference.

Jooby's MCP module: interesting, and not the way in

Jooby 3 includes jooby-mcp, which makes a Jooby application an MCP server — @McpTool, @McpPrompt, @McpResource on service methods, over streamable HTTP, SSE or WebSocket. The obvious thought is that spicemine could expose repository search, timelines and file reads to an agent for nothing.

It cannot be used as designed, for a structural reason. The module "relies on APT to generate high-performance routing code with zero reflection overhead": the Jooby Annotation Processor reads annotated Java or Kotlin source at compile time and generates a dispatcher class whose name ends in Mcp_, and McpModule is constructed from an instance of that generated class. Our handlers are not Java source. They are Lisp closures wrapped in classes generated by jnew-runtime-class at runtime (chapter 3), long after any annotation processor has finished. There is no source for APT to read and no documented builder API to register a tool without it.

Using it anyway would mean writing the MCP surface as annotated Java service classes — a second implementation of logic the Lisp already has, in the language this project exists to avoid writing. It also means taking Jackson 3 plus the MCP SDK plus the inspector, which roughly doubles the per-feature side of a 28-jar dependency list for one feature, against a JSON codec that is currently 76 lines of Lisp.

The cheaper path, if MCP is wanted, is to speak it directly. MCP is JSON-RPC over HTTP. This project has a JSON reader and writer (json-parse, jval, jobj), a route table, and an OpenAPI description already generated from that route table. An MCP endpoint that advertises a handful of tools — list spaces, list repositories, read a timeline, read a file — is a few hundred lines of Lisp and no new jars. That is the same trade the project has made for Markdown, HTML, JSON and migrations, and the same reason: a few hundred lines is affordable, a transitive graph is not.

The part that is a product question, not a technical one

spicemine's premise is in its README: you can't look at everyone's code without signing in. The Guardian is the feature that enforces it. An MCP server is a machine-readable interface to repository content designed for consumption by AI agents, which is, read uncharitably, the thing the Guardian exists to stop.

The resolution is that MCP done properly is the opposite of scraping, because it is attributable. A crawler is anonymous, unthrottled and unaccountable. An MCP session authenticated with a personal token is a named person's agent: rate-limited, logged, revocable, and attached to somebody who agreed to terms. GUARDIAN.md already says tokens are how every legitimate robot opts in. An MCP endpoint behind a token is that sentence with an implementation.

So: not a near-term priority, not through Jooby's module, and when it happens it is authenticated, logged and revocable from the first commit — not added afterwards.

Not planned

A component model or server-pushed UI state. Datastar is a transport for fragments, not a reason to put session state on the server. A page — and now a fragment — stays a function of the request, and the only server state stays a session row. That is what makes a second instance a configuration change.

An ORM. spice-domain writes SQL and rows are lists. 563 lines for nine tables, and the Postgres bugs that did show up were decoder bugs an ORM would have hidden rather than prevented. The migration runner of 0.1.0 is the one piece of schema machinery that was worth building, and it writes no queries.

Lisp dependencies. The rule in chapter 1 is not a phase. If a problem needs a PDF library, an image pipeline or a numerical stack, it comes from a jar or it does not come.

Making startup faster. Done, as far as it goes: cold start was 9.4 seconds and was 2.0 once the two require forms that cost 8.2 of them went (chapter 11). It is 2.37 s on 0.71.0, of which 1.7 s is loading our files and starting the service - ours to look at - and on JDK 25 and later the JVM's AOT cache takes a quarter off. Chapter 12 is clear that if process startup is in your request path this is the wrong architecture rather than a performance problem.

A plugin system for SCM engines. Adding an engine is a file of defmethods and a keyword (chapter 14). A registry would be a worse version of (eql :fossil).