A worked example: spicemine

The framework was extracted from the product, not the other way round.

Every number in this book comes from this one system, which is serving

you this page.

mélangej did not start as a framework. It started as spicemine: a multi-organization, multi-tenant repository host that serves Fossil, Git, Mercurial and Lore over their own wire protocols, with a web UI, an HTTP API, ten skins and two languages. The framework is what was left when the parts that were not about hosting repositories were given their own files.

That order matters for everything you have just read. There is no reference application written to show the framework off, and no feature in this book that exists because a framework ought to have one. If a chapter described something, this system uses it in production — and this chapter is where the thirteen before it join up.

The whole thing, in one picture

The figure is generated: diagrams/architecture.mmd is Mermaid source, rendered to SVG at authoring time and committed beside it. Mermaid is a tool that ran once, not a dependency — the page that shows you this diagram loads no JavaScript to do it.

Two things to notice before the detail.

The repositories are not in the database. Postgres or SQLite holds nine tables of metadata — who exists, who may see what, which repository is where. The repositories are directories, in their own native formats, readable by the tools that made them. An instance that loses its database has lost its accounts, not its code.

One JVM, one process. There is no job runner, no cache server, no sidecar and no message bus. Chapter 11 measures what that one process costs: 294 MB resident on the two-core VPS this is served from.

The layers, with their sizes

  layer       lines  files                     what it is allowed to know
  --------------------------------------------------------------------------
  HTTP          307  spice-http                Jooby and Netty; nothing
                                               about spicemine
  routes      1 912  api-v0, spice-ui          requests, responses,
                                               permissions
  rendering   1 185  spice-html, spice-tmpl,   strings - no request,
                     spice-md, spice-banner    no database
  words       1 338  spice-i18n, spice-skin    keys and tokens - no HTML
  rules         207  spice-web, spice-auth     sessions, hashes,
                                               constant-time comparison
  data          787  spice-domain, spice-db    SQL and rows
  engines       451  spice-scm                 the four adapters
  wire          601  6 .java files             JGit, CGI, subprocesses

The rule that holds it together is a negative one, and it is enforced by where the Java is (chapter 2): the rendering and localization files contain no interop, no database access and no request object. They take data and return strings. That is why they can be tested by calling them, and why seven of them load on SBCL.

One request, all the way down

GET /ui/acme/widgets — a repository page, in a browser, in French, in the Paper skin.

1. Netty reads the socket on an I/O thread and hands the request to a worker thread. Nothing of ours has run yet.
2. Jooby matches /ui/{org}/{repo} and calls the handler that def-page registered. def-page is a macro over def-route, which is a macro over java:jnew-runtime-class — Jooby reads handler bytecode, so a Lisp closure is not enough and a class has to be generated. That is one of the 184 interop sites, it is in spice-http, and no page ever sees it.
3. spice-web reads the session cookie, looks up one row, and produces the viewer. There is no other server-side session state.
4. spice-i18n resolves the language from the cookie, then the Accept-Language header, then the default, and binds lang for the rest of the request.
5. spice-domain asks whether this viewer may see this repository, in SQL, with bound parameters. This is where a bigint id must be bound as an integer rather than a string, which DATABASE.md explains.
6. spice-scm dispatches on the repository's engine keyword to read the timeline and the README. (eql :git) goes to JGit; (eql :fossil) shells out through FossilCgi.
7. spice-md turns the README into HTML — escaping every character before it looks for markup, so a README cannot inject anything.
8. spice-ui builds the page as nested lists and spice-html emits it, escaping by default. spice-skin supplies the Paper tokens as CSS custom properties; every word on the page came from spice-i18n by key.
9. The bytes go back out through Jooby and Netty.

No step in that list knows about more than the next one. The page function cannot reach the socket, the HTML emitter cannot reach the database, and the Markdown renderer does not know what a request is.

Four design patterns worth re-using

Every string the UI can show lives in one file, by key. spice-i18n is 885 lines and the build fails if a key is missing a translation. Nothing else in the system holds a sentence. The cost is a lookup; the benefit is that adding a language is a column, and that no page can quietly fall back to English.

Every colour is a token, and skins are alists. Ten skins, no CSS frameworks, and the browser suite checks each one for AA contrast — which is how the Paper skin's 4.57:1 is known to be 4.57:1 rather than assumed to be fine.

The SCM adapter is (eql :keyword) dispatch and nothing else. Adding an engine is a file of methods. There is no registry, no plugin descriptor, no base class, and no engine can break another by existing.

Java is allowed, below a line that is written down. 601 lines of it, in six files, all of them protocol plumbing that would be worse in Lisp: Git's smart-HTTP, Fossil's CGI, Mercurial's hgweb, JWT signing, an event fan-out. Routing, rules and domain logic are Lisp. Chapter 2 counts the crossings.

Opinionated decisions

Each of these is a thing a web framework is usually expected to have. None of them is missing by accident, and each one costs something to leave out.

• No ORM. spice-domain writes SQL, and rows are lists. 563 lines for nine tables.
• No component model and no server-pushed UI state. A page — and, as of chapter 6, a fragment — is a function of the request. The only server state is a session row, which is what makes a second instance a configuration change rather than a project.
• No client framework. One vendored 33 KB script for fragments and no build step, because there is no front end to build.
• No cache. Chapter 11 says why: rendering costs about 1.5 CPU-ms for a 35 KB page, and a cache would be the most complex component in the system in exchange for a cost that is not the constraint.
• No savepoints. with-transaction is re-entrant — an inner form joins the outer one rather than nesting — because nothing yet needs to roll back part of an operation and keep the rest. The connection pool that this list used to say was missing arrived in 0.1.0; DATABASE.md has what it measured.

The living instance

What the diagram costs in practice, on the two-vCPU VPS that serves this book:

  resident memory          294 MB
  cold start               2.0 s  (was 9.4; chapter 11)
  Lisp                     7 457 lines, 18 files, 0 Lisp dependencies
  Java                       601 lines, 6 files, 28 jars, 39 MB
  tests                    232 Lisp checks, 134 browser checks,
                           both database backends, plus 10 shapes
                           of the NixOS module evaluated

Those are chapter 11's numbers and chapter 10's deployment. They are repeated here because an architecture diagram without a bill is a drawing.