Deployment

The whole artifact is a directory. There is no container, no application server, and nothing to install into.

  spicemine/
    bin/spicemine-server      generated launcher
    bin/spicemine-admin       the way back in
    lib/*.jar                 29 jars, 40 MB
    lisp/*.lisp               the sources, 1,964 KB
    lisp/*.abcl               the same files compiled
    book/*.md                 this book

./gradlew installDist produces it; installDist is finalized by compileLisp, so the packaged artifact is always the compiled one and nobody has to remember a flag.

The service

A systemd unit with Environment=JAVA_OPTS and a data directory. In this project it is a NixOS module, and the parts worth copying anywhere:

• The service user owns its data. /var/lib/spicemine is drwxr-x--- spicemine spicemine. Run the admin tool as that user or sqlite leaves root-owned journal files and the server loses write access to its own database.
• Secrets never enter the store. The Nix store is world-readable, so credentials live in a root-owned mode-600 file referenced by EnvironmentFile=, and three module assertions refuse to build if a config path points at /nix/store.
• TLS terminates in front. The app speaks plain HTTP on a loopback port and reads X-Forwarded-Proto to decide whether to set Secure on cookies.

Heap and threads

Netty sizes its pools from the core count of the machine it finds itself on, and on anything large that is the single biggest line in this process's memory. Set the pool sizes to the vCPUs you actually have and the heap can come down with no throughput penalty:

                                        peak RSS   req/s
  -Xmx2g, default Netty pools            1 575 MB   2 821
  -Xmx2g, io=2 worker=2                    975 MB   2 713
  -Xmx2g, io=2 worker=2, -Xss512k          973 MB   2 821
  -Xmx1g, io=2 worker=2, -Xss512k          714 MB   2 702
  -Xmx512m, io=2 worker=2, -Xss512k        584 MB   2 770

Native Memory Tracking on the 35 KB page says where it goes:

                     default pools   io=2 worker=2
  Java heap             1 210 MB        355 MB
  Thread stacks           455 MB         78 MB
  Direct buffers           45 MB          2 MB
  Class (metaspace)        28 MB         28 MB
  Code cache               25 MB         24 MB

Hundreds of threads cost 455 MB of stack reservation, and the concurrent in-flight work keeps the collector on a 1.2 GB heap. Cap the pools and both collapse.

Two things are not the cause, and both are worth knowing because both are the usual suspects:

• Direct buffers. 45 MB of 1,575. Netty 4.2 changed its default allocator to the adaptive one; for this workload it is better than the old pooled allocator, which measures 144 MB of direct memory against the adaptive one's 45 — so -Dio.netty.allocator.type=pooled makes this worse. -Dio.netty.tryReflectionSetAccessible=true with --add-opens java.base/java.nio=ALL-UNNAMED, the standard advice for pinning JDK 9+ direct memory, moves peak RSS by 4% and leaves direct memory unchanged. Neither is in the deployment.
• ABCL's generated classes. ABCL emits a JVM class per compiled function, so metaspace looks like the obvious suspect. It is 28 MB and it does not move.

What to set

  Environment=JAVA_OPTS=-Xmx512m -Xss512k \
    -Dspice.io.threads=2 -Dspice.worker.threads=2 \
    -Dspice.port=8643 -Dspice.home=/var/lib/spicemine

A real instance under that configuration sits at 511 MB resident while serving pages, so a 1 GB host is enough. The thread counts are not an expert tuning knob: leaving them unset is how you get half a gigabyte of thread stacks on a box with two cores to run them on.

-XX:+UseSerialGC also fits, at 578 MB, and halves throughput to 1,330 req/s. Not worth it unless memory is tighter than that.

Method and the full attribution are in core/bench/README.md.

Startup

Cold start was about 2.0 seconds to the first answered request when it was measured, at mélangej 0.2.0 with eighteen files, and almost none of it was yours:

  bare ABCL                      0.67 s
  + the eighteen fasls, Jooby    2.0  s

Loading the fasls is 0.17 s of that, measured on the installed distribution; the rest is the JVM, ABCL and Jooby starting, with a generated class per route. It was 9.4 seconds until 0.1.0, and 8.2 of those were (require :abcl-contrib) and (require :jss), loaded for one reader macro; chapter 11 has the before and after. Plan the restart window around two seconds, and do not expect it to shrink by writing less Lisp.

Releases

The deploy script runs on the host, not from your laptop, and it takes every location from the environment:

export SPICEMINE_HOST=your.host.example
ssh "root@$SPICEMINE_HOST" 'cd /root/spicemine && deploy/deploy.sh'

REPO=/root/spicemine   # where the checkout is
DEST=/opt/spicemine    # where releases go
PORT=8643  KEEP=5  UNIT=spicemine

No address, hostname or account appears in the repository. An address written into a file is correct until the day it is not, and is readable by everyone who can read the repository in the meantime.

What it does: update the checkout, build the distribution, copy it into a timestamped release directory, repoint the current symlink, restart the unit, health-check the result. A rollback is moving the symlink back.

It is forty lines of shell, and the reason it can be forty lines is the zero-Lisp-dependency rule: there is no package manager to run on the far side.

Can you upgrade a running system instead of restarting it?

Two seconds of startup still makes this a fair question, and the answer is yes, further than expected - including replacing Java classes that are already loaded. Everything below was run against a live server.

  works     redefine a Lisp function a live route calls
  works     load a changed .lisp file into the live image
  works     compile-file a changed file and load the fasl live
  works     add a jar to a running image's classpath
  works     load a second version of a Java class under its own loader
  works     redefine an already-loaded Java class (method bodies)
  refused   add or remove a Java field or method
  fails     add a new URL to a running Jooby app

The Lisp side: a file at a time

Redefining one function works, which is unsurprising:

(defun say () "VERSION-ONE")
(def-route (app "GET" "/named") (ctx) (say))
;; /named -> VERSION-ONE
(defun say () "VERSION-TWO")
;; /named -> VERSION-TWO      no restart, no dropped connection

So does loading a whole changed file, which is the useful unit:

(load "spice-ui.lisp")        ; interpreted, instant

and so does compiling it first, which is what you want for anything hot:

(load (compile-file "spice-ui.lisp"))   ; -> spice-ui.abcl, then loaded

Both were verified against a running server: a page function defined in a file, served, then replaced from source and again from a freshly compiled fasl, with the new text appearing on the next request each time. That is the whole development loop, available on a production image.

The one thing to arrange is indirection: a route body that calls a named function picks up a new definition, while a route whose logic is inline in the closure cannot be changed. spice-ui.lisp is written as thin handlers calling named functions anyway, for reasons that had nothing to do with this.

Jars: add freely, and replace with care

Adding a jar to a running image works:

(java:add-to-classpath "/path/to/gson-2.10.1.jar")
(java:jcall "toJson" (java:jnew "com.google.gson.Gson") "hi")   ; => "\"hi\""

A class that was not on the classpath at startup becomes loadable and usable. For adding a driver or a library to a long-running instance, that is genuine hot deployment.

Replacing a class that is already loaded is also possible, which is the part usually written off. A loaded class belongs to its classloader for the life of that loader, so there are two routes:

A new classloader. Load the new jar under its own URLClassLoader and use the class from there. Verified: v2 answers from its loader while the original instance still answers v1. The catch is type identity - the two demo.Greeter classes are different types, so you cannot pass one where the other is expected. Fine for a plugin, useless for patching something the framework already holds.

The Attach API, which needs no restart and no startup flag on the target. An agent can be loaded into an already running JVM and redefine bytecode in place:

(let* ((nm (java:jcall "getName"
                       (java:jstatic "getRuntimeMXBean"
                                     "java.lang.management.ManagementFactory")))
       (pid (subseq nm 0 (position #\@ nm)))
       (vm (java:jstatic "attach" "com.sun.tools.attach.VirtualMachine" pid)))
  (java:jcall "loadAgent" vm "/path/hotagent.jar"
              "demo.Greeter=/path/new/Greeter.class")
  (java:jcall "detach" vm))

The agent's entry point is agentmain (not premain, which is the -javaagent path) and it calls Instrumentation.redefineClasses. Measured result: the existing object's method returned the new value immediately - no new instance, no reload, no restart.

Two details that decide whether this works for you:

• Find the class through Instrumentation.getAllLoadedClasses(), not Class.forName. The agent runs in the system classloader, and anything ABCL loaded through add-to-classpath lives in org.armedbear.lisp.JavaClassLoader, invisible from there. Searching the loaded-class list finds every copy regardless of loader - in the test it found and redefined two, one per loader.
• Self-attach needs -Djdk.attach.allowAttachSelf=true on JDK 9+. Attaching from a separate process does not.

The hard limit is the class shape. Method bodies may change; the schema may not:

  change a method body          -> redefined, live instances pick it up
  add or remove a field/method  -> UnsupportedOperationException:
                                   "attempted to change the schema"

The refusal is clean - the class stays at its previous version rather than ending up half-applied. So this patches logic, not structure. For structural change you need a fresh classloader and a fresh object graph, which is what an application server does and what this architecture deliberately does not have.

The route table is the one real wall

Adding a URL to a running Jooby app fails, as does re-registering one:

java.lang.NullPointerException: Cannot invoke "java.util.LinkedList.getLast()"
  because "this.stack" is null

Jooby builds its route table during startup and the structure it builds with is gone afterwards. That is Jooby's design and a defensible one - a router mutable under load is a router with a concurrency bug waiting in it.

If hot-adding endpoints is a requirement, register one catch-all route at startup and dispatch inside Lisp, so a new URL is a hash-table write. That trades Jooby's routing for your own, and is the right trade only if you actually need it.

Jooby's own hot reload, and why we do not get it

Jooby has a development hot reload: mvn jooby:run or ./gradlew joobyRun watches the source tree, recompiles what changed, and restarts the application without exiting the JVM, using JBoss Modules to swap the application's classloader. Defaults worth knowing if you ever use it on a Java project: .java/.kt trigger a compile-then-restart, .conf/.properties/.class restart immediately, and it waits 500 ms after the last change so a multi-file save is one restart.

It does not apply to mélangej, for a reason that is structural rather than a missing feature. Jooby reloads by discarding a classloader and building a new application from freshly compiled classes. Our application is not classes. The Lisp files are data to ABCL, loaded into one long-lived image; joobyRun has no .lisp in compileExtensions and nothing to recompile if it did, and the thing it would throw away and rebuild — the classloader — is the thing holding the Lisp image we want to keep.

The two mechanisms also aim at different targets:

  joobyRun                          the ABCL REPL
  ---------------------------------------------------------------------
  restart the app, keep the JVM     keep the app, redefine a function
  granularity: the whole router     granularity: one function
  needs a compile step              needs nothing
  a dev loop                        a dev loop AND a live production
                                      instance you can inspect
  new routes: yes (it rebuilds)     new routes: no (see above)

So the comparison ends in a trade rather than a winner. Jooby's reload can add a route, because it rebuilds the router from scratch; ours cannot, because it does not. Ours can redefine one function in a process that is serving traffic and holding warmed state, which is a thing joobyRun is not trying to do.

What it does mean for us: joobyRun is the right answer if you add Java to a mélangej project. The six .java files below the waterline (chapter 14) are ordinary Java, and if you are iterating on GitHttp or FossilCgi rather than on Lisp, Jooby's plugin is a better loop than installDist and a nine-second restart. It is not in this project's build because nobody has needed it yet — the Java changes about once a quarter and the Lisp changes hourly.

Getting a REPL into the running server

None of the above is reachable without one. Set spice.swank.port and the server starts swank inside its own image (chapter 9):

JAVA_OPTS="-Dspice.swank.port=4005" spicemine-server
ssh -N -L 4005:127.0.0.1:4005 "$SPICEMINE_HOST"    # from your machine

then M-x slime-connect to localhost:4005 and you are inside the process serving traffic.

A swank port is unauthenticated remote code execution. It binds to localhost, it is reached through an SSH tunnel, and it is off unless the property is set. If you would not give someone a root shell, do not give them this port.

Should you?

Having established that it works: for a routine deploy, don't.

A restart costs two seconds and gives you a process whose state is exactly what the files say. A hot-patched image is a process whose behaviour is the files plus the history of what somebody typed into it, and that history is not in version control, not in the next deploy, and not in anybody's head after a fortnight. The failure is not that hot patching goes wrong; it is that it goes right, and three weeks later a restart silently reverts a fix nobody remembers making.

Where it earns its keep is the case restart cannot serve: reading live state to find out what is actually happening, and stopping the bleeding at 3am before the real fix goes through the normal path. Use it for the emergency and the autopsy. Deploy with the deploy script.

Backups

The backup takes a consistent copy and then proves it can be restored: sqlite integrity_check, fossil test-integrity on every repository, and a check that the signing key is present. It prints RESTORABLE or it fails loudly. A backup that has never been restored is a hope.

Offsite is one rclone destination of your choosing (S3, B2, R2, GCS, SFTP, MinIO), optionally encrypted to an age public key, with the upload read back and hashed against a sidecar. --audit re-reads everything on the remote and compares.

Two rules in it that are not specific to this project:

• the host name comes from uname -n, because the unit's PATH does not include hostname;
• only the newest N local archives are upload candidates, so an archive that local pruning removed is never offered again.

The migration story

Schema changes are additive and applied at boot: a list of (table column type) triples, each added with alter table add column if the column is not already there. Both sqlite and postgres do that cheaply and without rewriting rows, which is why every addition is nullable or defaulted — an upgrade must not need a maintenance window on somebody else's instance.

The migration test builds a database in the exact shape the deployed version had, runs the current schema against it, and asserts the old rows, tokens and repositories all still work. That test is the reason upgrades on the live instance have been uneventful.