CI for binary

.forgejo/workflows/zig-build.yaml

@@ -0,0 +1,92 @@
+name: Generic zig build
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - '*'
+env:
+  BUILD_TARGET: x86_64-linux-musl
+  BUILD_OPTIMIZATION: ReleaseSafe
+  BINARY_NAME: zfin-vestwell
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v4
+      - name: Setup Zig
+        uses: https://codeberg.org/mlugg/setup-zig@v2.2.1
+      # Needs network: build.zig.zon pins srf and zfin at git.lerch.org, and
+      # `zig build` resolves them here. The Docker build itself needs none.
+      - name: Build project
+        run: zig build --summary all
+      # Gates the image: `deploy` has `needs: build`, so a test failure means
+      # nothing is published.
+      - name: Run tests
+        run: zig build test --summary all
+      # Static musl is what makes the `FROM scratch` runtime image possible.
+      - name: Package
+        run: zig build -Dtarget="$BUILD_TARGET" -Doptimize="$BUILD_OPTIMIZATION"
+      - name: Upload
+        uses: actions/upload-artifact@v3
+        with:
+          name: ${{ env.BINARY_NAME }}
+          path: zig-out/bin/${{ env.BINARY_NAME }}
+      - name: Notify
+        uses: https://git.lerch.org/lobo/action-notify-ntfy@v2
+        if: always() && env.GITEA_ACTIONS == 'true'
+        with:
+          host: ${{ secrets.NTFY_HOST }}
+          topic: ${{ secrets.NTFY_TOPIC }}
+          status: ${{ job.status }}
+          user: ${{ secrets.NTFY_USER }}
+          password: ${{ secrets.NTFY_PASSWORD }}
+  deploy:
+    runs-on: ubuntu-latest
+    container:
+      image: ghcr.io/catthehacker/ubuntu:act-22.04
+    needs: build
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v4
+      - name: Download Artifact
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ env.BINARY_NAME }}
+      # The artifact lands beside the Dockerfile because the image build uses
+      # `context: docker` -- nothing outside that directory is visible to it.
+      - name: "Make executable actually executable"
+        run: chmod 755 ${{ env.BINARY_NAME }} && mv ${{ env.BINARY_NAME }} docker
+      - name: Get short ref
+        id: vars
+        run: echo "shortsha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+      - name: Login to Gitea
+        uses: docker/login-action@v2
+        with:
+          registry: git.lerch.org
+          username: ${{ github.actor }}
+          password: ${{ secrets.PACKAGE_PUSH }}
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: docker
+          push: true
+          tags: |
+            git.lerch.org/${{ github.repository }}:${{ steps.vars.outputs.shortsha }}
+            git.lerch.org/${{ github.repository }}:latest
+      # The Cronicle event pins an immutable tag rather than tracking `:latest`,
+      # because `:latest` moves on a push to any branch. Print the tag so it can
+      # be copied out of the run log when deploying.
+      - name: Deployable tag
+        run: |
+          echo "::notice::deploy tag git.lerch.org/${{ github.repository }}:${{ steps.vars.outputs.shortsha }}"
+          echo "git.lerch.org/${{ github.repository }}:${{ steps.vars.outputs.shortsha }}"
+      - name: Notify
+        uses: https://git.lerch.org/lobo/action-notify-ntfy@v2
+        if: always()
+        with:
+          host: ${{ secrets.NTFY_HOST }}
+          topic: ${{ secrets.NTFY_TOPIC }}
+          status: ${{ job.status }}
+          user: ${{ secrets.NTFY_USER }}
+          password: ${{ secrets.NTFY_PASSWORD }}

.gitignore

@@ -3,3 +3,8 @@ zig-out/
 zig-pkg/
 coverage/
 .tmp/
+
+# CI drops the built binary here before `docker build` (context: docker).
+# It is a build artifact, not source.
+docker/zfin-vestwell
+.stignore

README.md

@@ -154,19 +154,123 @@ There is deliberately no default for `--out`, so nothing is written anywhere by
 accident.
 
 
+### When the feed publishes
+
+Determined empirically, not guessed. The S3 object carries its own write time:
+
+```
+last-modified: Fri, 28 Aug 2026 12:00:39 GMT     <- OR-CSP.json
+last-modified: Fri, 28 Aug 2026 12:00:39 GMT     <- index.json (identical)
+navDate served: 2026-08-27
+```
+
+Identical timestamps across files means one pipeline writes them all. Checked
+against all eight Internet Archive captures of the feed, 8/8 consistent:
+
+| capture (UTC)    | dow | navDate    | dow | gap |
+|------------------|-----|------------|-----|-----|
+| 2025-02-15 06:03 | Sat | 2025-02-13 | Thu |   2 |
+| 2025-04-22 23:48 | Tue | 2025-04-21 | Mon |   1 |
+| 2025-08-28 13:57 | Thu | 2025-08-27 | Wed |   1 |
+| 2026-01-15 02:19 | Thu | 2026-01-13 | Tue |   2 |
+| 2026-03-31 15:46 | Tue | 2026-03-30 | Mon |   1 |
+| 2026-05-12 23:51 | Tue | 2026-05-11 | Mon |   1 |
+| 2026-08-02 07:45 | Sun | 2026-07-30 | Thu |   3 |
+| 2026-08-28 21:17 | Fri | 2026-08-27 | Thu |   1 |
+
+**Publish at ~12:00 UTC each weekday, carrying the previous business day's unit
+value.** The Sunday capture is the informative one: a three-day-old value only
+fits if weekends do not publish, so Friday's value does not appear until Monday.
+
+The practical consequence is that anything scheduled before 12:00 UTC reads the
+*previous* day's publish and is therefore two business days behind. 3:30am ET --
+07:30 UTC in EDT, 08:30 in EST -- is on the wrong side of that line.
+
 ### Steady state
 
-`record` is idempotent per navDate, so a daily job is safe -- the feed republishes
+Schedule at **13:00 UTC**, expressed in UTC rather than a local zone: the feed's
-the same value all weekend. Suggested cron/Cronicle shape:
+schedule is a UTC cron, so DST never enters into it. That leaves about an hour of
+margin after the publish.
 
-```
+```sh
-zfin-vestwell record && zfin-vestwell assemble --out "$ZFIN_CACHE_DIR"
+zfin-vestwell record   --data-dir /data && \
+zfin-vestwell assemble --data-dir /data --out /cache --offline
 ```
 
-`assemble` also fetches the feed itself, so the cache is current even on the run
+`record` does the one feed fetch and persists it; `assemble --offline` then merges
-where `record` had nothing new to add. Observed values always beat reconstructed
+purely from disk. Two steps rather than one so a failure is attributable, and
-ones for the same date, and `assemble` reports any date where they disagreed by
+`assemble` stays independently useful -- after a `reconstruct` on a workstation
-more than 0.1% -- a sustained count there means the model has drifted.
+syncs new `data/*.srf` across, `assemble` alone refreshes the cache.
+
+Running all seven days is slightly better than weekdays only. `record` is
+idempotent per navDate so weekend runs add nothing, but `assemble` re-stamps
+`#!expires` and keeps clients from falling through to the provider path.
+
+Observed values always beat reconstructed ones for the same date, and `assemble`
+reports any date where they disagreed by more than 0.1% -- a sustained count there
+means the model has drifted.
+
+### Running it on the NAS
+
+The container is `FROM scratch` with a static musl binary, ~10MB. Built and pushed
+by `.forgejo/workflows/zig-build.yaml` to
+`git.lerch.org/lobo/zfin-vestwell:{<shortsha>,latest}`.
+
+Two mounts. `/data` is the syncthing-mirrored repo, which is where the SRF inputs
+live and where `record` appends. `/cache` is zfin-server's cache directory, which
+is what `assemble` populates.
+
+```sh
+IMAGE=git.lerch.org/lobo/zfin-vestwell:<shortsha>
+DATA=/home/lobo/shared/zfin-vestwell/data
+CACHE=/data/zfin/cache
+
+docker run --rm -v "$DATA:/data" -v "$CACHE:/cache" "$IMAGE" \
+  record --data-dir /data && \
+docker run --rm -v "$DATA:/data" -v "$CACHE:/cache" "$IMAGE" \
+  assemble --data-dir /data --out /cache --offline
+```
+
+The Cronicle event pins `<shortsha>` rather than tracking `:latest`, because the
+workflow moves `:latest` on a push to any branch and this writes into a live
+cache. The deploy job prints the tag to copy.
+
+`USER 1000:1000` is baked into the image, and both host directories are owned by
+1000:1000, so no `--user` flag is needed. The uid must be numeric because
+`scratch` has no `/etc/passwd`.
+
+Writing into the cache while zfin-server serves from it is safe: zfin's `writeRaw`
+is atomic (tmp + fsync + rename), so a concurrent reader sees either the old file
+or the new one, never a torn one.
+
+### What runs where, and why
+
+| Command       | Needs zfin's candle cache        | Runs on     |
+|---------------|----------------------------------|-------------|
+| `reconstruct` | yes, five Vanguard funds         | workstation |
+| `verify`      | yes, cross-validation needs them | workstation |
+| `record`      | no                               | NAS         |
+| `assemble`    | no                               | NAS         |
+
+`reconstruct` and `verify` call the zfin-cache reader; `record` and `assemble` do
+not. That is not incidental -- it is why the NAS job needs no zfin cache of its
+own, which matters because the NAS tracks VSMPX, VTPSX and VBMPX but not VIPIX or
+VTIFX.
+
+There is therefore **no ordering dependency on the zfin NAV-refresh cron.** The
+only timing constraint is the Vestwell publish above.
+
+### Single writer
+
+**Only the NAS runs `record`.** `data/observed.srf` lives in the syncthing-mirrored
+tree so that it stays version-controlled and visible to the workstation, which
+means two writers would produce `.sync-conflict` files. Syncthing carries the
+appended log back for committing; avoid `git checkout`, `stash` or `reset` on that
+file while a run could be in flight.
+
+`.stignore` keeps the build artifacts out of the mirror. Without it syncthing would
+carry ~2GB (`.zig-cache` alone is 1.9GB) against 208K of actual data.
+
 
 ## Limitations
 

build.zig.zon

@@ -25,6 +25,7 @@
         "build.zig",
         "build.zig.zon",
         "build",
+        "docker",
         "src",
         "data",
         "tools",

docker/Dockerfile

@@ -0,0 +1,30 @@
+# Runtime image for zfin-vestwell.
+#
+# Single stage on `scratch`, matching zfin and zfin-server: the Zig build runs in
+# CI, produces a statically-linked musl binary, and the workflow drops it into
+# this directory before `docker build`. There is no toolchain in the image and no
+# network needed at image-build time.
+#
+# The CA bundle is mandatory, not optional cruft. `record` fetches the plan's feed
+# over HTTPS, and Zig's certificate loader scans a fixed list of host paths
+# (std/crypto/Certificate/Bundle.zig) of which `/etc/ssl/certs/ca-certificates.crt`
+# is the Debian/Ubuntu one. `scratch` has no CA store at all, so without this the
+# TLS handshake fails.
+FROM scratch
+COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
+COPY zfin-vestwell /zfin-vestwell
+
+# Bare ENTRYPOINT, following zfin rather than zfin-server: the subcommand varies
+# per invocation (`record` then `assemble`), so it cannot be baked in.
+#
+# WORKDIR is the data mount, so `--data-dir` can be omitted when the caller mounts
+# the repo's data/ directory at /data.
+WORKDIR /data
+
+# Numeric uid:gid is the only form that can work here -- `scratch` has no
+# /etc/passwd for a name to resolve against. 1000:1000 is what owns the
+# syncthing-mirrored data directory and the zfin cache on the NAS, and this
+# process writes to both.
+USER 1000:1000
+
+ENTRYPOINT ["/zfin-vestwell"]