lobo/zfin-vestwell
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
zfin-vestwell/README.md
Emil Lerch 5f47c95339
All checks were successful
Generic zig build / build (push) Successful in 1m16s
Details
Generic zig build / deploy (push) Successful in 16s
Details
json feeds do not publish NAV for Friday until Monday :(. This will derive that value temporarily
2026-08-31 10:47:24 -07:00

622 lines
31 KiB
Markdown
Raw Permalink Blame History

# zfin-vestwell
Daily unit-value history for the two Oregon College Savings Plan (Embark)
portfolios held in the finance repo: **ORCBI** and **ORC42**.
```
zfin-vestwell reconstruct # rebuild data/ORCBI.srf and data/ORC42.srf
zfin-vestwell verify # check them against every evidence source
zfin-vestwell record # append today's published value to the log
zfin-vestwell project # fill sessions the feed has not published yet
zfin-vestwell assemble --out DIR # write zfin cache files for it to serve
```
## The problem
`ORCBI` and `ORC42` are not securities in the ordinary sense. They are
**unitized fund-of-funds trust portfolios** inside a 529 plan: no ticker, no
CUSIP, no market-data provider coverage. Tiingo, Yahoo and TwelveData all 404
them.
The plan publishes a daily unit value, but only the **current** one. There is no
historical endpoint, no downloadable series, and no third-party archive of the
numbers. So history has to be reconstructed.
| Property | ORCBI | ORC42 |
|------------------------|----------------|------------------------------|
| Plan name | Balanced Index | College Enrollment Year 2042 |
| Kind | static | glidepath, steps quarterly |
| Feed fund id | `T6GF10080002` | `T6GFRD204202` |
| Inception | 2018-09-01 | 2023-07-01 |
| Sessions reconstructed | 2006 | 792 |
## Method
### 1. Anchors: the only observed values
Nine exact values per portfolio, in `data/anchors.srf`.
**Launch.** Every portfolio in the plan launched at exactly `$10.00`. Verified by
back-solving the feed's own `inception` field, which is the *annualized*
since-inception return: `nav / (1 + inception)^years` lands between 10.0005 and
10.0038 for all 37 priced portfolios. The Plan Disclosure Booklet agrees on the
date -- "The Portfolios commenced operations in September 2018."
**Seven Internet Archive captures** of the live feed, spanning 2025-02-13 to
2026-07-30, each carrying full 6-decimal values.
**The current feed value.**
> Keep `data/anchors.srf`. The wayback rows are not recoverable if the Internet
> Archive prunes those captures, and nothing else publishes these numbers.
### 2. Interpolation between anchors
Within each gap, walk the trading-day calendar compounding the weighted
**total return** of the underlying Vanguard funds, read from zfin's candle cache
(`adj_close`). Weights come from `data/model.srf`, per era, so a glidepath step
mid-gap needs no special handling.
Total return, not raw NAV, because the plan's own booklet says these portfolios
"reflect changes in value from income and gains and losses on the sale of the
Underlying Funds **solely by increasing or decreasing their Unit Value**." All
income compounds into the unit value; nothing is distributed. The underlying
mutual funds *do* distribute, so their raw NAV drops on every ex-date. A raw-NAV
basket under-tracks by roughly the blended yield -- measured at +1.5% to -1.6%
drift over four months on ORCBI. That is why `ticker::` + `price_ratio::` is the
wrong tool for these symbols.
### 3. Pinning
The walk lands near but not exactly on the closing anchor. Take the ratio and
distribute it geometrically across the gap, so both endpoints are exact.
**This step matters more than the model.** No fee term appears anywhere in this
project, and none is needed: the asset-based fee, cash drag, trade-date lag and
securities-lending income are all roughly constant-rate effects, and a
constant-rate misfit is exactly what geometric distribution removes. An unpinned
walk across ORCBI's 6.5-year pre-2025 gap is off by -1.29%; pinned, that same gap
predicts a held-out interior anchor to **+0.013%**.
## Accuracy
`verify` measures it two independent ways.
**Leave-one-out cross-validation.** Drop an interior anchor, pin its neighbours,
predict it. This is the number that matters, because it measures error in the
*middle* of an un-anchored gap.
| Fund | Worst interior error |
|-------|----------------------|
| ORCBI | 0.079% |
| ORC42 | 0.060% |
**Against hand-typed values.** 16 values per portfolio, recovered from the
finance repo's snapshots and **never fed into the model**, so agreement is
evidence rather than a fit. Compared at lag=1, since the portal publishes the
prior session's value.
| Fund | n | mean | stdev | max |
|-------|-----|---------|--------|----------------|
| ORCBI | 16 | +0.0075 | 0.0089 | 0.0287 (0.15%) |
| ORC42 | 16 | +0.0035 | 0.0075 | 0.0312 (0.19%) |
Those values are rounded to 2dp at source, which is already +/-0.005 of noise.
**One session forward, unpinned.** Both figures above measure *interpolation* --
error in the middle of a gap whose far end is anchored. `project` has no far end,
so neither bounds it. Walking from a published value one session forward and
comparing against the next published value:
| Fund | Projected | Published | Error |
|-------|------------|------------|---------|
| ORCBI | 19.092910 | 19.092721 | 0.0010% |
| ORC42 | 16.808301 | 16.808106 | 0.0012% |
About eighty times tighter than the interior interpolation bound, and two orders
of magnitude under the 0.1% reporting threshold. That is unsurprising -- one day of
tracking error on a basket with exactly known weights has very little room to go
wrong -- but it is measured rather than assumed, and `assemble` re-measures it
every time a published value lands on a date that was projected.
**Implied drag** is reported for information, not enforced. It should sit near
the plan's published cost table (0.234%/yr for ORCBI, 0.239% for ORC42); a fee
is not identifiable over a short gap, so per-gap figures are noisy by nature.
## Serving it to zfin
`ORC42` and `ORCBI` have no provider coverage, so zfin can only see them if
something puts files in its candle cache. That is what `assemble` does.
```
zfin-vestwell record # feed -> data/observed.srf (idempotent)
zfin-vestwell project # funds -> data/observed.srf (weekend fill)
zfin-vestwell assemble --out "$CACHE" # series + log + feed -> cache files
```
`assemble` writes `<out>/<SYMBOL>/candles_daily.srf` and `candles_meta.srf`, and
needs no zfin cache of its own -- only the committed series and the observation
log. Run it on whatever host owns the cache zfin reads. On ZFIN_SERVER that makes
the symbols available to every client, because a read never triggers a refetch:
the server serves a present file as-is.
Verified against a real zfin:
```
$ zfin diagnose ORC42
local newest 2026-08-27, TTL still in the future, external
adj basis 1970-01-01 - no dividends or splits cached, nothing to restate
tracked NO - no normal run fetches this symbol
server the server will never refresh this symbol on its own
provider tiingo FAILED: NotFound
```
Every line there is load-bearing. The cache is fresh and tagged `external`; the
adjustment-restatement path cannot fire because there are no corporate actions;
the symbol is untracked so no refresh sweep touches it; and no provider carries it,
which is precisely why the cache has to be populated this way.
### Deployment order matters
`assemble` writes `provider::external`. A zfin that does not know that enum
variant treats the meta file as a cache **miss**, takes the cold-start path, gets
a unanimous 404, and writes a negative-cache marker **over
`candles_daily.srf`** -- the history is then gone.
So deploy a zfin carrying the variant to both the client and ZFIN_SERVER *before*
populating a cache they read. That is an operational precondition, not something
this program checks: the enum is verified at compile time against the zfin this
links against, which says nothing about the zfin on the consuming side.
There is deliberately no default for `--out`, so nothing is written anywhere by
accident.
### When the feed publishes
Determined empirically, not guessed. The S3 object carries its own write time:
```
last-modified: Fri, 28 Aug 2026 12:00:39 GMT <- OR-CSP.json
last-modified: Fri, 28 Aug 2026 12:00:39 GMT <- index.json (identical)
navDate served: 2026-08-27
```
Identical timestamps across files means one pipeline writes them all. Checked
against all eight Internet Archive captures of the feed, 8/8 consistent:
| capture (UTC) | dow | navDate | dow | gap |
|------------------|-----|------------|-----|-----|
| 2025-02-15 06:03 | Sat | 2025-02-13 | Thu | 2 |
| 2025-04-22 23:48 | Tue | 2025-04-21 | Mon | 1 |
| 2025-08-28 13:57 | Thu | 2025-08-27 | Wed | 1 |
| 2026-01-15 02:19 | Thu | 2026-01-13 | Tue | 2 |
| 2026-03-31 15:46 | Tue | 2026-03-30 | Mon | 1 |
| 2026-05-12 23:51 | Tue | 2026-05-11 | Mon | 1 |
| 2026-08-02 07:45 | Sun | 2026-07-30 | Thu | 3 |
| 2026-08-28 21:17 | Fri | 2026-08-27 | Thu | 1 |
**Publish at ~12:00 UTC each weekday, carrying the previous business day's unit
value.** The Sunday capture is the informative one: a three-day-old value only
fits if weekends do not publish, so Friday's value does not appear until Monday.
Confirmed directly from three consecutive job runs at 13:00 UTC:
| Run (UTC) | dow | navDate served | outcome |
|---------------------|-----|----------------|----------------------------|
| 2026-08-29 13:00:00 | Sat | 2026-08-27 | already recorded, no-op |
| 2026-08-30 13:00:00 | Sun | 2026-08-27 | already recorded, no-op |
| 2026-08-31 13:00:00 | Mon | 2026-08-28 | recorded |
So the weekend blackout is real and is not a scheduling artefact. **Friday's unit
value does not exist in this feed until Monday around 12:00 UTC**, and no cron
time can change that:
```
Mon 12:00 -> Fri Thu 12:00 -> Wed
Tue 12:00 -> Mon Fri 12:00 -> Thu
Wed 12:00 -> Tue Sat (none) -> Thu
Sun (none) -> Thu
```
From Friday's publish until Monday's, the newest value the feed can offer is
Thursday's. `project` is what fills that; see below. The plan's own logged-in UI
does have Friday's value on Saturday, so the recordkeeper is ahead of this
marketing export -- but that path needs an authenticated session and this one does
not.
Two lesser hazards, neither of which has bitten yet:
- The object ships **no `cache-control` and no `expires`**, so CloudFront falls
back to a distribution default TTL. A POP serving a pre-publish copy at 13:00
would look exactly like the weekend blackout.
- That copy **cannot be forced**. A query-string cache-buster, `Cache-Control:
no-cache` and `Pragma: no-cache` all return a byte-identical cached response
with the `date` header frozen and `age` still climbing. The only available
response is to detect and retry later.
The practical consequence for scheduling is that anything before 12:00 UTC reads
the *previous* day's publish and is therefore two business days behind. 3:30am ET
-- 07:30 UTC in EDT, 08:30 in EST -- is on the wrong side of that line.
### Steady state
Schedule at **13:00 UTC**, expressed in UTC rather than a local zone: the feed's
schedule is a UTC cron, so DST never enters into it. That leaves about an hour of
margin after the publish -- measured at 59m23s against an observed `last-modified`
of 12:00:39.
```sh
zfin-vestwell record --data-dir /data && \
zfin-vestwell project --data-dir /data && \
zfin-vestwell assemble --data-dir /data --out /cache --offline
```
`record` does the one feed fetch and persists it; `project` fills any session the
underlying funds already have but the feed has not published; `assemble --offline`
then merges purely from disk. Three steps rather than one so a failure is
attributable, and `assemble` stays independently useful -- after a `reconstruct` on
a workstation syncs new `data/*.srf` across, `assemble` alone refreshes the cache.
Running all seven days is what makes `project` worth having: Saturday is the run
that fills Friday. `record` is idempotent per navDate and `project` per
`(date, value)`, so the extra runs append nothing they should not, and `assemble`
re-stamps `#!expires` and keeps clients from falling through to the provider path.
Observed values always beat reconstructed ones for the same date, and `assemble`
reports any date where they disagreed by more than 0.1% -- a sustained count there
means the model has drifted.
### Filling the weekend: `project`
The feed publishes on weekdays; the underlying Vanguard funds do not have that
gap. zfin's NAV refresh runs at 3:30am ET, so by Saturday 13:00 UTC the funds'
Friday closes are in its cache while the feed still reads Thursday. `project`
walks that difference: it takes the newest **feed** value and compounds the
weighted total return forward, exactly as the reconstruction does between anchors
but with nothing at the far end to pin against.
The trigger is a comparison, not a date calculation:
| Sat 13:00 UTC | funds newest | feed newest | outcome |
|---------------|--------------|-------------|--------------------------------|
| Tue-Fri | D-1 | D-1 | no gap, no-op |
| **Sat** | **Fri** | **Thu** | **project Fri** |
| Sun | Fri | Thu | same value recomputed, no-op |
| Mon | Fri | Fri | no gap; the publish supersedes |
Nothing in the code knows what a weekend is. A market holiday or an unscheduled
closure needs no handling either: a day that did not trade has no bar, so it never
enters the calendar. And if zfin's refresh fails, the funds do not advance, no gap
is seen, and behaviour degrades to what it was before this existed rather than to
a wrong number.
Projections live in `data/observed.srf` alongside the feed values, marked
`source::projected`. An absent `source` means `feed`, so every row written before
this existed still means what it did. Three things depend on the distinction, and
each would be silently wrong without it: `record` must not treat a projected date
as already recorded, or Monday's real value would be skipped; a projection must
not be walked forward from another projection; and `verify`'s live-feed check must
not read a projection as the feed being stale.
Precedence in `assemble` is `observed` > `reconstructed` > `projected` -- a value
pinned at both ends beats one pinned at a single end. So Monday's publish replaces
Saturday's estimate automatically, and `assemble` reports the realised error when
it does. That is the only measurement of extrapolation accuracy there is; the
cross-validation figures above bound *interpolation*.
Measured against a published anchor, one session forward is accurate to about
**0.001%** for both portfolios -- roughly eighty times tighter than the interior
interpolation bound, and well under the 0.1% reporting threshold.
Two deliberate limits:
- **A horizon past five sessions is refused**, and `project` exits non-zero. One
session is normal, two means the feed missed a weekday publish, and a week means
the feed is broken rather than closed. Failing stops the `&&` chain, so the last
good cache keeps being served instead of an ever-longer unpinned walk, and the
job goes red.
- **`project` alone depends on zfin's NAV refresh** having run. `record` and
`assemble` still do not, so the independence claimed under *What runs where* is
narrowed to this one command rather than given up.
Every other failure -- a fund missing from the cache, a navDate the funds have no
bar for, a walk that cannot be computed -- warns and exits 0. A projection is an
enrichment, and it must never stop the cache being written.
### Running it on the NAS
The container is `FROM scratch` with a static musl binary, ~10MB. Built and pushed
by `.forgejo/workflows/zig-build.yaml` to
`git.lerch.org/lobo/zfin-vestwell:{<shortsha>,latest}`.
Two mounts. `/data` is the syncthing-mirrored repo, which is where the SRF inputs
live and where `record` and `project` append. `/cache` is zfin-server's cache
directory: `assemble` populates it, and `project` reads the underlying funds out of
it.
`ZFIN_CACHE_DIR` is required rather than optional here. `project` resolves the
cache the way zfin does -- `$ZFIN_CACHE_DIR`, else `$XDG_CACHE_HOME/zfin`, else
`$HOME/.cache/zfin` -- and a `scratch` image has no `/etc/passwd`, so `$HOME` is
unset and the fallback chain has nothing to land on.
```sh
IMAGE=git.lerch.org/lobo/zfin-vestwell:<shortsha>
DATA=/home/lobo/shared/zfin-vestwell/data
CACHE=/data/zfin/cache
docker run --rm -v "$DATA:/data" -v "$CACHE:/cache" "$IMAGE" \
record --data-dir /data && \
docker run --rm -e ZFIN_CACHE_DIR=/cache -v "$DATA:/data" -v "$CACHE:/cache" "$IMAGE" \
project --data-dir /data && \
docker run --rm -v "$DATA:/data" -v "$CACHE:/cache" "$IMAGE" \
assemble --data-dir /data --out /cache --offline
```
Only the `project` invocation carries `-e ZFIN_CACHE_DIR=/cache`, because it is the
only one of the three that reads the candle cache. Passing it to all three would be
harmless, but naming it once makes it obvious which command would break without it.
`project` reads the funds from the same `/cache` it hands `assemble` as `--out`;
they are different symbols under one root, so no extra mount is needed. It reads
that cache directly rather than asking ZFIN_SERVER over HTTP: this job already runs
on the host that owns the cache, so a network call would fetch data off the disk it
has mounted while adding a live-service dependency and an API key to a job that
otherwise needs neither.
The Cronicle event pins `<shortsha>` rather than tracking `:latest`, because the
workflow moves `:latest` on a push to any branch and this writes into a live
cache. The deploy job prints the tag to copy.
`USER 1000:1000` is baked into the image, and both host directories are owned by
1000:1000, so no `--user` flag is needed. The uid must be numeric because
`scratch` has no `/etc/passwd`.
Writing into the cache while zfin-server serves from it is safe: zfin's `writeRaw`
is atomic (tmp + fsync + rename), so a concurrent reader sees either the old file
or the new one, never a torn one. That same guarantee is what makes `project`
reading the funds out from under a concurrent NAV refresh safe.
ORC42 needs **VIPIX** and **VTIFX** tracked on the NAS in addition to VSMPX, VTPSX
and VBMPX; they are 2% of its allocation each. ORCBI has been a single static era
of `VSMPX 36 / VTPSX 24 / VBMPX 40` since 2018-09-01, so it needs nothing beyond
what was already there. Without the two extra funds `project` declines ORC42 and
says so, rather than projecting from a partial basket.
### What runs where, and why
| Command | Needs zfin's candle cache | Runs on |
|---------------|----------------------------------|-------------|
| `reconstruct` | yes, five Vanguard funds | workstation |
| `verify` | yes, cross-validation needs them | workstation |
| `record` | no | NAS |
| `project` | yes, the in-force era's funds | NAS |
| `assemble` | no | NAS |
`reconstruct`, `verify` and `project` call the zfin-cache reader; `record` and
`assemble` do not. `record` and `assemble` between them are still enough to
maintain a correct cache, which is why a failure in `project` is not allowed to
stop them.
`project` is therefore **the one command with an ordering dependency on the zfin
NAV-refresh cron**, and it needs VIPIX and VTIFX tracked as well as VSMPX, VTPSX
and VBMPX. The dependency is comfortable rather than tight: the refresh runs at
3:30am ET and the job at 13:00 UTC, four and a half hours later. If it has not run,
the funds have not advanced, `project` sees no gap and does nothing.
For everything else the only timing constraint is the Vestwell publish above.
### Single writer
**Only the NAS runs `record` and `project`.** `data/observed.srf` lives in the
syncthing-mirrored tree so that it stays version-controlled and visible to the
workstation, which means two writers would produce `.sync-conflict` files.
Syncthing carries the appended log back for committing; avoid `git checkout`,
`stash` or `reset` on that file while a run could be in flight.
`.stignore` keeps the build artifacts out of the mirror. Without it syncthing would
carry ~2GB (`.zig-cache` alone is 1.9GB) against 208K of actual data.
## Limitations
- **The series is derived, not observed**, except at the nine anchors. Every row
carries `source::anchor` or `source::reconstructed` so no consumer can confuse
the two.
- **Between Friday's publish and Monday's, the newest bar is an estimate.** It
carries `source::projected` in `data/observed.srf`, `assemble` says so in its
output, and the published value replaces it when it arrives. The zfin cache
itself has no source column, so a consumer reading only the cache cannot tell a
projected bar from a published one -- check the log or `assemble`'s report if
that distinction matters to you.
- **It cannot be regenerated from this repo alone.** `reconstruct` needs about ten
years of dividend-adjusted daily prices for five institutional-class Vanguard
funds, read from zfin's cache -- which lives outside this repo, is not
versioned, and is populated through rate-limited provider APIs. A fresh clone on
a fresh machine cannot rebuild `data/ORCBI.srf` or `data/ORC42.srf`. That is why
they are committed rather than gitignored.
- **It is stable under re-basing, but not under a corporate-action correction.**
zfin re-bases its adjusted series whenever adjustments are restated, rescaling
every historical bar of a fund. The reconstruction consumes only per-day ratios
of adjusted closes, so any rescaling cancels exactly -- there is a test for this
(`output is invariant to rescaling the input price series`), including
independent per-fund factors. What is *not* stable: a provider adding or
restating a dividend changes the single daily ratio spanning its ex-date, and
scale invariance does not help there.
- **ORC42's projected eras are published intent, not fact.** The booklet's
glidepath table is a cross-section of every enrollment-year portfolio as of
2026-07-01, and because all of them follow one glidepath keyed on
years-to-enrollment it maps onto ORC42's future: ORC42 in calendar year `Y`
holds what enrollment-year row `E = 4068 - Y` holds today. `Y=2026` gives
`E=2042`, matching ORC42's fitted current era, so the mapping is confirmed at
the one point where that is possible. Everything past 2027-01-01 is marked
`basis::projected` and carries two approximations: the table is annual while
allocations actually step quarterly, and era boundaries are placed on 1
January. The booklet also reserves the right to "change the asset
allocations ... and change the selection of Underlying Funds", so `verify`
flags any projected era that has become current.
- **ORC42's fitted glidepath boundary is approximate.** `2026-01-01` was chosen
by grid search; every candidate from 2025-10-01 to 2026-04-01 gives rms <=
0.041%, so the result is insensitive to it.
- **ORCBI's pre-2025 weights are an assumption.** That stretch spans a
pre-Vestwell (Sumday-era) administrator change and no primary source for the
fund lineup that far back was located. Cross-validation says it reconstructs
fine, but the weights are unverified there.
- **These are instrument prices, not portfolio history.** The finance repo has
only held these funds since 2026-02-26. Pre-2026 values are the security's
price series and must never be written into `history/*-portfolio.srf`, which
records what was actually owned.
## Files
| Path | Role |
|------------------------------------|---------------------------------------------------------------|
| `data/anchors.srf` | The nine observed values, with provenance. **Irreplaceable.** |
| `data/model.srf` | Fund weights per era, prefilled to 2047. Generated. |
| `data/observed.srf` | Append-only log of values read from the feed. Grows daily. |
| `data/recorded.srf` | Hand-typed values, used only to verify. |
| `data/ORCBI.srf`, `data/ORC42.srf` | Generated. Daily unit values, per-row provenance. |
## Maintenance
**Going forward, no reconstruction is needed.** Append each day's published value
to `data/anchors.srf` as `source::feed` and the derived portion becomes a
progressively smaller fraction of the series. Once anchors are daily the
allocation model stops mattering entirely: pinning between two consecutive
sessions reproduces both endpoints exactly whatever the weights say, which the
test `weights are inert when anchors are one session apart` locks down. The model
then matters only for the historical backfill and for filling any gap where
forward recording lapsed.
**ORC42's glidepath is prefilled to 2047**, annually, from the booklet's forward
table. Nothing needs doing each quarter. When `verify` reports
```
model era 2027-01-01 is `projected` but is now in force ...
```
confirm that era against the booklet's current allocation table, correct it if the
plan changed, and flip it to `basis::fitted`. To regenerate the projected rows
after a booklet revision:
```
curl -sLo .tmp/booklet.html \
https://marcom.vestwell.com/program-description/oregon-college.html
python3 tools/gen_model.py .tmp/booklet.html > data/model.srf
```
`verify` cross-checks the last row against the live feed and fails if the
committed series has gone stale.
## The feed's broken Content-Encoding
The feed responds with `content-encoding: base64`. The body is **not** base64; it
is plain JSON. The header is simply wrong, it is set on every file in that S3
prefix, and it survives every request variant.
Zig 0.16's `std.http.Client` rejects it, correctly per spec, and reports it as
`error.HttpHeadersInvalid`. Worse, a head-*parse* failure never marks the
connection closing, so with keep-alive on a poisoned connection returns to the
pool and teardown can block.
`src/feed.zig` works around it without a hand-rolled HTTP stack: `receiveHead` is
a thin wrapper over two separately-public steps, and only the second fails. So it
takes the raw head bytes, drops the offending line, and hands them to the same
public parser -- keeping TLS, connection handling and body framing in the standard
library.
## Development
```
zig build # build the exe -- run this, see below
zig build test # 87 tests
zig build coverage -Dcoverage-threshold=80 # coverage floor
zlint --deny-warnings --fix <files>
zig fmt <files>
```
Reading the underlying prices requires zfin's cache to be populated for VSMPX,
VTPSX, VBMPX, VIPIX and VTIFX. `zfin quote <SYM>` fills it.
### Depending on zfin
`src/cache_files.zig` imports zfin and uses `zfin.Candle`, `zfin.cache.Store` and
`Store.cacheCandles` rather than re-declaring those types and serializing them
here. Two reasons, both about the destructive path:
- **`provider` is `Store.CandleProvider.external`, a compile-checked enum value**,
not the string `"external"`. A rename upstream is now a build error instead of a
cache file that destroys itself on next read.
- **zfin's own writer produces the bytes.** The directive block, field separators,
type tags, trailing newline and atomic rename are all its code, so the format
cannot drift from what its reader expects.
What zfin does not do is validate the series or report a write failure --
`cacheCandles` returns void and logs. Both gaps are covered: bars are checked
before the call, and afterwards the result is read back through zfin's own
`readCandleMeta`, which proves the bytes parse as well as exist.
The `zfin` library module is lean -- `srf`, `zeit`, `build_info` -- so none of the
CLI/TUI dependency tree comes with it.
This checks the producing side only. An *older* zfin reading the cache is still a
hazard, and the installed client and the deployed ZFIN_SERVER are upgraded
separately from this repo -- see "Deployment order matters" above.
### Writing SRF
Every file this project writes goes through `srf.fmt` from the `srf` library, not
through format strings. That is a correctness matter, not style: SRF
length-prefixes a string value containing a comma (`key:24:some, value`) because a
bare comma reads as a field separator. Hand-formatted records silently lost fields
whenever a value contained one -- which is how a comment written into an `evidence`
field first broke `anchors.srf`.
The library also owns the `#!srfv1` line, the `#!expires=`/`#!created=`
directives, the `:num:` type tags and the per-record trailing newline. For
`candles_meta.srf` those are exactly the things whose malformation makes zfin
treat the file as a cache miss and overwrite `candles_daily.srf`, so they are
much better handled by the library than by this project.
`srf.FormatOptions` has no precision control and Zig's `{d}` is
shortest-round-trip, which would spend 17 significant digits on a reconstructed
value. `src/srf_num.zig` solves that with SRF's own extension point: a one-field
wrapper whose `srfFormat` method renders fixed decimals. It is the only place in
the project that writes SRF field syntax by hand, and it does so because that is
the interface the library asks for.
`tools/gen_model.py` is the exception -- Python has no binding for the library --
so instead of emulating the escaping it refuses to emit any value that would need
it.
### Coverage, and what is deliberately not covered
Currently 88.84%. The split is not uniform, on purpose:
| Module | Coverage |
|--------------------------------------------------------------------|----------|
| recon, verify, data, series, assemble, cache_files, srf_num, civil | 100% |
| candles (`parse` tested, the file read is not) | 95% |
| feed (both parsers tested, `fetchBody` is not) | 82% |
| main (CLI wiring; `assemble` covered end to end) | 38% |
Every line of *logic* is tested. What is not tested is mostly the I/O boundary:
file reads, the one HTTP call, and CLI plumbing. That end of the program is also
verified by running it against real data and checking `verify` exits zero.
`cmdAssemble` is the exception and does have an integration test, against a
temp-directory fixture. It is the one function that writes into a cache nothing
else can rebuild, and the test reads the result back with `zfin.cache.Store`
rather than with our own parser -- proving the bytes parse, not merely that they
exist.
`src/main.zig` calls `std.testing.refAllDecls` so that `main` and the command
functions compile in the test binary. Without it they are dead-code eliminated,
and `zig build test` can pass while `zig build` fails on a type error inside
`main` -- which happened during development. Keeping it costs about 12 points of
coverage but makes the untested surface visible in the number rather than hiding
it. **Always run `zig build`, not just `zig build test`.** The pre-commit hook
runs both.
A similar harness for `cmdReconstruct`, `cmdVerify` and `cmdRecord` would close
most of the remaining gap and is a reasonable follow-up.
Reference in a new issue View git blame Copy permalink