zfin-vestwell/README.md

# zfin-vestwell

Daily unit-value history for the two Oregon College Savings Plan (Embark)
portfolios held in the finance repo: **ORCBI** and **ORC42**.

```
zfin-vestwell reconstruct        # rebuild data/ORCBI.srf and data/ORC42.srf
zfin-vestwell verify             # check them against every evidence source
zfin-vestwell record             # append today's published value to the log
zfin-vestwell assemble --out DIR # write zfin cache files for it to serve
```

## The problem

`ORCBI` and `ORC42` are not securities in the ordinary sense. They are
**unitized fund-of-funds trust portfolios** inside a 529 plan: no ticker, no
CUSIP, no market-data provider coverage. Tiingo, Yahoo and TwelveData all 404
them.

The plan publishes a daily unit value, but only the **current** one. There is no
historical endpoint, no downloadable series, and no third-party archive of the
numbers. So history has to be reconstructed.

| Property               | ORCBI          | ORC42                        |
|------------------------|----------------|------------------------------|
| Plan name              | Balanced Index | College Enrollment Year 2042 |
| Kind                   | static         | glidepath, steps quarterly   |
| Feed fund id           | `T6GF10080002` | `T6GFRD204202`               |
| Inception              | 2018-09-01     | 2023-07-01                   |
| Sessions reconstructed | 2006           | 792                          |

## Method

### 1. Anchors: the only observed values

Nine exact values per portfolio, in `data/anchors.srf`.

**Launch.** Every portfolio in the plan launched at exactly `$10.00`. Verified by
back-solving the feed's own `inception` field, which is the *annualized*
since-inception return: `nav / (1 + inception)^years` lands between 10.0005 and
10.0038 for all 37 priced portfolios. The Plan Disclosure Booklet agrees on the
date -- "The Portfolios commenced operations in September 2018."

**Seven Internet Archive captures** of the live feed, spanning 2025-02-13 to
2026-07-30, each carrying full 6-decimal values.

**The current feed value.**

> Keep `data/anchors.srf`. The wayback rows are not recoverable if the Internet
> Archive prunes those captures, and nothing else publishes these numbers.

### 2. Interpolation between anchors

Within each gap, walk the trading-day calendar compounding the weighted
**total return** of the underlying Vanguard funds, read from zfin's candle cache
(`adj_close`). Weights come from `data/model.srf`, per era, so a glidepath step
mid-gap needs no special handling.

Total return, not raw NAV, because the plan's own booklet says these portfolios
"reflect changes in value from income and gains and losses on the sale of the
Underlying Funds **solely by increasing or decreasing their Unit Value**." All
income compounds into the unit value; nothing is distributed. The underlying
mutual funds *do* distribute, so their raw NAV drops on every ex-date. A raw-NAV
basket under-tracks by roughly the blended yield -- measured at +1.5% to -1.6%
drift over four months on ORCBI. That is why `ticker::` + `price_ratio::` is the
wrong tool for these symbols.

### 3. Pinning

The walk lands near but not exactly on the closing anchor. Take the ratio and
distribute it geometrically across the gap, so both endpoints are exact.

**This step matters more than the model.** No fee term appears anywhere in this
project, and none is needed: the asset-based fee, cash drag, trade-date lag and
securities-lending income are all roughly constant-rate effects, and a
constant-rate misfit is exactly what geometric distribution removes. An unpinned
walk across ORCBI's 6.5-year pre-2025 gap is off by -1.29%; pinned, that same gap
predicts a held-out interior anchor to **+0.013%**.

## Accuracy

`verify` measures it two independent ways.

**Leave-one-out cross-validation.** Drop an interior anchor, pin its neighbours,
predict it. This is the number that matters, because it measures error in the
*middle* of an un-anchored gap.

| Fund  | Worst interior error |
|-------|----------------------|
| ORCBI |               0.079% |
| ORC42 |               0.060% |

**Against hand-typed values.** 16 values per portfolio, recovered from the
finance repo's snapshots and **never fed into the model**, so agreement is
evidence rather than a fit. Compared at lag=1, since the portal publishes the
prior session's value.

| Fund  |   n |    mean |  stdev | max            |
|-------|-----|---------|--------|----------------|
| ORCBI |  16 | +0.0075 | 0.0089 | 0.0287 (0.15%) |
| ORC42 |  16 | +0.0035 | 0.0075 | 0.0312 (0.19%) |

Those values are rounded to 2dp at source, which is already +/-0.005 of noise.

**Implied drag** is reported for information, not enforced. It should sit near
the plan's published cost table (0.234%/yr for ORCBI, 0.239% for ORC42); a fee
is not identifiable over a short gap, so per-gap figures are noisy by nature.

## Serving it to zfin

`ORC42` and `ORCBI` have no provider coverage, so zfin can only see them if
something puts files in its candle cache. That is what `assemble` does.

```
zfin-vestwell record                    # feed -> data/observed.srf (idempotent)
zfin-vestwell assemble --out "$CACHE"   # series + log + feed -> cache files
```

`assemble` writes `<out>/<SYMBOL>/candles_daily.srf` and `candles_meta.srf`, and
needs no zfin cache of its own -- only the committed series and the observation
log. Run it on whatever host owns the cache zfin reads. On ZFIN_SERVER that makes
the symbols available to every client, because a read never triggers a refetch:
the server serves a present file as-is.

Verified against a real zfin:

```
$ zfin diagnose ORC42
local      newest 2026-08-27, TTL still in the future, external
adj basis  1970-01-01 - no dividends or splits cached, nothing to restate
tracked    NO - no normal run fetches this symbol
server     the server will never refresh this symbol on its own
provider   tiingo FAILED: NotFound
```

Every line there is load-bearing. The cache is fresh and tagged `external`; the
adjustment-restatement path cannot fire because there are no corporate actions;
the symbol is untracked so no refresh sweep touches it; and no provider carries it,
which is precisely why the cache has to be populated this way.

### Deployment order matters

`assemble` writes `provider::external`. A zfin that does not know that enum
variant treats the meta file as a cache **miss**, takes the cold-start path, gets
a unanimous 404, and writes a negative-cache marker **over
`candles_daily.srf`** -- the history is then gone.

So deploy a zfin carrying the variant to both the client and ZFIN_SERVER *before*
populating a cache they read. That is an operational precondition, not something
this program checks: the enum is verified at compile time against the zfin this
links against, which says nothing about the zfin on the consuming side.

There is deliberately no default for `--out`, so nothing is written anywhere by
accident.


### Steady state

`record` is idempotent per navDate, so a daily job is safe -- the feed republishes
the same value all weekend. Suggested cron/Cronicle shape:

```
zfin-vestwell record && zfin-vestwell assemble --out "$ZFIN_CACHE_DIR"
```

`assemble` also fetches the feed itself, so the cache is current even on the run
where `record` had nothing new to add. Observed values always beat reconstructed
ones for the same date, and `assemble` reports any date where they disagreed by
more than 0.1% -- a sustained count there means the model has drifted.

## Limitations

- **The series is derived, not observed**, except at the nine anchors. Every row
  carries `source::anchor` or `source::reconstructed` so no consumer can confuse
  the two.
- **It cannot be regenerated from this repo alone.** `reconstruct` needs about ten
  years of dividend-adjusted daily prices for five institutional-class Vanguard
  funds, read from zfin's cache -- which lives outside this repo, is not
  versioned, and is populated through rate-limited provider APIs. A fresh clone on
  a fresh machine cannot rebuild `data/ORCBI.srf` or `data/ORC42.srf`. That is why
  they are committed rather than gitignored.
- **It is stable under re-basing, but not under a corporate-action correction.**
  zfin re-bases its adjusted series whenever adjustments are restated, rescaling
  every historical bar of a fund. The reconstruction consumes only per-day ratios
  of adjusted closes, so any rescaling cancels exactly -- there is a test for this
  (`output is invariant to rescaling the input price series`), including
  independent per-fund factors. What is *not* stable: a provider adding or
  restating a dividend changes the single daily ratio spanning its ex-date, and
  scale invariance does not help there.
- **ORC42's projected eras are published intent, not fact.** The booklet's
  glidepath table is a cross-section of every enrollment-year portfolio as of
  2026-07-01, and because all of them follow one glidepath keyed on
  years-to-enrollment it maps onto ORC42's future: ORC42 in calendar year `Y`
  holds what enrollment-year row `E = 4068 - Y` holds today. `Y=2026` gives
  `E=2042`, matching ORC42's fitted current era, so the mapping is confirmed at
  the one point where that is possible. Everything past 2027-01-01 is marked
  `basis::projected` and carries two approximations: the table is annual while
  allocations actually step quarterly, and era boundaries are placed on 1
  January. The booklet also reserves the right to "change the asset
  allocations ... and change the selection of Underlying Funds", so `verify`
  flags any projected era that has become current.
- **ORC42's fitted glidepath boundary is approximate.** `2026-01-01` was chosen
  by grid search; every candidate from 2025-10-01 to 2026-04-01 gives rms <=
  0.041%, so the result is insensitive to it.
- **ORCBI's pre-2025 weights are an assumption.** That stretch spans a
  pre-Vestwell (Sumday-era) administrator change and no primary source for the
  fund lineup that far back was located. Cross-validation says it reconstructs
  fine, but the weights are unverified there.
- **These are instrument prices, not portfolio history.** The finance repo has
  only held these funds since 2026-02-26. Pre-2026 values are the security's
  price series and must never be written into `history/*-portfolio.srf`, which
  records what was actually owned.

## Files

| Path                               | Role                                                          |
|------------------------------------|---------------------------------------------------------------|
| `data/anchors.srf`                 | The nine observed values, with provenance. **Irreplaceable.** |
| `data/model.srf`                   | Fund weights per era, prefilled to 2047. Generated.           |
| `data/observed.srf`                | Append-only log of values read from the feed. Grows daily.    |
| `data/recorded.srf`                | Hand-typed values, used only to verify.                       |
| `data/ORCBI.srf`, `data/ORC42.srf` | Generated. Daily unit values, per-row provenance.             |

## Maintenance

**Going forward, no reconstruction is needed.** Append each day's published value
to `data/anchors.srf` as `source::feed` and the derived portion becomes a
progressively smaller fraction of the series. Once anchors are daily the
allocation model stops mattering entirely: pinning between two consecutive
sessions reproduces both endpoints exactly whatever the weights say, which the
test `weights are inert when anchors are one session apart` locks down. The model
then matters only for the historical backfill and for filling any gap where
forward recording lapsed.

**ORC42's glidepath is prefilled to 2047**, annually, from the booklet's forward
table. Nothing needs doing each quarter. When `verify` reports

```
model era 2027-01-01 is `projected` but is now in force ...
```

confirm that era against the booklet's current allocation table, correct it if the
plan changed, and flip it to `basis::fitted`. To regenerate the projected rows
after a booklet revision:

```
curl -sLo .tmp/booklet.html \
    https://marcom.vestwell.com/program-description/oregon-college.html
python3 tools/gen_model.py .tmp/booklet.html > data/model.srf
```

`verify` cross-checks the last row against the live feed and fails if the
committed series has gone stale.

## The feed's broken Content-Encoding

The feed responds with `content-encoding: base64`. The body is **not** base64; it
is plain JSON. The header is simply wrong, it is set on every file in that S3
prefix, and it survives every request variant.

Zig 0.16's `std.http.Client` rejects it, correctly per spec, and reports it as
`error.HttpHeadersInvalid`. Worse, a head-*parse* failure never marks the
connection closing, so with keep-alive on a poisoned connection returns to the
pool and teardown can block.

`src/feed.zig` works around it without a hand-rolled HTTP stack: `receiveHead` is
a thin wrapper over two separately-public steps, and only the second fails. So it
takes the raw head bytes, drops the offending line, and hands them to the same
public parser -- keeping TLS, connection handling and body framing in the standard
library.

## Development

```
zig build                                   # build the exe -- run this, see below
zig build test                              # 87 tests
zig build coverage -Dcoverage-threshold=80  # coverage floor
zlint --deny-warnings --fix <files>
zig fmt <files>
```

Reading the underlying prices requires zfin's cache to be populated for VSMPX,
VTPSX, VBMPX, VIPIX and VTIFX. `zfin quote <SYM>` fills it.

### Depending on zfin

`src/cache_files.zig` imports zfin and uses `zfin.Candle`, `zfin.cache.Store` and
`Store.cacheCandles` rather than re-declaring those types and serializing them
here. Two reasons, both about the destructive path:

- **`provider` is `Store.CandleProvider.external`, a compile-checked enum value**,
  not the string `"external"`. A rename upstream is now a build error instead of a
  cache file that destroys itself on next read.
- **zfin's own writer produces the bytes.** The directive block, field separators,
  type tags, trailing newline and atomic rename are all its code, so the format
  cannot drift from what its reader expects.

What zfin does not do is validate the series or report a write failure --
`cacheCandles` returns void and logs. Both gaps are covered: bars are checked
before the call, and afterwards the result is read back through zfin's own
`readCandleMeta`, which proves the bytes parse as well as exist.

The `zfin` library module is lean -- `srf`, `zeit`, `build_info` -- so none of the
CLI/TUI dependency tree comes with it.

This checks the producing side only. An *older* zfin reading the cache is still a
hazard, and the installed client and the deployed ZFIN_SERVER are upgraded
separately from this repo -- see "Deployment order matters" above.

### Writing SRF

Every file this project writes goes through `srf.fmt` from the `srf` library, not
through format strings. That is a correctness matter, not style: SRF
length-prefixes a string value containing a comma (`key:24:some, value`) because a
bare comma reads as a field separator. Hand-formatted records silently lost fields
whenever a value contained one -- which is how a comment written into an `evidence`
field first broke `anchors.srf`.

The library also owns the `#!srfv1` line, the `#!expires=`/`#!created=`
directives, the `:num:` type tags and the per-record trailing newline. For
`candles_meta.srf` those are exactly the things whose malformation makes zfin
treat the file as a cache miss and overwrite `candles_daily.srf`, so they are
much better handled by the library than by this project.

`srf.FormatOptions` has no precision control and Zig's `{d}` is
shortest-round-trip, which would spend 17 significant digits on a reconstructed
value. `src/srf_num.zig` solves that with SRF's own extension point: a one-field
wrapper whose `srfFormat` method renders fixed decimals. It is the only place in
the project that writes SRF field syntax by hand, and it does so because that is
the interface the library asks for.

`tools/gen_model.py` is the exception -- Python has no binding for the library --
so instead of emulating the escaping it refuses to emit any value that would need
it.

### Coverage, and what is deliberately not covered

Currently 88.84%. The split is not uniform, on purpose:

| Module                                                             | Coverage |
|--------------------------------------------------------------------|----------|
| recon, verify, data, series, assemble, cache_files, srf_num, civil |     100% |
| candles (`parse` tested, the file read is not)                     |      95% |
| feed (both parsers tested, `fetchBody` is not)                     |      82% |
| main (CLI wiring; `assemble` covered end to end)                   |      38% |

Every line of *logic* is tested. What is not tested is mostly the I/O boundary:
file reads, the one HTTP call, and CLI plumbing. That end of the program is also
verified by running it against real data and checking `verify` exits zero.

`cmdAssemble` is the exception and does have an integration test, against a
temp-directory fixture. It is the one function that writes into a cache nothing
else can rebuild, and the test reads the result back with `zfin.cache.Store`
rather than with our own parser -- proving the bytes parse, not merely that they
exist.

`src/main.zig` calls `std.testing.refAllDecls` so that `main` and the command
functions compile in the test binary. Without it they are dead-code eliminated,
and `zig build test` can pass while `zig build` fails on a type error inside
`main` -- which happened during development. Keeping it costs about 12 points of
coverage but makes the untested surface visible in the number rather than hiding
it. **Always run `zig build`, not just `zig build test`.** The pre-commit hook
runs both.

A similar harness for `cmdReconstruct`, `cmdVerify` and `cmdRecord` would close
most of the remaining gap and is a reasonable follow-up.