zfin-vestwell/README.md

# zfin-vestwell

Daily unit-value history for the two Oregon College Savings Plan (Embark)
portfolios held in the finance repo: **ORCBI** and **ORC42**.

```
zfin-vestwell reconstruct   # rebuild data/ORCBI.srf and data/ORC42.srf
zfin-vestwell verify        # check them against every evidence source
```

## The problem

`ORCBI` and `ORC42` are not securities in the ordinary sense. They are
**unitized fund-of-funds trust portfolios** inside a 529 plan: no ticker, no
CUSIP, no market-data provider coverage. Tiingo, Yahoo and TwelveData all 404
them.

The plan publishes a daily unit value, but only the **current** one. There is no
historical endpoint, no downloadable series, and no third-party archive of the
numbers. So history has to be reconstructed.

| Property               | ORCBI          | ORC42                        |
|------------------------|----------------|------------------------------|
| Plan name              | Balanced Index | College Enrollment Year 2042 |
| Kind                   | static         | glidepath, steps quarterly   |
| Feed fund id           | `T6GF10080002` | `T6GFRD204202`               |
| Inception              | 2018-09-01     | 2023-07-01                   |
| Sessions reconstructed | 2006           | 792                          |

## Method

### 1. Anchors: the only observed values

Nine exact values per portfolio, in `data/anchors.srf`.

**Launch.** Every portfolio in the plan launched at exactly `$10.00`. Verified by
back-solving the feed's own `inception` field, which is the *annualized*
since-inception return: `nav / (1 + inception)^years` lands between 10.0005 and
10.0038 for all 37 priced portfolios. The Plan Disclosure Booklet agrees on the
date -- "The Portfolios commenced operations in September 2018."

**Seven Internet Archive captures** of the live feed, spanning 2025-02-13 to
2026-07-30, each carrying full 6-decimal values.

**The current feed value.**

> Keep `data/anchors.srf`. The wayback rows are not recoverable if the Internet
> Archive prunes those captures, and nothing else publishes these numbers.

### 2. Interpolation between anchors

Within each gap, walk the trading-day calendar compounding the weighted
**total return** of the underlying Vanguard funds, read from zfin's candle cache
(`adj_close`). Weights come from `data/model.srf`, per era, so a glidepath step
mid-gap needs no special handling.

Total return, not raw NAV, because the plan's own booklet says these portfolios
"reflect changes in value from income and gains and losses on the sale of the
Underlying Funds **solely by increasing or decreasing their Unit Value**." All
income compounds into the unit value; nothing is distributed. The underlying
mutual funds *do* distribute, so their raw NAV drops on every ex-date. A raw-NAV
basket under-tracks by roughly the blended yield -- measured at +1.5% to -1.6%
drift over four months on ORCBI. That is why `ticker::` + `price_ratio::` is the
wrong tool for these symbols.

### 3. Pinning

The walk lands near but not exactly on the closing anchor. Take the ratio and
distribute it geometrically across the gap, so both endpoints are exact.

**This step matters more than the model.** No fee term appears anywhere in this
project, and none is needed: the asset-based fee, cash drag, trade-date lag and
securities-lending income are all roughly constant-rate effects, and a
constant-rate misfit is exactly what geometric distribution removes. An unpinned
walk across ORCBI's 6.5-year pre-2025 gap is off by -1.29%; pinned, that same gap
predicts a held-out interior anchor to **+0.013%**.

## Accuracy

`verify` measures it two independent ways.

**Leave-one-out cross-validation.** Drop an interior anchor, pin its neighbours,
predict it. This is the number that matters, because it measures error in the
*middle* of an un-anchored gap.

| Fund  | Worst interior error |
|-------|----------------------|
| ORCBI |               0.079% |
| ORC42 |               0.060% |

**Against hand-typed values.** 16 values per portfolio, recovered from the
finance repo's snapshots and **never fed into the model**, so agreement is
evidence rather than a fit. Compared at lag=1, since the portal publishes the
prior session's value.

| Fund  |   n |    mean |  stdev | max            |
|-------|-----|---------|--------|----------------|
| ORCBI |  16 | +0.0075 | 0.0089 | 0.0287 (0.15%) |
| ORC42 |  16 | +0.0035 | 0.0075 | 0.0312 (0.19%) |

Those values are rounded to 2dp at source, which is already +/-0.005 of noise.

**Implied drag** is reported for information, not enforced. It should sit near
the plan's published cost table (0.234%/yr for ORCBI, 0.239% for ORC42); a fee
is not identifiable over a short gap, so per-gap figures are noisy by nature.

## Limitations

- **The series is derived, not observed**, except at the nine anchors. Every row
  carries `source::anchor` or `source::reconstructed` so no consumer can confuse
  the two.
- **It cannot be regenerated from this repo alone.** `reconstruct` needs about ten
  years of dividend-adjusted daily prices for five institutional-class Vanguard
  funds, read from zfin's cache -- which lives outside this repo, is not
  versioned, and is populated through rate-limited provider APIs. A fresh clone on
  a fresh machine cannot rebuild `data/ORCBI.srf` or `data/ORC42.srf`. That is why
  they are committed rather than gitignored.
- **It is stable under re-basing, but not under a corporate-action correction.**
  zfin re-bases its adjusted series whenever adjustments are restated, rescaling
  every historical bar of a fund. The reconstruction consumes only per-day ratios
  of adjusted closes, so any rescaling cancels exactly -- there is a test for this
  (`output is invariant to rescaling the input price series`), including
  independent per-fund factors. What is *not* stable: a provider adding or
  restating a dividend changes the single daily ratio spanning its ex-date, and
  scale invariance does not help there.
- **ORC42's projected eras are published intent, not fact.** The booklet's
  glidepath table is a cross-section of every enrollment-year portfolio as of
  2026-07-01, and because all of them follow one glidepath keyed on
  years-to-enrollment it maps onto ORC42's future: ORC42 in calendar year `Y`
  holds what enrollment-year row `E = 4068 - Y` holds today. `Y=2026` gives
  `E=2042`, matching ORC42's fitted current era, so the mapping is confirmed at
  the one point where that is possible. Everything past 2027-01-01 is marked
  `basis::projected` and carries two approximations: the table is annual while
  allocations actually step quarterly, and era boundaries are placed on 1
  January. The booklet also reserves the right to "change the asset
  allocations ... and change the selection of Underlying Funds", so `verify`
  flags any projected era that has become current.
- **ORC42's fitted glidepath boundary is approximate.** `2026-01-01` was chosen
  by grid search; every candidate from 2025-10-01 to 2026-04-01 gives rms <=
  0.041%, so the result is insensitive to it.
- **ORCBI's pre-2025 weights are an assumption.** That stretch spans a
  pre-Vestwell (Sumday-era) administrator change and no primary source for the
  fund lineup that far back was located. Cross-validation says it reconstructs
  fine, but the weights are unverified there.
- **These are instrument prices, not portfolio history.** The finance repo has
  only held these funds since 2026-02-26. Pre-2026 values are the security's
  price series and must never be written into `history/*-portfolio.srf`, which
  records what was actually owned.

## Files

| Path                               | Role                                                          |
|------------------------------------|---------------------------------------------------------------|
| `data/anchors.srf`                 | The nine observed values, with provenance. **Irreplaceable.** |
| `data/model.srf`                   | Fund weights per era, prefilled to 2047. Generated.           |
| `data/recorded.srf`                | Hand-typed values, used only to verify.                       |
| `data/ORCBI.srf`, `data/ORC42.srf` | Generated. Daily unit values, per-row provenance.             |

## Maintenance

**Going forward, no reconstruction is needed.** Append each day's published value
to `data/anchors.srf` as `source::feed` and the derived portion becomes a
progressively smaller fraction of the series. Once anchors are daily the
allocation model stops mattering entirely: pinning between two consecutive
sessions reproduces both endpoints exactly whatever the weights say, which the
test `weights are inert when anchors are one session apart` locks down. The model
then matters only for the historical backfill and for filling any gap where
forward recording lapsed.

**ORC42's glidepath is prefilled to 2047**, annually, from the booklet's forward
table. Nothing needs doing each quarter. When `verify` reports

```
model era 2027-01-01 is `projected` but is now in force ...
```

confirm that era against the booklet's current allocation table, correct it if the
plan changed, and flip it to `basis::fitted`. To regenerate the projected rows
after a booklet revision:

```
curl -sLo .tmp/booklet.html \
    https://marcom.vestwell.com/program-description/oregon-college.html
python3 tools/gen_model.py .tmp/booklet.html > data/model.srf
```

`verify` cross-checks the last row against the live feed and fails if the
committed series has gone stale.

## The feed's broken Content-Encoding

The feed responds with `content-encoding: base64`. The body is **not** base64; it
is plain JSON. The header is simply wrong, it is set on every file in that S3
prefix, and it survives every request variant.

Zig 0.16's `std.http.Client` rejects it, correctly per spec, and reports it as
`error.HttpHeadersInvalid`. Worse, a head-*parse* failure never marks the
connection closing, so with keep-alive on a poisoned connection returns to the
pool and teardown can block.

`src/feed.zig` works around it without a hand-rolled HTTP stack: `receiveHead` is
a thin wrapper over two separately-public steps, and only the second fails. So it
takes the raw head bytes, drops the offending line, and hands them to the same
public parser -- keeping TLS, connection handling and body framing in the standard
library.

## Development

```
zig build                                   # build the exe -- run this, see below
zig build test                              # 55 tests
zig build coverage -Dcoverage-threshold=80  # coverage floor
zlint --deny-warnings --fix <files>
zig fmt <files>
```

Reading the underlying prices requires zfin's cache to be populated for VSMPX,
VTPSX, VBMPX, VIPIX and VTIFX. `zfin quote <SYM>` fills it.

### Coverage, and what is deliberately not covered

Currently 87.86%. The split is not uniform, on purpose:

| Module                                         | Coverage |
|------------------------------------------------|----------|
| recon, verify, data, series, civil             |     100% |
| candles (`parse` tested, the file read is not) |      95% |
| feed (both parsers tested, `fetchBody` is not) |      82% |
| main (CLI wiring and file I/O)                 |      26% |

Every line of *logic* is tested. What is not tested is the I/O boundary: reading
files, the one HTTP call, and CLI plumbing. That end of the program is verified by
running it against real data and checking `verify` exits zero, which is stronger
evidence than a fixture test would give.

`src/main.zig` calls `std.testing.refAllDecls` so that `main` and the command
functions compile in the test binary. Without it they are dead-code eliminated,
and `zig build test` can pass while `zig build` fails on a type error inside
`main` -- which happened during development. Keeping it costs about 12 points of
coverage but makes the untested surface visible in the number rather than hiding
it. **Always run `zig build`, not just `zig build test`.** The pre-commit hook
runs both.

A temp-directory integration harness for `cmdReconstruct` and `cmdVerify
--offline` would close most of the remaining gap and is a reasonable follow-up.