zfin/docs/explanation/snapshots-model.md

# The snapshot model

A snapshot is the file [`zfin snapshot`](../reference/cli/snapshot.md)
writes to `history/<date>-portfolio.srf`, and the thing every
history- and comparison-oriented command reads back. This page explains
what a snapshot is *for*, which is also an explanation of what it
deliberately leaves out. For the day-to-day workflow, see
[Snapshots and history](../guides/snapshots-and-history.md).

## A snapshot is a valuation record

A snapshot answers exactly one question:

> What was this portfolio worth on date D, decomposed by symbol,
> account, and tax type?

That is the whole contract. A snapshot is **not** a faithful record of
your lots. It records enough per-lot detail to *decompose a valuation*,
and stops there.

This is a deliberate choice rather than an oversight, and the rest of
this page is the reasoning -- partly so the decision is auditable, and
partly so the same field-by-field extension proposals don't have to be
re-litigated every time someone notices something missing.

## Two layers, and which one wins

A snapshot file has two layers that are computed independently, and it
matters which you read:

1. **The totals layer** -- the `total`, `tax_type`, and `account` rows.
   This is the **authoritative** valuation. It comes from
   `valuation.portfolioSummary`, the same code path that produces the
   headline numbers in `zfin portfolio`, and it includes
   portfolio-level adjustments.
2. **The lot layer** -- the `lot` rows. This is a **decomposition
   aid**. It comes from a separate, direct walk over the portfolio's
   lots, and it deliberately carries no portfolio-level adjustments.

When the two disagree, **the totals layer wins**. See "Known
shortcoming" below for the one case where they measurably do.

## Why not a faithful lot record?

Because zfin already has one, and it is better at the job.

The git history of your `portfolio.srf` *is* the faithful lot record.
It has complete fidelity (every field of every lot, not a projection of
some of them), and it has true temporal resolution (one revision per
edit, rather than one file per day).
[`zfin contributions`](../reference/cli/contributions.md) is built
directly on it: it walks git revisions, deserializes complete lots, and
matches them across revisions by a composite key.

Making snapshots into a *second* faithful lot record would create two
records of lot identity with different cadences, different
completeness, and different provenance -- and therefore an open-ended
obligation to reconcile them whenever they disagree. Nothing currently
asks for that.

There is also a harder lesson embedded in the contributions code. Even
with the *complete* lot available, matching lots across time is
unreliable enough that it needs a fuzzy fallback key alongside the
strict one, because ordinary bookkeeping breaks strict identity:
certificate-of-deposit auto-renewals rewrite the open date, account
renames change the account, and reconciliation tweaks move the open
price. Cross-time lot matching is a genuinely hard problem that the
git-revision pipeline solves with real machinery. A daily valuation
file is not going to solve it as a side effect of carrying one more
column.

## Proposals that were considered and rejected

If you are here because a field you want is missing, check this list
first.

### `open_date`

The strongest-looking candidate, because without it a snapshot's lots
have no temporal identity at all, so they cannot be matched to lots in
any other snapshot.

Rejected, because it doesn't actually deliver that. `open_date` is one
component of the strict lot key; a snapshot carrying it would still
lack everything the fuzzy fallback needs when the strict key breaks --
which, per above, is routine. The result would *look* like it enabled
cross-snapshot lot matching while quietly not doing so, which is worse
than the honest absence. If cross-snapshot lot keying is genuinely
needed, the answer is to extend the git-revision pipeline.

There is one non-identity argument for `open_date`: holding-period
analysis, such as decomposing unrealized gains into short- and
long-term at a past date. `cost_basis` is already emitted, so
`open_date` is the only missing input. This is a legitimately
valuation-shaped use, and it is the argument to make if the field is
ever wanted -- but it is currently hypothetical. zfin has no
holding-period logic anywhere, and tax-loss harvesting is deliberately
hand-declared in `accounts.srf` rather than computed from lots. Adding
a field for a consumer that does not exist is how formats rot.

### `split_factor`

A stock lot's `shares` is written raw (as transacted) while its `value`
is split-adjusted, so the two are not related by the obvious
`shares * price` identity. It is tempting to emit the split factor to
close that gap.

Rejected, because it carries no information. All three quantities are
already recoverable by algebra from what *is* emitted:

```
raw shares       == cost_basis / open_price
effective shares == value / price
split_factor     == value / (shares * price)
```

`compare.aggregateSnapshotStocks` uses the second of these and
documents it. Emitting the split factor as well would introduce a
*second* route to effective shares within a single record: identical
whenever the writer is correct, divergent exactly when it is buggy, and
with no rule for which one a reader should trust. That is the same
two-sources-of-truth problem as the lot-record proposal, in miniature.

The "it would let us validate the invariant" argument does not survive
either: a check comparing the writer's output against the writer's own
inputs, assigned in the same function moments earlier, cannot detect a
wrong split factor, a wrong price, or a wrong share count. It can only
catch a typo in a single expression, which is a unit test's job.

### `drip`

Rejected as a category error. `zfin contributions` classifies dividend
reinvestment by reading the `drip` flag from git revisions of
`portfolio.srf`; it never reads snapshots at all. Adding `drip` here
would not affect that classification, or anything else.

## Known shortcoming: the covered-call gap

`sum(stock lot.value)` can slightly **exceed** `total::liquid` in the
same file.

The cause is the two-layer split. When you hold an open, in-the-money
sold call, the totals layer caps the covered underlying's market value
at the option's strike price -- the shares are effectively committed at
that price, so valuing them at the higher market price would overstate
the portfolio. The lot layer applies no such cap; each lot is marked at
plain market value.

Consequences, in ascending order of obscurity:

- `total::liquid` is correct and remains the figure to read.
- Summing lot values yourself will overstate the liquid total whenever
  such a call was open on the snapshot date. The overstatement is
  bounded by `(market - strike) * covered_shares` across affected
  underlyings.
- `history.aggregateSnapshotAllocations` derives per-symbol weights by
  dividing summed lot values by the totals-layer liquid figure, so
  those weights can sum to slightly more than 1 in the same
  circumstance.

This is documented rather than fixed because it is immaterial in
practice: it requires an open ITM sold call on the snapshot date, and
the affected consumer is an allocation weighting whose downstream use
tolerates the error. Note also that it is **not diagnosable from a
snapshot alone** -- deciding whether a given gap is a legitimate
covered-call cap or a writer bug requires the option's `strike`, which
the format does not emit. Emitting `strike` and `multiplier` is
therefore the one field addition with a concrete, valuation-shaped
consumer, should this ever need fixing properly.

## Wire-format compatibility

The format is [SRF](https://git.lerch.org/lobo/srf), and its
compatibility behavior is what makes the format safe to extend:

- **Default-valued fields are elided on write.** A field equal to its
  default does not appear in the file at all.
- **Unknown fields are ignored on read.** A reader skips fields it
  doesn't know about.
- Together these give compatibility in both directions: old readers
  tolerate new files, and new readers tolerate old files.

Two constraints follow for anyone extending a record type:

- **Every field must have a default.** Fields are matched by name and
  absent ones are filled from their default; a field with no default
  makes every previously-written snapshot fail to parse.
- **`kind` must stay first.** It is the union discriminator, and the
  reader requires it as the first field on the line.

### `snapshot_version` and ambiguous absence

Each snapshot carries a `snapshot_version`, currently `1`. Nothing
reads it to gate behavior, so bumping it is normally documentary.

There is one case where it becomes load-bearing. Because default-valued
fields are elided, a new field whose value happens to equal its default
is absent from the file -- and a reader cannot tell that apart from "the
writer predates this field entirely." **Bump `snapshot_version` when,
and only when, a new field's absence is semantically ambiguous.**
Novelty alone is not a reason.

Worked examples: a `drip: bool = false` or an `open_date: ?Date = null`
would each require a bump, because absence could mean either "false /
unknown" or "old writer." A `split_factor: f64 = 1.0` would not, since
`1.0` means "no adjustment" either way.

### Writers and readers are separate deployments

Snapshots are often written by a scheduled job and read by interactive
commands, and those two can be running different builds of zfin for
weeks at a time. So a newly added field appears only in files written
after the *writer's* build is refreshed, and never appears in files
already on disk. `zfin snapshot` never reads existing snapshots, so an
older writer is safe; just don't assume a field's presence based on the
version of the binary you happen to be reading with.