836 lines
39 KiB
Markdown
836 lines
39 KiB
Markdown
# Future Work
|
||
|
||
No work here is blocking — we're in a good state. Items below are
|
||
ordered roughly by priority within each section. Priority labels
|
||
(`HIGH` / `MEDIUM` / `LOW`) mark items that deserve explicit
|
||
ranking; unlabeled items are "someday, if the mood strikes."
|
||
|
||
## Projections: future enhancements
|
||
|
||
- **Configurable return cap per position — priority MEDIUM.**
|
||
Default: none; cap outliers like NVDA. Should route through
|
||
`projections.srf` cleanly.
|
||
- **Accumulation-mode SWR rate column is misleading — priority LOW.**
|
||
When `retirement_age`/`retirement_at` is configured, the "Safe
|
||
Withdrawal" table's % column divides the SWR amount by the
|
||
CURRENT portfolio value, not the post-accumulation portfolio
|
||
value. The dollar amount is correct (it's the safe spending in
|
||
retirement, given the projected accumulation), but the % rate
|
||
comes out absurdly high (e.g., 22% of today's portfolio). The
|
||
Accumulation phase block already shows the median portfolio at
|
||
retirement, so the user can compute the real rate themselves —
|
||
but the SWR table's rate column should ideally divide by the
|
||
median post-accumulation value, or be suppressed when accumulation
|
||
is active. Decide which.
|
||
- **Chart vertical line at retirement boundary — priority LOW.**
|
||
The accumulation-phase spec called this "mandatory" but it was
|
||
explicitly deferred during implementation. The chart currently
|
||
shows the full `accumulation_years + horizon` span without a
|
||
visual marker for where accumulation ends and distribution
|
||
begins. Easier to add to the kitty-graphics chart than the braille
|
||
one.
|
||
- **Goal-seek over distribution horizon for W1 — priority LOW.**
|
||
Today the W1 ("set spending, find date") workflow reports the
|
||
earliest retirement at each user-configured `(horizon, confidence)`
|
||
cell. The philosophically correct version asks "when have I
|
||
accumulated enough wealth that the projection shows a 95%
|
||
probability of success withdrawing X per year from retirement
|
||
until age-of-death?" — i.e. goal-seek across both `accumulation_years`
|
||
AND `distribution_years` simultaneously, anchored to a configured
|
||
age-of-death. NP-shaped search; not worth optimizing until
|
||
someone wants it.
|
||
- **Per-person retirement_age — priority LOW.**
|
||
V1 of the accumulation-phase spec chose Option A: a single
|
||
household retirement boundary derived from the oldest configured
|
||
birthdate. Households where one earner retires significantly
|
||
earlier than the other would benefit from per-person
|
||
`retirement_age` fields on each `type::birthdate` record, with
|
||
contributions stopped per-person.
|
||
- **Configurable max_accumulation_years — priority LOW.**
|
||
Hardcoded at 50 years. Route through `projections.srf` if anyone
|
||
hits the cap.
|
||
- Configurable MIN period selection (currently 3Y/5Y/10Y, exclude 1Y)
|
||
- Multiple spending models: flat (current), decreasing (1-2% real annual decrease,
|
||
Blanchett "spending smile"). Late-life healthcare better modeled as a life event.
|
||
- Unclassified position handling in allocation split (warn user)
|
||
- **Historical projection overlay follow-ups.** The base
|
||
`--overlay-actuals` overlay shipped (CLI tip + TUI primary surface).
|
||
Open enhancements:
|
||
- Historical `metadata.srf` / `projections.srf` for back-dated
|
||
runs. Today the overlay re-runs against current classifications
|
||
and assumptions; for historically faithful what-the-model-said-then
|
||
output we'd check out the git-tracked versions of those files
|
||
at the as-of commit and load those instead. Edge case until
|
||
classifications materially drift.
|
||
- Contribution-attribution overlay. Today's actuals line includes
|
||
contributions implicitly; the bands assume modeled contributions
|
||
that may or may not match reality. A "decompose actuals into
|
||
market return vs contributions" annotation would clarify how
|
||
much of the trajectory was the model being right vs new money
|
||
arriving on schedule.
|
||
- Mosaic mode: overlay multiple as-of starting points on one chart
|
||
("show me 1Y, 3Y, 5Y, 10Y projections all at once") so the user
|
||
can see how the projection envelope tightened as data came in.
|
||
- **Better composition basis for imported-only as-of.** Today
|
||
the imported-only path uses today's allocations scaled by
|
||
`imported_liquid / today_total_liquid`. That's the simplest
|
||
thing that could work, but it's "today's mix back-dated" —
|
||
it ignores everything we know about the historical context.
|
||
Specifically: `imported_values.srf` already carries an
|
||
`expected_return` field per row that the user captured at
|
||
that date in their source spreadsheet. We could:
|
||
- Use the imported `expected_return` as a sanity check
|
||
against the simulation's per-position weighted return
|
||
(warn or clamp if they diverge wildly — the spreadsheet's
|
||
number reflects what the user actually saw at the time).
|
||
- Use the imported `expected_return` to bias the
|
||
stock/bond split inference: a higher expected return
|
||
implies a higher historical equity weighting than today's
|
||
mix probably reflects.
|
||
- Reach further: derive a synthetic stock/bond split from
|
||
the imported `expected_return` directly, treating it as
|
||
a weighted average of SPY and AGG returns at that date
|
||
and solving for the weights. That gives a per-imported-
|
||
row composition that's locally faithful instead of
|
||
one-mix-fits-all.
|
||
None of these are urgent — the current "today's mix scaled"
|
||
approximation is documented as such and the bands still
|
||
render meaningfully — but each would tighten the historical
|
||
faithfulness one notch. Pick whichever has the highest
|
||
payoff vs. complexity when this gets revisited.
|
||
|
||
## `--export-chart` follow-ups — priority LOW
|
||
|
||
V1 of `--export-chart <PATH>` shipped for `quote` and `projections`
|
||
(default bands mode only). Several adjacent surfaces still don't
|
||
have PNG export and were deferred:
|
||
|
||
- **`history --export-chart`.** The `history` command renders a
|
||
single-series braille chart of portfolio value over time
|
||
(synthesized into `Candle` records and fed to
|
||
`format.computeBrailleChart`). It doesn't share the z2d
|
||
pipeline that `quote` (`tui/chart.zig`) and `projections`
|
||
(`tui/projection_chart.zig`) use. To export, options:
|
||
- **A.** Pipe the synthesized candles through
|
||
`tui/chart.zig`'s `renderChart` — but that draws Bollinger
|
||
Bands and an RSI panel, both meaningless on a portfolio-
|
||
value series.
|
||
- **B.** Add a minimal "single-series line chart" z2d
|
||
renderer (a slimmed-down `projection_chart.zig` without
|
||
bands). ~150 lines. Same renderToSurface shape so PNG
|
||
export is trivial after.
|
||
- **C.** Skip it permanently; the braille chart is fine for
|
||
what `history` is. Document as "not exportable".
|
||
B is the right answer if PNG export of the history chart is
|
||
ever requested.
|
||
- **`projections --convergence` / `--return-backtest`.** Both
|
||
render forecast-evaluation charts via `tui/forecast_chart.zig`.
|
||
Not refactored to expose a `renderToSurface` seam yet —
|
||
parser rejects `--export-chart` in those modes today. Low
|
||
effort to add (mirror the `tui/chart.zig` pattern).
|
||
- **`projections --vs <DATE>`.** No chart at all in this mode
|
||
(text-only delta table); `--export-chart` rejected at parse
|
||
time. Could grow a side-by-side bands comparison chart, but
|
||
that's a feature of its own — not just an export plumbing job.
|
||
- **Theme overrides at export time.** Today the export always
|
||
uses `theme.default_theme`. A `--theme <PATH>` flag at export
|
||
time would let users render with their configured theme or a
|
||
presentation-friendly one. Out of scope for V1; gate when
|
||
someone asks for it.
|
||
- **File format alternatives.** SVG / PDF / WebP — `z2d` only
|
||
exports PNG natively today; would need an external dependency
|
||
or a pixel-buffer-to-format conversion.
|
||
|
||
## `zfin doctor` health-check command — priority LOW
|
||
|
||
Front-door command for the file constellation and environment.
|
||
Answers "is my zfin setup sane?" without making any changes.
|
||
|
||
The configuration surface has grown organically and only the README
|
||
explains the file layout:
|
||
|
||
- `portfolio.srf` — lots
|
||
- `metadata.srf` — classifications
|
||
- `accounts.srf` — tax types
|
||
- `projections.srf` — retirement config
|
||
- `transaction_log.srf` — internal transfers
|
||
- `imported_values.srf` — back-history
|
||
- `history/*-portfolio.srf` — snapshots
|
||
- `~/.config/zfin/keys.srf` — keybinds
|
||
- `~/.config/zfin/theme.srf` — colors
|
||
|
||
Plus 5 API keys, a cache directory, and an optional `ZFIN_SERVER`.
|
||
|
||
### v1 scope (full health check)
|
||
|
||
- **File inventory + parse check.** For each of the files above:
|
||
does it exist, where was it resolved from (cwd vs `ZFIN_HOME` vs
|
||
`~/.config/zfin`), and does it parse cleanly. No fixes.
|
||
- **Cross-reference checks.** Every account in `portfolio.srf`
|
||
has an `accounts.srf` entry. Every held symbol has a
|
||
`metadata.srf` entry (or is opted out). Every snapshot file
|
||
parses as a portfolio. `transaction_log.srf` records reference
|
||
real account names.
|
||
- **Environment audit.** Which API keys are set (presence only,
|
||
never the value). Cache size + symbol count. `ZFIN_SERVER`
|
||
reachability if set (HEAD/OPTIONS, low timeout). Staleness of
|
||
hand-maintained data files (T-bill rates, Shiller `ie_data.csv`)
|
||
— same registry as `data/staleness.zig`.
|
||
- **Output shape.** Sectioned, color-coded. Every check is
|
||
`OK` / `WARN` / `FAIL` with a one-line context. Exit code 0 on
|
||
all-OK or warnings only; non-zero only on FAIL. Suitable for
|
||
CI / cron / pre-commit.
|
||
|
||
### Driver
|
||
|
||
The file constellation has grown to nine files in three locations,
|
||
plus five API keys, plus the cache, plus the optional server.
|
||
Today only the README explains the structure. A `doctor` command
|
||
surfaces it, catches regressions, and is the obvious place to point
|
||
new users (or to point future-self after a long break).
|
||
|
||
### Open question for when this is picked up
|
||
|
||
Does this *replace* the portfolio-hygiene portion of `audit`, or
|
||
live alongside it? Probably alongside — `audit` is reconciliation
|
||
against external broker exports, `doctor` is internal-state
|
||
validation. But worth confirming the boundary before implementing
|
||
to avoid duplicated checks.
|
||
|
||
## Split `audit.zig` into per-broker reconcilers — priority LOW
|
||
|
||
`src/commands/audit.zig` is now 2856 lines (was 3438) after the
|
||
brokerage parsers moved to per-broker files under `src/brokerage/`.
|
||
It still bundles three logically distinct responsibilities:
|
||
|
||
- Portfolio hygiene check (no-flag mode)
|
||
- Fidelity positions CSV reconciler (`--fidelity`)
|
||
- Schwab per-account positions CSV reconciler (`--schwab`) and
|
||
Schwab account-summary stdin reconciler (`--schwab-summary`)
|
||
|
||
The brokerage parsers themselves are split per broker:
|
||
`src/brokerage/types.zig` (shared `BrokeragePosition` +
|
||
`parseDollarAmount`), `src/brokerage/fidelity.zig` (Fidelity CSV +
|
||
option-symbol matcher), `src/brokerage/schwab.zig` (per-account
|
||
CSV + summary paste). Adding a new broker is a one-file add next
|
||
to those. What's left is splitting the *reconciler*
|
||
(compare-portfolio-vs-brokerage) and *display* code in audit.zig
|
||
into per-broker files that consume those parsers.
|
||
|
||
### Sketch
|
||
|
||
```
|
||
src/commands/audit/
|
||
mod.zig ← thin dispatcher; current public `run()` lives here
|
||
hygiene.zig ← portfolio hygiene check (no-flag mode)
|
||
fidelity.zig ← --fidelity reconciler (uses brokerage/fidelity.zig)
|
||
schwab.zig ← --schwab + --schwab-summary reconcilers
|
||
common.zig ← shared types (Discrepancy, ReconcileResult), formatters
|
||
```
|
||
|
||
The hygiene check can be referenced from `zfin doctor` (above)
|
||
without pulling in reconciler baggage.
|
||
|
||
### Driver
|
||
|
||
Maintenance friction. The split makes the audit-bug investigations
|
||
already in this TODO file (phantom discrepancy on freshly-added
|
||
lots) easier to localize, and lets a `zfin doctor` command reuse
|
||
hygiene without inheriting the reconciliation surface.
|
||
|
||
Pure internal refactor; no user-visible change.
|
||
|
||
## Refactor: trim `src/format.zig` once Money / Date have absorbed their helpers — priority LOW
|
||
|
||
`src/format.zig` is still a ~1700-line grab-bag, but the money- and
|
||
date-shaped helpers that used to live there have been moved out:
|
||
money formatting now lives in `src/Money.zig` (with `{f}` /
|
||
`whole()` / `trim()` / `signed()` / `padRight(N)` / `padLeft(N)`),
|
||
and date formatting lives in `src/Date.zig` (with `{f}` /
|
||
`padRight(N)` / `padLeft(N)`). What's left in `format.zig` is the
|
||
genuinely-format-domain stuff: braille charts, return formatters,
|
||
allocation notes, signed-percent rendering.
|
||
|
||
If the file ever grows enough to be annoying again, consider
|
||
renaming to `src/render.zig` to better describe what's left, or
|
||
splitting the braille chart out (it's ~600 lines on its own).
|
||
Not blocking — file it as cleanup if and when it bites.
|
||
|
||
### `projections --vs <date>` doesn't support imported-only as-of dates — priority MEDIUM
|
||
|
||
The crash that used to happen when `--vs <date>` resolved to an
|
||
imported-only date is fixed: `loadAsOfContext` now branches on
|
||
`resolution.source` and emits a graceful "no snapshot at that
|
||
date" error instead of panicking with `FileNotFound`. But the
|
||
feature itself is still missing - back-dating a `--vs` comparison
|
||
to a date that's only covered by `imported_values.srf` (no real
|
||
snapshot) is rejected outright.
|
||
|
||
The `runBands` path (`projections --as-of <imported_date>`)
|
||
handles the imported-only case by loading today's portfolio
|
||
composition + scaling to the imported liquid total, then calling
|
||
`view.loadProjectionContextFromImported`. `loadAsOfContext`
|
||
needs the same plumbing - but as outparams, since the caller
|
||
(`computeKeyComparison`) needs to own `live_loaded` and
|
||
`live_pf_data` for the lifetime of the returned context.
|
||
|
||
Two implementation shapes:
|
||
|
||
A. **Add outparams to `loadAsOfContext`.** New
|
||
`live_loaded_out: *?cli.LoadedPortfolio` and
|
||
`live_pf_data_out: *?cli.PortfolioData` parameters. Caller
|
||
declares them and `defer`s their deinit. ~30 lines, but
|
||
duplicates the imported-only loading code (already lives in
|
||
`runBands`'s `else` branch around line 392-429 of
|
||
`src/commands/projections.zig`).
|
||
|
||
B. **Extract a shared helper.** Pull the snapshot-vs-imported
|
||
branching from both `runBands` and `loadAsOfContext` into one
|
||
`loadProjectionContextForResolution` that returns a
|
||
discriminated union (snapshot ctx with snap_bundle owned, or
|
||
imported ctx with live_loaded + live_pf_data owned). Both
|
||
call sites use it. ~60 lines but eliminates the duplication
|
||
that AGENTS.md warns about (the two paths drifting causes
|
||
`compare --projections` to disagree with standalone
|
||
`projections --as-of`).
|
||
|
||
Recommendation: B. The duplication risk is real - the
|
||
`computeKeyComparison` doc-block already calls out that "if you
|
||
change inputs to either of these loaders, change them in BOTH
|
||
places." Adding a third copy of the imported-only loader code
|
||
makes that worse.
|
||
|
||
## Audit: manual-check accounts mechanism — priority HIGH
|
||
|
||
Some accounts/positions can't be reconciled from broker CSVs and need a
|
||
human-in-the-loop reminder at the audit step. Two recurring shapes:
|
||
|
||
- **No-CSV-export accounts** (e.g. some insurance / annuity products)
|
||
where values only live in periodic statements. Git can't detect a
|
||
"change" because nothing changes locally; the user has to log in
|
||
to see the new value.
|
||
- **Payroll-deduction-then-purchase accounts** (e.g. ESPP) where
|
||
payroll-deducted cash doesn't appear in the broker positions CSV
|
||
until the purchase date hits (typically every 6 months). Between
|
||
purchases the cash is a real contribution that `zfin audit` can't
|
||
see.
|
||
|
||
The existing `update_cadence::weekly|monthly|quarterly|none` field already
|
||
sort-of covers this, but has two gaps:
|
||
|
||
1. It fires off the last *git-detected change*, not the last *human
|
||
review*. For statement-only accounts, the value sometimes hasn't
|
||
changed in months — so git never fires, cadence never trips.
|
||
2. Payroll-deduction accounts need weekly-ish attention while
|
||
accumulating cash between purchases, but the accrued balance is
|
||
invisible to the CSV audit.
|
||
|
||
Drift symptom seen in practice: several accounts on
|
||
`update_cadence::weekly` in `accounts.srf` weren't flagged as overdue
|
||
despite no changes in two weeks, because the cadence reads
|
||
git-detected change time rather than human-review time. The cadence
|
||
values themselves may also be wrong for these accounts — revisit
|
||
whether weekly is the right cadence vs. monthly/quarterly given how
|
||
rarely they actually change.
|
||
|
||
### Options
|
||
|
||
A. **New `update_cadence::manual` variant** — always fires every audit
|
||
run until silenced. Blunt but zero design work.
|
||
|
||
B. **`last_refreshed::YYYY-MM-DD` field on `accounts.srf`** — explicit
|
||
human-review timestamp, decoupled from git-detected changes. Audit
|
||
compares `today - last_refreshed` against the cadence. User bumps
|
||
the field when they check the statement. Probably the most
|
||
correct fit for statement-only accounts.
|
||
|
||
C. **Sticky TODO list** — a `todos.srf` or `todo::` field on accounts
|
||
that audit always surfaces until cleared. General-purpose; also
|
||
covers "remember to rebalance on 5/15".
|
||
|
||
### ESPP-style accrual follow-through
|
||
|
||
Payroll-deduction accounts are also a contribution-attribution blind
|
||
spot. If a paystub deducts $X/week but the cash lot doesn't reach
|
||
`portfolio.srf` until the purchase date, the attribution math is
|
||
under-counting contributions and over-counting the purchase-week
|
||
gain. Possible fixes are discussed in the "Contributions diff" TODO
|
||
below — option C there (per-account `cash_is_contribution`) would
|
||
make manually-entered ESPP-style cash additions count correctly.
|
||
|
||
## Investigate: detailed 401(k) contributions data source
|
||
|
||
Found a more detailed contributions screen on at least one
|
||
employer-sponsored 401(k) provider portal — distinct from the
|
||
standard positions/holdings view we already pull from. Worth
|
||
investigating whether this unlocks better attribution than what
|
||
we get from the positions CSV alone, and whether other 401(k)
|
||
providers expose similar screens.
|
||
|
||
Open questions to answer when picking this up:
|
||
|
||
- Which screen specifically (path / URL within the portal)? Is there
|
||
an export option, or is it view-only / scrape territory?
|
||
- What fields does it expose (employee pre-tax, employer match,
|
||
after-tax / mega-backdoor, by-pay-period dates, per-fund
|
||
allocations)?
|
||
- Refresh cadence — per-paycheck, daily, on-demand?
|
||
- Can it be auto-discovered like the existing audit CSVs, or
|
||
is it manual-entry territory?
|
||
|
||
If the export is structured and recurring, this could feed a
|
||
401(k)-specific contributions classifier that bypasses the lot-diff
|
||
heuristic for that account, similar to how `cash_is_contribution`
|
||
opts ESPP/HSA accounts into cash-based attribution.
|
||
|
||
Related: ESPP-style accrual blind spot in the "Audit: manual-check
|
||
accounts mechanism" section above.
|
||
|
||
## In-kind transfer support (`type::in_kind`) — priority MEDIUM
|
||
|
||
`transaction_log.srf` parses `type::in_kind` records but the
|
||
contributions matcher always rejects them with "in-kind transfers
|
||
not yet supported in v1." In-kind movements need per-symbol
|
||
matching across accounts: an in-kind transfer of 100 VTI shares
|
||
from Acct A to Acct B shows up as `lot_removed` on A + `new_stock`
|
||
on B (or a `rollup_delta` share increase if B already had a VTI
|
||
lot), neither of which can be matched by the current
|
||
amount-based cash matcher.
|
||
|
||
Proposed: a second pass in `matchTransfers` that iterates
|
||
`type::in_kind` records and looks for same-symbol matches across
|
||
`lot_removed` on `from` + `new_stock`/`rollup_delta` on `to`
|
||
within the window. Gated on share-count and open_price sanity so
|
||
a partial transfer doesn't false-positive against an unrelated
|
||
edit.
|
||
|
||
Driver: when the user starts moving positions between accounts
|
||
directly (e.g. Roth conversion of already-held shares, 401k →
|
||
rollover IRA in-kind) rather than liquidating and re-buying.
|
||
|
||
## Torn SRF files from server sync (root cause unknown)
|
||
|
||
**Status:** Root cause still unidentified. We have mitigations and
|
||
diagnostics in place that keep torn responses from corrupting the
|
||
cache, but we don't yet know *why* responses arrive torn. Until we
|
||
have a root cause, this is not resolved — it's mitigated.
|
||
|
||
Mitigations landed so far:
|
||
|
||
- `syncFromServer` (`src/service.zig`) validates responses via
|
||
`cache.Store.looksCompleteSrf` before `writeRaw`. Torn HTTP bodies
|
||
(empty, missing `#!srfv1` header, or no trailing newline) are
|
||
rejected with a warn-level log and NOT written to cache.
|
||
- HTTP responses are checked for an `ETag` sha256 header; on mismatch
|
||
we retry the request once before giving up and falling back to the
|
||
provider.
|
||
- Read-path self-heal: on SRF parse failure during read, the cache
|
||
entry is invalidated so a subsequent refresh can repair without
|
||
user intervention.
|
||
- Diagnostics: richer error capture around the sync path. So far,
|
||
HTTP transit is the dominant source of torn responses — but that's
|
||
an observation, not a root cause.
|
||
|
||
**Remaining work:**
|
||
|
||
- Identify root cause. Candidates to investigate: proxy/load-balancer
|
||
behavior, HTTP keepalive reuse, partial reads on the server side,
|
||
client-side buffer handling. The etag retry tells us whether the
|
||
problem is per-request or persistent; dig into the diagnostics
|
||
output when the next occurrence is captured.
|
||
- Once root cause is known, decide whether the current mitigations
|
||
are sufficient or whether a targeted fix is needed. The
|
||
mitigations may end up being the whole answer, but we can't
|
||
conclude that without understanding the underlying cause.
|
||
|
||
(Content-Length validation was considered and rejected: once the
|
||
server starts compressing response bodies, Content-Length reflects
|
||
the compressed byte count, not the decoded payload, so it's not a
|
||
reliable integrity check.)
|
||
|
||
## Market-aware cache TTL for daily candles
|
||
|
||
Daily candle TTL is currently 23h45m, but candle data only becomes meaningful
|
||
after the market close. Investigate keying the cache freshness to ~4:30 PM
|
||
Eastern rather than a rolling window. This would avoid unnecessary refetches
|
||
during the trading day and ensure a fetch shortly after close gets fresh data.
|
||
Probably alleviated by the cron job approach.
|
||
|
||
## Cache TTL semantics on merge writes — priority LOW
|
||
|
||
The `writeMerged` primitive in `cache/store.zig` rewrites `dividends.srf` /
|
||
`splits.srf` with `expires = now + ttl` whenever it adds a new record or
|
||
upgrades fields on an existing one. This is conceptually wrong: TTL should
|
||
reflect "when do we expect new information from the primary provider?",
|
||
which is a property of the conversation with that provider — not of the
|
||
file's last-modification time. Adding a 25-year-old historical dividend
|
||
that Tiingo just supplied tells us nothing about Polygon's freshness; we
|
||
shouldn't bump the file's expiry as a side effect.
|
||
|
||
The cleaner design:
|
||
|
||
- Cache file's `#!expires=` reflects "when did Polygon (the primary) last
|
||
say `here's everything I have`?"
|
||
- Tiingo merge writes preserve the existing expires, only rewriting records.
|
||
- Only `fetchCached`'s post-Polygon-fetch write bumps expires.
|
||
|
||
In practice the current behavior caused exactly one observable problem: a
|
||
one-time TTL herd on 2026-06-04 when the new merge code's first run added
|
||
pre-2010 Tiingo backfill across 23+ symbols in a single overnight burst,
|
||
and they all inherited that day's clock for `expires = now + 14d`. We
|
||
manually re-staggered (`stagger_cache_ttls.py`) and moved on.
|
||
|
||
Steady-state risk: minimal. The merge primitive's "skip if nothing changed"
|
||
branch means no-op refreshes don't bump expires. New entries from genuinely
|
||
new dividends are spread across the calendar by the dividends themselves
|
||
(quarterly cadence varies per ticker). Field upgrades stop firing once
|
||
Polygon's metadata is in place.
|
||
|
||
When this could matter again:
|
||
- Adding a third source for div/splits (TTL semantics get murkier).
|
||
- Wiping and rebuilding the server cache (one-time herd recurs).
|
||
- A long pause in nightly refreshes followed by a backlog of merge writes.
|
||
|
||
Fix would be small: thread `?expires_override` into `writeMerged` and have
|
||
the merge path call `serializeWithMeta` with the existing expires (from the
|
||
read) when source_hint isn't the primary.
|
||
|
||
## On-demand server-side fetch for new symbols
|
||
|
||
Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure
|
||
cache reads — they 404 if the data isn't already on disk. New symbols only get
|
||
populated when added to the portfolio and picked up by the next cron refresh.
|
||
|
||
Consider: on a cache miss, instead of blocking the HTTP response with a
|
||
multi-second provider fetch, kick off an async background fetch (or just
|
||
auto-add the symbol to the portfolio) and return 404 as usual. The next
|
||
request — or the next cron run — would then have the data. This gives
|
||
"instant-ish gratification" for new symbols without the downsides of
|
||
synchronous fetch-on-miss (latency, rate limit contention, unbounded cache
|
||
growth from arbitrary tickers).
|
||
|
||
Note that this process doesn't do anything to eliminate all the API keys
|
||
that are necessary for a fully functioning system. A more aggressive view
|
||
would be to treat ZFIN_SERVER as a 100% source of record, but that would
|
||
introduce some opacity to the process as we wait for candles (for example) to
|
||
populate. This could be solved on the server by spawning a thread to fetch the
|
||
data, then returning 202 Accepted, which could then be polled client side. Maybe
|
||
this is a better long term approach?
|
||
|
||
## Audit: em-dash sentinel usage across all tables — priority LOW
|
||
|
||
The codebase uses `—` (em-dash) as the canonical "no data" sentinel
|
||
in several table cells, but the rendering rules and alignment
|
||
choices are inconsistent. AGENTS.md now warns against em-dash
|
||
overuse generally; this audit is the second half — pick a
|
||
consistent treatment and apply it everywhere.
|
||
|
||
Known em-dash sites:
|
||
|
||
- `src/views/projections.zig` (back-test): hard-coded `dash_cell`
|
||
literal in 10-col cells — pre-shaped at compile time so no
|
||
helper is involved. Numeric cells use Zig's `{s:>10}` byte-
|
||
padding (safe since they're pure ASCII).
|
||
- `src/commands/history.zig` / `src/tui/history_tab.zig`: centered
|
||
via `fmt.centerDash` in 31-col cells (illiquid totals on
|
||
imported-only history rows).
|
||
- `src/commands/milestones.zig`: right-padded via
|
||
`fmt.padRightToCols` in the "days since prev" cell. Mixes
|
||
with ASCII cells like `"42 days"`.
|
||
- `src/commands/perf.zig` / `src/tui/performance_tab.zig`:
|
||
emitted via `{s:>13}` byte-padding — under-padded by 2 cols
|
||
per em-dash. Either hard-code a `dash_cell` literal (cell
|
||
width is static) or migrate to `fmt.centerDash` /
|
||
`fmt.padRightToCols`.
|
||
|
||
Decisions to make:
|
||
|
||
1. **Centered vs right-aligned in numeric columns.** Back-test
|
||
centers; perf right-aligns (or would, if it weren't broken).
|
||
Centering reads as a more deliberate sentinel; right-aligning
|
||
keeps the visual right-edge of the column smooth. Pick one.
|
||
2. **Should some tables drop the em-dash entirely** in favor of
|
||
ASCII `-`? Rule of thumb: if the column header makes the
|
||
meaning unambiguous AND no rows contain bare `-` for other
|
||
reasons (signed values use `-2.21%` which is multi-char, so
|
||
a lone `-` is unambiguous), `-` is fine. If the column also
|
||
carries dates or strings where a stray `-` could read as
|
||
part of the value, keep `—`.
|
||
3. **Helper vs literal.** When the cell width is fixed and the
|
||
dash position is static, a hard-coded literal const string
|
||
(like back-test's `dash_cell`) is simpler than calling a
|
||
helper at runtime. Use helpers when width or position varies.
|
||
|
||
Once decisions are made, sweep all four sites + add a regression
|
||
alignment test per table that mixes a fully-populated row with
|
||
an em-dash-heavy row and verifies `displayCols` matches.
|
||
|
||
## TUI: numeric keypad input not handled — priority LOW
|
||
|
||
Numeric-keypad keys (Num0-Num9, decimal point, minus) don't reach
|
||
the modal text-input handlers. Reproduced on the projections tab
|
||
as-of date prompt (`d`): typing the date with the numpad produces
|
||
no input, while the digit keys on the main keyboard row work fine.
|
||
Affects every modal that takes numeric input — symbol-input is
|
||
unaffected because it's letters.
|
||
|
||
Likely cause: the modal handlers route through `vaxis.Key.codepoint`
|
||
matching, but vaxis emits keypad keys with a distinct keycode (kitty
|
||
keyboard protocol) rather than the codepoint of the equivalent ASCII
|
||
digit. The fix is in the modal key paths (`handleDateInputKey` in
|
||
`src/tui/projections_tab.zig` and the symbol-input handler in
|
||
`src/tui.zig` — though that one is letters-only in practice) and
|
||
possibly the shared `input_buffer.zig` if that's where character
|
||
gathering lives. Worth surveying both files plus any other tab that
|
||
will grow numeric input (the CLI options command's near-the-money
|
||
strike count would be a candidate if migrated to a modal).
|
||
|
||
Verification: open the TUI, press `d` on projections, try to type a
|
||
date with the keypad. Then try the keyboard row. Both should commit
|
||
identical input.
|
||
|
||
## CLI dispatch / arg-parsing bugs (found May 2026)
|
||
|
||
Found during a post-framework-refactor sanity check of all 20
|
||
commands plus interactive. The framework dispatch itself is
|
||
working correctly; these are gaps in command-level behavior or in
|
||
the global `--refresh-data` flag's coverage.
|
||
|
||
### `zfin interactive --default-keys`/`--default-theme` swallow trailing flags — priority LOW
|
||
|
||
`zfin interactive --default-keys --bogus-flag` is silently
|
||
accepted: `--default-keys` prints its output and `return`s from
|
||
`tui.run` before the rest of the args are parsed. The flag
|
||
parser itself now rejects unknown flags (`error.InvalidArgs`
|
||
+ exit 1), but only when control reaches the parsing loop —
|
||
which it doesn't if `--default-keys` or `--default-theme`
|
||
short-circuits first.
|
||
|
||
Fix: validate the entire arg list before honoring the
|
||
print-and-exit flags, or restructure so the parser runs first
|
||
and the print-and-exit flags fire from inside the loop after
|
||
all args have been validated.
|
||
|
||
### `etf <SYMBOL>` warns `failed to serialize ETF profile: WriteFailed` — priority LOW
|
||
|
||
Every `zfin etf VTI` invocation prints
|
||
`warning(cache): VTI: failed to serialize ETF profile: WriteFailed`
|
||
to stderr before the foreground output. The ETF profile renders
|
||
correctly; just the cache write fails.
|
||
|
||
`src/cache/store.zig:209` calls `serializeEtfProfile` which fails
|
||
on the `aw.writer.print(...)` call inside it (line 1007). Likely
|
||
a Zig 0.16 stdlib quirk in the SRF writer path or a missing
|
||
`flush()` somewhere in the writer chain.
|
||
|
||
Investigate by replacing the `print` with a manual `print` +
|
||
`flush` to see if it's a buffer-not-flushed issue, or by
|
||
serializing a known-good fixture in isolation.
|
||
|
||
## Analysis: umbrella-insurance exposure indicator — priority MEDIUM
|
||
|
||
In the `analysis` command and TUI tab, surface how much of the
|
||
liquid portfolio is *exposed to lawsuit / creditor risk* and
|
||
therefore should be covered by umbrella insurance. The number
|
||
the user actually wants is "how much could be lost if I'm sued
|
||
and lose" — which is liquid assets minus the legally-shielded
|
||
buckets.
|
||
|
||
### Shielding rules (US, broad strokes)
|
||
|
||
- **401(k) and other ERISA-qualified employer plans**: shielded
|
||
by ERISA, federal-law-level protection. Effectively untouchable
|
||
in lawsuits. Always exclude from the umbrella-coverage figure.
|
||
- **IRAs (traditional + Roth)**: protection is **state-by-state**.
|
||
- Federal bankruptcy protection up to ~$1.5M (BAPCPA, indexed)
|
||
applies in bankruptcy court only.
|
||
- Outside bankruptcy (i.e., civil judgments), it's state law.
|
||
Some states give full protection (e.g., TX, FL); some give
|
||
none; many give partial / "reasonably necessary for support."
|
||
- **Inherited IRAs**: a separate category. Less protected than
|
||
contributory IRAs in many states post-*Clark v. Rameker* (2014).
|
||
Ideally tracked separately if relevant.
|
||
- **HSAs**: typically protected if held in an HSA-qualified
|
||
account; varies by state.
|
||
- **Pensions / annuities**: usually protected, varies.
|
||
- **Trusts**: depends on trust type; generally outside the scope
|
||
of this TODO unless we get clean trust metadata.
|
||
- **Brokerage / taxable**: never shielded. Always counts as
|
||
exposed.
|
||
- **Cash in personal bank accounts**: never shielded.
|
||
|
||
### Implementation options
|
||
|
||
Two extremes:
|
||
|
||
1. **Fully automatic** — infer shielding from `account_type` in
|
||
`accounts.srf` (e.g., `401k`, `roth_ira`, `traditional_ira`)
|
||
plus a static state-default table. Risk: state-by-state IRA
|
||
law is genuinely complex, and a wrong default could produce
|
||
a misleading number. The user would need to KNOW the table's
|
||
assumptions.
|
||
2. **Fully manual** — add a `shielded` boolean (or
|
||
`shielded_fraction`: 0.0–1.0) to each account in `accounts.srf`.
|
||
User decides per account. Most accurate, more upfront work.
|
||
|
||
### Recommended hybrid
|
||
|
||
- Add an `account_type` field if not already present (the
|
||
metadata for tax-treatment is likely already there for
|
||
contributions tracking). Map types to a default shielding:
|
||
- `401k`, `403b`, `457`, `pension` → `shielded = true`
|
||
- `roth_401k`, `roth_403b` → `shielded = true`
|
||
- `traditional_ira`, `roth_ira`, `sep_ira`, `simple_ira` →
|
||
`shielded = depends_on_state` (unknown without state info)
|
||
- `hsa` → `shielded = true` (with caveat)
|
||
- `brokerage`, `bank`, `joint`, `trust` → `shielded = false`
|
||
- Add an optional per-account `shielded` override in
|
||
`accounts.srf` (e.g. `shielded::true` / `shielded::false` /
|
||
`shielded::partial`) that wins over the default. This is the
|
||
escape hatch for "my state of residence treats IRAs as
|
||
shielded" or "this trust isn't protected."
|
||
- Add a `state` field at the user level (perhaps
|
||
`metadata.srf` or a config), with a built-in lookup table of
|
||
state IRA protection (`full`, `partial`, `none`) that
|
||
populates the default for IRA-type accounts when no override
|
||
is set.
|
||
|
||
### Output
|
||
|
||
In the analysis tab / command, add a section like:
|
||
|
||
```
|
||
Umbrella exposure
|
||
Total liquid value: $X,XXX,XXX
|
||
Shielded (401k/ERISA): $XXX,XXX
|
||
Shielded (IRA, NY default = partial): $XXX,XXX (per state default)
|
||
Shielded (override): $XXX,XXX
|
||
Net exposure: $X,XXX,XXX ← umbrella target
|
||
```
|
||
|
||
Show it both as an absolute dollar amount and as a percent of
|
||
liquid total. Include the assumption text (state, default for
|
||
that state) inline so the user knows what the number means and
|
||
when it might be wrong.
|
||
|
||
### Tests
|
||
|
||
- Default shielding for each `account_type`.
|
||
- Override wins over default.
|
||
- State table consistency (no missing states).
|
||
- "I don't know my state" path should refuse to guess for IRAs
|
||
and instead report the IRA-shielded portion as `unknown`,
|
||
forcing the user to choose between "treat as shielded" /
|
||
"treat as exposed" via override.
|
||
- Inherited IRAs flagged separately if we track that.
|
||
|
||
### Open questions
|
||
|
||
- Where should the state field live — `metadata.srf` (per
|
||
portfolio file) or a global user config (one user, one state)?
|
||
- Should we surface a "needed umbrella amount = net_exposure +
|
||
[home equity, vehicles]" figure, or strictly stay liquid? The
|
||
user asked about liquid assets specifically, so default to that
|
||
but leave a note.
|
||
- How to handle joint accounts where one spouse's lawsuit could
|
||
reach the other's share. State-specific (community property
|
||
vs separate property states). Probably out of scope for v1.
|
||
|
||
|
||
|
||
The following items are acknowledged but not prioritized. Listed here
|
||
so they don't get lost; pick up opportunistically.
|
||
|
||
### UX
|
||
|
||
- **CLI options command UX.** The `options` command auto-expands only
|
||
the nearest monthly expiration and lists others collapsed. Reconsider
|
||
the interaction model — e.g. allow specifying an expiration date,
|
||
showing all monthlies expanded by default, or filtering by strategy
|
||
(covered calls, spreads).
|
||
- **TUI: toggle to last symbol keybind.** A single-key toggle that
|
||
flips between the current symbol and the previously selected one
|
||
(like `cd -` in bash or `Ctrl+^` in vim). Store `last_symbol` on
|
||
`App`; on symbol change, stash the previous. Useful for
|
||
eyeball-comparing performance/risk data between two symbols.
|
||
|
||
### Data quality
|
||
|
||
- **Fix `enrich` command for international funds.** `deriveMetadata`
|
||
in `src/commands/enrich.zig` misclassifies international ETFs:
|
||
1. `geo` uses Alpha Vantage's `Country` field, which is the *fund
|
||
issuer's* domicile (USA for all US-listed ETFs), not the fund's
|
||
investment geography. Every US-domiciled international fund gets
|
||
`geo::US`.
|
||
2. `asset_class` short-circuits to `"ETF"` when
|
||
`asset_type == "ETF"`, or falls through to a US-market-cap
|
||
heuristic that always produces `"US Large Cap"` /
|
||
`"US Mid Cap"` / `"US Small Cap"`.
|
||
|
||
Known misclassified tickers (all came back as
|
||
`geo::US, asset_class::US Large Cap`):
|
||
- **FRDM** — Freedom 100 Emerging Markets ETF → should be
|
||
`geo::Emerging Markets, asset_class::Emerging Markets`
|
||
- **HFXI** — NYLI FTSE International Equity Currency Neutral ETF
|
||
→ should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
- **IDMO** — Invesco S&P International Developed Momentum ETF →
|
||
should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
- **IVLU** — iShares MSCI International Developed Value Factor
|
||
ETF → should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
|
||
The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography
|
||
data. Options: use the ETF_PROFILE holdings/country data to infer
|
||
geography, parse the fund name for keywords ("International",
|
||
"Emerging", "ex-US"), or accept that `enrich` is a scaffold and
|
||
emit a `# TODO` comment for ETFs instead of silently misclassifying.
|
||
|
||
### Options / valuation
|
||
|
||
- **Per-account covered call adjustment.** `adjustForCoveredCalls` in
|
||
`valuation.zig` operates on portfolio-wide aggregated allocations.
|
||
It matches sold calls against total underlying shares across all
|
||
accounts. This is wrong — calls in one account can only cover
|
||
shares in that same account. Fixing means restructuring
|
||
`portfolioSummary`, since `Allocation` is currently
|
||
account-agnostic. Low priority — naked calls are rare, and calls
|
||
are typically in the same account as the underlying.
|
||
- **Covered call adjustment O(N*M) loop.** `adjustForCoveredCalls`
|
||
has a nested loop — for each allocation, it iterates all lots to
|
||
find matching option contracts. Fine for personal portfolios
|
||
(<1000 lots). Pre-indexing options by underlying would help if
|
||
someone had a very large options-heavy portfolio.
|
||
|
||
### Analysis / correctness
|
||
|
||
- **Analysis account/asset-class total mismatch.** The "By Account"
|
||
and "By Tax Type" sections in the analysis command sum to slightly
|
||
more than "Asset Class" (~0.6% error). Likely a discrepancy between
|
||
how the lot-level account loop values cash, CDs, or options vs how
|
||
the asset-class section computes them via `portfolio.totalCash()` /
|
||
`totalCdFaceValue()`. Per-account values themselves are correct
|
||
after the price_ratio fix.
|
||
|
||
### Audit
|
||
|
||
- **Audit large-lot threshold tuning.** `src/commands/audit.zig` uses
|
||
`audit_large_lot_threshold: f64 = 10_000.0` as the cutoff for
|
||
"surface this new lot for confirmation." Revisit if $10k proves too
|
||
aggressive (ESPP accruals spam the report) or too permissive (large
|
||
DRIP confirmations slip past). If runtime tuning becomes necessary,
|
||
a `--large-lot <amount>` flag or a global
|
||
`audit_large_lot_threshold` field on `accounts.srf` would be
|
||
reasonable extensions.
|
||
|
||
### Infra / performance
|
||
|
||
- **HTTP connection pooling.** Parallel server sync in `loadAllPrices`
|
||
spawns up to 8 threads, each with its own HTTP connection. Could
|
||
reuse connections to reduce TCP handshake overhead. Only matters
|
||
with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
|
||
- **Streaming cache deserialization.** Cache store reads entire files
|
||
into memory (`readFileAlloc` with 50MB limit). For portfolios with
|
||
10+ years of daily candles, this could use significant memory.
|
||
Keep current approach unless memory becomes a real problem.
|