zfin/TODO.md

Future Work

No work here is blocking — we're in a good state. Items below are ordered roughly by priority within each section. Priority labels (HIGH / MEDIUM / LOW) mark items that deserve explicit ranking; unlabeled items are "someday, if the mood strikes."

Next up: configurable benchmark symbols (low-effort win) and the manual-check accounts mechanism (medium effort, real user value).

Projections: future enhancements

• Configurable benchmark symbols — priority MEDIUM. Currently hardcoded SPY + AGG. Route through projections.srf as a type::config,benchmark::SYMBOL record (or similar). Low effort.
• Configurable return cap per position — priority MEDIUM. Default: none; cap outliers like NVDA. Should route through projections.srf cleanly.
• Accumulation-mode SWR rate column is misleading — priority LOW. When retirement_age/retirement_at is configured, the "Safe Withdrawal" table's % column divides the SWR amount by the CURRENT portfolio value, not the post-accumulation portfolio value. The dollar amount is correct (it's the safe spending in retirement, given the projected accumulation), but the % rate comes out absurdly high (e.g., 22% of today's portfolio). The Accumulation phase block already shows the median portfolio at retirement, so the user can compute the real rate themselves — but the SWR table's rate column should ideally divide by the median post-accumulation value, or be suppressed when accumulation is active. Decide which.
• Chart vertical line at retirement boundary — priority LOW. The accumulation-phase spec called this "mandatory" but it was explicitly deferred during implementation. The chart currently shows the full accumulation_years + horizon span without a visual marker for where accumulation ends and distribution begins. Easier to add to the kitty-graphics chart than the braille one.
• Goal-seek over distribution horizon for W1 — priority LOW. Today the W1 ("set spending, find date") workflow reports the earliest retirement at each user-configured (horizon, confidence) cell. The philosophically correct version asks "when have I accumulated enough wealth that the projection shows a 95% probability of success withdrawing X per year from retirement until age-of-death?" — i.e. goal-seek across both accumulation_years AND distribution_years simultaneously, anchored to a configured age-of-death. NP-shaped search; not worth optimizing until someone wants it.
• Per-person retirement_age — priority LOW. V1 of the accumulation-phase spec chose Option A: a single household retirement boundary derived from the oldest configured birthdate. Households where one earner retires significantly earlier than the other would benefit from per-person retirement_age fields on each type::birthdate record, with contributions stopped per-person.
• Configurable max_accumulation_years — priority LOW. Hardcoded at 50 years. Route through projections.srf if anyone hits the cap.
• Configurable MIN period selection (currently 3Y/5Y/10Y, exclude 1Y)
• Multiple spending models: flat (current), decreasing (1-2% real annual decrease, Blanchett "spending smile"). Late-life healthcare better modeled as a life event.
• Unclassified position handling in allocation split (warn user)
• Historical projection overlay follow-ups. The base --overlay-actuals overlay shipped (CLI tip + TUI primary surface). Open enhancements:
  • Historical metadata.srf / projections.srf for back-dated runs. Today the overlay re-runs against current classifications and assumptions; for historically faithful what-the-model-said-then output we'd check out the git-tracked versions of those files at the as-of commit and load those instead. Edge case until classifications materially drift.
  • Contribution-attribution overlay. Today's actuals line includes contributions implicitly; the bands assume modeled contributions that may or may not match reality. A "decompose actuals into market return vs contributions" annotation would clarify how much of the trajectory was the model being right vs new money arriving on schedule.
  • Mosaic mode: overlay multiple as-of starting points on one chart ("show me 1Y, 3Y, 5Y, 10Y projections all at once") so the user can see how the projection envelope tightened as data came in.
  • Better composition basis for imported-only as-of. Today the imported-only path uses today's allocations scaled by imported_liquid / today_total_liquid. That's the simplest thing that could work, but it's "today's mix back-dated" — it ignores everything we know about the historical context. Specifically: imported_values.srf already carries an expected_return field per row that the user captured at that date in their source spreadsheet. We could:
    • Use the imported expected_return as a sanity check against the simulation's per-position weighted return (warn or clamp if they diverge wildly — the spreadsheet's number reflects what the user actually saw at the time).
    • Use the imported expected_return to bias the stock/bond split inference: a higher expected return implies a higher historical equity weighting than today's mix probably reflects.
    • Reach further: derive a synthetic stock/bond split from the imported expected_return directly, treating it as a weighted average of SPY and AGG returns at that date and solving for the weights. That gives a per-imported- row composition that's locally faithful instead of one-mix-fits-all. None of these are urgent — the current "today's mix scaled" approximation is documented as such and the bands still render meaningfully — but each would tighten the historical faithfulness one notch. Pick whichever has the highest payoff vs. complexity when this gets revisited.
  • Chart zoom for short-history overlays. With a 50-year projection horizon and only ~10 years of imported actuals, the actuals line is squashed into the first 20% of the chart and the comparison-against-bands story is hard to read. Two design directions:
    • Auto-zoom: when the overlay is on, the chart's x-axis defaults to [as_of, today + N years] (where N is small, e.g. 2x the actuals span) instead of [as_of, as_of + horizon]. The bands beyond `today
      • N` are still computed but clipped from view. The tradeoff: the user loses the long-tail terminal-value context unless they toggle back out.
    • Toggle: a separate keybind (e.g. z for zoom) flips between full-horizon and zoomed views. Default off so the bands tell their full story; user opts in when they want overlay legibility. Auto-zoom is more invasive (changes the default chart semantics for everyone running with overlay-on) but better matches what the user actually wants when they toggle the overlay. Toggle is safer but requires the user to know the feature exists. Probably do auto-zoom but expose a toggle to escape it ("show full horizon").

Export chart as PNG (--export-chart <path>) — priority MEDIUM

z2d already supports PNG export natively. Today the chart-bearing commands (quote, history, projections, plus the equivalent TUI tabs) render to braille (CLI) or Kitty graphics (TUI). Adding a --export-chart <path> flag would land just the chart (not the surrounding text output) as a PNG file at the given path, at full fidelity, regardless of which surface invoked it.

Driver: when reviewing a back-dated projection or a notable price move, capturing the chart as an image (e.g. for a write-up, an email to the household, or a wiki page) is currently a screenshot-and-crop chore. PNG export makes it a one-shot CLI invocation.

Sketch:

• zfin quote AAPL --export-chart aapl.png → just the price+ Bollinger chart as a PNG, no other output.
• zfin projections --as-of 1Y --overlay-actuals --export-chart projection.png → the projection-bands chart plus overlay, no other output.
• The chart code already produces RGB pixel buffers via z2d; replace the transmitPreEncodedImage call (TUI) or the braille text path (CLI) with a Surface.write_png call when the flag is present.

Plumbing: a thin "chart-only render" entry point in each chart module (projection_chart.zig, chart.zig for symbols), called from the relevant command's run() when --export-chart is set. Exits before the rest of the text output renders.

Out of scope for V1: file-format alternatives (SVG, PDF), themed color overrides for export (always uses the active terminal theme), non-chart command output as PNG.

Refactor: trim src/format.zig once Money / Date have absorbed their helpers — priority LOW

src/format.zig is still a ~1700-line grab-bag, but the money- and date-shaped helpers that used to live there have been moved out: money formatting now lives in src/Money.zig (with {f} / whole() / trim() / signed() / padRight(N) / padLeft(N)), and date formatting lives in src/Date.zig (with {f} / padRight(N) / padLeft(N)). What's left in format.zig is the genuinely-format-domain stuff: braille charts, return formatters, allocation notes, signed-percent rendering.

If the file ever grows enough to be annoying again, consider renaming to src/render.zig to better describe what's left, or splitting the braille chart out (it's ~600 lines on its own). Not blocking — file it as cleanup if and when it bites.

Refactor: TUI App struct knows too much about each tab — priority MEDIUM

src/tui.zig's App struct currently has dozens of tab-specific fields scattered across its top level — projections_loaded, projections_disabled, projections_config, projections_ctx, projections_horizon_idx, projections_image_id, projections_image_width, projections_image_height, projections_chart_dirty, projections_chart_visible, projections_events_enabled, projections_value_min, projections_value_max, projections_as_of, projections_as_of_requested, projections_overlay_actuals, plus equivalents for portfolio, history, options, earnings, analysis, perf, and quote tabs.

This couples App to the implementation details of every tab. Touching a single tab's state shape requires editing the central struct, which makes refactors noisy and discourages tab modules from owning their own state cleanly.

Proposal: each tab gets exactly ONE field on App — a struct (or pointer to a struct) defined in that tab's own file. The tab module owns its state shape; App only carries the top-level reference.

Sketch:

// src/tui/projections_tab.zig
pub const State = struct {
    loaded: bool = false,
    disabled: bool = false,
    config: projections.UserConfig = .{},
    ctx: ?ProjectionContext = null,
    horizon_idx: usize = 0,
    chart: ChartState = .{},
    as_of: AsOfState = .{},
    overlay_actuals: bool = false,
    // ...
};

// src/tui.zig
pub const App = struct {
    // ... shared cross-tab state (today, allocator, vx_app, ...) ...
    portfolio: portfolio_tab.State = .{},
    quote: quote_tab.State = .{},
    perf: perf_tab.State = .{},
    history: history_tab.State = .{},
    projections: projections_tab.State = .{},
    options: options_tab.State = .{},
    earnings: earnings_tab.State = .{},
    analysis: analysis_tab.State = .{},
};

After the migration, app.projections_overlay_actuals becomes app.projections.overlay_actuals, etc. Each tab's State struct documents its own invariants without polluting App.

Benefits:

• Tab modules become genuinely self-contained. Adding new state to a tab only touches that tab's file.
• App's field count drops from ~80 to ~10 (cross-cutting state
  • 8 tab-state fields).
• Clearer ownership: when reading App, you can tell at a glance which fields are shared vs tab-private.
• Easier to reason about lifetimes — each tab's freeLoaded() operates on its own struct rather than reaching into App.
• Onboarding new contributors: "to add a tab feature, define state in the tab file" instead of "edit two files in lockstep."

Migration approach:

This is a large mechanical refactor (~80+ field renames across all tab files plus tui.zig). Best done as one focused PR:

1. Define State in each tab file with all current fields.
2. Add the eight new fields to App; flip them on as defaults (so old app.projections_* accesses temporarily still work if we add accessor shims, but cleaner to just rip the bandaid).
3. Sweep all app.<tab>_<field> references → app.<tab>.<field>.
4. Delete the old top-level fields from App.
5. Verify zig build test is unchanged. The refactor should be strictly behavior-preserving.

Risks: large diff (touches every tab file plus tui.zig), but mechanical — no logic changes, no tests should move. The pre- commit hooks catch any miss instantly.

While we're at it: action handler bodies should also move. The same shape problem shows up in tui.zig's keybind-action dispatch — sort_reverse, toggle_chart, toggle_events, account_filter, and the per-tab branches inside them are ~100 lines of tab-specific logic living in the central event loop. Concretely (line numbers as of writing):

• sort_reverse (~line 1328) dual-dispatches by active tab: portfolio flips sort direction + calls sortPortfolioAllocations / rebuildPortfolioRows; projections toggles overlay-actuals + reloads data + sets status. None of that body is App-level concern; it's two separate tabs' private state mutations.
• toggle_chart (~line 1382) flips projections_chart_visible and resets scroll_offset. Pure projections-tab business.
• toggle_events (~line 1390) flips projections_events_enabled and triggers a reload. Same.
• account_filter (~line 1366) opens the account picker mode with portfolio-tab-specific cursor positioning.

The cleaner shape: each tab module exposes a handleAction(app, action) bool (or similar) that returns true when it consumed the action. tui.zig's dispatch becomes a thin "ask the active tab if it wants this action; otherwise fall through to global handlers." The body of each case shrinks to a one-liner that dispatches to tab_module.handleAction(self).

This pairs naturally with the State-struct migration above — the tab module's handleAction operates on its own State struct rather than reaching into App. Some keybinds are genuinely cross-cutting (quit, refresh, tab navigation, scroll) and stay in the central handler. The split is "App owns chrome; tab owns content."

Driver: every overlay-actuals-shaped feature added to a tab recently has involved adding 1–2 fields to App, and the struct keeps growing. Eventually it becomes unreviewable.

Bug: braille charts use raw close, not adj_close — cliff at splits

Reproduction: zfin quote SOXX (or the TUI quote tab). The braille chart drops sharply on 2024-03-07, which is the iShares Semiconductor ETF's 3-for-1 split date:

• 2024-03-06 close: $689.60
• 2024-03-07 close: $237.75 (≈ $689.60 / 2.9)

The adj_close column in ~/.cache/zfin/SOXX/candles_daily.srf tracks correctly through the split (~$226 → ~$234), so the provider data is fine. The bug is purely cosmetic: the chart renders the unadjusted close price.

Root cause: computeBrailleChart in src/format.zig:888 indexes data[i].close instead of data[i].adj_close. Lines 901, 902, 904, 905, 935 all use .close.

Independent confirmation: zfin splits SOXX returns 2024-03-07 3:1 from Polygon. So the split data exists in the provider layer (and gets cached as splits.srf once requested), but the charting code path doesn't consult it.

Fix candidates:

A. Switch computeBrailleChart to consume adj_close directly. Simplest. Affects every chart caller (quote, history, projections median band, TUI quote/projections tabs). Cosmetic only — no computation depends on it. The price-axis labels would render adjusted prices, which may surprise users used to seeing the raw last-close. Mitigate with a comment in the chart's right-edge label region or a header note.

B. Pass a flag to computeBrailleChart selecting close vs adj_close. Default to adjusted; let the quote tab show raw. More flexible, marginally more code.

C. Add a chart_close accessor to Candle that returns adj_close if non-zero, else close. Same effect as (A) with a defensive fallback.

D. Apply split adjustments at chart-data prep time using splits.srf. Walk the candle slice with the split history and pre-multiply pre-split closes by the cumulative ratio. More work, but produces a chart-axis dollar value the user expects ("today's last close was $X, the chart starts at $Y from N years ago"). This is what most charting libraries do. Requires plumbing DataService.getSplits into the chart-prep path on every chart caller, OR doing the adjustment once in the service layer alongside candle fetching. Not all callers have a DataService reference today (e.g., runProjection's synthetic median-band candles).

Recommendation: Start with (A) or (C) — single-line fix, gets the cliff out of all charts immediately. (D) is the "correct" fix but a bigger refactor; file as a follow-up if (A)/(C) lands first.

Other affected symbols: Any held position with a split in the last 10 years will have the same artifact. Check NVDA (10:1 split on 2024-06-10) for a louder example.

Priority: LOW. Cosmetic only — analytics already use adj_close correctly via the per-position trailing-returns path. But it's confusing when scanning a chart and seeing a 50% drop that isn't real.

Audit: manual-check accounts mechanism (NYL, Kelly's ESPP, etc.) — priority HIGH

Some accounts/positions can't be reconciled from broker CSVs and need a human-in-the-loop reminder at the audit step. Examples:

• NY Life — no CSV export at all. Values only live in periodic statements.
• Kelly's ESPP — accrued payroll-deduction cash doesn't appear in the Fidelity positions CSV until the purchase date hits (typically every 6 months). Between purchases the cash is a real contribution that zfin audit can't see.
• Future: treat as an open category.

The existing update_cadence::weekly|monthly|quarterly|none field already sort-of covers this, but has two gaps:

1. It fires off the last git-detected change, not the last human review. For NYL, the value sometimes hasn't changed in months — so git never fires, cadence never trips.
2. ESPP needs weekly-ish attention while accumulating cash between purchases, but the accrued balance is invisible to the CSV audit.

Options

A. New update_cadence::manual variant — always fires every audit run until silenced. Blunt but zero design work.

B. last_refreshed::YYYY-MM-DD field on accounts.srf — explicit human-review timestamp, decoupled from git-detected changes. Audit compares today - last_refreshed against the cadence. User bumps the field when they check the statement. Probably the most correct fit for NYL.

C. Sticky TODO list — a todos.srf or todo:: field on accounts that audit always surfaces until cleared. General-purpose; also covers "remember to rebalance on 5/15".

ESPP-specific follow-through

ESPP is also a contribution-attribution blind spot. If Kelly's paystub deducts $X/week but the cash lot doesn't reach portfolio.srf until the purchase date, the attribution math is under-counting contributions and over-counting the purchase-week gain. Possible fixes are discussed in the "Contributions diff" TODO below — option C there (per-account cash_is_contribution) would make manually-entered ESPP cash additions count correctly.

In-kind transfer support (type::in_kind) — priority MEDIUM

transaction_log.srf parses type::in_kind records but the contributions matcher always rejects them with "in-kind transfers not yet supported in v1." In-kind movements need per-symbol matching across accounts: an in-kind transfer of 100 VTI shares from Acct A to Acct B shows up as lot_removed on A + new_stock on B (or a rollup_delta share increase if B already had a VTI lot), neither of which can be matched by the current amount-based cash matcher.

Proposed: a second pass in matchTransfers that iterates type::in_kind records and looks for same-symbol matches across lot_removed on from + new_stock/rollup_delta on to within the window. Gated on share-count and open_price sanity so a partial transfer doesn't false-positive against an unrelated edit.

Driver: when the user starts moving positions between accounts directly (e.g. Roth conversion of already-held shares, 401k → rollover IRA in-kind) rather than liquidating and re-buying.

Torn SRF files from server sync (root cause unknown)

Status: Root cause still unidentified. We have mitigations and diagnostics in place that keep torn responses from corrupting the cache, but we don't yet know why responses arrive torn. Until we have a root cause, this is not resolved — it's mitigated.

Mitigations landed so far:

• syncFromServer (src/service.zig) validates responses via cache.Store.looksCompleteSrf before writeRaw. Torn HTTP bodies (empty, missing #!srfv1 header, or no trailing newline) are rejected with a warn-level log and NOT written to cache.
• HTTP responses are checked for an ETag sha256 header; on mismatch we retry the request once before giving up and falling back to the provider.
• Read-path self-heal: on SRF parse failure during read, the cache entry is invalidated so a subsequent refresh can repair without user intervention.
• Diagnostics: richer error capture around the sync path. So far, HTTP transit is the dominant source of torn responses — but that's an observation, not a root cause.

Remaining work:

• Identify root cause. Candidates to investigate: proxy/load-balancer behavior, HTTP keepalive reuse, partial reads on the server side, client-side buffer handling. The etag retry tells us whether the problem is per-request or persistent; dig into the diagnostics output when the next occurrence is captured.
• Once root cause is known, decide whether the current mitigations are sufficient or whether a targeted fix is needed. The mitigations may end up being the whole answer, but we can't conclude that without understanding the underlying cause.

(Content-Length validation was considered and rejected: once the server starts compressing response bodies, Content-Length reflects the compressed byte count, not the decoded payload, so it's not a reliable integrity check.)

Market-aware cache TTL for daily candles

Daily candle TTL is currently 23h45m, but candle data only becomes meaningful after the market close. Investigate keying the cache freshness to ~4:30 PM Eastern rather than a rolling window. This would avoid unnecessary refetches during the trading day and ensure a fetch shortly after close gets fresh data. Probably alleviated by the cron job approach.

On-demand server-side fetch for new symbols

Currently the server's SRF endpoints (/candles, /dividends, etc.) are pure cache reads — they 404 if the data isn't already on disk. New symbols only get populated when added to the portfolio and picked up by the next cron refresh.

Consider: on a cache miss, instead of blocking the HTTP response with a multi-second provider fetch, kick off an async background fetch (or just auto-add the symbol to the portfolio) and return 404 as usual. The next request — or the next cron run — would then have the data. This gives "instant-ish gratification" for new symbols without the downsides of synchronous fetch-on-miss (latency, rate limit contention, unbounded cache growth from arbitrary tickers).

Note that this process doesn't do anything to eliminate all the API keys that are necessary for a fully functioning system. A more aggressive view would be to treat ZFIN_SERVER as a 100% source of record, but that would introduce some opacity to the process as we wait for candles (for example) to populate. This could be solved on the server by spawning a thread to fetch the data, then returning 202 Accepted, which could then be polled client side. Maybe this is a better long term approach?

Low-priority items

The following items are acknowledged but not prioritized. Listed here so they don't get lost; pick up opportunistically.

UX

• CLI options command UX. The options command auto-expands only the nearest monthly expiration and lists others collapsed. Reconsider the interaction model — e.g. allow specifying an expiration date, showing all monthlies expanded by default, or filtering by strategy (covered calls, spreads).
• TUI: toggle to last symbol keybind. A single-key toggle that flips between the current symbol and the previously selected one (like cd - in bash or Ctrl+^ in vim). Store last_symbol on App; on symbol change, stash the previous. Useful for eyeball-comparing performance/risk data between two symbols.

Data quality

• Fix enrich command for international funds. deriveMetadata in src/commands/enrich.zig misclassifies international ETFs:

  1. geo uses Alpha Vantage's Country field, which is the fund issuer's domicile (USA for all US-listed ETFs), not the fund's investment geography. Every US-domiciled international fund gets geo::US.
  2. asset_class short-circuits to "ETF" when asset_type == "ETF", or falls through to a US-market-cap heuristic that always produces "US Large Cap" / "US Mid Cap" / "US Small Cap".

  Known misclassified tickers (all came back as geo::US, asset_class::US Large Cap):

  • FRDM — Freedom 100 Emerging Markets ETF → should be geo::Emerging Markets, asset_class::Emerging Markets
  • HFXI — NYLI FTSE International Equity Currency Neutral ETF → should be geo::International Developed, asset_class::International Developed
  • IDMO — Invesco S&P International Developed Momentum ETF → should be geo::International Developed, asset_class::International Developed
  • IVLU — iShares MSCI International Developed Value Factor ETF → should be geo::International Developed, asset_class::International Developed

  The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography data. Options: use the ETF_PROFILE holdings/country data to infer geography, parse the fund name for keywords ("International", "Emerging", "ex-US"), or accept that enrich is a scaffold and emit a # TODO comment for ETFs instead of silently misclassifying.

Options / valuation

• Per-account covered call adjustment. adjustForCoveredCalls in valuation.zig operates on portfolio-wide aggregated allocations. It matches sold calls against total underlying shares across all accounts. This is wrong — calls in one account can only cover shares in that same account. Fixing means restructuring portfolioSummary, since Allocation is currently account-agnostic. Low priority — naked calls are rare, and calls are typically in the same account as the underlying.
• Covered call adjustment O(N*M) loop. adjustForCoveredCalls has a nested loop — for each allocation, it iterates all lots to find matching option contracts. Fine for personal portfolios (<1000 lots). Pre-indexing options by underlying would help if someone had a very large options-heavy portfolio.

Analysis / correctness

• Analysis account/asset-class total mismatch. The "By Account" and "By Tax Type" sections in the analysis command sum to slightly more than "Asset Class" (~0.6% error). Likely a discrepancy between how the lot-level account loop values cash, CDs, or options vs how the asset-class section computes them via portfolio.totalCash() / totalCdFaceValue(). Per-account values themselves are correct after the price_ratio fix.

Audit

• Audit large-lot threshold tuning. src/commands/audit.zig uses audit_large_lot_threshold: f64 = 10_000.0 as the cutoff for "surface this new lot for confirmation." Revisit if $10k proves too aggressive (ESPP accruals spam the report) or too permissive (large DRIP confirmations slip past). If runtime tuning becomes necessary, a --large-lot <amount> flag or a global audit_large_lot_threshold field on accounts.srf would be reasonable extensions.

Infra / performance

• HTTP connection pooling. Parallel server sync in loadAllPrices spawns up to 8 threads, each with its own HTTP connection. Could reuse connections to reduce TCP handshake overhead. Only matters with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
• Streaming cache deserialization. Cache store reads entire files into memory (readFileAlloc with 50MB limit). For portfolios with 10+ years of daily candles, this could use significant memory. Keep current approach unless memory becomes a real problem.