update todo.md

TODO.md

@@ -1,19 +1,33 @@
 # Future Work
 
+No work here is blocking — we're in a good state. Items below are
+ordered roughly by priority within each section. Priority labels
+(`HIGH` / `MEDIUM` / `LOW`) mark items that deserve explicit
+ranking; unlabeled items are "someday, if the mood strikes."
+
+**Next up:** configurable benchmark symbols (low-effort win) and the
+manual-check accounts mechanism (medium effort, real user value).
+
 ## Projections: future enhancements
 
-- Configurable return cap per position (default: none; cap outliers like NVDA)
+- **Configurable benchmark symbols — priority MEDIUM.** Currently
+  hardcoded SPY + AGG. Route through `projections.srf` as a
+  `type::config,benchmark::SYMBOL` record (or similar). Low effort.
+- **Configurable return cap per position — priority MEDIUM.**
+  Default: none; cap outliers like NVDA. Should route through
+  `projections.srf` cleanly.
+- **"Not Retired Yet" accumulation-phase mode — priority HIGH.**
+  Accumulation phase with contributions before retirement.
+  Separate from life events — the simulation has two distinct phases:
+  accumulation (base spending = 0, contributions applied) and
+  distribution (searched-for withdrawal + spending model). Config:
+  `retirement_age::60` or `retirement_in::10`, plus
+  `annual_contribution::100000`. Safe withdrawal search only applies
+  to the distribution phase. Chart shows portfolio growing during
+  accumulation, peaking at retirement, then drawing down.
 - Configurable MIN period selection (currently 3Y/5Y/10Y, exclude 1Y)
-- "Not Retired Yet" mode: accumulation phase with contributions before retirement.
-  Separate from life events — the simulation has two distinct phases: accumulation
-  (base spending = 0, contributions applied) and distribution (searched-for withdrawal
-  + spending model). Config: `retirement_age::60` or `retirement_in::10`, plus
-  `annual_contribution::100000`. Safe withdrawal search only applies to the
-  distribution phase. Chart shows portfolio growing during accumulation, peaking
-  at retirement, then drawing down.
 - Multiple spending models: flat (current), decreasing (1-2% real annual decrease,
   Blanchett "spending smile"). Late-life healthcare better modeled as a life event.
-- Configurable benchmark symbols (currently hardcoded SPY + AGG)
 - Unclassified position handling in allocation split (warn user)
 - Historical projection comparison: re-run projections from any past snapshot date,
   overlay actual portfolio trajectory from subsequent snapshots onto the projected
@@ -39,129 +53,7 @@
   the current versions of both; historical git-tracked versions could
   be checked out and loaded instead. Edge case for now.
 
-## Analysis account/asset-class total mismatch
+## Audit: manual-check accounts mechanism (NYL, Riley's ESPP, etc.) — priority HIGH
-
-The "By Account" and "By Tax Type" sections in the analysis command sum to slightly
-more than "Asset Class" (~0.6% error). Likely a discrepancy between how the lot-level
-account loop values cash, CDs, or options vs how the asset-class section computes them
-via `portfolio.totalCash()` / `totalCdFaceValue()`. Low priority — the per-account
-values themselves are correct after the price_ratio fix.
-
-## Upgrade to 0.16.0
-
-Pending dependencies:
-
-* SRF: complete (use 0.15.2 tag if needed)
-* VAxis: work seems to be in this PR: https://github.com/rockorager/libvaxis/pull/316
-* z2d: 0.16.0 branch exists, but not sure if this is fully compatible with last commit in March
-
-## CLI options command UX
-
-The `options` command auto-expands only the nearest monthly expiration and
-lists others collapsed. Reconsider the interaction model — e.g. allow
-specifying an expiration date, showing all monthlies expanded by default,
-or filtering by strategy (covered calls, spreads).
-
-## TUI: toggle to last symbol keybind
-
-Add a single-key toggle that flips between the current symbol and the
-previously selected one (like `cd -` in bash or `Ctrl+^` in vim). Store
-`last_symbol` on `App`; on symbol change, stash the previous. The toggle
-key swaps current and last. Works on any tab — particularly useful for
-eyeball-comparing performance/risk data between two symbols.
-
-## Fix `enrich` command for international funds
-
-`deriveMetadata` in `src/commands/enrich.zig` misclassifies international ETFs:
-
-1. **`geo`** uses Alpha Vantage's `Country` field, which is the *fund issuer's*
-   domicile (USA for all US-listed ETFs), not the fund's investment geography.
-   Every US-domiciled international fund gets `geo::US`.
-
-2. **`asset_class`** short-circuits to `"ETF"` when `asset_type == "ETF"`, or
-   falls through to a US-market-cap heuristic that always produces
-   `"US Large Cap"` / `"US Mid Cap"` / `"US Small Cap"`.
-
-Known misclassified tickers (all came back as `geo::US, asset_class::US Large Cap`):
-- **FRDM** — Freedom 100 Emerging Markets ETF → should be `geo::Emerging Markets, asset_class::Emerging Markets`
-- **HFXI** — NYLI FTSE International Equity Currency Neutral ETF → should be `geo::International Developed, asset_class::International Developed`
-- **IDMO** — Invesco S&P International Developed Momentum ETF → should be `geo::International Developed, asset_class::International Developed`
-- **IVLU** — iShares MSCI International Developed Value Factor ETF → should be `geo::International Developed, asset_class::International Developed`
-
-The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography data.
-Options: use the ETF_PROFILE holdings/country data to infer geography, parse
-the fund name for keywords ("International", "Emerging", "ex-US"), or accept
-that `enrich` is a scaffold and emit a `# TODO` comment for ETFs instead of
-silently misclassifying.
-
-## Market-aware cache TTL for daily candles
-
-Daily candle TTL is currently 23h45m, but candle data only becomes meaningful
-after the market close. Investigate keying the cache freshness to ~4:30 PM
-Eastern rather than a rolling window. This would avoid unnecessary refetches
-during the trading day and ensure a fetch shortly after close gets fresh data.
-Probably alleviated by the cron job approach.
-
-## On-demand server-side fetch for new symbols
-
-Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure
-cache reads — they 404 if the data isn't already on disk. New symbols only get
-populated when added to the portfolio and picked up by the next cron refresh.
-
-Consider: on a cache miss, instead of blocking the HTTP response with a
-multi-second provider fetch, kick off an async background fetch (or just
-auto-add the symbol to the portfolio) and return 404 as usual. The next
-request — or the next cron run — would then have the data. This gives
-"instant-ish gratification" for new symbols without the downsides of
-synchronous fetch-on-miss (latency, rate limit contention, unbounded cache
-growth from arbitrary tickers).
-
-Note that this process doesn't do anything to eliminate all the API keys
-that are necessary for a fully functioning system. A more aggressive view
-would be to treat ZFIN_SERVER as a 100% source of record, but that would
-introduce some opacity to the process as we wait for candles (for example) to
-populate. This could be solved on the server by spawning a thread to fetch the
-data, then returning 202 Accepted, which could then be polled client side. Maybe
-this is a better long term approach?
-
-## Per-account covered call adjustment
-
-`adjustForCoveredCalls` in `valuation.zig` operates on portfolio-wide aggregated
-allocations. It matches sold calls against total underlying shares across all
-accounts. This is wrong — calls in one account can only cover shares in that
-same account. If NVDA calls are sold in Emil IRA, they shouldn't cap NVDA
-shares held in Joint Account.
-
-Fixing this means restructuring `portfolioSummary`, since `Allocation` is
-currently account-agnostic. Approach: compute per-account reductions using
-`positionsForAccount` + account-filtered option lots, then sum into
-portfolio-wide reductions. Each account's reduction capped by that account's
-shares, not the global total.
-
-Low priority — naked calls are rare, and calls are typically in the same
-account as the underlying.
-
-## Covered call adjustment optimization
-
-`adjustForCoveredCalls` has a nested loop — for each allocation, it iterates
-all lots to find matching option contracts. O(N*M) is fine for personal
-portfolios (<1000 lots). Pre-indexing options by underlying would help if
-someone had a very large options-heavy portfolio.
-
-## HTTP connection pooling
-
-Parallel server sync in `loadAllPrices` spawns up to 8 threads, each with its
-own HTTP connection. Could reuse connections to reduce TCP handshake overhead.
-Only matters with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
-8 concurrent connections is fine for now.
-
-## Streaming cache deserialization
-
-Cache store reads entire files into memory (`readFileAlloc` with 50MB limit).
-For portfolios with 10+ years of daily candles, this could use significant
-memory. Keep current approach unless memory becomes a real problem.
-
-## Audit: manual-check accounts mechanism (NYL, Riley's ESPP, etc.)
 
 Some accounts/positions can't be reconciled from broker CSVs and need a
 human-in-the-loop reminder at the audit step. Examples:
@@ -208,25 +100,6 @@ in the "Contributions diff" TODO below — option C there (per-account
 `cash_is_contribution`) would make manually-entered ESPP
 cash additions count correctly.
 
-## Audit: tax-loss account persistent discrepancy (~-$897.59)
-
-**Status (2026-05-02):** Resolved via the `direct_indexing::true`
-flag in `accounts.srf`. When set on an account:
-
-- Audit now emits a suggested `price_ratio` adjustment even for
-  lots with ratio == 1.0 (previously skipped). The Schwab-summary
-  path picks up direct-indexing accounts with a single stock lot
-  and computes the suggested ratio from the account-level value
-  delta. Copy the suggestion into `portfolio.srf` and next audit
-  is clean.
-- Contributions swallows sub-1% share drift on lots in the account
-  as tracking-error noise instead of emitting `rollup_delta` /
-  `drip_negative`. Real contributions (>1% of account value) still
-  surface.
-
-See `src/analytics/analysis.zig` `AccountTaxEntry.direct_indexing`
-and `src/commands/audit.zig` `displaySchwabSummaryRatioSuggestions`.
-
 ## In-kind transfer support (`type::in_kind`) — priority MEDIUM
 
 `transaction_log.srf` parses `type::in_kind` records but the
@@ -249,27 +122,14 @@ Driver: when the user starts moving positions between accounts
 directly (e.g. Roth conversion of already-held shares, 401k →
 rollover IRA in-kind) rather than liquidating and re-buying.
 
-## Audit large-lot threshold tuning — priority LOW
+## Torn SRF files from server sync (root cause unknown)
 
-`src/commands/audit.zig` uses `audit_large_lot_threshold: f64 =
+**Status:** Root cause still unidentified. We have mitigations and
-10_000.0` as the cutoff for "surface this new lot for confirmation."
+diagnostics in place that keep torn responses from corrupting the
-The value is a judgment call; revisit if:
+cache, but we don't yet know *why* responses arrive torn. Until we
+have a root cause, this is not resolved — it's mitigated.
 
-- $10k proves too aggressive (weekly payroll ESPP accruals near
+Mitigations landed so far:
-  the limit spam the report),
-- $10k proves too permissive (large DRIP confirmations or CD
-  maturities slip past uncalled),
-- the user wants per-account thresholds (e.g. $25k for
-  high-turnover brokerage, $5k for IRAs).
-
-If runtime tuning becomes necessary, either a `--large-lot
-<amount>` flag on `zfin audit` or a global `audit_large_lot_threshold`
-field on `accounts.srf`-the-file would be reasonable extensions.
-Until then the constant is the single knob.
-
-## Torn SRF files from server sync (recurring bug)
-
-**Status:** Multi-layer defense in place.
 
 - `syncFromServer` (`src/service.zig`) validates responses via
   `cache.Store.looksCompleteSrf` before `writeRaw`. Torn HTTP bodies
@@ -281,17 +141,160 @@ Until then the constant is the single knob.
 - Read-path self-heal: on SRF parse failure during read, the cache
   entry is invalidated so a subsequent refresh can repair without
   user intervention.
-- Diagnostics: richer error capture around the sync path to pinpoint
+- Diagnostics: richer error capture around the sync path. So far,
-  where torn responses originate (HTTP transit has been the dominant
+  HTTP transit is the dominant source of torn responses — but that's
-  source so far).
+  an observation, not a root cause.
 
 **Remaining work:**
 
-- Continue monitoring — if torn responses persist despite the etag
+- Identify root cause. Candidates to investigate: proxy/load-balancer
-  retry, investigate lower-level transport issues (proxy, keepalive,
+  behavior, HTTP keepalive reuse, partial reads on the server side,
-  partial reads on the server side).
+  client-side buffer handling. The etag retry tells us whether the
+  problem is per-request or persistent; dig into the diagnostics
+  output when the next occurrence is captured.
+- Once root cause is known, decide whether the current mitigations
+  are sufficient or whether a targeted fix is needed. The
+  mitigations may end up being the whole answer, but we can't
+  conclude that without understanding the underlying cause.
 
 (Content-Length validation was considered and rejected: once the
 server starts compressing response bodies, Content-Length reflects
 the compressed byte count, not the decoded payload, so it's not a
 reliable integrity check.)
+
+## Upgrade to 0.16.0
+
+Pending dependencies:
+
+* SRF: complete (use 0.15.2 tag if needed)
+* VAxis: work seems to be in this PR: https://github.com/rockorager/libvaxis/pull/316
+* z2d: implemented in 0.11.0 of z2d
+
+## Market-aware cache TTL for daily candles
+
+Daily candle TTL is currently 23h45m, but candle data only becomes meaningful
+after the market close. Investigate keying the cache freshness to ~4:30 PM
+Eastern rather than a rolling window. This would avoid unnecessary refetches
+during the trading day and ensure a fetch shortly after close gets fresh data.
+Probably alleviated by the cron job approach.
+
+## On-demand server-side fetch for new symbols
+
+Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure
+cache reads — they 404 if the data isn't already on disk. New symbols only get
+populated when added to the portfolio and picked up by the next cron refresh.
+
+Consider: on a cache miss, instead of blocking the HTTP response with a
+multi-second provider fetch, kick off an async background fetch (or just
+auto-add the symbol to the portfolio) and return 404 as usual. The next
+request — or the next cron run — would then have the data. This gives
+"instant-ish gratification" for new symbols without the downsides of
+synchronous fetch-on-miss (latency, rate limit contention, unbounded cache
+growth from arbitrary tickers).
+
+Note that this process doesn't do anything to eliminate all the API keys
+that are necessary for a fully functioning system. A more aggressive view
+would be to treat ZFIN_SERVER as a 100% source of record, but that would
+introduce some opacity to the process as we wait for candles (for example) to
+populate. This could be solved on the server by spawning a thread to fetch the
+data, then returning 202 Accepted, which could then be polled client side. Maybe
+this is a better long term approach?
+
+## Low-priority items
+
+The following items are acknowledged but not prioritized. Listed here
+so they don't get lost; pick up opportunistically.
+
+### UX
+
+- **CLI options command UX.** The `options` command auto-expands only
+  the nearest monthly expiration and lists others collapsed. Reconsider
+  the interaction model — e.g. allow specifying an expiration date,
+  showing all monthlies expanded by default, or filtering by strategy
+  (covered calls, spreads).
+- **TUI: toggle to last symbol keybind.** A single-key toggle that
+  flips between the current symbol and the previously selected one
+  (like `cd -` in bash or `Ctrl+^` in vim). Store `last_symbol` on
+  `App`; on symbol change, stash the previous. Useful for
+  eyeball-comparing performance/risk data between two symbols.
+
+### Data quality
+
+- **Fix `enrich` command for international funds.** `deriveMetadata`
+  in `src/commands/enrich.zig` misclassifies international ETFs:
+    1. `geo` uses Alpha Vantage's `Country` field, which is the *fund
+       issuer's* domicile (USA for all US-listed ETFs), not the fund's
+       investment geography. Every US-domiciled international fund gets
+       `geo::US`.
+    2. `asset_class` short-circuits to `"ETF"` when
+       `asset_type == "ETF"`, or falls through to a US-market-cap
+       heuristic that always produces `"US Large Cap"` /
+       `"US Mid Cap"` / `"US Small Cap"`.
+
+  Known misclassified tickers (all came back as
+  `geo::US, asset_class::US Large Cap`):
+    - **FRDM** — Freedom 100 Emerging Markets ETF → should be
+      `geo::Emerging Markets, asset_class::Emerging Markets`
+    - **HFXI** — NYLI FTSE International Equity Currency Neutral ETF
+      → should be
+      `geo::International Developed, asset_class::International Developed`
+    - **IDMO** — Invesco S&P International Developed Momentum ETF →
+      should be
+      `geo::International Developed, asset_class::International Developed`
+    - **IVLU** — iShares MSCI International Developed Value Factor
+      ETF → should be
+      `geo::International Developed, asset_class::International Developed`
+
+  The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography
+  data. Options: use the ETF_PROFILE holdings/country data to infer
+  geography, parse the fund name for keywords ("International",
+  "Emerging", "ex-US"), or accept that `enrich` is a scaffold and
+  emit a `# TODO` comment for ETFs instead of silently misclassifying.
+
+### Options / valuation
+
+- **Per-account covered call adjustment.** `adjustForCoveredCalls` in
+  `valuation.zig` operates on portfolio-wide aggregated allocations.
+  It matches sold calls against total underlying shares across all
+  accounts. This is wrong — calls in one account can only cover
+  shares in that same account. Fixing means restructuring
+  `portfolioSummary`, since `Allocation` is currently
+  account-agnostic. Low priority — naked calls are rare, and calls
+  are typically in the same account as the underlying.
+- **Covered call adjustment O(N*M) loop.** `adjustForCoveredCalls`
+  has a nested loop — for each allocation, it iterates all lots to
+  find matching option contracts. Fine for personal portfolios
+  (<1000 lots). Pre-indexing options by underlying would help if
+  someone had a very large options-heavy portfolio.
+
+### Analysis / correctness
+
+- **Analysis account/asset-class total mismatch.** The "By Account"
+  and "By Tax Type" sections in the analysis command sum to slightly
+  more than "Asset Class" (~0.6% error). Likely a discrepancy between
+  how the lot-level account loop values cash, CDs, or options vs how
+  the asset-class section computes them via `portfolio.totalCash()` /
+  `totalCdFaceValue()`. Per-account values themselves are correct
+  after the price_ratio fix.
+
+### Audit
+
+- **Audit large-lot threshold tuning.** `src/commands/audit.zig` uses
+  `audit_large_lot_threshold: f64 = 10_000.0` as the cutoff for
+  "surface this new lot for confirmation." Revisit if $10k proves too
+  aggressive (ESPP accruals spam the report) or too permissive (large
+  DRIP confirmations slip past). If runtime tuning becomes necessary,
+  a `--large-lot <amount>` flag or a global
+  `audit_large_lot_threshold` field on `accounts.srf` would be
+  reasonable extensions.
+
+### Infra / performance
+
+- **HTTP connection pooling.** Parallel server sync in `loadAllPrices`
+  spawns up to 8 threads, each with its own HTTP connection. Could
+  reuse connections to reduce TCP handshake overhead. Only matters
+  with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
+- **Streaming cache deserialization.** Cache store reads entire files
+  into memory (`readFileAlloc` with 50MB limit). For portfolios with
+  10+ years of daily candles, this could use significant memory.
+  Keep current approach unless memory becomes a real problem.