From b75381a9bd9fd69290a97e166c738439d8b566fa Mon Sep 17 00:00:00 2001 From: Emil Lerch Date: Fri, 8 May 2026 15:02:44 -0700 Subject: [PATCH] update todo.md --- TODO.md | 355 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 179 insertions(+), 176 deletions(-) diff --git a/TODO.md b/TODO.md index 92854aa..d1f48a6 100644 --- a/TODO.md +++ b/TODO.md @@ -1,19 +1,33 @@ # Future Work +No work here is blocking — we're in a good state. Items below are +ordered roughly by priority within each section. Priority labels +(`HIGH` / `MEDIUM` / `LOW`) mark items that deserve explicit +ranking; unlabeled items are "someday, if the mood strikes." + +**Next up:** configurable benchmark symbols (low-effort win) and the +manual-check accounts mechanism (medium effort, real user value). + ## Projections: future enhancements -- Configurable return cap per position (default: none; cap outliers like NVDA) +- **Configurable benchmark symbols — priority MEDIUM.** Currently + hardcoded SPY + AGG. Route through `projections.srf` as a + `type::config,benchmark::SYMBOL` record (or similar). Low effort. +- **Configurable return cap per position — priority MEDIUM.** + Default: none; cap outliers like NVDA. Should route through + `projections.srf` cleanly. +- **"Not Retired Yet" accumulation-phase mode — priority HIGH.** + Accumulation phase with contributions before retirement. + Separate from life events — the simulation has two distinct phases: + accumulation (base spending = 0, contributions applied) and + distribution (searched-for withdrawal + spending model). Config: + `retirement_age::60` or `retirement_in::10`, plus + `annual_contribution::100000`. Safe withdrawal search only applies + to the distribution phase. Chart shows portfolio growing during + accumulation, peaking at retirement, then drawing down. - Configurable MIN period selection (currently 3Y/5Y/10Y, exclude 1Y) -- "Not Retired Yet" mode: accumulation phase with contributions before retirement. - Separate from life events — the simulation has two distinct phases: accumulation - (base spending = 0, contributions applied) and distribution (searched-for withdrawal - + spending model). Config: `retirement_age::60` or `retirement_in::10`, plus - `annual_contribution::100000`. Safe withdrawal search only applies to the - distribution phase. Chart shows portfolio growing during accumulation, peaking - at retirement, then drawing down. - Multiple spending models: flat (current), decreasing (1-2% real annual decrease, Blanchett "spending smile"). Late-life healthcare better modeled as a life event. -- Configurable benchmark symbols (currently hardcoded SPY + AGG) - Unclassified position handling in allocation split (warn user) - Historical projection comparison: re-run projections from any past snapshot date, overlay actual portfolio trajectory from subsequent snapshots onto the projected @@ -39,129 +53,7 @@ the current versions of both; historical git-tracked versions could be checked out and loaded instead. Edge case for now. -## Analysis account/asset-class total mismatch - -The "By Account" and "By Tax Type" sections in the analysis command sum to slightly -more than "Asset Class" (~0.6% error). Likely a discrepancy between how the lot-level -account loop values cash, CDs, or options vs how the asset-class section computes them -via `portfolio.totalCash()` / `totalCdFaceValue()`. Low priority — the per-account -values themselves are correct after the price_ratio fix. - -## Upgrade to 0.16.0 - -Pending dependencies: - -* SRF: complete (use 0.15.2 tag if needed) -* VAxis: work seems to be in this PR: https://github.com/rockorager/libvaxis/pull/316 -* z2d: 0.16.0 branch exists, but not sure if this is fully compatible with last commit in March - -## CLI options command UX - -The `options` command auto-expands only the nearest monthly expiration and -lists others collapsed. Reconsider the interaction model — e.g. allow -specifying an expiration date, showing all monthlies expanded by default, -or filtering by strategy (covered calls, spreads). - -## TUI: toggle to last symbol keybind - -Add a single-key toggle that flips between the current symbol and the -previously selected one (like `cd -` in bash or `Ctrl+^` in vim). Store -`last_symbol` on `App`; on symbol change, stash the previous. The toggle -key swaps current and last. Works on any tab — particularly useful for -eyeball-comparing performance/risk data between two symbols. - -## Fix `enrich` command for international funds - -`deriveMetadata` in `src/commands/enrich.zig` misclassifies international ETFs: - -1. **`geo`** uses Alpha Vantage's `Country` field, which is the *fund issuer's* - domicile (USA for all US-listed ETFs), not the fund's investment geography. - Every US-domiciled international fund gets `geo::US`. - -2. **`asset_class`** short-circuits to `"ETF"` when `asset_type == "ETF"`, or - falls through to a US-market-cap heuristic that always produces - `"US Large Cap"` / `"US Mid Cap"` / `"US Small Cap"`. - -Known misclassified tickers (all came back as `geo::US, asset_class::US Large Cap`): -- **FRDM** — Freedom 100 Emerging Markets ETF → should be `geo::Emerging Markets, asset_class::Emerging Markets` -- **HFXI** — NYLI FTSE International Equity Currency Neutral ETF → should be `geo::International Developed, asset_class::International Developed` -- **IDMO** — Invesco S&P International Developed Momentum ETF → should be `geo::International Developed, asset_class::International Developed` -- **IVLU** — iShares MSCI International Developed Value Factor ETF → should be `geo::International Developed, asset_class::International Developed` - -The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography data. -Options: use the ETF_PROFILE holdings/country data to infer geography, parse -the fund name for keywords ("International", "Emerging", "ex-US"), or accept -that `enrich` is a scaffold and emit a `# TODO` comment for ETFs instead of -silently misclassifying. - -## Market-aware cache TTL for daily candles - -Daily candle TTL is currently 23h45m, but candle data only becomes meaningful -after the market close. Investigate keying the cache freshness to ~4:30 PM -Eastern rather than a rolling window. This would avoid unnecessary refetches -during the trading day and ensure a fetch shortly after close gets fresh data. -Probably alleviated by the cron job approach. - -## On-demand server-side fetch for new symbols - -Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure -cache reads — they 404 if the data isn't already on disk. New symbols only get -populated when added to the portfolio and picked up by the next cron refresh. - -Consider: on a cache miss, instead of blocking the HTTP response with a -multi-second provider fetch, kick off an async background fetch (or just -auto-add the symbol to the portfolio) and return 404 as usual. The next -request — or the next cron run — would then have the data. This gives -"instant-ish gratification" for new symbols without the downsides of -synchronous fetch-on-miss (latency, rate limit contention, unbounded cache -growth from arbitrary tickers). - -Note that this process doesn't do anything to eliminate all the API keys -that are necessary for a fully functioning system. A more aggressive view -would be to treat ZFIN_SERVER as a 100% source of record, but that would -introduce some opacity to the process as we wait for candles (for example) to -populate. This could be solved on the server by spawning a thread to fetch the -data, then returning 202 Accepted, which could then be polled client side. Maybe -this is a better long term approach? - -## Per-account covered call adjustment - -`adjustForCoveredCalls` in `valuation.zig` operates on portfolio-wide aggregated -allocations. It matches sold calls against total underlying shares across all -accounts. This is wrong — calls in one account can only cover shares in that -same account. If NVDA calls are sold in Emil IRA, they shouldn't cap NVDA -shares held in Joint trust. - -Fixing this means restructuring `portfolioSummary`, since `Allocation` is -currently account-agnostic. Approach: compute per-account reductions using -`positionsForAccount` + account-filtered option lots, then sum into -portfolio-wide reductions. Each account's reduction capped by that account's -shares, not the global total. - -Low priority — naked calls are rare, and calls are typically in the same -account as the underlying. - -## Covered call adjustment optimization - -`adjustForCoveredCalls` has a nested loop — for each allocation, it iterates -all lots to find matching option contracts. O(N*M) is fine for personal -portfolios (<1000 lots). Pre-indexing options by underlying would help if -someone had a very large options-heavy portfolio. - -## HTTP connection pooling - -Parallel server sync in `loadAllPrices` spawns up to 8 threads, each with its -own HTTP connection. Could reuse connections to reduce TCP handshake overhead. -Only matters with very large portfolios (100+ symbols) hitting ZFIN_SERVER. -8 concurrent connections is fine for now. - -## Streaming cache deserialization - -Cache store reads entire files into memory (`readFileAlloc` with 50MB limit). -For portfolios with 10+ years of daily candles, this could use significant -memory. Keep current approach unless memory becomes a real problem. - -## Audit: manual-check accounts mechanism (NYL, Kelly's ESPP, etc.) +## Audit: manual-check accounts mechanism (NYL, Kelly's ESPP, etc.) — priority HIGH Some accounts/positions can't be reconciled from broker CSVs and need a human-in-the-loop reminder at the audit step. Examples: @@ -208,25 +100,6 @@ in the "Contributions diff" TODO below — option C there (per-account `cash_is_contribution`) would make manually-entered ESPP cash additions count correctly. -## Audit: tax-loss account persistent discrepancy (~-$897.59) - -**Status (2026-05-02):** Resolved via the `direct_indexing::true` -flag in `accounts.srf`. When set on an account: - -- Audit now emits a suggested `price_ratio` adjustment even for - lots with ratio == 1.0 (previously skipped). The Schwab-summary - path picks up direct-indexing accounts with a single stock lot - and computes the suggested ratio from the account-level value - delta. Copy the suggestion into `portfolio.srf` and next audit - is clean. -- Contributions swallows sub-1% share drift on lots in the account - as tracking-error noise instead of emitting `rollup_delta` / - `drip_negative`. Real contributions (>1% of account value) still - surface. - -See `src/analytics/analysis.zig` `AccountTaxEntry.direct_indexing` -and `src/commands/audit.zig` `displaySchwabSummaryRatioSuggestions`. - ## In-kind transfer support (`type::in_kind`) — priority MEDIUM `transaction_log.srf` parses `type::in_kind` records but the @@ -249,27 +122,14 @@ Driver: when the user starts moving positions between accounts directly (e.g. Roth conversion of already-held shares, 401k → rollover IRA in-kind) rather than liquidating and re-buying. -## Audit large-lot threshold tuning — priority LOW +## Torn SRF files from server sync (root cause unknown) -`src/commands/audit.zig` uses `audit_large_lot_threshold: f64 = -10_000.0` as the cutoff for "surface this new lot for confirmation." -The value is a judgment call; revisit if: +**Status:** Root cause still unidentified. We have mitigations and +diagnostics in place that keep torn responses from corrupting the +cache, but we don't yet know *why* responses arrive torn. Until we +have a root cause, this is not resolved — it's mitigated. -- $10k proves too aggressive (weekly payroll ESPP accruals near - the limit spam the report), -- $10k proves too permissive (large DRIP confirmations or CD - maturities slip past uncalled), -- the user wants per-account thresholds (e.g. $25k for - high-turnover brokerage, $5k for IRAs). - -If runtime tuning becomes necessary, either a `--large-lot -` flag on `zfin audit` or a global `audit_large_lot_threshold` -field on `accounts.srf`-the-file would be reasonable extensions. -Until then the constant is the single knob. - -## Torn SRF files from server sync (recurring bug) - -**Status:** Multi-layer defense in place. +Mitigations landed so far: - `syncFromServer` (`src/service.zig`) validates responses via `cache.Store.looksCompleteSrf` before `writeRaw`. Torn HTTP bodies @@ -281,17 +141,160 @@ Until then the constant is the single knob. - Read-path self-heal: on SRF parse failure during read, the cache entry is invalidated so a subsequent refresh can repair without user intervention. -- Diagnostics: richer error capture around the sync path to pinpoint - where torn responses originate (HTTP transit has been the dominant - source so far). +- Diagnostics: richer error capture around the sync path. So far, + HTTP transit is the dominant source of torn responses — but that's + an observation, not a root cause. **Remaining work:** -- Continue monitoring — if torn responses persist despite the etag - retry, investigate lower-level transport issues (proxy, keepalive, - partial reads on the server side). +- Identify root cause. Candidates to investigate: proxy/load-balancer + behavior, HTTP keepalive reuse, partial reads on the server side, + client-side buffer handling. The etag retry tells us whether the + problem is per-request or persistent; dig into the diagnostics + output when the next occurrence is captured. +- Once root cause is known, decide whether the current mitigations + are sufficient or whether a targeted fix is needed. The + mitigations may end up being the whole answer, but we can't + conclude that without understanding the underlying cause. (Content-Length validation was considered and rejected: once the server starts compressing response bodies, Content-Length reflects the compressed byte count, not the decoded payload, so it's not a reliable integrity check.) + +## Upgrade to 0.16.0 + +Pending dependencies: + +* SRF: complete (use 0.15.2 tag if needed) +* VAxis: work seems to be in this PR: https://github.com/rockorager/libvaxis/pull/316 +* z2d: implemented in 0.11.0 of z2d + +## Market-aware cache TTL for daily candles + +Daily candle TTL is currently 23h45m, but candle data only becomes meaningful +after the market close. Investigate keying the cache freshness to ~4:30 PM +Eastern rather than a rolling window. This would avoid unnecessary refetches +during the trading day and ensure a fetch shortly after close gets fresh data. +Probably alleviated by the cron job approach. + +## On-demand server-side fetch for new symbols + +Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure +cache reads — they 404 if the data isn't already on disk. New symbols only get +populated when added to the portfolio and picked up by the next cron refresh. + +Consider: on a cache miss, instead of blocking the HTTP response with a +multi-second provider fetch, kick off an async background fetch (or just +auto-add the symbol to the portfolio) and return 404 as usual. The next +request — or the next cron run — would then have the data. This gives +"instant-ish gratification" for new symbols without the downsides of +synchronous fetch-on-miss (latency, rate limit contention, unbounded cache +growth from arbitrary tickers). + +Note that this process doesn't do anything to eliminate all the API keys +that are necessary for a fully functioning system. A more aggressive view +would be to treat ZFIN_SERVER as a 100% source of record, but that would +introduce some opacity to the process as we wait for candles (for example) to +populate. This could be solved on the server by spawning a thread to fetch the +data, then returning 202 Accepted, which could then be polled client side. Maybe +this is a better long term approach? + +## Low-priority items + +The following items are acknowledged but not prioritized. Listed here +so they don't get lost; pick up opportunistically. + +### UX + +- **CLI options command UX.** The `options` command auto-expands only + the nearest monthly expiration and lists others collapsed. Reconsider + the interaction model — e.g. allow specifying an expiration date, + showing all monthlies expanded by default, or filtering by strategy + (covered calls, spreads). +- **TUI: toggle to last symbol keybind.** A single-key toggle that + flips between the current symbol and the previously selected one + (like `cd -` in bash or `Ctrl+^` in vim). Store `last_symbol` on + `App`; on symbol change, stash the previous. Useful for + eyeball-comparing performance/risk data between two symbols. + +### Data quality + +- **Fix `enrich` command for international funds.** `deriveMetadata` + in `src/commands/enrich.zig` misclassifies international ETFs: + 1. `geo` uses Alpha Vantage's `Country` field, which is the *fund + issuer's* domicile (USA for all US-listed ETFs), not the fund's + investment geography. Every US-domiciled international fund gets + `geo::US`. + 2. `asset_class` short-circuits to `"ETF"` when + `asset_type == "ETF"`, or falls through to a US-market-cap + heuristic that always produces `"US Large Cap"` / + `"US Mid Cap"` / `"US Small Cap"`. + + Known misclassified tickers (all came back as + `geo::US, asset_class::US Large Cap`): + - **FRDM** — Freedom 100 Emerging Markets ETF → should be + `geo::Emerging Markets, asset_class::Emerging Markets` + - **HFXI** — NYLI FTSE International Equity Currency Neutral ETF + → should be + `geo::International Developed, asset_class::International Developed` + - **IDMO** — Invesco S&P International Developed Momentum ETF → + should be + `geo::International Developed, asset_class::International Developed` + - **IVLU** — iShares MSCI International Developed Value Factor + ETF → should be + `geo::International Developed, asset_class::International Developed` + + The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography + data. Options: use the ETF_PROFILE holdings/country data to infer + geography, parse the fund name for keywords ("International", + "Emerging", "ex-US"), or accept that `enrich` is a scaffold and + emit a `# TODO` comment for ETFs instead of silently misclassifying. + +### Options / valuation + +- **Per-account covered call adjustment.** `adjustForCoveredCalls` in + `valuation.zig` operates on portfolio-wide aggregated allocations. + It matches sold calls against total underlying shares across all + accounts. This is wrong — calls in one account can only cover + shares in that same account. Fixing means restructuring + `portfolioSummary`, since `Allocation` is currently + account-agnostic. Low priority — naked calls are rare, and calls + are typically in the same account as the underlying. +- **Covered call adjustment O(N*M) loop.** `adjustForCoveredCalls` + has a nested loop — for each allocation, it iterates all lots to + find matching option contracts. Fine for personal portfolios + (<1000 lots). Pre-indexing options by underlying would help if + someone had a very large options-heavy portfolio. + +### Analysis / correctness + +- **Analysis account/asset-class total mismatch.** The "By Account" + and "By Tax Type" sections in the analysis command sum to slightly + more than "Asset Class" (~0.6% error). Likely a discrepancy between + how the lot-level account loop values cash, CDs, or options vs how + the asset-class section computes them via `portfolio.totalCash()` / + `totalCdFaceValue()`. Per-account values themselves are correct + after the price_ratio fix. + +### Audit + +- **Audit large-lot threshold tuning.** `src/commands/audit.zig` uses + `audit_large_lot_threshold: f64 = 10_000.0` as the cutoff for + "surface this new lot for confirmation." Revisit if $10k proves too + aggressive (ESPP accruals spam the report) or too permissive (large + DRIP confirmations slip past). If runtime tuning becomes necessary, + a `--large-lot ` flag or a global + `audit_large_lot_threshold` field on `accounts.srf` would be + reasonable extensions. + +### Infra / performance + +- **HTTP connection pooling.** Parallel server sync in `loadAllPrices` + spawns up to 8 threads, each with its own HTTP connection. Could + reuse connections to reduce TCP handshake overhead. Only matters + with very large portfolios (100+ symbols) hitting ZFIN_SERVER. +- **Streaming cache deserialization.** Cache store reads entire files + into memory (`readFileAlloc` with 50MB limit). For portfolios with + 10+ years of daily candles, this could use significant memory. + Keep current approach unless memory becomes a real problem.