zfin/TODO.md

Future Work

No work here is blocking - we're in a good state. Items below are ordered roughly by priority within each section. Priority labels (HIGH / MEDIUM / LOW) mark items that deserve explicit ranking; unlabeled items are "someday, if the mood strikes."

Investigate: detailed 401(k) contributions data source

Found a more detailed contributions screen on at least one employer-sponsored 401(k) provider portal - distinct from the standard positions/holdings view we already pull from. Worth investigating whether this unlocks better attribution than what we get from the positions CSV alone, and whether other 401(k) providers expose similar screens.

Open questions to answer when picking this up:

• Which screen specifically (path / URL within the portal)? Is there an export option, or is it view-only / scrape territory?
• What fields does it expose (employee pre-tax, employer match, after-tax / mega-backdoor, by-pay-period dates, per-fund allocations)?
• Refresh cadence - per-paycheck, daily, on-demand?
• Can it be auto-discovered like the existing audit CSVs, or is it manual-entry territory?

If the export is structured and recurring, this could feed a 401(k)-specific contributions classifier that bypasses the lot-diff heuristic for that account, similar to how cash_is_contribution opts ESPP/HSA accounts into cash-based attribution.

Related: ESPP-style accrual blind spot in the "Audit: manual-check accounts mechanism" section above.

On-demand server-side fetch for new symbols

Currently the server's SRF endpoints (/candles, /dividends, etc.) are pure cache reads - they 404 if the data isn't already on disk. New symbols only get populated when added to the portfolio and picked up by the next cron refresh.

Consider: on a cache miss, instead of blocking the HTTP response with a multi-second provider fetch, kick off an async background fetch (or just auto-add the symbol to the portfolio) and return 404 as usual. The next request - or the next cron run - would then have the data. This gives "instant-ish gratification" for new symbols without the downsides of synchronous fetch-on-miss (latency, rate limit contention, unbounded cache growth from arbitrary tickers).

Note that this process doesn't do anything to eliminate all the API keys that are necessary for a fully functioning system. A more aggressive view would be to treat ZFIN_SERVER as a 100% source of record, but that would introduce some opacity to the process as we wait for candles (for example) to populate. This could be solved on the server by spawning a thread to fetch the data, then returning 202 Accepted, which could then be polled client side. Maybe this is a better long term approach?

Support Tiingo paid plan - priority LOW

zfin hardwires Tiingo to free-tier assumptions: the provider is constructed with RateLimiter.perHour(io, 50) in Tiingo.init (providers/tiingo.zig), and the only Tiingo surface is end-of-day candles plus the corporate actions that ride along in the same response. A user who pays for a Tiingo plan ($30/mo Power tier and up) gets nothing for it today - the same 50/hour throttle, the same EOD-only data. "Support the paid plan" is the umbrella for unlocking what that subscription actually buys: higher rate limits and real-time IEX quotes. The two are coupled (real-time polling only makes sense once the limit is raised), which is why they belong in one entry rather than two.

Tier-aware rate limiting

The 50/hour cap is hardcoded in Tiingo.init (RateLimiter.perHour(io, 50)), and the module docstring bakes in "Free tier: 50 requests/hour and 1,000 requests/day." Paid tiers raise both ceilings substantially, so a paying subscriber is being throttled far below their entitlement. Today only the hourly bucket is wired; the daily ceiling isn't enforced at all (the docstring notes it's "far from binding" for bursty EOD usage - real-time polling changes that calculus).

Work:

• Make the Tiingo limits configurable instead of hardcoded. Options: explicit ZFIN_TIINGO_RATE_PER_HOUR (and per-day) numeric env knobs, or a coarser ZFIN_TIINGO_PLAN = free (default) | power | ... that maps to known limits. Lean toward explicit numeric overrides so we aren't chasing Tiingo's published per-tier numbers as they drift.
• RateLimiter already supports arbitrary init(io, max, window_ns) plus perDay/perHour convenience ctors, so the limiter side is cheap. Decide whether a paid plan needs both an hourly and a daily bucket enforced, or whether hourly alone stays sufficient.
• Caveat from RateLimiter's own docs: the bucket is in-memory and per-process - it caps a single run's burst, not usage across separate launches in the same window. Sustained real-time polling (below) makes cross-process usage likelier, so revisit whether per-process accounting is still good enough.

Real-time IEX quotes (was: configurable live-quote provider)

The TUI refresh key (r) values the portfolio with live intraday quotes via DataService.loadLiveQuotes (service.zig), which is Yahoo-only: Yahoo is keyless, consolidated, and stays off every rate-limit budget, so bursty refresh traffic costs nothing. The tradeoffs are that Yahoo's unofficial feed is ~15-minute delayed and "can break without notice."

Tiingo's IEX endpoint (/iex/?tickers=A,B,C) is a strong opt-in alternative for a paid subscriber: it's genuinely real-time (IEX last-sale, no 15-min delay), official/keyed, and bills per HTTP request - one call returns the whole portfolio (confirmed empirically: a 2-ticker batch decrements the daily quota by 1, not 2). Fields map cleanly: tngoLast to price, prevClose to day-change. Caveats: IEX is a single venue (~2-3% of volume), so tngoLast can sit stale between prints on illiquid names, and IEX doesn't trade mutual funds, so those fall back to the candle close.

Proposal: a config knob (env var, e.g. ZFIN_LIVE_QUOTE_PROVIDER = yahoo (default) | tiingo) that switches loadLiveQuotes to a new Tiingo.fetchQuotes(tickers) batched call. A paid subscriber who wants real-time and mashes r a lot (or once we add streaming) reuses their existing TIINGO_API_KEY and gets real-time coverage; everyone else keeps the keyless Yahoo default.

Implementation notes:

• Tiingo.fetchQuotes returns an array whose order is NOT guaranteed to match the request order, so key results by the returned ticker field, not by position.
• Live quotes share Tiingo's token bucket, so this is the concrete reason the tier-aware rate-limiting work above has to land first (or alongside): a batched quote call is only 1 request, but heavy r use plus candle refreshes draining the free 50/hour bucket is exactly the contention that raising the paid-tier limit relieves.

Websocket streaming (follow-on)

Tiingo's IEX websocket would be the natural follow-on for true push-based real-time, replacing poll-on-r entirely. Materially bigger than the REST quote path (persistent connection, reconnect handling, a background task feeding the TUI) and squarely a paid-plan feature. Sequence it after the REST quote path proves out.

Infra / performance

• HTTP connection pooling. Parallel server sync in loadAllPrices spawns up to 8 threads, each with its own HTTP connection. Could reuse connections to reduce TCP handshake overhead. Only matters with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
• Streaming cache deserialization. Cache store reads entire files into memory (readFileAlloc with 50MB limit). For portfolios with 10+ years of daily candles, this could use significant memory. Keep current approach unless memory becomes a real problem.