zfin/TODO.md

# Future Work

No work here is blocking - we're in a good state. Items below are
ordered roughly by priority within each section. Priority labels
(`HIGH` / `MEDIUM` / `LOW`) mark items that deserve explicit
ranking; unlabeled items are "someday, if the mood strikes."

## Investigate: detailed 401(k) contributions data source

Found a more detailed contributions screen on at least one
employer-sponsored 401(k) provider portal - distinct from the
standard positions/holdings view we already pull from. Worth
investigating whether this unlocks better attribution than what
we get from the positions CSV alone, and whether other 401(k)
providers expose similar screens.

Open questions to answer when picking this up:

- Which screen specifically (path / URL within the portal)? Is there
  an export option, or is it view-only / scrape territory?
- What fields does it expose (employee pre-tax, employer match,
  after-tax / mega-backdoor, by-pay-period dates, per-fund
  allocations)?
- Refresh cadence - per-paycheck, daily, on-demand?
- Can it be auto-discovered like the existing audit CSVs, or
  is it manual-entry territory?

If the export is structured and recurring, this could feed a
401(k)-specific contributions classifier that bypasses the lot-diff
heuristic for that account, similar to how `cash_is_contribution`
opts ESPP/HSA accounts into cash-based attribution.

Related: ESPP-style accrual blind spot in the "Audit: manual-check
accounts mechanism" section above.

## On-demand server-side fetch for new symbols

Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure
cache reads - they 404 if the data isn't already on disk. New symbols only get
populated when added to the portfolio and picked up by the next cron refresh.

Consider: on a cache miss, instead of blocking the HTTP response with a
multi-second provider fetch, kick off an async background fetch (or just
auto-add the symbol to the portfolio) and return 404 as usual. The next
request - or the next cron run - would then have the data. This gives
"instant-ish gratification" for new symbols without the downsides of
synchronous fetch-on-miss (latency, rate limit contention, unbounded cache
growth from arbitrary tickers).

Note that this process doesn't do anything to eliminate all the API keys
that are necessary for a fully functioning system. A more aggressive view
would be to treat ZFIN_SERVER as a 100% source of record, but that would
introduce some opacity to the process as we wait for candles (for example) to
populate. This could be solved on the server by spawning a thread to fetch the
data, then returning 202 Accepted, which could then be polled client side. Maybe
this is a better long term approach?

## Support Tiingo paid plan - priority LOW

zfin hardwires Tiingo to free-tier assumptions: the provider is
constructed with `RateLimiter.perHour(io, 50)` in `Tiingo.init`
(`providers/tiingo.zig`), and the only Tiingo surface is end-of-day
candles plus the corporate actions that ride along in the same
response. A user who pays for a Tiingo plan ($30/mo Power tier and
up) gets nothing for it today - the same 50/hour throttle, the same
EOD-only data. "Support the paid plan" is the umbrella for unlocking
what that subscription actually buys: higher rate limits and
real-time IEX quotes. The two are coupled (real-time polling only
makes sense once the limit is raised), which is why they belong in
one entry rather than two.

### Tier-aware rate limiting

The 50/hour cap is hardcoded in `Tiingo.init`
(`RateLimiter.perHour(io, 50)`), and the module docstring bakes in
"Free tier: 50 requests/hour and 1,000 requests/day." Paid tiers
raise both ceilings substantially, so a paying subscriber is being
throttled far below their entitlement. Today only the hourly bucket
is wired; the daily ceiling isn't enforced at all (the docstring
notes it's "far from binding" for bursty EOD usage - real-time
polling changes that calculus).

Work:

- Make the Tiingo limits configurable instead of hardcoded. Options:
  explicit `ZFIN_TIINGO_RATE_PER_HOUR` (and per-day) numeric env
  knobs, or a coarser `ZFIN_TIINGO_PLAN` = `free` (default) |
  `power` | ... that maps to known limits. Lean toward explicit
  numeric overrides so we aren't chasing Tiingo's published per-tier
  numbers as they drift.
- `RateLimiter` already supports arbitrary `init(io, max, window_ns)`
  plus `perDay`/`perHour` convenience ctors, so the limiter side is
  cheap. Decide whether a paid plan needs both an hourly and a daily
  bucket enforced, or whether hourly alone stays sufficient.
- Caveat from `RateLimiter`'s own docs: the bucket is in-memory and
  per-process - it caps a single run's burst, not usage across
  separate launches in the same window. Sustained real-time polling
  (below) makes cross-process usage likelier, so revisit whether
  per-process accounting is still good enough.

### Real-time IEX quotes (was: configurable live-quote provider)

The TUI refresh key (`r`) values the portfolio with live intraday
quotes via `DataService.loadLiveQuotes` (`service.zig`), which is
Yahoo-only: Yahoo is keyless, consolidated, and stays off every
rate-limit budget, so bursty refresh traffic costs nothing. The
tradeoffs are that Yahoo's unofficial feed is ~15-minute delayed and
"can break without notice."

Tiingo's IEX endpoint (`/iex/?tickers=A,B,C`) is a strong opt-in
alternative for a paid subscriber: it's genuinely real-time (IEX
last-sale, no 15-min delay), official/keyed, and bills per HTTP
request - one call returns the whole portfolio (confirmed
empirically: a 2-ticker batch decrements the daily quota by 1, not
2). Fields map cleanly: `tngoLast` to price, `prevClose` to
day-change. Caveats: IEX is a single venue (~2-3% of volume), so
`tngoLast` can sit stale between prints on illiquid names, and IEX
doesn't trade mutual funds, so those fall back to the candle close.

Proposal: a config knob (env var, e.g. `ZFIN_LIVE_QUOTE_PROVIDER` =
`yahoo` (default) | `tiingo`) that switches `loadLiveQuotes` to a new
`Tiingo.fetchQuotes(tickers)` batched call. A paid subscriber who
wants real-time and mashes `r` a lot (or once we add streaming)
reuses their existing `TIINGO_API_KEY` and gets real-time coverage;
everyone else keeps the keyless Yahoo default.

Implementation notes:

- `Tiingo.fetchQuotes` returns an array whose order is NOT guaranteed
  to match the request order, so key results by the returned
  `ticker` field, not by position.
- Live quotes share Tiingo's token bucket, so this is the concrete
  reason the tier-aware rate-limiting work above has to land first
  (or alongside): a batched quote call is only 1 request, but heavy
  `r` use plus candle refreshes draining the free 50/hour bucket is
  exactly the contention that raising the paid-tier limit relieves.

### Websocket streaming (follow-on)

Tiingo's IEX websocket would be the natural follow-on for true
push-based real-time, replacing poll-on-`r` entirely. Materially
bigger than the REST quote path (persistent connection, reconnect
handling, a background task feeding the TUI) and squarely a
paid-plan feature. Sequence it after the REST quote path proves out.

## Infra / performance

- **HTTP connection pooling.** Parallel server sync in `loadAllPrices`
  spawns up to 8 threads, each with its own HTTP connection. Could
  reuse connections to reduce TCP handshake overhead. Only matters
  with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
- **Streaming cache deserialization.** Cache store reads entire files
  into memory (`readFileAlloc` with 50MB limit). For portfolios with
  10+ years of daily candles, this could use significant memory.
  Keep current approach unless memory becomes a real problem.