From baf5e2a86f783933f58bc9f30355fb74fac03266 Mon Sep 17 00:00:00 2001 From: Emil Lerch Date: Tue, 30 Jun 2026 08:32:56 -0700 Subject: [PATCH] update todo and docs --- TODO.md | 88 ---------------------------- docs/explanation/data-providers.md | 22 +++++++ docs/reference/cli/quote.md | 20 ++++++- docs/reference/config/environment.md | 35 +++++++++++ docs/reference/providers.md | 37 +++++++----- docs/reference/tui.md | 21 ++++++- 6 files changed, 118 insertions(+), 105 deletions(-) diff --git a/TODO.md b/TODO.md index 7bb4bc8..8ec3c22 100644 --- a/TODO.md +++ b/TODO.md @@ -55,94 +55,6 @@ populate. This could be solved on the server by spawning a thread to fetch the data, then returning 202 Accepted, which could then be polled client side. Maybe this is a better long term approach? -## Support Tiingo paid plan - priority LOW - -zfin hardwires Tiingo to free-tier assumptions: the provider is -constructed with `RateLimiter.perHour(io, 50)` in `Tiingo.init` -(`providers/tiingo.zig`), and the only Tiingo surface is end-of-day -candles plus the corporate actions that ride along in the same -response. A user who pays for a Tiingo plan ($30/mo Power tier and -up) gets nothing for it today - the same 50/hour throttle, the same -EOD-only data. "Support the paid plan" is the umbrella for unlocking -what that subscription actually buys: higher rate limits and -real-time IEX quotes. The two are coupled (real-time polling only -makes sense once the limit is raised), which is why they belong in -one entry rather than two. - -### Tier-aware rate limiting - -The 50/hour cap is hardcoded in `Tiingo.init` -(`RateLimiter.perHour(io, 50)`), and the module docstring bakes in -"Free tier: 50 requests/hour and 1,000 requests/day." Paid tiers -raise both ceilings substantially, so a paying subscriber is being -throttled far below their entitlement. Today only the hourly bucket -is wired; the daily ceiling isn't enforced at all (the docstring -notes it's "far from binding" for bursty EOD usage - real-time -polling changes that calculus). - -Work: - -- Make the Tiingo limits configurable instead of hardcoded. Options: - explicit `ZFIN_TIINGO_RATE_PER_HOUR` (and per-day) numeric env - knobs, or a coarser `ZFIN_TIINGO_PLAN` = `free` (default) | - `power` | ... that maps to known limits. Lean toward explicit - numeric overrides so we aren't chasing Tiingo's published per-tier - numbers as they drift. -- `RateLimiter` already supports arbitrary `init(io, max, window_ns)` - plus `perDay`/`perHour` convenience ctors, so the limiter side is - cheap. Decide whether a paid plan needs both an hourly and a daily - bucket enforced, or whether hourly alone stays sufficient. -- Caveat from `RateLimiter`'s own docs: the bucket is in-memory and - per-process - it caps a single run's burst, not usage across - separate launches in the same window. Sustained real-time polling - (below) makes cross-process usage likelier, so revisit whether - per-process accounting is still good enough. - -### Real-time IEX quotes (was: configurable live-quote provider) - -The TUI refresh key (`r`) values the portfolio with live intraday -quotes via `DataService.loadLiveQuotes` (`service.zig`), which is -Yahoo-only: Yahoo is keyless, consolidated, and stays off every -rate-limit budget, so bursty refresh traffic costs nothing. The -tradeoffs are that Yahoo's unofficial feed is ~15-minute delayed and -"can break without notice." - -Tiingo's IEX endpoint (`/iex/?tickers=A,B,C`) is a strong opt-in -alternative for a paid subscriber: it's genuinely real-time (IEX -last-sale, no 15-min delay), official/keyed, and bills per HTTP -request - one call returns the whole portfolio (confirmed -empirically: a 2-ticker batch decrements the daily quota by 1, not -2). Fields map cleanly: `tngoLast` to price, `prevClose` to -day-change. Caveats: IEX is a single venue (~2-3% of volume), so -`tngoLast` can sit stale between prints on illiquid names, and IEX -doesn't trade mutual funds, so those fall back to the candle close. - -Proposal: a config knob (env var, e.g. `ZFIN_LIVE_QUOTE_PROVIDER` = -`yahoo` (default) | `tiingo`) that switches `loadLiveQuotes` to a new -`Tiingo.fetchQuotes(tickers)` batched call. A paid subscriber who -wants real-time and mashes `r` a lot (or once we add streaming) -reuses their existing `TIINGO_API_KEY` and gets real-time coverage; -everyone else keeps the keyless Yahoo default. - -Implementation notes: - -- `Tiingo.fetchQuotes` returns an array whose order is NOT guaranteed - to match the request order, so key results by the returned - `ticker` field, not by position. -- Live quotes share Tiingo's token bucket, so this is the concrete - reason the tier-aware rate-limiting work above has to land first - (or alongside): a batched quote call is only 1 request, but heavy - `r` use plus candle refreshes draining the free 50/hour bucket is - exactly the contention that raising the paid-tier limit relieves. - -### Websocket streaming (follow-on) - -Tiingo's IEX websocket would be the natural follow-on for true -push-based real-time, replacing poll-on-`r` entirely. Materially -bigger than the REST quote path (persistent connection, reconnect -handling, a background task feeding the TUI) and squarely a -paid-plan feature. Sequence it after the REST quote path proves out. - ## Infra / performance - **HTTP connection pooling.** Parallel server sync in `loadAllPrices` diff --git a/docs/explanation/data-providers.md b/docs/explanation/data-providers.md index 0e0670c..61f0f43 100644 --- a/docs/explanation/data-providers.md +++ b/docs/explanation/data-providers.md @@ -24,6 +24,28 @@ behind one interface -- commands ask for "candles for VTI," not "call Tiingo." That's also what makes the cache and rate limiting uniform across providers. +## Live quotes: free/delayed vs real-time + +Intraday quotes -- the `r` refresh and the opt-in live stream (`L` in +the TUI, or [`quote --live`](../reference/cli/quote.md)) -- can come from +either feed, chosen by +[`ZFIN_LIVE_QUOTE_PROVIDER`](../reference/config/environment.md#live-quotes-and-streaming): + +- **Yahoo** (default): keyless, consolidated, all tiers -- but ~15-min + delayed and unofficial. +- **Tiingo**: real-time IEX last-sale (`tngoLast`). Needs + `TIINGO_API_KEY` but works on **any** Tiingo tier -- the IEX + *reference price* (Tiingo's `thresholdLevel: 6`) needs no exchange + license, unlike raw IEX market data. Caveats: it's a single venue, so + a thinly-traded name can sit stale between prints, and IEX doesn't + carry mutual funds (those fall back to the candle close). + +The same choice drives both the one-shot refresh and the websocket +stream. A stable stream costs a single request to connect, so it's +practical even on Tiingo's free 50/hour tier; the Power tier (10,000/hour) +just adds headroom for heavy use, which is why it's the auto-default +there. + ## Fallback, not single-point-of-failure Where a second source can stand in, zfin uses it: diff --git a/docs/reference/cli/quote.md b/docs/reference/cli/quote.md index 2f23101..0ebdfa2 100644 --- a/docs/reference/cli/quote.md +++ b/docs/reference/cli/quote.md @@ -4,7 +4,7 @@ Show the latest quote for a symbol, with a price chart and recent history. ``` -Usage: zfin quote [--since ] [--export-chart ] +Usage: zfin quote [--since ] [--export-chart ] [--live] ``` Prints the last price, the day's open/high/low, volume, and the @@ -29,6 +29,24 @@ Supports `--export-chart ` to render the chart as a 1920x1080 PNG instead of text (see [export charts](../../guides/offline-and-refresh.md) and the projections page). +## Live streaming (`--live`) + +`zfin quote --live ` opens a websocket and prints the price +continuously (one line per second) instead of a one-shot quote. It runs +until interrupted (Ctrl-C) or killed -- handy with `timeout`: + +```bash +timeout 12 zfin quote --live BTC-USD # watch live ticks for 12s +``` + +The transport follows +[`ZFIN_LIVE_QUOTE_PROVIDER`](../config/environment.md#live-quotes-and-streaming): +free, keyless **Yahoo** by default, or real-time IEX via **Tiingo** when +selected (works on any Tiingo tier with a key). Outside market hours, +US equities stream a closing snapshot and then go quiet; 24/7 symbols +(e.g. `BTC-USD` on Yahoo) keep ticking. The chart / `--since` / +`--export-chart` options don't apply in `--live` mode. + ## Example ```bash diff --git a/docs/reference/config/environment.md b/docs/reference/config/environment.md index 4e06bba..84e3c47 100644 --- a/docs/reference/config/environment.md +++ b/docs/reference/config/environment.md @@ -30,12 +30,47 @@ unavailable. Quotes (Yahoo) and options (CBOE) need no key. See [Data providers and API keys](../providers.md) for signup links and free-tier limits. +## Tiingo plan + +`ZFIN_TIINGO_PLAN` is your Tiingo subscription tier (not a key). It +sizes the hourly request budget for **all** Tiingo usage -- candle +history (the main consumer), real-time IEX quotes, and the live stream +-- so a paying subscriber isn't throttled to free-tier limits. + +| Variable | Values | Default | Effect | +|--------------------|------------------|---------|-----------------------------------------------------------------| +| `ZFIN_TIINGO_PLAN` | `free` / `power` | `free` | Hourly Tiingo request cap: free = 50/hour, power = 10,000/hour. | + +An unrecognized value logs a warning and falls back to `free`. + ## Contact email | Variable | Used for | |-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `ZFIN_USER_EMAIL` | The contact address SEC EDGAR requires in its `User-Agent` header. Enables ETF profiles and [`zfin enrich`](../cli/enrich.md). Not a key -- just your email. Without it, ETF profiles and enrichment are unavailable. | +## Live quotes and streaming + +`ZFIN_LIVE_QUOTE_PROVIDER` chooses where intraday quotes come from -- +both the `r`/F5 refresh and the opt-in live stream (the `L` toggle on +the TUI's Quote and Portfolio tabs, or [`quote --live`](../cli/quote.md)). + +| Variable | Values | Default | Purpose | +|----------------------------|--------------------|-----------|--------------------------------------------------------------------------------------------------------------------------| +| `ZFIN_LIVE_QUOTE_PROVIDER` | `yahoo` / `tiingo` | (derived) | Provider for live quotes + streaming. Unset: derived from the [Tiingo plan](#tiingo-plan) (Power -> Tiingo, else Yahoo). | + +- **Yahoo** (default): keyless, works on all tiers, but ~15-minute + delayed and unofficial (no guarantees). +- **Tiingo**: real-time IEX last-sale (the `tngoLast` reference price); + requires `TIINGO_API_KEY`. The IEX reference feed works on **any** + Tiingo tier, free included -- set `ZFIN_LIVE_QUOTE_PROVIDER=tiingo` to + use it. It's the *automatic* default only on the Power plan, which has + the request headroom for heavy refresh/streaming use; on the free tier + it's opt-in (each websocket connect spends one of the 50/hour, but a + stable stream then costs nothing). + +An unrecognized value logs a warning and falls back to the default. + ## Paths and directories | Variable | Default | Purpose | diff --git a/docs/reference/providers.md b/docs/reference/providers.md index 88eb1af..eb947a6 100644 --- a/docs/reference/providers.md +++ b/docs/reference/providers.md @@ -10,18 +10,19 @@ how keys are configured see ## Summary -| Data | Provider | Auth | Free-tier limit | Cache TTL | -|-----------------------|------------------|-------------------------------|----------------------------|------------| -| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | ~24h | -| Real-time quotes | Yahoo | none | unofficial | never | -| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never | -| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | -| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | -| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour | -| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days | -| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days | -| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived | -| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite | +| Data | Provider | Auth | Free-tier limit | Cache TTL | +|-----------------------|------------------|-------------------------------|------------------------------------------------|------------| +| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h | +| Real-time quotes | Yahoo | none | unofficial | never | +| Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never | +| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never | +| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | +| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | +| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour | +| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days | +| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days | +| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived | +| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite | ## Where to get a key @@ -40,8 +41,16 @@ how keys are configured see funds. Candles are fetched from a fixed 2000-01-01 start so the cache supports long `--as-of` projections. Its price series also carries per-row dividend/split data that zfin merges into the Polygon view. -- **Yahoo** -- primary, keyless quote source, and the candle fallback - when Tiingo is unavailable. + Also serves **real-time IEX quotes** (the `tngoLast` reference price) + for the `r` refresh and the live stream when + [`ZFIN_LIVE_QUOTE_PROVIDER=tiingo`](config/environment.md#live-quotes-and-streaming); + the IEX reference feed works on any tier (no exchange license). Rate + limits are tier-aware via `ZFIN_TIINGO_PLAN` (free 50/hour, Power + 10,000/hour) -- one batched quote call or one stream connect is a + single request. +- **Yahoo** -- primary, keyless quote source, the candle fallback when + Tiingo is unavailable, and the default live-quote/stream feed + (~15-min delayed, unofficial). - **TwelveData** -- quote fallback only. It is no longer used for candles (its split-adjusted closes proved unreliable for return math). - **Polygon** -- primary for dividends and splits, including diff --git a/docs/reference/tui.md b/docs/reference/tui.md index 8b5e7ba..721446e 100644 --- a/docs/reference/tui.md +++ b/docs/reference/tui.md @@ -62,8 +62,25 @@ current bindings). The global defaults: | `page_down` / `Ctrl-f`, `page_up` / `Ctrl-b` | Page down / up | Individual tabs add their own actions (e.g. the Projections tab's `d` -to set an as-of date and `o` to toggle the overlay). The `?` overlay -and `zfin i --default-keys` list every binding, global and tab-scoped. +to set an as-of date and `o` to toggle the overlay, or the Quote and +Portfolio tabs' `L` to toggle live streaming). The `?` overlay and +`zfin i --default-keys` list every binding, global and tab-scoped. + +## Live streaming + +On the **Quote** and **Portfolio** tabs, press `L` to toggle a live +price stream (off by default). While on, prices update in place every +~100ms: the Quote tab tracks the selected symbol; the Portfolio tab +re-values the whole position table and totals as held symbols tick. +Press `L` again (or switch tabs) to stop. + +The feed follows +[`ZFIN_LIVE_QUOTE_PROVIDER`](config/environment.md#live-quotes-and-streaming): +keyless **Yahoo** (default, ~15-min delayed) or real-time IEX via +**Tiingo** (any keyed tier). Streaming covers what each feed prices +live -- stocks/ETFs (and crypto on Yahoo); mutual funds, CDs, options, +and cash hold their last value. The historical-return blocks stay as of +the last full refresh. ## Customizing