lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

update todo and docs

Browse source
This commit is contained in:
Emil Lerch 2026-06-30 08:32:56 -07:00
parent bb04d7babd
commit baf5e2a86f
Signed by: lobo
GPG key ID: A7B62D657EF764F8
6 changed files with 118 additions and 105 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
TODO.md
View file

@ -55,94 +55,6 @@ populate. This could be solved on the server by spawning a thread to fetch the
data, then returning 202 Accepted, which could then be polled client side. Maybe data, then returning 202 Accepted, which could then be polled client side. Maybe
this is a better long term approach? this is a better long term approach?
## Support Tiingo paid plan - priority LOW
zfin hardwires Tiingo to free-tier assumptions: the provider is
constructed with `RateLimiter.perHour(io, 50)` in `Tiingo.init`
(`providers/tiingo.zig`), and the only Tiingo surface is end-of-day
candles plus the corporate actions that ride along in the same
response. A user who pays for a Tiingo plan ($30/mo Power tier and
up) gets nothing for it today - the same 50/hour throttle, the same
EOD-only data. "Support the paid plan" is the umbrella for unlocking
what that subscription actually buys: higher rate limits and
real-time IEX quotes. The two are coupled (real-time polling only
makes sense once the limit is raised), which is why they belong in
one entry rather than two.
### Tier-aware rate limiting
The 50/hour cap is hardcoded in `Tiingo.init`
(`RateLimiter.perHour(io, 50)`), and the module docstring bakes in
"Free tier: 50 requests/hour and 1,000 requests/day." Paid tiers
raise both ceilings substantially, so a paying subscriber is being
throttled far below their entitlement. Today only the hourly bucket
is wired; the daily ceiling isn't enforced at all (the docstring
notes it's "far from binding" for bursty EOD usage - real-time
polling changes that calculus).
Work:
- Make the Tiingo limits configurable instead of hardcoded. Options:
explicit `ZFIN_TIINGO_RATE_PER_HOUR` (and per-day) numeric env
knobs, or a coarser `ZFIN_TIINGO_PLAN` = `free` (default) |
`power` | ... that maps to known limits. Lean toward explicit
numeric overrides so we aren't chasing Tiingo's published per-tier
numbers as they drift.
- `RateLimiter` already supports arbitrary `init(io, max, window_ns)`
plus `perDay`/`perHour` convenience ctors, so the limiter side is
cheap. Decide whether a paid plan needs both an hourly and a daily
bucket enforced, or whether hourly alone stays sufficient.
- Caveat from `RateLimiter`'s own docs: the bucket is in-memory and
per-process - it caps a single run's burst, not usage across
separate launches in the same window. Sustained real-time polling
(below) makes cross-process usage likelier, so revisit whether
per-process accounting is still good enough.
### Real-time IEX quotes (was: configurable live-quote provider)
The TUI refresh key (`r`) values the portfolio with live intraday
quotes via `DataService.loadLiveQuotes` (`service.zig`), which is
Yahoo-only: Yahoo is keyless, consolidated, and stays off every
rate-limit budget, so bursty refresh traffic costs nothing. The
tradeoffs are that Yahoo's unofficial feed is ~15-minute delayed and
"can break without notice."
Tiingo's IEX endpoint (`/iex/?tickers=A,B,C`) is a strong opt-in
alternative for a paid subscriber: it's genuinely real-time (IEX
last-sale, no 15-min delay), official/keyed, and bills per HTTP
request - one call returns the whole portfolio (confirmed
empirically: a 2-ticker batch decrements the daily quota by 1, not
2). Fields map cleanly: `tngoLast` to price, `prevClose` to
day-change. Caveats: IEX is a single venue (~2-3% of volume), so
`tngoLast` can sit stale between prints on illiquid names, and IEX
doesn't trade mutual funds, so those fall back to the candle close.
Proposal: a config knob (env var, e.g. `ZFIN_LIVE_QUOTE_PROVIDER` =
`yahoo` (default) | `tiingo`) that switches `loadLiveQuotes` to a new
`Tiingo.fetchQuotes(tickers)` batched call. A paid subscriber who
wants real-time and mashes `r` a lot (or once we add streaming)
reuses their existing `TIINGO_API_KEY` and gets real-time coverage;
everyone else keeps the keyless Yahoo default.
Implementation notes:
- `Tiingo.fetchQuotes` returns an array whose order is NOT guaranteed
to match the request order, so key results by the returned
`ticker` field, not by position.
- Live quotes share Tiingo's token bucket, so this is the concrete
reason the tier-aware rate-limiting work above has to land first
(or alongside): a batched quote call is only 1 request, but heavy
`r` use plus candle refreshes draining the free 50/hour bucket is
exactly the contention that raising the paid-tier limit relieves.
### Websocket streaming (follow-on)
Tiingo's IEX websocket would be the natural follow-on for true
push-based real-time, replacing poll-on-`r` entirely. Materially
bigger than the REST quote path (persistent connection, reconnect
handling, a background task feeding the TUI) and squarely a
paid-plan feature. Sequence it after the REST quote path proves out.
## Infra / performance ## Infra / performance
- **HTTP connection pooling.** Parallel server sync in `loadAllPrices` - **HTTP connection pooling.** Parallel server sync in `loadAllPrices`

22
docs/explanation/data-providers.md
View file

@ -24,6 +24,28 @@ behind one interface -- commands ask for "candles for VTI," not "call
Tiingo." That's also what makes the cache and rate limiting uniform Tiingo." That's also what makes the cache and rate limiting uniform
across providers. across providers.
## Live quotes: free/delayed vs real-time
Intraday quotes -- the `r` refresh and the opt-in live stream (`L` in
the TUI, or [`quote --live`](../reference/cli/quote.md)) -- can come from
either feed, chosen by
[`ZFIN_LIVE_QUOTE_PROVIDER`](../reference/config/environment.md#live-quotes-and-streaming):
- **Yahoo** (default): keyless, consolidated, all tiers -- but ~15-min
delayed and unofficial.
- **Tiingo**: real-time IEX last-sale (`tngoLast`). Needs
`TIINGO_API_KEY` but works on **any** Tiingo tier -- the IEX
*reference price* (Tiingo's `thresholdLevel: 6`) needs no exchange
license, unlike raw IEX market data. Caveats: it's a single venue, so
a thinly-traded name can sit stale between prints, and IEX doesn't
carry mutual funds (those fall back to the candle close).
The same choice drives both the one-shot refresh and the websocket
stream. A stable stream costs a single request to connect, so it's
practical even on Tiingo's free 50/hour tier; the Power tier (10,000/hour)
just adds headroom for heavy use, which is why it's the auto-default
there.
## Fallback, not single-point-of-failure ## Fallback, not single-point-of-failure
Where a second source can stand in, zfin uses it: Where a second source can stand in, zfin uses it:

20
docs/reference/cli/quote.md
View file

@ -4,7 +4,7 @@ Show the latest quote for a symbol, with a price chart and recent
history. history.
``` ```
Usage: zfin quote <SYMBOL> [--since <WHEN>] [--export-chart <PATH>] Usage: zfin quote <SYMBOL> [--since <WHEN>] [--export-chart <PATH>] [--live]
``` ```
Prints the last price, the day's open/high/low, volume, and the Prints the last price, the day's open/high/low, volume, and the
@ -29,6 +29,24 @@ Supports `--export-chart <PATH>` to render the chart as a 1920x1080
PNG instead of text (see [export charts](../../guides/offline-and-refresh.md) PNG instead of text (see [export charts](../../guides/offline-and-refresh.md)
and the projections page). and the projections page).
## Live streaming (`--live`)
`zfin quote --live <SYMBOL>` opens a websocket and prints the price
continuously (one line per second) instead of a one-shot quote. It runs
until interrupted (Ctrl-C) or killed -- handy with `timeout`:
```bash
timeout 12 zfin quote --live BTC-USD # watch live ticks for 12s
```
The transport follows
[`ZFIN_LIVE_QUOTE_PROVIDER`](../config/environment.md#live-quotes-and-streaming):
free, keyless **Yahoo** by default, or real-time IEX via **Tiingo** when
selected (works on any Tiingo tier with a key). Outside market hours,
US equities stream a closing snapshot and then go quiet; 24/7 symbols
(e.g. `BTC-USD` on Yahoo) keep ticking. The chart / `--since` /
`--export-chart` options don't apply in `--live` mode.
## Example ## Example
```bash ```bash

35
docs/reference/config/environment.md
View file

@ -30,12 +30,47 @@ unavailable. Quotes (Yahoo) and options (CBOE) need no key. See
[Data providers and API keys](../providers.md) for signup links and [Data providers and API keys](../providers.md) for signup links and
free-tier limits. free-tier limits.
## Tiingo plan
`ZFIN_TIINGO_PLAN` is your Tiingo subscription tier (not a key). It
sizes the hourly request budget for **all** Tiingo usage -- candle
history (the main consumer), real-time IEX quotes, and the live stream
-- so a paying subscriber isn't throttled to free-tier limits.
| Variable | Values | Default | Effect |
|--------------------|------------------|---------|-----------------------------------------------------------------|
| `ZFIN_TIINGO_PLAN` | `free` / `power` | `free` | Hourly Tiingo request cap: free = 50/hour, power = 10,000/hour. |
An unrecognized value logs a warning and falls back to `free`.
## Contact email ## Contact email
| Variable | Used for | | Variable | Used for |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ZFIN_USER_EMAIL` | The contact address SEC EDGAR requires in its `User-Agent` header. Enables ETF profiles and [`zfin enrich`](../cli/enrich.md). Not a key -- just your email. Without it, ETF profiles and enrichment are unavailable. | | `ZFIN_USER_EMAIL` | The contact address SEC EDGAR requires in its `User-Agent` header. Enables ETF profiles and [`zfin enrich`](../cli/enrich.md). Not a key -- just your email. Without it, ETF profiles and enrichment are unavailable. |
## Live quotes and streaming
`ZFIN_LIVE_QUOTE_PROVIDER` chooses where intraday quotes come from --
both the `r`/F5 refresh and the opt-in live stream (the `L` toggle on
the TUI's Quote and Portfolio tabs, or [`quote --live`](../cli/quote.md)).
| Variable | Values | Default | Purpose |
|----------------------------|--------------------|-----------|--------------------------------------------------------------------------------------------------------------------------|
| `ZFIN_LIVE_QUOTE_PROVIDER` | `yahoo` / `tiingo` | (derived) | Provider for live quotes + streaming. Unset: derived from the [Tiingo plan](#tiingo-plan) (Power -> Tiingo, else Yahoo). |
- **Yahoo** (default): keyless, works on all tiers, but ~15-minute
delayed and unofficial (no guarantees).
- **Tiingo**: real-time IEX last-sale (the `tngoLast` reference price);
requires `TIINGO_API_KEY`. The IEX reference feed works on **any**
Tiingo tier, free included -- set `ZFIN_LIVE_QUOTE_PROVIDER=tiingo` to
use it. It's the *automatic* default only on the Power plan, which has
the request headroom for heavy refresh/streaming use; on the free tier
it's opt-in (each websocket connect spends one of the 50/hour, but a
stable stream then costs nothing).
An unrecognized value logs a warning and falls back to the default.
## Paths and directories ## Paths and directories
| Variable | Default | Purpose | | Variable | Default | Purpose |

37
docs/reference/providers.md
View file

@ -10,18 +10,19 @@ how keys are configured see
## Summary ## Summary
| Data | Provider | Auth | Free-tier limit | Cache TTL | | Data | Provider | Auth | Free-tier limit | Cache TTL |
|-----------------------|------------------|-------------------------------|----------------------------|------------| |-----------------------|------------------|-------------------------------|------------------------------------------------|------------|
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | ~24h | | Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h |
| Real-time quotes | Yahoo | none | unofficial | never | | Real-time quotes | Yahoo | none | unofficial | never |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never | | Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | | Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never |
| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | | Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour | | Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days | | Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days | | Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived | | ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite | | Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite |
## Where to get a key ## Where to get a key
@ -40,8 +41,16 @@ how keys are configured see
funds. Candles are fetched from a fixed 2000-01-01 start so the cache funds. Candles are fetched from a fixed 2000-01-01 start so the cache
supports long `--as-of` projections. Its price series also carries supports long `--as-of` projections. Its price series also carries
per-row dividend/split data that zfin merges into the Polygon view. per-row dividend/split data that zfin merges into the Polygon view.
- **Yahoo** -- primary, keyless quote source, and the candle fallback Also serves **real-time IEX quotes** (the `tngoLast` reference price)
when Tiingo is unavailable. for the `r` refresh and the live stream when
[`ZFIN_LIVE_QUOTE_PROVIDER=tiingo`](config/environment.md#live-quotes-and-streaming);
the IEX reference feed works on any tier (no exchange license). Rate
limits are tier-aware via `ZFIN_TIINGO_PLAN` (free 50/hour, Power
10,000/hour) -- one batched quote call or one stream connect is a
single request.
- **Yahoo** -- primary, keyless quote source, the candle fallback when
Tiingo is unavailable, and the default live-quote/stream feed
(~15-min delayed, unofficial).
- **TwelveData** -- quote fallback only. It is no longer used for - **TwelveData** -- quote fallback only. It is no longer used for
candles (its split-adjusted closes proved unreliable for return math). candles (its split-adjusted closes proved unreliable for return math).
- **Polygon** -- primary for dividends and splits, including - **Polygon** -- primary for dividends and splits, including

21
docs/reference/tui.md
View file

@ -62,8 +62,25 @@ current bindings). The global defaults:
| `page_down` / `Ctrl-f`, `page_up` / `Ctrl-b` | Page down / up | | `page_down` / `Ctrl-f`, `page_up` / `Ctrl-b` | Page down / up |
Individual tabs add their own actions (e.g. the Projections tab's `d` Individual tabs add their own actions (e.g. the Projections tab's `d`
to set an as-of date and `o` to toggle the overlay). The `?` overlay to set an as-of date and `o` to toggle the overlay, or the Quote and
and `zfin i --default-keys` list every binding, global and tab-scoped. Portfolio tabs' `L` to toggle live streaming). The `?` overlay and
`zfin i --default-keys` list every binding, global and tab-scoped.
## Live streaming
On the **Quote** and **Portfolio** tabs, press `L` to toggle a live
price stream (off by default). While on, prices update in place every
~100ms: the Quote tab tracks the selected symbol; the Portfolio tab
re-values the whole position table and totals as held symbols tick.
Press `L` again (or switch tabs) to stop.
The feed follows
[`ZFIN_LIVE_QUOTE_PROVIDER`](config/environment.md#live-quotes-and-streaming):
keyless **Yahoo** (default, ~15-min delayed) or real-time IEX via
**Tiingo** (any keyed tier). Streaming covers what each feed prices
live -- stocks/ETFs (and crypto on Yahoo); mutual funds, CDs, options,
and cash hold their last value. The historical-return blocks stay as of
the last full refresh.
## Customizing ## Customizing