lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

update todo and docs

Browse source
This commit is contained in:
Emil Lerch 2026-06-30 08:32:56 -07:00
parent bb04d7babd
commit baf5e2a86f
Signed by: lobo
GPG key ID: A7B62D657EF764F8
6 changed files with 118 additions and 105 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
TODO.md
View file

@ -55,94 +55,6 @@ populate. This could be solved on the server by spawning a thread to fetch the
data, then returning 202 Accepted, which could then be polled client side. Maybe
this is a better long term approach?
## Support Tiingo paid plan - priority LOW
zfin hardwires Tiingo to free-tier assumptions: the provider is
constructed with `RateLimiter.perHour(io, 50)` in `Tiingo.init`
(`providers/tiingo.zig`), and the only Tiingo surface is end-of-day
candles plus the corporate actions that ride along in the same
response. A user who pays for a Tiingo plan ($30/mo Power tier and
up) gets nothing for it today - the same 50/hour throttle, the same
EOD-only data. "Support the paid plan" is the umbrella for unlocking
what that subscription actually buys: higher rate limits and
real-time IEX quotes. The two are coupled (real-time polling only
makes sense once the limit is raised), which is why they belong in
one entry rather than two.
### Tier-aware rate limiting
The 50/hour cap is hardcoded in `Tiingo.init`
(`RateLimiter.perHour(io, 50)`), and the module docstring bakes in
"Free tier: 50 requests/hour and 1,000 requests/day." Paid tiers
raise both ceilings substantially, so a paying subscriber is being
throttled far below their entitlement. Today only the hourly bucket
is wired; the daily ceiling isn't enforced at all (the docstring
notes it's "far from binding" for bursty EOD usage - real-time
polling changes that calculus).
Work:
- Make the Tiingo limits configurable instead of hardcoded. Options:
explicit `ZFIN_TIINGO_RATE_PER_HOUR` (and per-day) numeric env
knobs, or a coarser `ZFIN_TIINGO_PLAN` = `free` (default) |
`power` | ... that maps to known limits. Lean toward explicit
numeric overrides so we aren't chasing Tiingo's published per-tier
numbers as they drift.
- `RateLimiter` already supports arbitrary `init(io, max, window_ns)`
plus `perDay`/`perHour` convenience ctors, so the limiter side is
cheap. Decide whether a paid plan needs both an hourly and a daily
bucket enforced, or whether hourly alone stays sufficient.
- Caveat from `RateLimiter`'s own docs: the bucket is in-memory and
per-process - it caps a single run's burst, not usage across
separate launches in the same window. Sustained real-time polling
(below) makes cross-process usage likelier, so revisit whether
per-process accounting is still good enough.
### Real-time IEX quotes (was: configurable live-quote provider)
The TUI refresh key (`r`) values the portfolio with live intraday
quotes via `DataService.loadLiveQuotes` (`service.zig`), which is
Yahoo-only: Yahoo is keyless, consolidated, and stays off every
rate-limit budget, so bursty refresh traffic costs nothing. The
tradeoffs are that Yahoo's unofficial feed is ~15-minute delayed and
"can break without notice."
Tiingo's IEX endpoint (`/iex/?tickers=A,B,C`) is a strong opt-in
alternative for a paid subscriber: it's genuinely real-time (IEX
last-sale, no 15-min delay), official/keyed, and bills per HTTP
request - one call returns the whole portfolio (confirmed
empirically: a 2-ticker batch decrements the daily quota by 1, not
2). Fields map cleanly: `tngoLast` to price, `prevClose` to
day-change. Caveats: IEX is a single venue (~2-3% of volume), so
`tngoLast` can sit stale between prints on illiquid names, and IEX
doesn't trade mutual funds, so those fall back to the candle close.
Proposal: a config knob (env var, e.g. `ZFIN_LIVE_QUOTE_PROVIDER` =
`yahoo` (default) | `tiingo`) that switches `loadLiveQuotes` to a new
`Tiingo.fetchQuotes(tickers)` batched call. A paid subscriber who
wants real-time and mashes `r` a lot (or once we add streaming)
reuses their existing `TIINGO_API_KEY` and gets real-time coverage;
everyone else keeps the keyless Yahoo default.
Implementation notes:
- `Tiingo.fetchQuotes` returns an array whose order is NOT guaranteed
to match the request order, so key results by the returned
`ticker` field, not by position.
- Live quotes share Tiingo's token bucket, so this is the concrete
reason the tier-aware rate-limiting work above has to land first
(or alongside): a batched quote call is only 1 request, but heavy
`r` use plus candle refreshes draining the free 50/hour bucket is
exactly the contention that raising the paid-tier limit relieves.
### Websocket streaming (follow-on)
Tiingo's IEX websocket would be the natural follow-on for true
push-based real-time, replacing poll-on-`r` entirely. Materially
bigger than the REST quote path (persistent connection, reconnect
handling, a background task feeding the TUI) and squarely a
paid-plan feature. Sequence it after the REST quote path proves out.
## Infra / performance
- **HTTP connection pooling.** Parallel server sync in `loadAllPrices`

22
docs/explanation/data-providers.md
View file

@ -24,6 +24,28 @@ behind one interface -- commands ask for "candles for VTI," not "call
Tiingo." That's also what makes the cache and rate limiting uniform
across providers.
## Live quotes: free/delayed vs real-time
Intraday quotes -- the `r` refresh and the opt-in live stream (`L` in
the TUI, or [`quote --live`](../reference/cli/quote.md)) -- can come from
either feed, chosen by
[`ZFIN_LIVE_QUOTE_PROVIDER`](../reference/config/environment.md#live-quotes-and-streaming):
- **Yahoo** (default): keyless, consolidated, all tiers -- but ~15-min
delayed and unofficial.
- **Tiingo**: real-time IEX last-sale (`tngoLast`). Needs
`TIINGO_API_KEY` but works on **any** Tiingo tier -- the IEX
*reference price* (Tiingo's `thresholdLevel: 6`) needs no exchange
license, unlike raw IEX market data. Caveats: it's a single venue, so
a thinly-traded name can sit stale between prints, and IEX doesn't
carry mutual funds (those fall back to the candle close).
The same choice drives both the one-shot refresh and the websocket
stream. A stable stream costs a single request to connect, so it's
practical even on Tiingo's free 50/hour tier; the Power tier (10,000/hour)
just adds headroom for heavy use, which is why it's the auto-default
there.
## Fallback, not single-point-of-failure
Where a second source can stand in, zfin uses it:

20
docs/reference/cli/quote.md
View file

@ -4,7 +4,7 @@ Show the latest quote for a symbol, with a price chart and recent
history.
```
Usage: zfin quote <SYMBOL> [--since <WHEN>] [--export-chart <PATH>]
Usage: zfin quote <SYMBOL> [--since <WHEN>] [--export-chart <PATH>] [--live]
```
Prints the last price, the day's open/high/low, volume, and the
@ -29,6 +29,24 @@ Supports `--export-chart <PATH>` to render the chart as a 1920x1080
PNG instead of text (see [export charts](../../guides/offline-and-refresh.md)
and the projections page).
## Live streaming (`--live`)
`zfin quote --live <SYMBOL>` opens a websocket and prints the price
continuously (one line per second) instead of a one-shot quote. It runs
until interrupted (Ctrl-C) or killed -- handy with `timeout`:
```bash
timeout 12 zfin quote --live BTC-USD # watch live ticks for 12s
```
The transport follows
[`ZFIN_LIVE_QUOTE_PROVIDER`](../config/environment.md#live-quotes-and-streaming):
free, keyless **Yahoo** by default, or real-time IEX via **Tiingo** when
selected (works on any Tiingo tier with a key). Outside market hours,
US equities stream a closing snapshot and then go quiet; 24/7 symbols
(e.g. `BTC-USD` on Yahoo) keep ticking. The chart / `--since` /
`--export-chart` options don't apply in `--live` mode.
## Example
```bash

35
docs/reference/config/environment.md
View file

@ -30,12 +30,47 @@ unavailable. Quotes (Yahoo) and options (CBOE) need no key. See
[Data providers and API keys](../providers.md) for signup links and
free-tier limits.
## Tiingo plan
`ZFIN_TIINGO_PLAN` is your Tiingo subscription tier (not a key). It
sizes the hourly request budget for **all** Tiingo usage -- candle
history (the main consumer), real-time IEX quotes, and the live stream
-- so a paying subscriber isn't throttled to free-tier limits.
| Variable | Values | Default | Effect |
|--------------------|------------------|---------|-----------------------------------------------------------------|
| `ZFIN_TIINGO_PLAN` | `free` / `power` | `free` | Hourly Tiingo request cap: free = 50/hour, power = 10,000/hour. |
An unrecognized value logs a warning and falls back to `free`.
## Contact email
| Variable | Used for |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ZFIN_USER_EMAIL` | The contact address SEC EDGAR requires in its `User-Agent` header. Enables ETF profiles and [`zfin enrich`](../cli/enrich.md). Not a key -- just your email. Without it, ETF profiles and enrichment are unavailable. |
## Live quotes and streaming
`ZFIN_LIVE_QUOTE_PROVIDER` chooses where intraday quotes come from --
both the `r`/F5 refresh and the opt-in live stream (the `L` toggle on
the TUI's Quote and Portfolio tabs, or [`quote --live`](../cli/quote.md)).
| Variable | Values | Default | Purpose |
|----------------------------|--------------------|-----------|--------------------------------------------------------------------------------------------------------------------------|
| `ZFIN_LIVE_QUOTE_PROVIDER` | `yahoo` / `tiingo` | (derived) | Provider for live quotes + streaming. Unset: derived from the [Tiingo plan](#tiingo-plan) (Power -> Tiingo, else Yahoo). |
- **Yahoo** (default): keyless, works on all tiers, but ~15-minute
delayed and unofficial (no guarantees).
- **Tiingo**: real-time IEX last-sale (the `tngoLast` reference price);
requires `TIINGO_API_KEY`. The IEX reference feed works on **any**
Tiingo tier, free included -- set `ZFIN_LIVE_QUOTE_PROVIDER=tiingo` to
use it. It's the *automatic* default only on the Power plan, which has
the request headroom for heavy refresh/streaming use; on the free tier
it's opt-in (each websocket connect spends one of the 50/hour, but a
stable stream then costs nothing).
An unrecognized value logs a warning and falls back to the default.
## Paths and directories
| Variable | Default | Purpose |

37
docs/reference/providers.md
View file

@ -10,18 +10,19 @@ how keys are configured see
## Summary
| Data | Provider | Auth | Free-tier limit | Cache TTL |
|-----------------------|------------------|-------------------------------|----------------------------|------------|
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | ~24h |
| Real-time quotes | Yahoo | none | unofficial | never |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite |
| Data | Provider | Auth | Free-tier limit | Cache TTL |
|-----------------------|------------------|-------------------------------|------------------------------------------------|------------|
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h |
| Real-time quotes | Yahoo | none | unofficial | never |
| Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite |
## Where to get a key
@ -40,8 +41,16 @@ how keys are configured see
funds. Candles are fetched from a fixed 2000-01-01 start so the cache
supports long `--as-of` projections. Its price series also carries
per-row dividend/split data that zfin merges into the Polygon view.
- **Yahoo** -- primary, keyless quote source, and the candle fallback
when Tiingo is unavailable.
Also serves **real-time IEX quotes** (the `tngoLast` reference price)
for the `r` refresh and the live stream when
[`ZFIN_LIVE_QUOTE_PROVIDER=tiingo`](config/environment.md#live-quotes-and-streaming);
the IEX reference feed works on any tier (no exchange license). Rate
limits are tier-aware via `ZFIN_TIINGO_PLAN` (free 50/hour, Power
10,000/hour) -- one batched quote call or one stream connect is a
single request.
- **Yahoo** -- primary, keyless quote source, the candle fallback when
Tiingo is unavailable, and the default live-quote/stream feed
(~15-min delayed, unofficial).
- **TwelveData** -- quote fallback only. It is no longer used for
candles (its split-adjusted closes proved unreliable for return math).
- **Polygon** -- primary for dividends and splits, including

21
docs/reference/tui.md
View file

@ -62,8 +62,25 @@ current bindings). The global defaults:
| `page_down` / `Ctrl-f`, `page_up` / `Ctrl-b` | Page down / up |
Individual tabs add their own actions (e.g. the Projections tab's `d`
to set an as-of date and `o` to toggle the overlay). The `?` overlay
and `zfin i --default-keys` list every binding, global and tab-scoped.
to set an as-of date and `o` to toggle the overlay, or the Quote and
Portfolio tabs' `L` to toggle live streaming). The `?` overlay and
`zfin i --default-keys` list every binding, global and tab-scoped.
## Live streaming
On the **Quote** and **Portfolio** tabs, press `L` to toggle a live
price stream (off by default). While on, prices update in place every
~100ms: the Quote tab tracks the selected symbol; the Portfolio tab
re-values the whole position table and totals as held symbols tick.
Press `L` again (or switch tabs) to stop.
The feed follows
[`ZFIN_LIVE_QUOTE_PROVIDER`](config/environment.md#live-quotes-and-streaming):
keyless **Yahoo** (default, ~15-min delayed) or real-time IEX via
**Tiingo** (any keyed tier). Streaming covers what each feed prices
live -- stocks/ETFs (and crypto on Yahoo); mutual funds, CDs, options,
and cash hold their last value. The historical-return blocks stay as of
the last full refresh.
## Customizing