lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

update documentation on dividend smart refresh

Browse source
This commit is contained in:
Emil Lerch 2026-09-20 07:19:40 -07:00
parent ca3bd11110
commit 88deae226a
Signed by: lobo
GPG key ID: A7B62D657EF764F8
7 changed files with 155 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
AGENTS.md
View file

@ -882,6 +882,13 @@ command.
- **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`).
- **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes a second comptime hook, `needsRefresh: ?*const fn ([]const T, Date) bool`, for types where TTL cannot express "is this entry missing something it should have by now". `getDividends` passes `dividendsNeedRefresh`; `getSplits` and `getOptions` pass `null`. Three rules if you add one:
- It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it.
- `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that.
- It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure.
The branch is comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the dividend rationale and its three documented blind spots.
- **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means.
- **The `color` parameter flows through everything.** CLI commands accept a `color: bool` parameter. Don't use ANSI escapes unconditionally - always gate on the `color` flag.

13
README.md
View file

@ -84,13 +84,24 @@ it does best, with aggressive caching to stay within free-tier limits.
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | 23h45m |
| Real-time quotes | Yahoo Finance | None required | Unofficial | Never cached |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8 req/min, 800/day | Never cached |
| Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days |
| Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 6 days\*\* |
| Splits | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days |
| Options chains | CBOE | None required | ~30 req/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days* |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10 req/sec | ~90 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | No per-day quota | Long-lived |
\* **Earnings smart refresh** -- cached earnings re-fetch inside the
30-day window once a report date has passed but the actual result is
still missing.
\*\* **Dividend smart refresh** -- the TTL is only a backstop. Cached
dividends also re-fetch whenever a distribution the symbol's own
schedule says is due has not arrived. An ETF's amount is not published
until roughly its own ex-date and it pays 1-7 days later, so a long TTL
can be written before the record exists and still be fresh after the
cash has landed.
Not all keys are required; without a given key, that data type is
simply unavailable. For per-provider notes, signup links, the full
caching/TTL model, and the complete environment-variable list, see

94
docs/dev/caching-implementation.md
View file

@ -146,8 +146,8 @@ in `DataType.ttl()`.
|----------------------|------------------|--------|--------------------------------------------------|
| Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)|
| Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable |
| Dividends | 14 days | 11% | Declared well in advance |
| Splits | 14 days | 11% | Rare corporate events |
| Dividends | 6 days | 11% | Backstop only; see dividend smart refresh below |
| Splits | 14 days | 11% | Rare corporate events, announced weeks ahead |
| Options | 1 hour | 0 | Move continuously during market hours |
| Earnings | 30 days | 8% | Smart-refresh after an announcement date passes |
| Classification | 90 days | 8% | Sector/industry/country from Wikidata |
@ -309,14 +309,97 @@ Everything that is not candles flows through the generic `fetchCached`,
which is simpler because each type is a single file:
1. `.fresh_only` read; a fresh entry (including a negative one) returns
immediately.
immediately -- **unless** the type supplies a `needsRefresh` hook that
says the fresh entry is incomplete (see below).
2. `skip_network`: return any cached entry, even stale; else
`FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns.
3. Server sync (if configured); a fresh synced entry returns, subject to
the same `needsRefresh` check.
4. Provider fetch; on success write with the type's TTL; on
`NotFound` write a negative entry; on transient error return
`FetchFailed` without poisoning the cache.
#### The two comptime hooks
`fetchCached` takes two optional comptime function pointers. Both default
to "absent" by passing `null`, which is the behaviour every type had
before they existed.
| Hook | Signature | Purpose |
|-----------------|--------------------------------------------|-----------------------------------------------------|
| `postProcess` | `fn (*T, Allocator) anyerror!void` | Recompute a derived field after parse |
| `needsRefresh` | `fn ([]const T, Date) bool` | "This entry is fresh, but is it *complete*?" |
`postProcess` is **not** for duping strings -- `Store.read` already does
that. Today only `EarningsEvent` uses it (to rebuild `surprise`).
`needsRefresh` exists because freshness and completeness are different
questions. It is consulted at **both** fresh-read sites (step 1 and step
3) through `serveFreshOrDiscard`, which owns the free so a caller cannot
decide to discard and then leak. Covering step 3 is not optional: with a
`ZFIN_SERVER` configured, a sync can leave a still-incomplete entry on
disk that step 3 would otherwise serve, defeating the mechanism
entirely. The hook is suppressed under `skip_network`, since offline
mode never refetches and discarding could only turn a served result into
a failure.
The hook's branch is comptime-elided when the argument is `null`, which
is what lets types without a `freeSlice` (such as `Split`) keep
compiling.
#### Dividend smart refresh
`dividendsNeedRefresh` is the hook `getDividends` passes. It answers
true when at least one full cadence period has elapsed since the newest
cached ex-date and the distribution that should have landed in it is
still absent.
The problem it solves is that dividend TTL and dividend *availability*
are uncorrelated. An operating company declares weeks ahead of the
ex-date and pays about a month after it. An ETF cannot: its distribution
is the pass-through of whatever the fund earned, so the amount is not
published until roughly its own ex-date, and it pays 1-7 days later. The
window in which the record exists but the cash has not yet landed is
about a week, so any longer TTL can be written before the record exists
and still be fresh after the payment. Quarter-end clustering makes that
systematic rather than unlucky -- a dozen funds go ex within a few days
of each other, so one refresh pass puts all of them on the same expiry
shelf.
Cadence comes from `dividendCadenceDays`: the **median** of up to the
five newest gaps between cached ex-dates. Median rather than mean (a
special distribution landing days after a regular one drags the mean) and
rather than minimum (the same outlier destroys the minimum outright,
predicting a next ex-date days away and firing for the whole chase window
after every payment). `newestExDates` sorts rather than trusting file
order, because `writeSupplement`'s merge interleaves two providers'
records and a cadence computed from negative gaps would forecast into the
past forever.
Three properties are load-bearing and each has its own test:
- **A full period must elapse.** Otherwise it refires immediately after
a successful fetch, on the record it just stored.
- **It rolls forward.** The expected date is the latest one at or before
today, not the first one after the newest cached record -- so a symbol
two periods behind stays recoverable instead of being permanently past
its window.
- **The chase window is bounded** (`dividend_chase_days`, 14). An
unbounded chase refetches forever whenever a sponsor skips a period or
the cadence estimate is wrong.
Known blind spots, all three deliberately left to the TTL backstop
rather than papered over with a more eager statistic that would cost
requests on every symbol to protect against a rare event on one:
- A **cadence change** (quarterly to monthly) keeps the old median for
several periods, so the forecast runs late.
- An **off-cycle special** is on no schedule, so the predicate never
fires for it.
- A symbol with **fewer than three cached records** has no cadence at
all -- which is the right direction, since a newly-bought holding has
no schedule to be late against.
### `loadAllPrices` (portfolio + watchlist price load)
The portfolio price load batches all symbols through three phases.
@ -559,6 +642,9 @@ the upstream provider on every request.
|---------------------------------|-----------------------------------------------------|
| Data-access entry point | `DataService` - `src/service.zig` |
| Per-type generic fetch | `fetchCached` - `src/service.zig` |
| Fresh-but-incomplete decision | `serveFreshOrDiscard` - `src/service.zig` |
| Dividend schedule check | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` |
| Earnings smart refresh | `earningsNeedsRefresh` - `src/service.zig` |
| Candle fetch + incremental | `getCandles` - `src/service.zig` |
| Batch price load | `loadAllPrices` - `src/service.zig` |
| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` |

32
docs/explanation/caching.md
View file

@ -42,14 +42,38 @@ Different data ages at different rates, so each type has its own TTL:
| Data type | TTL | Why |
|---------------|---------------|----------------------------------------------------------------------------------|
| Daily candles | market-aware | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
| Dividends | 14 days | Declared well in advance |
| Splits | 14 days | Rare corporate events |
| Dividends | 6 days\* | Short: an ETF distribution is only knowable for a few days before it pays |
| Splits | 14 days | Rare corporate events, announced weeks ahead |
| Options | 1 hour | Prices move continuously when markets are open |
| Earnings | 30 days\* | Quarterly; smart-refreshed around announcements |
| Earnings | 30 days\*\* | Quarterly; smart-refreshed around announcements |
| ETF profiles | ~30 days | Holdings and weights change slowly |
| Quotes | never cached | Meant to be a live price check |
\* **Earnings smart refresh:** even inside the 30-day window, cached
\* **Dividend smart refresh:** even inside the 6-day window, cached
dividends re-fetch automatically once a distribution the symbol's own
schedule says is due has not arrived. This exists because a dividend
cache being *fresh* and being *complete* are different things.
An operating company declares a dividend weeks ahead of the ex-date and
pays about a month after it, so there is plenty of time to notice. An
**ETF does not work that way.** Its distribution is the pass-through of
whatever the fund earned, so the amount is not published until roughly
its own ex-date -- and it pays 1 to 7 days later. The entire window in
which the record exists but the cash has not yet landed is about a week.
A long TTL straddles that window whole: a cache written a few days
before the ex-date stays "fresh" until well after the payment, so the
distribution is invisible to the run that needed it. Quarter-end makes
it systematic rather than unlucky, because a dozen funds go ex within
the same few days and one refresh pass puts all of them on the same
expiry shelf.
So the schedule is the gate, and the TTL is only the backstop. It still
matters, because the schedule check cannot see three things: a holding
too new to have an established cadence, an off-cycle special
distribution, and a symbol with fewer than three cached records.
\*\* **Earnings smart refresh:** even inside the 30-day window, cached
earnings re-fetch automatically once an earnings date has passed but
the cache still lacks the actual result -- so numbers appear promptly
after an announcement without daily polling.

10
docs/explanation/data-providers.md
View file

@ -14,11 +14,19 @@ Each data type has a primary provider chosen for coverage and quality:
|-------------------------------|----------------------|-----------------------------------------------|
| Daily candles | Tiingo | Deep history; stocks, ETFs, mutual funds |
| Real-time quotes | Yahoo | No key required |
| Dividends / splits | Polygon | Carries forward-looking declared events |
| Dividends / splits | Polygon | Carries forward-looking declared events\* |
| Options chains | CBOE | No key; 15-minute delayed |
| Earnings | FMP | Actuals + analyst estimates |
| ETF profiles / classification | SEC EDGAR + Wikidata | Authoritative holdings; needs a contact email |
\* Forward-looking for **splits and operating-company dividends**, which
are declared weeks to months ahead. It is not a promise about ETF
distributions: a fund's per-share amount is the pass-through of what it
earned, so it does not exist until roughly the fund's own ex-date no
matter which provider you ask. That is why dividends carry a short TTL
plus a schedule check -- see
[Caching and data freshness](caching.md).
The [data service](concepts.md#the-data-service-and-caching) hides this
behind one interface -- commands ask for "candles for VTI," not "call
Tiingo." That's also what makes the cache and rate limiting uniform

3
docs/guides/offline-and-refresh.md
View file

@ -66,7 +66,8 @@ In `auto` mode, each data type has its own time-to-live:
| Data | TTL |
|--------------------|------------------------------------------------|
| Daily candles | ~24 hours |
| Dividends / splits | 14 days |
| Dividends | 6 days (refreshed early once a payout is due) |
| Splits | 14 days |
| Options | 1 hour |
| Earnings | 30 days (refreshed early once a result is due) |
| ETF profiles | ~30 days |

8
docs/reference/providers.md
View file

@ -16,7 +16,7 @@ how keys are configured see
| Real-time quotes | Yahoo | none | unofficial | never |
| Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 6 days\* |
| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days |
@ -24,6 +24,12 @@ how keys are configured see
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite |
\* The dividend TTL is only a backstop. Cached dividends also re-fetch
whenever a distribution the symbol's own schedule says is due has not
arrived, because an ETF's amount is not published until roughly its
ex-date and it pays within a week -- see
[Caching and data freshness](../explanation/caching.md).
## Where to get a key
| Provider | Sign up | Notes |