diff --git a/AGENTS.md b/AGENTS.md index 12904a7..a7863a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -882,6 +882,13 @@ command. - **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`). +- **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes a second comptime hook, `needsRefresh: ?*const fn ([]const T, Date) bool`, for types where TTL cannot express "is this entry missing something it should have by now". `getDividends` passes `dividendsNeedRefresh`; `getSplits` and `getOptions` pass `null`. Three rules if you add one: + - It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it. + - `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that. + - It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure. + + The branch is comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the dividend rationale and its three documented blind spots. + - **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means. - **The `color` parameter flows through everything.** CLI commands accept a `color: bool` parameter. Don't use ANSI escapes unconditionally - always gate on the `color` flag. diff --git a/README.md b/README.md index fb69353..7f4062c 100644 --- a/README.md +++ b/README.md @@ -84,13 +84,24 @@ it does best, with aggressive caching to stay within free-tier limits. | Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | 23h45m | | Real-time quotes | Yahoo Finance | None required | Unofficial | Never cached | | Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8 req/min, 800/day | Never cached | -| Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days | +| Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 6 days\*\* | | Splits | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days | | Options chains | CBOE | None required | ~30 req/min (self-imposed) | 1 hour | | Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days* | | ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10 req/sec | ~90 days | | Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | No per-day quota | Long-lived | +\* **Earnings smart refresh** -- cached earnings re-fetch inside the +30-day window once a report date has passed but the actual result is +still missing. + +\*\* **Dividend smart refresh** -- the TTL is only a backstop. Cached +dividends also re-fetch whenever a distribution the symbol's own +schedule says is due has not arrived. An ETF's amount is not published +until roughly its own ex-date and it pays 1-7 days later, so a long TTL +can be written before the record exists and still be fresh after the +cash has landed. + Not all keys are required; without a given key, that data type is simply unavailable. For per-provider notes, signup links, the full caching/TTL model, and the complete environment-variable list, see diff --git a/docs/dev/caching-implementation.md b/docs/dev/caching-implementation.md index 83c5070..903612e 100644 --- a/docs/dev/caching-implementation.md +++ b/docs/dev/caching-implementation.md @@ -146,8 +146,8 @@ in `DataType.ttl()`. |----------------------|------------------|--------|--------------------------------------------------| | Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)| | Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable | -| Dividends | 14 days | 11% | Declared well in advance | -| Splits | 14 days | 11% | Rare corporate events | +| Dividends | 6 days | 11% | Backstop only; see dividend smart refresh below | +| Splits | 14 days | 11% | Rare corporate events, announced weeks ahead | | Options | 1 hour | 0 | Move continuously during market hours | | Earnings | 30 days | 8% | Smart-refresh after an announcement date passes | | Classification | 90 days | 8% | Sector/industry/country from Wikidata | @@ -309,14 +309,97 @@ Everything that is not candles flows through the generic `fetchCached`, which is simpler because each type is a single file: 1. `.fresh_only` read; a fresh entry (including a negative one) returns - immediately. + immediately -- **unless** the type supplies a `needsRefresh` hook that + says the fresh entry is incomplete (see below). 2. `skip_network`: return any cached entry, even stale; else `FetchFailed`. -3. Server sync (if configured); a fresh synced entry returns. +3. Server sync (if configured); a fresh synced entry returns, subject to + the same `needsRefresh` check. 4. Provider fetch; on success write with the type's TTL; on `NotFound` write a negative entry; on transient error return `FetchFailed` without poisoning the cache. +#### The two comptime hooks + +`fetchCached` takes two optional comptime function pointers. Both default +to "absent" by passing `null`, which is the behaviour every type had +before they existed. + +| Hook | Signature | Purpose | +|-----------------|--------------------------------------------|-----------------------------------------------------| +| `postProcess` | `fn (*T, Allocator) anyerror!void` | Recompute a derived field after parse | +| `needsRefresh` | `fn ([]const T, Date) bool` | "This entry is fresh, but is it *complete*?" | + +`postProcess` is **not** for duping strings -- `Store.read` already does +that. Today only `EarningsEvent` uses it (to rebuild `surprise`). + +`needsRefresh` exists because freshness and completeness are different +questions. It is consulted at **both** fresh-read sites (step 1 and step +3) through `serveFreshOrDiscard`, which owns the free so a caller cannot +decide to discard and then leak. Covering step 3 is not optional: with a +`ZFIN_SERVER` configured, a sync can leave a still-incomplete entry on +disk that step 3 would otherwise serve, defeating the mechanism +entirely. The hook is suppressed under `skip_network`, since offline +mode never refetches and discarding could only turn a served result into +a failure. + +The hook's branch is comptime-elided when the argument is `null`, which +is what lets types without a `freeSlice` (such as `Split`) keep +compiling. + +#### Dividend smart refresh + +`dividendsNeedRefresh` is the hook `getDividends` passes. It answers +true when at least one full cadence period has elapsed since the newest +cached ex-date and the distribution that should have landed in it is +still absent. + +The problem it solves is that dividend TTL and dividend *availability* +are uncorrelated. An operating company declares weeks ahead of the +ex-date and pays about a month after it. An ETF cannot: its distribution +is the pass-through of whatever the fund earned, so the amount is not +published until roughly its own ex-date, and it pays 1-7 days later. The +window in which the record exists but the cash has not yet landed is +about a week, so any longer TTL can be written before the record exists +and still be fresh after the payment. Quarter-end clustering makes that +systematic rather than unlucky -- a dozen funds go ex within a few days +of each other, so one refresh pass puts all of them on the same expiry +shelf. + +Cadence comes from `dividendCadenceDays`: the **median** of up to the +five newest gaps between cached ex-dates. Median rather than mean (a +special distribution landing days after a regular one drags the mean) and +rather than minimum (the same outlier destroys the minimum outright, +predicting a next ex-date days away and firing for the whole chase window +after every payment). `newestExDates` sorts rather than trusting file +order, because `writeSupplement`'s merge interleaves two providers' +records and a cadence computed from negative gaps would forecast into the +past forever. + +Three properties are load-bearing and each has its own test: + +- **A full period must elapse.** Otherwise it refires immediately after + a successful fetch, on the record it just stored. +- **It rolls forward.** The expected date is the latest one at or before + today, not the first one after the newest cached record -- so a symbol + two periods behind stays recoverable instead of being permanently past + its window. +- **The chase window is bounded** (`dividend_chase_days`, 14). An + unbounded chase refetches forever whenever a sponsor skips a period or + the cadence estimate is wrong. + +Known blind spots, all three deliberately left to the TTL backstop +rather than papered over with a more eager statistic that would cost +requests on every symbol to protect against a rare event on one: + +- A **cadence change** (quarterly to monthly) keeps the old median for + several periods, so the forecast runs late. +- An **off-cycle special** is on no schedule, so the predicate never + fires for it. +- A symbol with **fewer than three cached records** has no cadence at + all -- which is the right direction, since a newly-bought holding has + no schedule to be late against. + ### `loadAllPrices` (portfolio + watchlist price load) The portfolio price load batches all symbols through three phases. @@ -559,6 +642,9 @@ the upstream provider on every request. |---------------------------------|-----------------------------------------------------| | Data-access entry point | `DataService` - `src/service.zig` | | Per-type generic fetch | `fetchCached` - `src/service.zig` | +| Fresh-but-incomplete decision | `serveFreshOrDiscard` - `src/service.zig` | +| Dividend schedule check | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` | +| Earnings smart refresh | `earningsNeedsRefresh` - `src/service.zig` | | Candle fetch + incremental | `getCandles` - `src/service.zig` | | Batch price load | `loadAllPrices` - `src/service.zig` | | Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` | diff --git a/docs/explanation/caching.md b/docs/explanation/caching.md index 8532e03..7838d10 100644 --- a/docs/explanation/caching.md +++ b/docs/explanation/caching.md @@ -42,14 +42,38 @@ Different data ages at different rates, so each type has its own TTL: | Data type | TTL | Why | |---------------|---------------|----------------------------------------------------------------------------------| | Daily candles | market-aware | Keyed to the next time a fresh bar is expected, not a rolling window (see below) | -| Dividends | 14 days | Declared well in advance | -| Splits | 14 days | Rare corporate events | +| Dividends | 6 days\* | Short: an ETF distribution is only knowable for a few days before it pays | +| Splits | 14 days | Rare corporate events, announced weeks ahead | | Options | 1 hour | Prices move continuously when markets are open | -| Earnings | 30 days\* | Quarterly; smart-refreshed around announcements | +| Earnings | 30 days\*\* | Quarterly; smart-refreshed around announcements | | ETF profiles | ~30 days | Holdings and weights change slowly | | Quotes | never cached | Meant to be a live price check | -\* **Earnings smart refresh:** even inside the 30-day window, cached +\* **Dividend smart refresh:** even inside the 6-day window, cached +dividends re-fetch automatically once a distribution the symbol's own +schedule says is due has not arrived. This exists because a dividend +cache being *fresh* and being *complete* are different things. + +An operating company declares a dividend weeks ahead of the ex-date and +pays about a month after it, so there is plenty of time to notice. An +**ETF does not work that way.** Its distribution is the pass-through of +whatever the fund earned, so the amount is not published until roughly +its own ex-date -- and it pays 1 to 7 days later. The entire window in +which the record exists but the cash has not yet landed is about a week. + +A long TTL straddles that window whole: a cache written a few days +before the ex-date stays "fresh" until well after the payment, so the +distribution is invisible to the run that needed it. Quarter-end makes +it systematic rather than unlucky, because a dozen funds go ex within +the same few days and one refresh pass puts all of them on the same +expiry shelf. + +So the schedule is the gate, and the TTL is only the backstop. It still +matters, because the schedule check cannot see three things: a holding +too new to have an established cadence, an off-cycle special +distribution, and a symbol with fewer than three cached records. + +\*\* **Earnings smart refresh:** even inside the 30-day window, cached earnings re-fetch automatically once an earnings date has passed but the cache still lacks the actual result -- so numbers appear promptly after an announcement without daily polling. diff --git a/docs/explanation/data-providers.md b/docs/explanation/data-providers.md index 61f0f43..ff678d5 100644 --- a/docs/explanation/data-providers.md +++ b/docs/explanation/data-providers.md @@ -14,11 +14,19 @@ Each data type has a primary provider chosen for coverage and quality: |-------------------------------|----------------------|-----------------------------------------------| | Daily candles | Tiingo | Deep history; stocks, ETFs, mutual funds | | Real-time quotes | Yahoo | No key required | -| Dividends / splits | Polygon | Carries forward-looking declared events | +| Dividends / splits | Polygon | Carries forward-looking declared events\* | | Options chains | CBOE | No key; 15-minute delayed | | Earnings | FMP | Actuals + analyst estimates | | ETF profiles / classification | SEC EDGAR + Wikidata | Authoritative holdings; needs a contact email | +\* Forward-looking for **splits and operating-company dividends**, which +are declared weeks to months ahead. It is not a promise about ETF +distributions: a fund's per-share amount is the pass-through of what it +earned, so it does not exist until roughly the fund's own ex-date no +matter which provider you ask. That is why dividends carry a short TTL +plus a schedule check -- see +[Caching and data freshness](caching.md). + The [data service](concepts.md#the-data-service-and-caching) hides this behind one interface -- commands ask for "candles for VTI," not "call Tiingo." That's also what makes the cache and rate limiting uniform diff --git a/docs/guides/offline-and-refresh.md b/docs/guides/offline-and-refresh.md index 3907a85..69b4eab 100644 --- a/docs/guides/offline-and-refresh.md +++ b/docs/guides/offline-and-refresh.md @@ -66,7 +66,8 @@ In `auto` mode, each data type has its own time-to-live: | Data | TTL | |--------------------|------------------------------------------------| | Daily candles | ~24 hours | -| Dividends / splits | 14 days | +| Dividends | 6 days (refreshed early once a payout is due) | +| Splits | 14 days | | Options | 1 hour | | Earnings | 30 days (refreshed early once a result is due) | | ETF profiles | ~30 days | diff --git a/docs/reference/providers.md b/docs/reference/providers.md index eb947a6..8ba6d6a 100644 --- a/docs/reference/providers.md +++ b/docs/reference/providers.md @@ -16,7 +16,7 @@ how keys are configured see | Real-time quotes | Yahoo | none | unofficial | never | | Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never | | Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never | -| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | +| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 6 days\* | | Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | | Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour | | Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days | @@ -24,6 +24,12 @@ how keys are configured see | Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived | | CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite | +\* The dividend TTL is only a backstop. Cached dividends also re-fetch +whenever a distribution the symbol's own schedule says is due has not +arrived, because an ETF's amount is not published until roughly its +ex-date and it pays within a week -- see +[Caching and data freshness](../explanation/caching.md). + ## Where to get a key | Provider | Sign up | Notes |