lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

update docs with clearer explanation of what is actually happening
All checks were successful
Generic zig build / build (push) Successful in 7m6s
Details
Generic zig build / publish-macos (push) Successful in 12s
Details
Generic zig build / deploy (push) Successful in 20s
Details

Browse source
This commit is contained in:
Emil Lerch 2026-09-20 13:12:03 -07:00
parent cbb90f1b1a
commit 41dd290648
Signed by: lobo
GPG key ID: A7B62D657EF764F8
8 changed files with 1057 additions and 224 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
AGENTS.md
View file

@ -882,12 +882,15 @@ command.
- **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`). - **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`).
- **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes a second comptime hook, `needsRefresh: ?*const fn ([]const T, Date) bool`, for types where TTL cannot express "is this entry missing something it should have by now". `getDividends` passes `dividendsNeedRefresh`; `getSplits` and `getOptions` pass `null`. Three rules if you add one: - **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes three optional comptime hooks: `postProcess` (recompute a derived field), `needsRefresh: ?*const fn ([]const T, Date) bool` ("fresh, but is it complete?"), and `ttlFor: ?*const fn ([]const T) cache.TtlSpec` ("how long should this stay fresh, given these records?"). `getDividends` uses `needsRefresh` + `ttlFor`; `getEarnings` uses `postProcess` + `needsRefresh`; `getSplits` and `getOptions` pass `null` for all three. Four rules if you add a `needsRefresh`:
- It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it. - It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it. This was observed live, not theorised.
- `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that. - `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that.
- It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure. - It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure.
- It is floored at one ask per symbol per 12 hours (`smart_refresh_recheck_interval` / `askedRecently`). Without that, a hook that stays true because the provider has nothing yet re-fires on **every command**, not once a day.
The branch is comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the dividend rationale and its three documented blind spots. **The floor reads `#!expires=`, never `#!created=`.** `serializeWithMeta` re-stamps `created` on every write, and dividend rows arrive as a side effect of Tiingo candle fetches via `writeSupplement` - so `created` means "file last touched" and a candle refresh would fake "we just asked the provider". Only a primary fetch moves `expires` (`.bump`); a supplement restores the old value (`.preserve`). There is a regression test named for this; don't "simplify" it to `created`.
All three hook branches are comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the mechanics and `docs/explanation/caching.md` for the worked example.
- **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means. - **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means.

37
README.md
View file

@ -79,28 +79,31 @@ around the runnable example portfolios in [`examples/`](examples/).
zfin aggregates data from multiple free-tier APIs, using each for what zfin aggregates data from multiple free-tier APIs, using each for what
it does best, with aggressive caching to stay within free-tier limits. it does best, with aggressive caching to stay within free-tier limits.
| Data type | Provider | Auth | Free-tier limit | Cache TTL | | Data type | Provider | Auth | Free-tier limit | Cache TTL |
|-----------------------|------------------|----------------------|----------------------------|--------------| |-----------------------|------------------|----------------------|----------------------------|------------------|
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | 23h45m | | Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000 req/day, 50 req/hour | 23h45m |
| Real-time quotes | Yahoo Finance | None required | Unofficial | Never cached | | Real-time quotes | Yahoo Finance | None required | Unofficial | Never cached |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8 req/min, 800/day | Never cached | | Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8 req/min, 800/day | Never cached |
| Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 6 days\*\* | | Dividends | Polygon | `POLYGON_API_KEY` | 5 req/min | 6 or 14 days\*\* |
| Splits | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days | | Splits | Polygon | `POLYGON_API_KEY` | 5 req/min | 14 days |
| Options chains | CBOE | None required | ~30 req/min (self-imposed) | 1 hour | | Options chains | CBOE | None required | ~30 req/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days* | | Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days* |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10 req/sec | ~90 days | | ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10 req/sec | ~90 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | No per-day quota | Long-lived | | Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | No per-day quota | Long-lived |
\* **Earnings smart refresh** -- cached earnings re-fetch inside the \* **Earnings smart refresh** -- cached earnings re-fetch inside the
30-day window once a report date has passed but the actual result is 30-day window once a report date has passed but the actual result is
still missing. still missing.
\*\* **Dividend smart refresh** -- the TTL is only a backstop. Cached \*\* **Dividends get three freshness checks, not one.** An ETF's payment
dividends also re-fetch whenever a distribution the symbol's own amount does not exist until roughly the day it goes ex, and it pays within
schedule says is due has not arrived. An ETF's amount is not published a week, so a fixed expiry time can sail right over the whole event. So
until roughly its own ex-date and it pays 1-7 days later, so a long TTL after the expiry clock says an entry is still fresh, a second check asks
can be written before the record exists and still be fresh after the whether a payment the symbol's own schedule predicts is missing from it -
cash has landed. and a third stops that from asking the provider more than once every 12
hours. The expiry clock is 14 days when the schedule is known and 6 days
when it isn't. See
[Caching and data freshness](docs/explanation/caching.md#dividends-a-three-level-freshness-check).
Not all keys are required; without a given key, that data type is Not all keys are required; without a given key, that data type is
simply unavailable. For per-provider notes, signup links, the full simply unavailable. For per-provider notes, signup links, the full

287
docs/dev/caching-implementation.md
View file

@ -310,16 +310,51 @@ which is simpler because each type is a single file:
1. `.fresh_only` read; a fresh entry (including a negative one) returns 1. `.fresh_only` read; a fresh entry (including a negative one) returns
immediately -- **unless** the type supplies a `needsRefresh` hook that immediately -- **unless** the type supplies a `needsRefresh` hook that
says the fresh entry is incomplete (see below). says the fresh entry is incomplete (see the freshness levels below).
2. `skip_network`: return any cached entry, even stale; else 2. `skip_network`: return any cached entry, even stale; else
`FetchFailed`. `FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns, subject to 3. Server sync (if configured); a fresh synced entry returns, subject to
the same `needsRefresh` check. the same `needsRefresh` check.
4. Provider fetch; on success write with the type's TTL; on 4. Provider fetch; on success write with the TTL from `ttlSpecFor`; on
`NotFound` write a negative entry; on `NoApiKey` return `NoApiKey`; `NotFound` write a negative entry; on `NoApiKey` return `NoApiKey`;
on any other transient error return `FetchFailed`. Neither of the on any other transient error return `FetchFailed`. Neither of the
last two poisons the cache. last two poisons the cache.
#### The three freshness levels
Dividends and earnings do not get a single freshness verdict; they get
three, in a fixed order. The user-facing explanation is in
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check);
this is where each level lives in code.
| Level | Name | Question | Implementation |
|-------|--------------------|-----------------------------------------------------------|------------------------------------------------------------|
| **1** | expiry clock | "Is this entry old?" | `#!expires=` vs now, enforced by `Store.read(.fresh_only)` |
| **2** | completeness check | "Is this entry missing something it should already have?" | `needsRefresh` hook, via `serveFreshOrDiscard` |
| **3** | recheck floor | "Did we already ask the provider recently?" | `askedRecently`, 12h |
**The ordering is the part to get right, and it is the opposite of the
natural reading.** Level 2 is a **second opinion on a "still fresh" verdict
from Level 1**, not an extra gate in front of a refresh:
- Level 1 says **stale** -> `Store.read(.fresh_only)` returns `null`, the
`if` body in step 1 never executes, and `serveFreshOrDiscard` is never
called. **Levels 2 and 3 do not run at all.** There is nothing for them
to contribute: a refetch is already happening.
- Level 1 says **fresh** -> Level 2 runs, and its whole job is to be able
to overrule that with "no it isn't, a payment has happened that this
entry doesn't contain."
- Level 2 says **incomplete** -> Level 3 runs, and can overrule Level 2
back to "serve the cache anyway" if we asked too recently.
So each level can only be reached by the previous one declining to
decide. If you ever find yourself reasoning about Level 2 firing on a
stale entry, the model is inverted.
`skip_network` short-circuits between Levels 1 and 2: offline mode serves
the fresh entry without consulting either later level, since it can never
refetch and discarding could only turn a served result into a failure.
`NoApiKey` is propagated rather than collapsed into `FetchFailed` `NoApiKey` is propagated rather than collapsed into `FetchFailed`
because it is the one distinction callers act on differently: because it is the one distinction callers act on differently:
`commands/earnings.zig` and `tui/earnings_tab.zig` both name the `commands/earnings.zig` and `tui/earnings_tab.zig` both name the
@ -333,67 +368,114 @@ implementation: it short-circuits mutual funds (no quarterly earnings
exist, so there is nothing to cache or fetch) and then delegates, exist, so there is nothing to cache or fetch) and then delegates,
supplying `earningsPostProcess` and `earningsNeedsRefreshHook`. supplying `earningsPostProcess` and `earningsNeedsRefreshHook`.
#### The two comptime hooks #### The three comptime hooks
`fetchCached` takes two optional comptime function pointers. Both default `fetchCached` takes three optional comptime function pointers. Passing
to "absent" by passing `null`, which is the behaviour every type had `null` for all three gives the behaviour every type had before they
before they existed. existed, which is what `getSplits` and `getOptions` do.
| Hook | Signature | Purpose | | Hook | Signature | Question it answers |
|-----------------|--------------------------------------------|-----------------------------------------------------| |----------------|--------------------------------------|-----------------------------------------------|
| `postProcess` | `fn (*T, Allocator) anyerror!void` | Recompute a derived field after parse | | `postProcess` | `fn (*T, Allocator) anyerror!void` | Any derived field to recompute after parsing? |
| `needsRefresh` | `fn ([]const T, Date) bool` | "This entry is fresh, but is it *complete*?" | | `needsRefresh` | `fn ([]const T, Date) bool` | This entry is fresh, but is it *complete*? |
| `ttlFor` | `fn ([]const T) cache.TtlSpec` | How long should this data stay fresh? |
Who passes what:
| Getter | postProcess | needsRefresh | ttlFor |
|---------------|------------------------|-----------------------------|----------------|
| `getDividends`| - | `dividendsNeedRefresh` | `dividendTtl` |
| `getEarnings` | `earningsPostProcess` | `earningsNeedsRefreshHook` | - |
| `getSplits` | - | - | - |
| `getOptions` | - | - | - |
`postProcess` is **not** for duping strings -- `Store.read` already does `postProcess` is **not** for duping strings -- `Store.read` already does
that. Today only `EarningsEvent` uses it (to rebuild `surprise`). that. Only `EarningsEvent` uses it, to rebuild `surprise`.
`needsRefresh` exists because freshness and completeness are different `needsRefresh` is consulted at **both** fresh-read sites (steps 1 and 3
questions. It is consulted at **both** fresh-read sites (step 1 and step above) through `serveFreshOrDiscard`, which owns the free so a caller
3) through `serveFreshOrDiscard`, which owns the free so a caller cannot cannot decide to discard and then leak. Covering step 3 is not optional:
decide to discard and then leak. Covering step 3 is not optional: with a with a `ZFIN_SERVER` configured, a sync can leave a still-incomplete entry
`ZFIN_SERVER` configured, a sync can leave a still-incomplete entry on on disk that step 3 would otherwise serve, defeating the mechanism
disk that step 3 would otherwise serve, defeating the mechanism entirely. (This is not theoretical - it was observed against a live
entirely. The hook is suppressed under `skip_network`, since offline server whose copy was missing the same record the local cache was.) The
mode never refetches and discarding could only turn a served result into hook is suppressed under `skip_network`, since offline mode never
a failure. refetches and discarding could only turn a served result into a failure.
The hook's branch is comptime-elided when the argument is `null`, which `ttlFor` exists because the right expiry time for dividends depends on the
is what lets types without a `freeSlice` (such as `Split`) keep records being written, not just on their type - see `dividendTtl`. It is
compiling. applied through `ttlSpecFor` at both write sites (the normal path and the
rate-limit retry) so a retry cannot stamp a different clock than the
first attempt would have.
#### Dividend smart refresh All three branches are comptime-elided when the argument is `null`, which
is what lets types without a `freeSlice` (such as `Split`) keep compiling.
`dividendsNeedRefresh` is the hook `getDividends` passes. It answers #### The recheck floor
true when at least one full cadence period has elapsed since the newest
cached ex-date and the distribution that should have landed in it is
still absent.
The problem it solves is that dividend TTL and dividend *availability* A `needsRefresh` hook has a failure mode that is easy to miss. When the
are uncorrelated. An operating company declares weeks ahead of the hook says something is overdue and the provider does not have it yet -
ex-date and pays about a month after it. An ETF cannot: its distribution the *normal* case, not an error - the cache is rewritten with a fresh
is the pass-through of whatever the fund earned, so the amount is not expiry but the same records. The hook therefore still says "overdue", and
published until roughly its own ex-date, and it pays 1-7 days later. The the next command asks again. On a quarter-end run touching fifteen funds
window in which the record exists but the cash has not yet landed is that is fifteen provider calls per command, repeated every time anything
about a week, so any longer TTL can be written before the record exists runs.
and still be fresh after the payment. Quarter-end clustering makes that
systematic rather than unlucky -- a dozen funds go ex within a few days
of each other, so one refresh pass puts all of them on the same expiry
shelf.
Cadence comes from `dividendCadenceDays`: the **median** of up to the `smart_refresh_recheck_interval` (12 hours) is the floor, enforced by
five newest gaps between cached ex-dates. Median rather than mean (a `askedRecently`. It applies to any type with a `needsRefresh` hook.
special distribution landing days after a regular one drags the mean) and
rather than minimum (the same outlier destroys the minimum outright, **It reads `#!expires=`, not `#!created=`, and that distinction is the
predicting a next ex-date days away and firing for the whole chase window whole trick.** `serializeWithMeta` re-stamps `created` on every write, and
after every payment). `newestExDates` sorts rather than trusting file dividend rows also arrive as a side effect of a Tiingo candle fetch via
order, because `writeSupplement`'s merge interleaves two providers' `writeSupplement`. So `created` means "file last touched", which a candle
records and a cadence computed from negative gaps would forecast into the refresh changes without anyone asking Polygon anything - keying the floor
past forever. off it would suppress a genuinely due refetch for 12 hours. `expires` is
moved only by a primary-provider fetch (`ExpiryPolicy.bump`); a supplement
deliberately puts the old value back (`.preserve`). There is a regression
test for exactly this.
Recovering the write time from `expires` is *exact*, not approximate,
because `computeExpires` derives its jitter from a hash of the symbol -
the same symbol always gets the same offset. Calling it with a zero "now"
returns precisely the TTL that was applied, so `expires - that` is the
write time to the second.
Two cases fall back to "ask again", which is the safe direction: a file
with no `#!expires=` at all, and a symbol that crossed from
unknown-schedule to known-schedule since it was written (stored under the
short TTL, recomputed with the long one, so the write looks older than it
was).
#### Dividend schedule prediction
`dividendsNeedRefresh` returns true when at least one full cadence period
has elapsed since the newest cached ex-date and the payment that should
have landed in it is still absent.
It exists because a dividend cache being *fresh* and being *complete* are
unrelated. An operating company declares weeks ahead of the ex-date and
pays about a month later. An ETF cannot: its distribution is the
pass-through of whatever the fund earned, so the amount does not exist
until roughly its own ex-date, and it pays 1-7 days later. The period in
which the record exists but the cash has not landed is about a week, so
any longer expiry can be written before the record exists and still be
fresh after the payment. Quarter-end clustering makes it systematic -- a
dozen funds go ex within a few days, so one refresh pass puts all of them
on the same expiry date with the same blind spot.
Cadence comes from `dividendCadenceDays`: the **median** of up to the five
newest gaps between cached ex-dates. Median rather than mean (a special
distribution landing days after a regular one drags the mean) and rather
than minimum (the same outlier destroys the minimum outright, predicting a
next ex-date days away and firing for the whole chase window after every
payment). `newestExDates` sorts rather than trusting file order, because
`writeSupplement`'s merge interleaves two providers' records and a cadence
computed from negative gaps would forecast into the past forever.
Three properties are load-bearing and each has its own test: Three properties are load-bearing and each has its own test:
- **A full period must elapse.** Otherwise it refires immediately after - **A full period must elapse.** Otherwise it refires immediately after a
a successful fetch, on the record it just stored. successful fetch, on the record it just stored.
- **It rolls forward.** The expected date is the latest one at or before - **It rolls forward.** The expected date is the latest one at or before
today, not the first one after the newest cached record -- so a symbol today, not the first one after the newest cached record -- so a symbol
two periods behind stays recoverable instead of being permanently past two periods behind stays recoverable instead of being permanently past
@ -402,17 +484,66 @@ Three properties are load-bearing and each has its own test:
unbounded chase refetches forever whenever a sponsor skips a period or unbounded chase refetches forever whenever a sponsor skips a period or
the cadence estimate is wrong. the cadence estimate is wrong.
Known blind spots, all three deliberately left to the TTL backstop `dividendTtl` then picks the expiry from the same cadence calculation:
rather than papered over with a more eager statistic that would cost `Ttl.dividends_scheduled` (14 days) when a cadence exists, because
requests on every symbol to protect against a rare event on one: prediction is what finds due payments; `Ttl.dividends` (6 days) when it
does not, because then the clock is the only mechanism and it has to
expire before the next weekly review. Six is the largest value that
guarantees that -- 6 x 1.11 jitter = 6.66 days, under 7, where 7 would
reach 7.77. `"TTL constants are reasonable"` in store.zig asserts that
property rather than just the number.
Known blind spots, all left to the expiry time rather than papered over
with a more eager statistic that would cost requests on every symbol to
protect against a rare event on one:
- A **cadence change** (quarterly to monthly) keeps the old median for - A **cadence change** (quarterly to monthly) keeps the old median for
several periods, so the forecast runs late. several periods, so the forecast runs late.
- An **off-cycle special** is on no schedule, so the predicate never - An **off-cycle special** is on no schedule, so prediction never fires
fires for it. for it.
- A symbol with **fewer than three cached records** has no cadence at - A symbol with **fewer than three cached records** has no cadence at all
all -- which is the right direction, since a newly-bought holding has -- which is exactly the case that gets the 6-day expiry.
no schedule to be late against.
#### Why Tiingo can never discover a new dividend
Tiingo returns dividends and splits alongside candles in a single request
(`populateAllFromTiingo`), written through `writeSupplement` so the merge
lands new records without moving the primary provider's `#!expires=`
clock. That makes it look like a free second source for Level 2, and it is
worth being explicit that it is not - for two independent reasons.
**Structural.** Tiingo carries a dividend as a `divCash` field on a daily
price bar, so a payment can only appear once a bar exists for it. A
declared-but-not-yet-ex dividend has no bar and therefore cannot be
represented at all. This is exactly why Polygon is the primary source -
see the comment in `fetchFromProvider`, which cites a forward-dated ARCC
ex-date as the case Tiingo's response cannot express.
**Circular.** `populateAllFromTiingo` is reached from only two places:
1. A **cold candle cache** - the symbol has no bars at all
(`getCandles`'s full-history path). Here Tiingo genuinely earns its
keep: one request bulk-loads the symbol's entire dividend history.
2. An **adjustment-basis restatement** - cached `adj_close` values predate
a corporate action they should reflect, so the series is refetched in
full.
Trigger 2 is decided by `freshness.adjustmentBasisStale` fed by
`freshness.newestCorporateAction`, and that function reads the **cached
dividend file** to find the newest action. So zfin only knows to restate
*after* the dividend record is already on disk - which means Polygon must
have supplied it first. By the time Tiingo's rows merge in, they add
nothing about that payment.
Net: Tiingo bulk-loads history and fills field-level gaps in old records.
Every *new* dividend arrives via Polygon. This is also why the Level 3
floor reads `#!expires=` and not `#!created=` - a Tiingo supplement
rewrites the file, and therefore `created`, without having asked Polygon
anything.
For the user-facing version of all this, including a worked quarterly
example, see
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
### `loadAllPrices` (portfolio + watchlist price load) ### `loadAllPrices` (portfolio + watchlist price load)
@ -652,26 +783,28 @@ the upstream provider on every request.
## Key code references ## Key code references
| Concern | Location | | Concern | Location |
|---------------------------------|-----------------------------------------------------| |---------------------------------|------------------------------------------------------------------------|
| Data-access entry point | `DataService` - `src/service.zig` | | Data-access entry point | `DataService` - `src/service.zig` |
| Per-type generic fetch | `fetchCached` - `src/service.zig` | | Per-type generic fetch | `fetchCached` - `src/service.zig` |
| Fresh-but-incomplete decision | `serveFreshOrDiscard` - `src/service.zig` | | Fresh-but-incomplete decision | `serveFreshOrDiscard` - `src/service.zig` |
| Dividend schedule check | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` | | Recheck floor (12h) | `askedRecently`, `smart_refresh_recheck_interval` - `src/service.zig` |
| Dividend schedule prediction | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` |
| Dividend expiry choice | `dividendTtl`, `ttlSpecFor` - `src/service.zig` |
| Earnings smart refresh | `earningsNeedsRefresh`, `earningsNeedsRefreshHook` - `src/service.zig` | | Earnings smart refresh | `earningsNeedsRefresh`, `earningsNeedsRefreshHook` - `src/service.zig` |
| Candle fetch + incremental | `getCandles` - `src/service.zig` | | Candle fetch + incremental | `getCandles` - `src/service.zig` |
| Batch price load | `loadAllPrices` - `src/service.zig` | | Batch price load | `loadAllPrices` - `src/service.zig` |
| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` | | Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` |
| Cache store, read/write | `Store` - `src/cache/store.zig` | | Cache store, read/write | `Store` - `src/cache/store.zig` |
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` | | Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` | | TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` | | Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| Negative-cache veto | `shouldNegativeCache` - `src/service.zig` | | Negative-cache veto | `shouldNegativeCache` - `src/service.zig` |
| Candle provenance labels | `CandleProvider` - `src/cache/store.zig` | | Candle provenance labels | `CandleProvider` - `src/cache/store.zig` |
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` | | NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` | | Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` | | Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
| Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` | | Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
For the user-facing summary and the `--refresh-data` walkthrough, see For the user-facing summary and the `--refresh-data` walkthrough, see
[Caching and data freshness](../explanation/caching.md). [Caching and data freshness](../explanation/caching.md).

194
docs/explanation/caching.md
View file

@ -39,44 +39,172 @@ The `--refresh-data` policy decides which tiers run:
Different data ages at different rates, so each type has its own TTL: Different data ages at different rates, so each type has its own TTL:
| Data type | TTL | Why | | Data type | TTL | Why |
|---------------|---------------|----------------------------------------------------------------------------------| |---------------|--------------------|----------------------------------------------------------------------------------|
| Daily candles | market-aware | Keyed to the next time a fresh bar is expected, not a rolling window (see below) | | Daily candles | market-aware | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
| Dividends | 6 days\* | Short: an ETF distribution is only knowable for a few days before it pays | | Dividends | 6 or 14 days\* | Depends on whether we know the symbol's payment schedule |
| Splits | 14 days | Rare corporate events, announced weeks ahead | | Splits | 14 days | Rare, and announced weeks ahead |
| Options | 1 hour | Prices move continuously when markets are open | | Options | 1 hour | Prices move continuously when markets are open |
| Earnings | 30 days\*\* | Quarterly; smart-refreshed around announcements | | Earnings | 30 days\*\* | Quarterly, with an early re-check around announcements |
| ETF profiles | ~30 days | Holdings and weights change slowly | | ETF profiles | ~30 days | Holdings and weights change slowly |
| Quotes | never cached | Meant to be a live price check | | Quotes | never cached | Meant to be a live price check |
\* **Dividend smart refresh:** even inside the 6-day window, cached \* The expiry time is only the first of three checks. See
dividends re-fetch automatically once a distribution the symbol's own [Dividends: a three-level freshness check](#dividends-a-three-level-freshness-check).
schedule says is due has not arrived. This exists because a dividend
cache being *fresh* and being *complete* are different things.
An operating company declares a dividend weeks ahead of the ex-date and \*\* **Earnings** use the same three-level check as dividends, with a
pays about a month after it, so there is plenty of time to notice. An different Level 2: instead of asking "is a payment overdue?", it asks "has
**ETF does not work that way.** Its distribution is the pass-through of a report date passed with the actual result still missing?". So results
whatever the fund earned, so the amount is not published until roughly appear promptly after an announcement without daily polling. Level 3
its own ex-date -- and it pays 1 to 7 days later. The entire window in applies too - one re-check per symbol per 12 hours.
which the record exists but the cash has not yet landed is about a week.
A long TTL straddles that window whole: a cache written a few days ## Dividends: a three-level freshness check
before the ex-date stays "fresh" until well after the payment, so the
distribution is invisible to the run that needed it. Quarter-end makes
it systematic rather than unlucky, because a dozen funds go ex within
the same few days and one refresh pass puts all of them on the same
expiry shelf.
So the schedule is the gate, and the TTL is only the backstop. It still For most data a plain expiry time is enough. For dividends it is not, so
matters, because the schedule check cannot see three things: a holding dividends (and earnings) go through **three checks instead of one**. The
too new to have an established cadence, an off-cycle special order matters, and it is the thing most easily got backwards:
distribution, and a symbol with fewer than three cached records.
| Level | Name | Question it answers | Runs when |
|-------|------------------------|-----------------------------------------------------------|---------------------------------------------------|
| **1** | **expiry clock** | "Is this cached entry old?" | always |
| **2** | **completeness check** | "Is this entry missing something it should already have?" | **only if Level 1 said the entry is still fresh** |
| **3** | **recheck floor** | "Did we already ask the provider about this recently?" | only if Level 2 said something is missing |
**Read that middle row carefully, because the intuition runs the other
way.** Level 2 is *not* an extra hurdle before a refresh. It is a
**second opinion on a "no" from Level 1**. If the expiry clock says the
entry is old, zfin just refetches and Level 2 is never consulted at all -
there is nothing for it to add, because a refresh is already happening.
Level 2 only earns its keep in the opposite case: the clock says *"this
is still fine, serve it"*, and Level 2 gets to answer *"no it isn't -
a payment has happened that this entry doesn't contain."*
Level 3 then stops Level 2 from being too eager, by refusing to ask the
provider about the same symbol more than once every 12 hours.
### Why one expiry time isn't enough
A regular company announces a dividend weeks before the ex-date and pays
about a month after it. There is plenty of time to notice. An ETF is
different. Its distribution is simply whatever the fund earned, passed
through to you, so the amount does not exist until roughly the fund's own
ex-date -- and it pays one to seven days later. The whole period between
"the payment becomes knowable" and "the cash is in your account" is about
a week.
A two-week expiry time sails straight over that. Suppose the cache is
refreshed a few days before an ETF goes ex. It is then considered fresh
for another eleven days, which covers the ex-date, the payment, *and* the
weekly review that was supposed to explain the deposit. The dividend is
invisible exactly when it matters. And it is not bad luck: dozens of
funds go ex within the same few days at each quarter end, so one refresh
pass puts all of them on the same expiry date and every one of them has
the same blind spot.
Level 2 exists to catch precisely that: it works out the symbol's own
payment schedule from the ex-dates already in the cache, and speaks up
when a payment is due that the cache does not have.
### Two things that are easy to picture wrongly
**The expiry clock is a duration, not a date.** When a record is cached,
zfin stamps "expires 14 days from now". It does not store "next check due
on 30 September". Each refetch re-stamps the duration from scratch.
**The expected payment date is never stored at all.** Level 2 recomputes
it on every single cache read, from whatever ex-dates are currently
cached. Nothing on disk holds it.
So capturing a payment does not "set the clock to" the next expected
payment. The two mechanisms run independently, on different schedules.
### A worked example
Take an ETF that has paid on 1 January, 1 April, 1 July and 1 October for
the last five years. The newest payment in the cache is 1 July 2026.
Level 2 measures the gaps between the most recent ex-dates -- 91, 90, 92,
92 and 91 days -- and takes the middle value: **91 days**. Added to 1
July, that makes **30 September** the date the next payment is expected.
(The real date is 1 October. Being a day out does not matter, as you will
see.)
| When | What the levels say | Result |
|------------------|---------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| 2 July - 29 Sept | L1: fresh, then stale every 14 days. L2: nothing due. | Nothing is expected, so only the clock causes fetches: **about six over the quarter** (91 days, at 14 days a time). |
| 30 Sept - 14 Oct | L1: fresh. L2: **a payment is overdue.** L3: allows one request/12 hours. | The provider is asked, at most twice a day no matter how many commands you run. Stops the moment the new record arrives. |
| 15 Oct onwards | L2: gave up -- two weeks past the expected date. | Back to the clock alone. Chasing a schedule that is clearly wrong would refetch forever. |
On 30 September the provider probably has nothing yet, because the fund
has not gone ex. That request is wasted, and it is the price of the estimate
being a day early. On 1 October the record appears, gets cached, and
Level 2 immediately goes quiet again -- it now expects the next payment
around 1 January.
The estimate also corrects itself. Once the 1 October record is cached,
the newest gaps median to 92 days rather than 91, which puts the next
expected date exactly on 1 January 2027.
### Why Level 1 has two different lengths
How long the expiry clock runs depends on whether Level 2 can do its job
for that symbol:
- **14 days when the payment schedule is known.** A symbol with a few
years of history is found by Level 2, so the clock only has to cover
the one thing a schedule cannot predict: a one-off special
distribution. Making it short as well would re-check every symbol
every few days to guard against a problem already solved.
- **6 days when the schedule is not known.** With fewer than three
cached payments there is no pattern to work from -- typically a holding
you have just bought. Level 2 declines, so the clock is the *only*
thing that will ever make zfin look again, and it has to expire before
a weekly review. Six days guarantees that; seven would not, because
the +/-11% spread applied to expiry times can stretch seven days to 7.8.
### Why Level 3 exists
When Level 2 says a payment is due and the provider does not have it yet,
nothing in the cache changes except the expiry time. Level 2 therefore
still says "due", so the next command would ask again -- and the one
after that. On a quarter-end run touching fifteen funds, that is fifteen
provider calls per command, every time you run anything.
So Level 3 allows a symbol to be asked about at most once every 12 hours.
Twelve hours sits well inside the shortest gap between a fund going ex
and paying (one day, in a real portfolio), while still being far more
frequent than the weekly reconcile that actually uses the answer.
### What about Tiingo?
Tiingo returns dividend data as a side effect of fetching prices, which
raises the obvious question of whether it can spot a new payment for
free. It cannot, and it is worth knowing that it never will.
Tiingo reports a dividend as a field on a daily price bar, so the payment
only shows up once it has already traded -- there is no such thing as a
forward-looking dividend in that data. Polygon is the primary source
precisely because it *does* carry declared-but-not-yet-paid dividends.
And the code path that merges Tiingo's dividend rows in only runs on a
cold cache or when a known corporate action forces a price-history
restatement, which means Polygon has already supplied the record by then.
So Tiingo is genuinely useful for bulk-loading a new symbol's entire
dividend history in one request, and for filling gaps in old records. It
is simply never the thing that notices a *new* payment.
### What none of this catches
Three cases where Level 2 cannot help, all left to Level 1:
- **A one-off special distribution.** It is on no schedule, so it is
found only when the expiry clock runs out.
- **A fund that changes its schedule** -- quarterly to monthly, say. The
old pattern persists for a few periods, so the estimate runs late.
- **A symbol with fewer than three cached payments.** No pattern to work
from, which is exactly why that case gets the 6-day clock.
\*\* **Earnings smart refresh:** even inside the 30-day window, cached
earnings re-fetch automatically once an earnings date has passed but
the cache still lacks the actual result -- so numbers appear promptly
after an announcement without daily polling.
## Market-aware candle freshness ## Market-aware candle freshness

17
docs/guides/offline-and-refresh.md
View file

@ -66,7 +66,7 @@ In `auto` mode, each data type has its own time-to-live:
| Data | TTL | | Data | TTL |
|--------------------|------------------------------------------------| |--------------------|------------------------------------------------|
| Daily candles | ~24 hours | | Daily candles | ~24 hours |
| Dividends | 6 days (refreshed early once a payout is due) | | Dividends | 6 or 14 days (see note below) |
| Splits | 14 days | | Splits | 14 days |
| Options | 1 hour | | Options | 1 hour |
| Earnings | 30 days (refreshed early once a result is due) | | Earnings | 30 days (refreshed early once a result is due) |
@ -74,8 +74,19 @@ In `auto` mode, each data type has its own time-to-live:
| Quotes | never cached | | Quotes | never cached |
So a second `portfolio` run the same day is instant and network-free So a second `portfolio` run the same day is instant and network-free
without any flag. For the full rationale, see without any flag.
[Caching and data freshness](../explanation/caching.md).
Dividends and earnings are the exception to "the clock decides". For
those, once the clock says an entry is still fresh, a second check asks
whether it is nonetheless *incomplete* -- for dividends, whether a payment
the symbol's own schedule predicts is missing from it. An ETF's amount is
only knowable for a few days before the cash arrives, so a fixed expiry
can miss the whole event. A third check stops the second one asking the
provider more than once every 12 hours. The dividend clock itself is 14
days when the schedule is known and 6 days when it is not.
For the full rationale and a worked example, see
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
## Inspecting and clearing the cache ## Inspecting and clearing the cache

37
docs/reference/providers.md
View file

@ -10,25 +10,26 @@ how keys are configured see
## Summary ## Summary
| Data | Provider | Auth | Free-tier limit | Cache TTL | | Data | Provider | Auth | Free-tier limit | Cache TTL |
|-----------------------|------------------|-------------------------------|------------------------------------------------|------------| |-----------------------|------------------|-------------------------------|------------------------------------------------|-------------|
| Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h | | Daily candles (OHLCV) | Tiingo | `TIINGO_API_KEY` | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h |
| Real-time quotes | Yahoo | none | unofficial | never | | Real-time quotes | Yahoo | none | unofficial | never |
| Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never | | Real-time IEX quotes | Tiingo | `TIINGO_API_KEY` | any tier (1 req per connect/batch) | never |
| Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never | | Quote fallback | TwelveData | `TWELVEDATA_API_KEY` | 8/min, 800/day | never |
| Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 6 days\* | | Dividends | Polygon | `POLYGON_API_KEY` | 5/min | 6 or 14 d\* |
| Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days | | Splits | Polygon | `POLYGON_API_KEY` | 5/min | 14 days |
| Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour | | Options chains | CBOE | none | ~30/min (self-imposed) | 1 hour |
| Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days | | Earnings | FMP | `FMP_API_KEY` | 250 req/day | 30 days |
| ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days | | ETF profiles | SEC EDGAR | `ZFIN_USER_EMAIL` | 10/sec | ~90 days |
| Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived | | Classification | Wikidata + EDGAR | `ZFIN_USER_EMAIL` | no daily quota | long-lived |
| CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite | | CUSIP lookup | OpenFIGI | `OPENFIGI_API_KEY` (optional) | higher with key | indefinite |
\* The dividend TTL is only a backstop. Cached dividends also re-fetch \* Dividends get three freshness checks rather than one. The expiry time
whenever a distribution the symbol's own schedule says is due has not shown here is only the first; when it says an entry is still fresh, a
arrived, because an ETF's amount is not published until roughly its second check asks whether a payment the symbol's schedule predicts is
ex-date and it pays within a week -- see missing from it, and a third limits how often that can hit Polygon. 14
[Caching and data freshness](../explanation/caching.md). days when the schedule is known, 6 days when it is not. See
[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
## Where to get a key ## Where to get a key

99
src/cache/store.zig vendored
View file

@ -35,27 +35,44 @@ pub const Ttl = struct {
const s_per_day = std.time.s_per_day; const s_per_day = std.time.s_per_day;
/// Historical candles older than 1 day never expire /// Historical candles older than 1 day never expire
pub const candles_historical: i64 = -1; // infinite pub const candles_historical: i64 = -1; // infinite
/// Dividend data. Six days, and the short value is load-bearing: /// How long a dividend cache entry stays fresh when we DON'T know the
/// an ETF's distribution is not retrievable until roughly its own /// symbol's payment schedule. Six days.
/// ex-date, and it pays 1-7 days later, so the entire window in
/// which a new record exists but the cash has not yet landed is
/// about a week. A 14-day TTL straddles that window whole - a cache
/// written days before the ex-date stays "fresh" until well after
/// the payment, and the distribution is invisible for the run that
/// needed it.
/// ///
/// Quarter-end clustering is what makes it systematic rather than /// This is the cautious case. With no schedule to predict from, the
/// unlucky: a dozen funds go ex within the same three days, so one /// clock is the only thing that will ever make us look again, so it
/// refresh pass writes a dozen entries onto the same expiry shelf /// has to tick faster than the weekly review - otherwise a newly
/// and every one of them straddles the next quarter's payment. /// bought holding's first distribution can be paid and spent before
/// anything notices. Six days is the largest value that still
/// guarantees it: the +/-11% jitter tops out at 6.66 days, which is
/// under 7. Seven days would reach 7.77 and could stay "fresh"
/// straight through a review.
/// ///
/// This is the BACKSTOP, not the mechanism. `dividendsNeedRefresh` /// It is also what `DataType.dividends.ttl()` returns, which is what
/// in service.zig does the real work by asking whether a scheduled /// `writeSupplement` falls back to when there is no existing file -
/// distribution has come due; the TTL covers the three cases that /// i.e. the first time we ever see a symbol. That is exactly the case
/// predicate cannot see - a holding too new to have a cadence, an /// this value is for.
/// off-cycle special, and a symbol with fewer than three cached ///
/// records. /// See `dividends_scheduled` for the case where we DO know the
/// schedule, and `dividendsNeedRefresh` in service.zig for how the
/// schedule is worked out.
pub const dividends: i64 = 6 * s_per_day; pub const dividends: i64 = 6 * s_per_day;
/// How long a dividend cache entry stays fresh when we DO know the
/// symbol's payment schedule. Fourteen days.
///
/// Long on purpose. Once a symbol has enough history to establish a
/// cadence, `dividendsNeedRefresh` predicts when the next payment is
/// due and asks the provider then - so the clock is no longer how we
/// find out. Making it short as well would re-check every symbol
/// every few days to protect against a problem the prediction has
/// already solved.
///
/// What the clock still covers here is the one thing a schedule
/// cannot predict: an off-cycle special distribution. That can be up
/// to 14 days late being noticed, which is the deliberate price of
/// not re-checking every symbol constantly.
pub const dividends_scheduled: i64 = 14 * s_per_day;
/// Split data refreshes biweekly. Deliberately NOT shortened /// Split data refreshes biweekly. Deliberately NOT shortened
/// alongside `dividends`: splits are announced weeks to months /// alongside `dividends`: splits are announced weeks to months
/// ahead of their effective date, so a forward-looking record is /// ahead of their effective date, so a forward-looking record is
@ -249,16 +266,16 @@ pub const DataType = enum {
/// ///
/// Jitter assignments: /// Jitter assignments:
/// ///
/// - 11% on dividends (6d base, ~1.3d total spread) and splits /// - 11% on dividends and splits. Dividends have two base values
/// (14d base, ~3d). The two no longer share a base - see /// (6d unscheduled / 14d scheduled - see `Ttl.dividends`) and
/// `Ttl.dividends` for why only dividends were shortened - but /// splits one (14d); all three keep the same percentage, because
/// they keep the same percentage because the goal is the same: /// the goal is the same: a daily cron should see a portfolio's
/// a daily cron should see a portfolio's symbols expire across /// symbols expire across several days rather than all at once.
/// several days instead of all in lockstep. Jitter is only a ///
/// load-spreading measure here, never a correctness one: /// Jitter is a load-spreading measure only, never a correctness
/// `dividendsNeedRefresh` is what guarantees a due /// one. `dividendsNeedRefresh` is what guarantees a due payment
/// distribution is seen, and a run that depended on jitter /// is noticed; a run that depended on jitter landing favourably
/// landing favourably would be relying on luck. /// would be relying on luck.
/// ///
/// - 8% on the longer-TTL types (classification 90d, /// - 8% on the longer-TTL types (classification 90d,
/// etf_metrics 90d, entity_facts 30d, earnings 30d, /// etf_metrics 90d, entity_facts 30d, earnings 30d,
@ -3408,20 +3425,24 @@ test "TTL constants are reasonable" {
// Latest candles use a market-aware expiry computed per-write by // Latest candles use a market-aware expiry computed per-write by
// market.nextCandleExpiry (no fixed TTL constant here anymore). // market.nextCandleExpiry (no fixed TTL constant here anymore).
// Dividends refresh every six days, splits biweekly. The two used // Dividends have two values depending on whether we know the
// to share a 14-day constant; they were deliberately split apart // symbol's payment schedule; splits have one. See `Ttl.dividends`.
// because an ETF distribution is only retrievable for the few days
// between its ex-date and its payment, while a split is announced
// weeks ahead. See `Ttl.dividends`.
try std.testing.expectEqual(@as(i64, 6 * std.time.s_per_day), Ttl.dividends); try std.testing.expectEqual(@as(i64, 6 * std.time.s_per_day), Ttl.dividends);
try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.dividends_scheduled);
try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.splits); try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.splits);
// The relationship, not just the values: a dividend TTL longer than // THE PROPERTY, not just the value. When we don't know a symbol's
// the ex-to-pay window is what made distributions invisible for the // schedule the clock is the only thing that will make us look again,
// run that needed them. Every observed ETF pays within 7 days of // so it has to expire before the next weekly review - including at
// going ex, so this bound is the property the constant exists to // the top of the jitter band. 6 days * 1.11 = 6.66, under 7. At 7
// satisfy - if someone lengthens it, this fails and says why. // days the band reaches 7.77 and an entry could stay fresh straight
try std.testing.expect(Ttl.dividends <= 7 * std.time.s_per_day); // through a review, which is the bug this guards against.
//
// Deliberately NOT asserted for `dividends_scheduled`: there the
// prediction finds due payments, so a long clock is correct.
const jitter = DataType.dividends.ttl().jitter_pct;
const worst_case = Ttl.dividends + @divFloor(Ttl.dividends * @as(i64, jitter), 100);
try std.testing.expect(worst_case < 7 * std.time.s_per_day);
// Options refresh hourly // Options refresh hourly
try std.testing.expectEqual(@as(i64, std.time.s_per_hour), Ttl.options); try std.testing.expectEqual(@as(i64, std.time.s_per_hour), Ttl.options);

601
src/service.zig
View file

@ -489,38 +489,165 @@ pub const DataService = struct {
return cache.Store.init(self.io, self.allocator, self.config.cache_dir); return cache.Store.init(self.io, self.allocator, self.config.cache_dir);
} }
/// Whether a FRESH cache read may be served as-is. Returns false - /// LEVEL 3 of the three-level freshness check: the shortest gap
/// having already freed `data` - when the smart-refresh hook says the /// between two provider asks for the same symbol, when the ask is
/// entry is fresh but incomplete. /// driven by the Level 2 completeness check rather than by the Level 1
/// expiry clock. Applies to every type with a `needsRefresh` hook -
/// dividends and earnings today.
///
/// WHY THIS EXISTS. When a hook says something is overdue, we ask the
/// provider. Often the provider does not have it yet - that is the
/// normal case, not a failure, because an ETF publishes its amount
/// around its own ex-date and FMP posts an actual some hours after the
/// report. The answer is "nothing new", and the cache gets rewritten
/// with a fresh clock but the SAME records. So the hook still says
/// "overdue", and the very next command asks again.
///
/// Without a floor that is one provider call per command, not per day.
/// A run touching fifteen symbols in their due windows makes fifteen
/// calls, and the next run ninety seconds later makes fifteen more. At
/// Polygon's four-per-minute limit that is four minutes of waiting,
/// repeated.
///
/// WHY TWELVE HOURS. The floor is also the worst case for how late we
/// notice something, so it has to stay well inside the shortest gap
/// between an event and the money moving. For dividends the tightest
/// real ex-to-pay gap is one day (QTUM), so twelve hours leaves half a
/// day of margin. Longer buys almost nothing: asking once a day is
/// already far more often than the weekly reconcile that uses the
/// answer.
const smart_refresh_recheck_interval: i64 = 12 * std.time.s_per_hour;
/// LEVELS 2 AND 3 of the three-level freshness check. Returns true to
/// serve the cached entry as-is; returns false - having already freed
/// `data` - to discard it and refetch.
///
/// ── The three levels, and why the order reads backwards ──────
///
/// Level 1 expiry clock "Is this entry old?"
/// Level 2 completeness check "Is it missing something it should
/// already have?"
/// Level 3 recheck floor "Did we just ask the provider?"
///
/// LEVEL 2 IS A SECOND OPINION ON A "STILL FRESH" VERDICT, not an
/// extra gate in front of a refresh. That is the thing to get right,
/// because the intuition runs the other way.
///
/// This function is only ever called on a FRESH entry, because
/// `fetchCached` reaches it through `Store.read(.fresh_only)` - which
/// returns null when Level 1 says the entry is stale. So when Level 1
/// says "stale", Levels 2 and 3 never run at all; a refetch is already
/// happening and they have nothing to contribute. Level 2 only earns
/// its keep in the opposite case: Level 1 says "this is fine, serve
/// it", and Level 2 gets to answer "no it is not - a payment has
/// happened that this entry does not contain".
///
/// Level 3 then exists to stop Level 2 being too eager, and can
/// overrule it back to "serve the cache".
///
/// `skip_network` short-circuits between Levels 1 and 2: offline mode
/// never refetches, so discarding a usable entry could only turn a
/// served result into a failure.
/// ///
/// One function rather than two inline blocks because `fetchCached` /// One function rather than two inline blocks because `fetchCached`
/// has two fresh-cache read sites (local, and post-server-sync) and /// has two fresh-cache read sites (local, and post-server-sync) and
/// the rule must be identical at both. It owns the free so a caller /// the rule must be identical at both. It owns the free so a caller
/// cannot decide to discard and then leak. /// cannot decide to discard and then leak.
///
/// Suppressed under `skip_network`: offline mode never refetches, so
/// discarding a usable entry could only turn a served result into a
/// failure. Same rule `getEarnings` applies.
fn serveFreshOrDiscard( fn serveFreshOrDiscard(
self: *DataService, self: *DataService,
comptime T: type, comptime T: type,
comptime needsRefresh: ?*const fn ([]const T, Date) bool, comptime needsRefresh: ?*const fn ([]const T, Date) bool,
data: []const T, comptime ttlFor: ?*const fn ([]const T) cache.TtlSpec,
symbol: []const u8,
cached: cache.Store.CacheResult(T),
skip_network: bool, skip_network: bool,
) bool { ) bool {
if (needsRefresh) |hook| { if (needsRefresh) |hook| {
if (skip_network) return true;
// wall-clock required: the hook asks whether a scheduled // wall-clock required: the hook asks whether a scheduled
// event has come due, which is a question about the actual // event has come due, which is a question about the actual
// current day. Threading `today` down would put a date // current day. Threading `today` down would put a date
// parameter on four public getters for one type's benefit. // parameter on four public getters for one type's benefit.
if (!skip_network and hook(data, fmt.todayDate(self.io))) { if (!hook(cached.data, fmt.todayDate(self.io))) return true;
T.freeSlice(self.allocator, data); if (self.askedRecently(T, ttlFor, symbol, cached)) {
return false; log.debug("{s}: {s} is overdue but we asked within the last {d}h; serving cache", .{ symbol, @tagName(comptime cache.Store.dataTypeFor(T)), @divFloor(smart_refresh_recheck_interval, std.time.s_per_hour) });
return true;
} }
T.freeSlice(self.allocator, cached.data);
return false;
} }
return true; return true;
} }
/// LEVEL 3 of the three-level freshness check: did we already ask the
/// provider about this symbol inside `smart_refresh_recheck_interval`?
///
/// Only reached when Level 2 has already said the entry is incomplete,
/// so a true answer here means "incomplete, but asking again this soon
/// would be wasted" - see `serveFreshOrDiscard` for the full ordering.
///
/// WHY `#!expires=` AND NOT `#!created=`. Both directives sit in the
/// same file and `created` looks like the obvious choice, but it is
/// the wrong one. `serializeWithMeta` re-stamps `created` on EVERY
/// write, and dividend rows also arrive as a side effect of a Tiingo
/// candle fetch (`writeSupplement`). So `created` answers "when was
/// this file last touched", which a candle refresh changes without
/// anyone asking Polygon anything.
///
/// `expires` answers the question we actually want. Only a fetch from
/// the primary provider moves it (`ExpiryPolicy.bump`); a supplement
/// deliberately puts the old value back (`.preserve`). So it marks the
/// last time we really asked.
///
/// Recovering the write time from it is exact rather than approximate,
/// because `computeExpires` derives its jitter from a hash of the
/// symbol - the same symbol always gets the same offset. Passing a
/// zero "now" therefore returns precisely the TTL that was applied,
/// and subtracting it from `expires` gives the write time to the
/// second.
///
/// Two cases fall back to "no, ask again", which is the safe
/// direction:
/// - No `#!expires=` in the file at all.
/// - The symbol crossed from unknown-schedule to known-schedule
/// since it was written, so it was stored under the short TTL and
/// we recompute with the long one. That makes the write look
/// older than it was, so we ask sooner than strictly needed.
fn askedRecently(
self: *DataService,
comptime T: type,
comptime ttlFor: ?*const fn ([]const T) cache.TtlSpec,
symbol: []const u8,
cached: cache.Store.CacheResult(T),
) bool {
const expires = cached.expires orelse return false;
const spec = if (ttlFor) |f| f(cached.data) else comptime cache.Store.dataTypeFor(T).ttl();
// `computeExpires` from a zero epoch returns the jittered TTL
// itself, which is what makes this exact - see the doc above.
const applied_ttl = cache.computeExpires(0, spec, symbol);
const written_at = expires - applied_ttl;
// wall-clock required: "recently" is relative to now.
const now_s = std.Io.Timestamp.now(self.io, .real).toSeconds();
return now_s - written_at < smart_refresh_recheck_interval;
}
/// The TTL to stamp on a write: from the records when the type
/// supplies a `ttlFor` hook, otherwise from the type alone.
///
/// Separate from the call sites because `fetchCached` writes in two
/// places (the normal path and the rate-limit retry) and they must
/// agree - a retry that stamped a different clock than the first
/// attempt would be a silently different freshness policy for
/// nobody's benefit.
fn ttlSpecFor(
comptime T: type,
comptime ttlFor: ?*const fn ([]const T) cache.TtlSpec,
items: cache.Store.DataFor(T),
) cache.TtlSpec {
if (ttlFor) |f| return f(items);
return comptime cache.Store.dataTypeFor(T).ttl();
}
/// Generic fetch-or-cache for simple data types (dividends, splits, options). /// Generic fetch-or-cache for simple data types (dividends, splits, options).
/// Checks cache first; on miss, fetches from the appropriate provider, /// Checks cache first; on miss, fetches from the appropriate provider,
/// writes to cache, and returns. On permanent fetch failure, writes a negative /// writes to cache, and returns. On permanent fetch failure, writes a negative
@ -530,20 +657,35 @@ pub const DataService = struct {
/// returns FetchFailed on cache miss without touching the network. /// returns FetchFailed on cache miss without touching the network.
/// `opts.force_refresh = true` -> treats cache as stale and fetches. /// `opts.force_refresh = true` -> treats cache as stale and fetches.
/// ///
/// `needsRefresh` is the smart-refresh hook: given a FRESH cache /// `needsRefresh` is LEVEL 2 of the three-level freshness check: given
/// entry it answers "is this nonetheless incomplete?". It exists /// a FRESH cache entry it answers "is this nonetheless incomplete?".
/// because freshness and completeness are different questions - /// It exists because freshness and completeness are different
/// `dividendsNeedRefresh` documents the case that forced it. Null /// questions - `dividendsNeedRefresh` documents the case that forced
/// means TTL is the only gate, which is the behaviour every type had /// it. Null means the expiry clock is the only gate, which is the
/// before the hook existed. The hook's branch is comptime-elided for /// behaviour every type had before the hook existed.
/// a null argument, so types with no `freeSlice` (Split) stay ///
/// compilable. /// NOTE WHERE IT SITS. Levels 2 and 3 live behind the
/// `Store.read(.fresh_only)` calls below, which return null when Level
/// 1 (the expiry clock) says the entry is stale. So a stale entry
/// bypasses them entirely - they can only ever overrule a "still
/// fresh" verdict, never reinforce a stale one. `serveFreshOrDiscard`
/// has the full ordering.
///
/// The hook's branch is comptime-elided for a null argument, so types
/// with no `freeSlice` (Split) stay compilable.
///
/// `ttlFor` sets LEVEL 1's length from the records being written
/// rather than from the type alone. Only dividends need it, because
/// the right clock depends on whether the records establish a payment
/// schedule - see `dividendTtl`. Null means "use `DataType.ttl()`",
/// which is what every other type does.
fn fetchCached( fn fetchCached(
self: *DataService, self: *DataService,
comptime T: type, comptime T: type,
symbol: []const u8, symbol: []const u8,
comptime postProcess: ?*const fn (*T, std.mem.Allocator) anyerror!void, comptime postProcess: ?*const fn (*T, std.mem.Allocator) anyerror!void,
comptime needsRefresh: ?*const fn ([]const T, Date) bool, comptime needsRefresh: ?*const fn ([]const T, Date) bool,
comptime ttlFor: ?*const fn ([]const T) cache.TtlSpec,
opts_in: FetchOptions, opts_in: FetchOptions,
) DataError!FetchResult(T) { ) DataError!FetchResult(T) {
// See `getCandles` - one fold, covering every type routed through here. // See `getCandles` - one fold, covering every type routed through here.
@ -556,7 +698,7 @@ pub const DataService = struct {
// returns cached even if stale, never touches the network. // returns cached even if stale, never touches the network.
if (!opts.force_refresh) { if (!opts.force_refresh) {
if (s.read(self.allocator, T, symbol, postProcess, .fresh_only)) |cached| { if (s.read(self.allocator, T, symbol, postProcess, .fresh_only)) |cached| {
if (self.serveFreshOrDiscard(T, needsRefresh, cached.data, opts.skip_network)) { if (self.serveFreshOrDiscard(T, needsRefresh, ttlFor, symbol, cached, opts.skip_network)) {
log.debug("{s}: {s} fresh in local cache", .{ symbol, @tagName(data_type) }); log.debug("{s}: {s} fresh in local cache", .{ symbol, @tagName(data_type) });
return .{ .data = cached.data, .source = .cached, .timestamp = cached.timestamp, .allocator = self.allocator }; return .{ .data = cached.data, .source = .cached, .timestamp = cached.timestamp, .allocator = self.allocator };
} }
@ -585,7 +727,7 @@ pub const DataService = struct {
// server DID have the newer record - that is the tier's // server DID have the newer record - that is the tier's
// reason for existing, and we return without spending a // reason for existing, and we return without spending a
// provider request. // provider request.
if (self.serveFreshOrDiscard(T, needsRefresh, cached.data, opts.skip_network)) { if (self.serveFreshOrDiscard(T, needsRefresh, ttlFor, symbol, cached, opts.skip_network)) {
log.debug("{s}: {s} synced from server and fresh", .{ symbol, @tagName(data_type) }); log.debug("{s}: {s} synced from server and fresh", .{ symbol, @tagName(data_type) });
return .{ .data = cached.data, .source = .cached, .timestamp = cached.timestamp, .allocator = self.allocator }; return .{ .data = cached.data, .source = .cached, .timestamp = cached.timestamp, .allocator = self.allocator };
} }
@ -603,7 +745,7 @@ pub const DataService = struct {
log.warn("{s}: {s} fetch failed after rate-limit retry: {t}", .{ symbol, @tagName(data_type), retry_err }); log.warn("{s}: {s} fetch failed after rate-limit retry: {t}", .{ symbol, @tagName(data_type), retry_err });
return DataError.FetchFailed; return DataError.FetchFailed;
}; };
s.writeWithSource(T, symbol, retried, data_type.ttl(), sourceHintFor(T)); s.writeWithSource(T, symbol, retried, ttlSpecFor(T, ttlFor, retried), sourceHintFor(T));
return .{ .data = retried, .source = .fetched, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator }; return .{ .data = retried, .source = .fetched, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator };
} }
// A MISSING KEY IS A CONFIGURATION FAULT, not a data fault, // A MISSING KEY IS A CONFIGURATION FAULT, not a data fault,
@ -638,7 +780,7 @@ pub const DataService = struct {
return DataError.FetchFailed; return DataError.FetchFailed;
}; };
s.writeWithSource(T, symbol, fetched, data_type.ttl(), sourceHintFor(T)); s.writeWithSource(T, symbol, fetched, ttlSpecFor(T, ttlFor, fetched), sourceHintFor(T));
return .{ .data = fetched, .source = .fetched, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator }; return .{ .data = fetched, .source = .fetched, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator };
} }
@ -1464,13 +1606,13 @@ pub const DataService = struct {
/// Fetch dividend history for a symbol. /// Fetch dividend history for a symbol.
/// ///
/// Carries a schedule-aware refresh hook (`dividendsNeedRefresh`) /// The only type that uses both extra hooks, and for one reason: a
/// because a fresh dividend cache can still be *incomplete*: an /// dividend cache can be fresh and still be missing a payment that
/// ETF's distribution is not retrievable until roughly its own /// has already happened. `dividendsNeedRefresh` notices that from the
/// ex-date and pays within a week of it, so TTL alone cannot /// symbol's own payment schedule, and `dividendTtl` sets the clock
/// guarantee the record is seen before the cash lands. /// according to whether that schedule is known.
pub fn getDividends(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(Dividend) { pub fn getDividends(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(Dividend) {
return self.fetchCached(Dividend, symbol, null, dividendsNeedRefresh, opts); return self.fetchCached(Dividend, symbol, null, dividendsNeedRefresh, dividendTtl, opts);
} }
/// Fetch split history for a symbol. /// Fetch split history for a symbol.
@ -1478,12 +1620,12 @@ pub const DataService = struct {
/// No refresh hook: splits are announced weeks to months ahead, so /// No refresh hook: splits are announced weeks to months ahead, so
/// the forward-looking record is cached long before it matters. /// the forward-looking record is cached long before it matters.
pub fn getSplits(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(Split) { pub fn getSplits(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(Split) {
return self.fetchCached(Split, symbol, null, null, opts); return self.fetchCached(Split, symbol, null, null, null, opts);
} }
/// Fetch options chain for a symbol (all expirations, no API key needed). /// Fetch options chain for a symbol (all expirations, no API key needed).
pub fn getOptions(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(OptionsChain) { pub fn getOptions(self: *DataService, symbol: []const u8, opts: FetchOptions) DataError!FetchResult(OptionsChain) {
return self.fetchCached(OptionsChain, symbol, null, null, opts); return self.fetchCached(OptionsChain, symbol, null, null, null, opts);
} }
/// Days after an expected ex-date during which a still-absent /// Days after an expected ex-date during which a still-absent
@ -1618,6 +1760,35 @@ pub const DataService = struct {
return today.days - expected.days <= dividend_chase_days; return today.days - expected.days <= dividend_chase_days;
} }
/// LEVEL 1's length for dividends, chosen from the records themselves
/// rather than fixed per type.
///
/// Two clocks, because the clock is doing a different job in each
/// case:
///
/// - **Schedule known** (a cadence can be worked out). The
/// prediction is what finds a due payment, so the clock only has
/// to cover what a schedule cannot predict - an off-cycle special
/// distribution. Fourteen days.
///
/// - **Schedule unknown** (fewer than three records, or no usable
/// cadence - typically a newly bought holding). There is nothing
/// to predict from, so the clock is the ONLY thing that will ever
/// make us look again. Six days, which guarantees the entry is
/// stale by the next weekly review.
///
/// Sizing them separately is the whole point. One short clock for
/// everything would re-check long-established payers every few days
/// to guard against a problem they do not have; one long clock would
/// let a new holding's first payment go unnoticed for a fortnight.
fn dividendTtl(divs: []const Dividend) cache.TtlSpec {
const known = dividendCadenceDays(divs) != null;
const base = if (known) cache.Ttl.dividends_scheduled else cache.Ttl.dividends;
// Jitter policy stays with the rest of it in `DataType.ttl()`;
// only the base differs here.
return .{ .seconds = base, .jitter_pct = comptime cache.DataType.dividends.ttl().jitter_pct };
}
/// Days after an earnings report date during which a still-missing /// Days after an earnings report date during which a still-missing
/// `actual` is worth chasing with a re-fetch. Past this window the /// `actual` is worth chasing with a re-fetch. Past this window the
/// gap is treated as permanent (FMP won't backfill it; earnings has /// gap is treated as permanent (FMP won't backfill it; earnings has
@ -1677,7 +1848,7 @@ pub const DataService = struct {
if (market.classify(symbol) == .mutual_fund) { if (market.classify(symbol) == .mutual_fund) {
return .{ .data = &.{}, .source = .cached, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator }; return .{ .data = &.{}, .source = .cached, .timestamp = std.Io.Timestamp.now(self.io, .real).toSeconds(), .allocator = self.allocator };
} }
return self.fetchCached(EarningsEvent, symbol, earningsPostProcess, earningsNeedsRefreshHook, opts); return self.fetchCached(EarningsEvent, symbol, earningsPostProcess, earningsNeedsRefreshHook, null, opts);
} }
/// Fetch ETF profile for a symbol. Assembles a unified /// Fetch ETF profile for a symbol. Assembles a unified
@ -4752,6 +4923,31 @@ fn corpusAged(today: Date, age_days: i32) [6]Dividend {
return quarterlySeries(6, today.addDays(-age_days), 91); return quarterlySeries(6, today.addDays(-age_days), 91);
} }
/// Write a cache entry as though the write had happened `age_s` ago.
///
/// Needed because `askedRecently` recovers the write time from
/// `#!expires=` minus the TTL that was applied. A freshly written entry
/// therefore always looks like "we just asked", which suppresses the
/// refresh hook - correct in production, useless in a test that wants to
/// exercise the hook. Back-dating the expires by the TTL is the only way
/// to make a test entry look old without sleeping.
///
/// `spec` must be the TTL the production path would have used for this
/// data, since that is what `askedRecently` will recompute.
fn writeAged(
comptime T: type,
store: *cache.Store,
symbol: []const u8,
items: cache.Store.DataFor(T),
spec: cache.TtlSpec,
age_s: i64,
) void {
// What the production write would have stamped, jitter included.
const applied = cache.computeExpires(0, spec, symbol);
// Exact expires, no jitter of our own, so the recompute lines up.
store.writeWithSource(T, symbol, items, .{ .seconds = applied - age_s }, null);
}
test "fetchCached hook: a null hook leaves the fresh-cache path untouched" { test "fetchCached hook: a null hook leaves the fresh-cache path untouched" {
// The regression guard for every other type. Splits pass no hook and // The regression guard for every other type. Splits pass no hook and
// have no `freeSlice`; if the hook branch were not comptime-elided // have no `freeSlice`; if the hook branch were not comptime-elided
@ -4855,7 +5051,11 @@ test "fetchCached hook: an overdue distribution discards the fresh entry and ref
const today = fmt.todayDate(io); const today = fmt.todayDate(io);
var divs = corpusAged(today, 95); var divs = corpusAged(today, 95);
var store = svc.store(); var store = svc.store();
store.write(Dividend, "TEST", divs[0..], cache.DataType.dividends.ttl()); // Aged two days: a just-written entry looks like "we already asked"
// to the recheck floor, which would suppress the refresh. The spec
// must match what production would have stamped (`dividendTtl`), since
// that is what the floor recomputes.
writeAged(Dividend, &store, "TEST", divs[0..], DataService.dividendTtl(divs[0..]), 2 * std.time.s_per_day);
// Sanity: the entry really is fresh, so TTL alone would have served // Sanity: the entry really is fresh, so TTL alone would have served
// it. That is the failure this replaces. // it. That is the failure this replaces.
@ -5039,10 +5239,14 @@ test "getEarnings: smart refresh survives the move onto fetchCached" {
// A report three days ago with no actual yet -> inside the chase // A report three days ago with no actual yet -> inside the chase
// window -> must refetch, which with no FMP key surfaces as NoApiKey. // window -> must refetch, which with no FMP key surfaces as NoApiKey.
//
// Aged deliberately: a just-written entry looks like "we already
// asked" to the recheck floor, which would suppress the very refresh
// this test is about.
var pending = [_]EarningsEvent{ var pending = [_]EarningsEvent{
.{ .symbol = "TEST", .date = today.addDays(-3), .estimate = 1.5 }, .{ .symbol = "TEST", .date = today.addDays(-3), .estimate = 1.5 },
}; };
store.write(EarningsEvent, "TEST", pending[0..], cache.DataType.earnings.ttl()); writeAged(EarningsEvent, &store, "TEST", pending[0..], cache.DataType.earnings.ttl(), 2 * std.time.s_per_day);
try std.testing.expectError(DataError.NoApiKey, svc.getEarnings("TEST", .{})); try std.testing.expectError(DataError.NoApiKey, svc.getEarnings("TEST", .{}));
// Same report with the actual posted -> nothing outstanding -> served // Same report with the actual posted -> nothing outstanding -> served
@ -5063,6 +5267,335 @@ test "getEarnings: smart refresh survives the move onto fetchCached" {
try std.testing.expectApproxEqAbs(@as(f64, 0.12), result.data[0].surprise.?, 1e-9); try std.testing.expectApproxEqAbs(@as(f64, 0.12), result.data[0].surprise.?, 1e-9);
} }
// ── The recheck floor, and the TTL that varies by schedule ────────
test "recheck floor: a just-asked symbol is not asked again" {
// The bug this fixes. When the schedule says a payment is due but the
// provider has nothing yet, the cache is rewritten with a fresh clock
// and the SAME records - so the schedule still says "due", and without
// a floor the next command asks again. Here the entry is written and
// immediately read back, which is exactly that situation.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var svc = DataService.init(io, allocator, Config{ .cache_dir = dir_path });
defer svc.deinit();
const today = fmt.todayDate(io);
var divs = corpusAged(today, 95);
// Confirm the schedule really does say "overdue", so the only thing
// that can stop the refetch is the floor.
try std.testing.expect(DataService.dividendsNeedRefresh(divs[0..], today));
var store = svc.store();
store.write(Dividend, "TEST", divs[0..], DataService.dividendTtl(divs[0..]));
// Reaching the provider would panic. Being served proves the floor held.
svc.panic_on_network_attempt = true;
const result = try svc.getDividends("TEST", .{});
defer result.deinit();
try std.testing.expectEqual(@as(usize, 6), result.data.len);
try std.testing.expectEqual(Source.cached, result.source);
}
test "recheck floor: once the interval has passed, the symbol is asked again" {
// The other side of the floor. Same overdue corpus, but the entry was
// written longer ago than the interval, so the refetch proceeds.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var svc = DataService.init(io, allocator, Config{ .cache_dir = dir_path });
defer svc.deinit();
const today = fmt.todayDate(io);
var divs = corpusAged(today, 95);
var store = svc.store();
const spec = DataService.dividendTtl(divs[0..]);
// One second inside the interval -> still served.
writeAged(Dividend, &store, "TEST", divs[0..], spec, DataService.smart_refresh_recheck_interval - 1);
{
svc.panic_on_network_attempt = true;
const held = try svc.getDividends("TEST", .{});
defer held.deinit();
try std.testing.expectEqual(Source.cached, held.source);
svc.panic_on_network_attempt = false;
}
// One second past it -> asked again (NoApiKey proves we got to the
// provider rather than being served the cache).
writeAged(Dividend, &store, "TEST", divs[0..], spec, DataService.smart_refresh_recheck_interval + 1);
try std.testing.expectError(DataError.NoApiKey, svc.getDividends("TEST", .{}));
}
test "recheck floor: a Tiingo supplement does not count as having asked" {
// WHY THE FLOOR READS `#!expires=` AND NOT `#!created=`.
//
// Dividend rows also arrive as a side effect of a Tiingo candle fetch,
// via `writeSupplement`. That rewrites the file - so `#!created=`
// moves - but it deliberately puts the old `#!expires=` back, because
// only the primary provider owns the freshness clock.
//
// If the floor keyed off `created`, any candle refresh would look like
// "we just asked Polygon" and suppress a genuinely due refetch for
// twelve hours. Keying off `expires` is immune.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var svc = DataService.init(io, allocator, Config{ .cache_dir = dir_path });
defer svc.deinit();
const today = fmt.todayDate(io);
var divs = corpusAged(today, 95);
var store = svc.store();
// Asked two days ago, so the floor has expired.
writeAged(Dividend, &store, "TEST", divs[0..], DataService.dividendTtl(divs[0..]), 2 * std.time.s_per_day);
// Now a candle fetch supplements the same file, moving `created` to
// now and leaving `expires` alone.
store.writeSupplement(Dividend, "TEST", divs[0..], "tiingo");
// Still asked. If this returns cached data, the floor is reading the
// wrong directive.
try std.testing.expectError(DataError.NoApiKey, svc.getDividends("TEST", .{}));
}
test "recheck floor: an entry with no expires directive is asked again" {
// Fails safe. A file carrying no `#!expires=` gives the floor nothing
// to measure from, and guessing "recently" would strand the symbol.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var svc = DataService.init(io, allocator, Config{ .cache_dir = dir_path });
defer svc.deinit();
const today = fmt.todayDate(io);
var divs = corpusAged(today, 95);
try std.testing.expect(svc.askedRecently(Dividend, DataService.dividendTtl, "TEST", .{
.data = divs[0..],
.timestamp = 0,
.expires = null,
}) == false);
}
test "recheck floor: the write time is recovered exactly, jitter and all" {
// The floor subtracts the applied TTL from `#!expires=` to get the
// write time. That is only exact because `computeExpires` derives its
// jitter from a hash of the symbol, so the same symbol always gets the
// same offset and it can be recomputed later.
//
// Checked across several symbols because a jitter bug would show up as
// a per-symbol discrepancy, not a uniform one.
for ([_][]const u8{ "TESTA", "TESTB", "TESTC", "TESTD" }) |sym| {
const spec = cache.DataType.dividends.ttl();
const applied = cache.computeExpires(0, spec, sym);
// Jitter must actually be doing something, or this proves nothing.
try std.testing.expect(applied != cache.Ttl.dividends or spec.jitter_pct == 0);
const now_s: i64 = 1_800_000_000;
const expires = cache.computeExpires(now_s, spec, sym);
try std.testing.expectEqual(now_s, expires - applied);
}
}
test "dividendTtl: six days when the schedule is unknown, fourteen when it is known" {
// The two clocks, and the reason they differ. A symbol with a
// established cadence is found by prediction, so its clock can be
// long; one without is found only by the clock, so it must be short
// enough to expire before the next weekly review.
const today = Date.fromYmd(2026, 9, 19);
// Five years of quarterly history -> schedule known -> long clock.
const established = quarterlySeries(6, today.addDays(-30), 91);
try std.testing.expectEqual(cache.Ttl.dividends_scheduled, DataService.dividendTtl(&established).seconds);
// Two records -> no cadence can be inferred -> short clock.
const sparse = divsFromYmd(.{ .{ 2026, 6, 15 }, .{ 2026, 3, 17 } });
try std.testing.expectEqual(cache.Ttl.dividends, DataService.dividendTtl(&sparse).seconds);
// Nothing at all -> short clock.
const none: []const Dividend = &.{};
try std.testing.expectEqual(cache.Ttl.dividends, DataService.dividendTtl(none).seconds);
// Duplicate ex-dates give a zero cadence, which is not a schedule.
const dup = divsFromYmd(.{ .{ 2026, 6, 15 }, .{ 2026, 6, 15 }, .{ 2026, 6, 15 } });
try std.testing.expectEqual(cache.Ttl.dividends, DataService.dividendTtl(&dup).seconds);
// Both keep the shared jitter policy.
const jitter = cache.DataType.dividends.ttl().jitter_pct;
try std.testing.expectEqual(jitter, DataService.dividendTtl(&established).jitter_pct);
try std.testing.expectEqual(jitter, DataService.dividendTtl(&sparse).jitter_pct);
}
test "ttlSpecFor: the hook wins when present, the type default otherwise" {
// The one-line dispatch that decides which clock a write gets. Only
// reachable in production after a successful provider fetch, so it is
// exercised directly here rather than left to a live-network path no
// test can take.
const today = Date.fromYmd(2026, 9, 19);
var established = quarterlySeries(6, today.addDays(-30), 91);
// With the dividend hook: the schedule is known, so the long clock.
try std.testing.expectEqual(
cache.Ttl.dividends_scheduled,
DataService.ttlSpecFor(Dividend, DataService.dividendTtl, established[0..]).seconds,
);
// Without a hook, the same records get the type default - which is the
// cautious short value. This is what every other type sees.
try std.testing.expectEqual(
cache.Ttl.dividends,
DataService.ttlSpecFor(Dividend, null, established[0..]).seconds,
);
// A type that has no hook at all.
var splits = [_]Split{.{ .date = Date.fromYmd(2024, 3, 7), .numerator = 3, .denominator = 1 }};
try std.testing.expectEqual(
cache.Ttl.splits,
DataService.ttlSpecFor(Split, null, splits[0..]).seconds,
);
}
test "dividendTtl: the chosen clock is what actually lands on disk" {
// `dividendTtl` returning the right number is worth nothing if the
// write path ignores it, so this reads `#!expires=` back off the file.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var store = cache.Store.init(io, allocator, dir_path);
const today = fmt.todayDate(io);
const now_s = std.Io.Timestamp.now(io, .real).toSeconds();
// Established payer -> ~14 days.
var established = quarterlySeries(6, today.addDays(-30), 91);
store.write(Dividend, "TESTA", established[0..], DataService.dividendTtl(&established));
const long_expires = readExpires(io, allocator, dir_path, "TESTA") orelse return error.TestUnexpectedResult;
try std.testing.expectEqual(
cache.computeExpires(0, DataService.dividendTtl(&established), "TESTA"),
long_expires - now_s,
);
// Sparse history -> ~6 days.
var sparse = divsFromYmd(.{ .{ 2026, 6, 15 }, .{ 2026, 3, 17 } });
store.write(Dividend, "TESTB", sparse[0..], DataService.dividendTtl(&sparse));
const short_expires = readExpires(io, allocator, dir_path, "TESTB") orelse return error.TestUnexpectedResult;
try std.testing.expectEqual(
cache.computeExpires(0, DataService.dividendTtl(&sparse), "TESTB"),
short_expires - now_s,
);
// And the long one really is longer, so nobody can "simplify" the two
// constants into one without this failing.
try std.testing.expect(long_expires > short_expires);
}
test "dividendTtl: a supplement to an unseen symbol gets the cautious clock" {
// `writeSupplement` has no records to reason about when the file does
// not exist yet, so it falls back to `DataType.dividends.ttl()`. That
// must be the SHORT value, because a symbol we have never seen before
// is precisely the unknown-schedule case.
try std.testing.expectEqual(cache.Ttl.dividends, cache.DataType.dividends.ttl().seconds);
}
test "full cycle: a quarterly ETF through one whole period" {
// The end-to-end story, on the pattern from the discussion that
// produced this code: ex-dates on 1/1, 4/1, 7/1 and 10/1, five years
// of history, newest cached record 2026-07-01.
//
// Median gap is 91 days, so the forecast for the next ex-date is
// 2026-09-30 - one day early, because Jul->Oct is actually 92. Being
// early is harmless; the window is fourteen days wide.
const newest = Date.fromYmd(2026, 7, 1);
var divs: [19]Dividend = undefined;
{
var i: usize = 0;
var y: i16 = 2026;
while (y >= 2022) : (y -= 1) {
for ([_]u8{ 10, 7, 4, 1 }) |m| {
const d = Date.fromYmd(y, m, 1);
if (!newest.lessThan(d) and i < divs.len) {
divs[i] = .{ .ex_date = d, .amount = 1.0, .type = .regular };
i += 1;
}
}
}
try std.testing.expectEqual(divs.len, i);
}
try std.testing.expectEqual(@as(i32, 91), DataService.dividendCadenceDays(&divs).?);
// Schedule known, so the clock is the long one.
try std.testing.expectEqual(cache.Ttl.dividends_scheduled, DataService.dividendTtl(&divs).seconds);
// NOT EXPECTING ANYTHING: the whole quarter between payments.
for ([_]Date{
Date.fromYmd(2026, 7, 2),
Date.fromYmd(2026, 8, 1),
Date.fromYmd(2026, 9, 1),
Date.fromYmd(2026, 9, 29),
}) |d| {
try std.testing.expect(!DataService.dividendsNeedRefresh(&divs, d));
}
// OVERDUE BY PREDICTION: from the forecast date to fourteen days on.
for ([_]Date{
Date.fromYmd(2026, 9, 30),
Date.fromYmd(2026, 10, 1),
Date.fromYmd(2026, 10, 7),
Date.fromYmd(2026, 10, 14),
}) |d| {
try std.testing.expect(DataService.dividendsNeedRefresh(&divs, d));
}
// GAVE UP: past the window the schedule is treated as wrong and the
// clock takes over, so we stop asking every run.
try std.testing.expect(!DataService.dividendsNeedRefresh(&divs, Date.fromYmd(2026, 10, 15)));
// The 10/1 record arrives. Asking stops immediately...
var with_oct: [20]Dividend = undefined;
with_oct[0] = .{ .ex_date = Date.fromYmd(2026, 10, 1), .amount = 1.0, .type = .regular };
@memcpy(with_oct[1..], &divs);
try std.testing.expect(!DataService.dividendsNeedRefresh(&with_oct, Date.fromYmd(2026, 10, 1)));
try std.testing.expect(!DataService.dividendsNeedRefresh(&with_oct, Date.fromYmd(2026, 12, 1)));
// ...and the forecast self-corrects: the newest gaps now median to 92,
// which lands the next expected ex-date exactly on 2027-01-01.
try std.testing.expectEqual(@as(i32, 92), DataService.dividendCadenceDays(&with_oct).?);
try std.testing.expect(DataService.dividendsNeedRefresh(&with_oct, Date.fromYmd(2027, 1, 1)));
}
/// Read the raw `#!expires=` directive off a cached dividend file.
fn readExpires(io: std.Io, allocator: std.mem.Allocator, dir_path: []const u8, symbol: []const u8) ?i64 {
const path = std.fs.path.join(allocator, &.{ dir_path, symbol, "dividends.srf" }) catch return null;
defer allocator.free(path);
const data = std.Io.Dir.cwd().readFileAlloc(io, path, allocator, .limited(1024 * 1024)) catch return null;
defer allocator.free(data);
var reader = std.Io.Reader.fixed(data);
var it = srf.iterator(&reader, allocator, .{ .parse_allocator = .none }) catch return null;
defer it.deinit();
return it.expires;
}
test "getEarnings: skip_network suppresses the smart refresh" { test "getEarnings: skip_network suppresses the smart refresh" {
// Offline mode must serve the incomplete-but-fresh entry rather than // Offline mode must serve the incomplete-but-fresh entry rather than
// discard it and fail. Same rule the dividend hook follows, and it now // discard it and fail. Same rule the dividend hook follows, and it now