zfin/docs/explanation/caching.md

# Caching and data freshness

zfin makes a lot of API calls on your behalf -- prices, dividends,
earnings, ETF holdings -- against providers with strict free-tier
limits. Aggressive caching is what keeps it fast and keeps you well
under those limits. This page explains how it works so the
[`--refresh-data`](../guides/offline-and-refresh.md) flag makes sense.

## The fetch path

Every data request walks the same tiers, stopping at the first one that
can satisfy it:

1. **Local cache.** Look for `~/.cache/zfin/<SYMBOL>/<type>.srf`. If the
   file exists and is within its TTL, deserialize and return -- no
   network at all.
2. **Shared server** *(optional)*. On a miss or stale entry, if
   `ZFIN_SERVER` is set, zfin asks that server before any provider; a
   hit is written into your local cache and served from there, so no
   provider call happens. See [Server sync](#server-sync-zfin_server).
3. **Provider.** Otherwise zfin fetches from the upstream provider,
   writes the result to the cache, and returns it.

Freshness is decided by an expiry timestamp written into each cache
file when it is fetched (an `#!expires=` line), compared against the
current time -- not the file's modification time. Each data type sets
its own TTL, which determines that expiry (see below). The cache
directory defaults to `~/.cache/zfin` and is set with `ZFIN_CACHE_DIR`.

The `--refresh-data` policy decides which tiers run:

- `auto` (default) walks all three.
- `force` skips the local cache and the server, going straight to the
  provider, then re-caches the result.
- `never` stops at the local cache: it returns cached data even if
  stale, and never touches the server or a provider.

## Time-to-live by data type

Different data ages at different rates, so each type has its own TTL:

| Data type     | TTL                | Why                                                                              |
|---------------|--------------------|----------------------------------------------------------------------------------|
| Daily candles | market-aware       | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
| Dividends     | 6 or 14 days\*     | Depends on whether we know the symbol's payment schedule                         |
| Splits        | 14 days            | Rare, and announced weeks ahead                                                  |
| Options       | 1 hour             | Prices move continuously when markets are open                                   |
| Earnings      | 30 days\*\*        | Quarterly, with an early re-check around announcements                           |
| ETF profiles  | ~30 days           | Holdings and weights change slowly                                               |
| Quotes        | never cached       | Meant to be a live price check                                                   |

\* The expiry time is only the first of three checks. See
[Dividends: a three-level freshness check](#dividends-a-three-level-freshness-check).

\*\* **Earnings** use the same three-level check as dividends, with a
different Level 2: instead of asking "is a payment overdue?", it asks "has
a report date passed with the actual result still missing?". So results
appear promptly after an announcement without daily polling. Level 3
applies too - one re-check per symbol per 12 hours.

## Dividends: a three-level freshness check

For most data a plain expiry time is enough. For dividends it is not, so
dividends (and earnings) go through **three checks instead of one**. The
order matters, and it is the thing most easily got backwards:

| Level | Name                   | Question it answers                                       | Runs when                                         |
|-------|------------------------|-----------------------------------------------------------|---------------------------------------------------|
| **1** | **expiry clock**       | "Is this cached entry old?"                               | always                                            |
| **2** | **completeness check** | "Is this entry missing something it should already have?" | **only if Level 1 said the entry is still fresh** |
| **3** | **recheck floor**      | "Did we already ask the provider about this recently?"    | only if Level 2 said something is missing         |

**Read that middle row carefully, because the intuition runs the other
way.** Level 2 is *not* an extra hurdle before a refresh. It is a
**second opinion on a "no" from Level 1**. If the expiry clock says the
entry is old, zfin just refetches and Level 2 is never consulted at all -
there is nothing for it to add, because a refresh is already happening.
Level 2 only earns its keep in the opposite case: the clock says *"this
is still fine, serve it"*, and Level 2 gets to answer *"no it isn't -
a payment has happened that this entry doesn't contain."*

Level 3 then stops Level 2 from being too eager, by refusing to ask the
provider about the same symbol more than once every 12 hours.

### Why one expiry time isn't enough

A regular company announces a dividend weeks before the ex-date and pays
about a month after it. There is plenty of time to notice. An ETF is
different. Its distribution is simply whatever the fund earned, passed
through to you, so the amount does not exist until roughly the fund's own
ex-date -- and it pays one to seven days later. The whole period between
"the payment becomes knowable" and "the cash is in your account" is about
a week.

A two-week expiry time sails straight over that. Suppose the cache is
refreshed a few days before an ETF goes ex. It is then considered fresh
for another eleven days, which covers the ex-date, the payment, *and* the
weekly review that was supposed to explain the deposit. The dividend is
invisible exactly when it matters. And it is not bad luck: dozens of
funds go ex within the same few days at each quarter end, so one refresh
pass puts all of them on the same expiry date and every one of them has
the same blind spot.

Level 2 exists to catch precisely that: it works out the symbol's own
payment schedule from the ex-dates already in the cache, and speaks up
when a payment is due that the cache does not have.

### Two things that are easy to picture wrongly

**The expiry clock is a duration, not a date.** When a record is cached,
zfin stamps "expires 14 days from now". It does not store "next check due
on 30 September". Each refetch re-stamps the duration from scratch.

**The expected payment date is never stored at all.** Level 2 recomputes
it on every single cache read, from whatever ex-dates are currently
cached. Nothing on disk holds it.

So capturing a payment does not "set the clock to" the next expected
payment. The two mechanisms run independently, on different schedules.

### A worked example

Take an ETF that has paid on 1 January, 1 April, 1 July and 1 October for
the last five years. The newest payment in the cache is 1 July 2026.

Level 2 measures the gaps between the most recent ex-dates -- 91, 90, 92,
92 and 91 days -- and takes the middle value: **91 days**. Added to 1
July, that makes **30 September** the date the next payment is expected.
(The real date is 1 October. Being a day out does not matter, as you will
see.)

| When             | What the levels say                                                       | Result                                                                                                                   |
|------------------|---------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| 2 July - 29 Sept | L1: fresh, then stale every 14 days. L2: nothing due.                     | Nothing is expected, so only the clock causes fetches: **about six over the quarter** (91 days, at 14 days a time).      |
| 30 Sept - 14 Oct | L1: fresh. L2: **a payment is overdue.** L3: allows one request/12 hours. | The provider is asked, at most twice a day no matter how many commands you run. Stops the moment the new record arrives. |
| 15 Oct onwards   | L2: gave up -- two weeks past the expected date.                          | Back to the clock alone. Chasing a schedule that is clearly wrong would refetch forever.                                 |

On 30 September the provider probably has nothing yet, because the fund
has not gone ex. That request is wasted, and it is the price of the estimate
being a day early. On 1 October the record appears, gets cached, and
Level 2 immediately goes quiet again -- it now expects the next payment
around 1 January.

The estimate also corrects itself. Once the 1 October record is cached,
the newest gaps median to 92 days rather than 91, which puts the next
expected date exactly on 1 January 2027.

### Why Level 1 has two different lengths

How long the expiry clock runs depends on whether Level 2 can do its job
for that symbol:

- **14 days when the payment schedule is known.** A symbol with a few
  years of history is found by Level 2, so the clock only has to cover
  the one thing a schedule cannot predict: a one-off special
  distribution. Making it short as well would re-check every symbol
  every few days to guard against a problem already solved.

- **6 days when the schedule is not known.** With fewer than three
  cached payments there is no pattern to work from -- typically a holding
  you have just bought. Level 2 declines, so the clock is the *only*
  thing that will ever make zfin look again, and it has to expire before
  a weekly review. Six days guarantees that; seven would not, because
  the +/-11% spread applied to expiry times can stretch seven days to 7.8.

### Why Level 3 exists

When Level 2 says a payment is due and the provider does not have it yet,
nothing in the cache changes except the expiry time. Level 2 therefore
still says "due", so the next command would ask again -- and the one
after that. On a quarter-end run touching fifteen funds, that is fifteen
provider calls per command, every time you run anything.

So Level 3 allows a symbol to be asked about at most once every 12 hours.
Twelve hours sits well inside the shortest gap between a fund going ex
and paying (one day, in a real portfolio), while still being far more
frequent than the weekly reconcile that actually uses the answer.

### What about Tiingo?

Tiingo returns dividend data as a side effect of fetching prices, which
raises the obvious question of whether it can spot a new payment for
free. It cannot, and it is worth knowing that it never will.

Tiingo reports a dividend as a field on a daily price bar, so the payment
only shows up once it has already traded -- there is no such thing as a
forward-looking dividend in that data. Polygon is the primary source
precisely because it *does* carry declared-but-not-yet-paid dividends.
And the code path that merges Tiingo's dividend rows in only runs on a
cold cache or when a known corporate action forces a price-history
restatement, which means Polygon has already supplied the record by then.

So Tiingo is genuinely useful for bulk-loading a new symbol's entire
dividend history in one request, and for filling gaps in old records. It
is simply never the thing that notices a *new* payment.

### What none of this catches

Three cases where Level 2 cannot help, all left to Level 1:

- **A one-off special distribution.** It is on no schedule, so it is
  found only when the expiry clock runs out.
- **A fund that changes its schedule** -- quarterly to monthly, say. The
  old pattern persists for a few periods, so the estimate runs late.
- **A symbol with fewer than three cached payments.** No pattern to work
  from, which is exactly why that case gets the 6-day clock.


## Market-aware candle freshness

A daily bar only becomes meaningful once the market settles, so candle
freshness is keyed to the market clock rather than a rolling 24-hour
window. Each cached candle's expiry is set to the next moment fresh data
should be available:

- **Equities and ETFs** settle shortly after the 16:00 ET close. Their
  bars expire at **16:55 ET** on the next trading day (weekends and NYSE
  holidays are skipped). This time allows for providers to become consistent
  with the market while also allowing a few minutes prior to a scheduled
  refresh task at the top of the hour.
- **Mutual funds** strike a single daily NAV that isn't reliably
  published until the next morning, so their bars expire at **03:25 ET**
  the morning after a trading session. Despite Tiingo's claims, NAVs only
  seem reliably available until about 3am Eastern. This is again timed such
  that scheduled jobs for the bottom of the hour can run reliably.

This keeps the expiry boundary out of trading hours, so a refresh fired
just after it always sees a finalized bar instead of a half-formed one,
and an interactive command run mid-session won't trigger a needless
refetch. If a refresh runs but the provider hasn't posted the just-closed
bar yet, the entry is retried in ~30 minutes rather than waiting a full
day.

**Un-modeled closures self-correct.** Some market closures aren't on the
modeled holiday calendar (Good Friday, which needs the Easter computus,
plus ad-hoc closures for national mourning or weather). On such a day the
calendar thinks a bar is due, the fetch keeps coming back empty, and the
~30-minute retry would otherwise repeat all day - and, for a Friday
closure, all weekend. To avoid that thrash, once an expected bar is ~90
minutes overdue the cache concludes the market was closed and falls back
to the normal next-session boundary - at most three 30-minute retries.
That window comfortably covers ordinary provider posting lag, so
genuinely-late data is still picked up by the short retry; only a true
closure trips the fallback.

**Warming a shared cache on a schedule.** If you run a cron to warm a
[server cache](#server-sync-zfin_server) (or your own local cache), the
boundaries above are also the natural cron times: a run shortly after
**17:00 ET** picks up the day's equity/ETF closes, and a run shortly
after **03:30 ET** picks up the prior session's mutual-fund NAVs. The
boundaries sit a couple of minutes before those times so the cron
reliably sees the cache already expired.

## Quotes are never cached

Because quotes exist to give you a live price, they're never served
from cache. The practical consequence: in offline mode
(`--refresh-data=never`) the [`quote`](../reference/cli/quote.md)
command has nothing to serve, while candle-based commands like
[`perf`](../reference/cli/perf.md) work fine from cached history.

## Incremental candle updates

Price history isn't re-downloaded wholesale. On a cache miss, zfin
fetches only candles newer than the last cached date and appends them,
using a small `candles_meta.srf` companion file to track the last date
and source provider. A ten-year history costs one big fetch the first
time and tiny top-ups thereafter.

With one exception. A provider's *adjusted* close prices bake in every
dividend and split that happened after each bar, so when a distribution
goes ex, all the bars behind it need marking down - and appending new
bars can't do that to bars already on disk. Left alone, total returns
spanning that ex-date read low by roughly the missed dividend.

So zfin tracks how current each series' adjustment basis is, and when a
dividend or split has gone ex behind it, re-downloads that symbol's full
history once to pick up the corrected values. For a quarterly dividend
payer that's about four full fetches a year, each within a day or so of
an ex-date. It's why a refresh run occasionally takes noticeably longer
than the usual top-up.

[`zfin diagnose SYMBOL`](../reference/cli/diagnose.md) reports the basis
for one symbol and says whether a restatement is pending.

## Negative caching

When a provider permanently fails for a symbol -- a nonexistent
ticker, say -- zfin records a negative cache entry so it doesn't retry
the same dead lookup on every run. (Transient failures like rate limits
are not cached this way; they're retried.)

One exception: a price history that was loaded into the cache from
outside zfin, rather than fetched from a provider, is never
negative-cached. Such a series has no provider to re-fetch it from, so
recording "nobody has this" would discard the only copy. `zfin diagnose
SYMBOL` names the source of a symbol's bars, and reports `external` for
these.

## Symbols without candle data (crypto)

A few holdings have no daily price history available from zfin's candle
provider -- cryptocurrencies (held as `DOGE-USD`, `BTC-USD`, and the
like) are the common case, and long-delisted tickers behave the same
way. zfin negative-caches these so it stops asking for candles that will
never arrive.

Such holdings are still valued, just not from price history:

- **In the TUI** they are priced from **live quotes** (Yahoo serves
  crypto), refreshed on load and on every streaming tick.
- A **manual price** wins when set: a `price::` field on the lot in
  `portfolio.srf` pins the value.
- Otherwise zfin falls back to your **average cost** for the holding and
  flags it in a warning color, so a candle-less position shows at
  break-even rather than disappearing.

Because the plain `portfolio` CLI command doesn't fetch live quotes, a
candle-less holding shows its average-cost fallback there; open the TUI
(or set a `price::`) to see a live value. History-based commands like
[`perf`](../reference/cli/perf.md) and the price charts simply have no
data for these symbols. To force a re-check (say a ticker started
trading), clear the entry with `--refresh-data=force` or `zfin cache
clear`.

## Rate limiting

Each provider has a client-side token-bucket limiter sized to its
free-tier ceiling (e.g. Polygon 5/min, FMP 250/day). When you'd exceed
the rate, zfin blocks until a token is available rather than firing a
request that would 429. This is why a `--refresh-data=force` run across
many symbols can pace itself instead of failing. Limits are listed in
[Data providers and API keys](../reference/providers.md).

## Server sync (`ZFIN_SERVER`)

`ZFIN_SERVER` points zfin at an optional
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance -- a
shared cache that sits between your local cache and the upstream
providers, and is the second tier of [the fetch path](#the-fetch-path).
On a local miss, zfin requests `GET {ZFIN_SERVER}/<SYMBOL>/<type>`
(candles, dividends, splits, options, earnings, classification, ETF
metrics, and EDGAR entity facts), and a hit is written straight into
your local cache.

Why bother: the server is warmed once -- say by a cron job on one
machine -- and then every client draws from it instead of each spending
its own provider quota, so a household or a fleet of machines shares one
set of API-key budgets and gets faster cold starts. For the portfolio
price load, the server is queried in parallel across symbols, with
per-symbol provider fallback only for what it can't supply.

It is entirely optional: when `ZFIN_SERVER` is unset, every server-sync
path silently no-ops and zfin runs local-cache-then-provider. Live
quotes are never served by the server (they aren't cached anywhere), and
`--refresh-data=force` bypasses the server to re-fetch from the provider.

## Controlling it

You rarely need to intervene -- `auto` does the right thing. When you
do:

- `--refresh-data=force` re-fetches everything (after a close, or to
  clear suspected bad data).
- `--refresh-data=never` goes fully offline.
- [`zfin cache stats`](../reference/cli/cache.md) shows what's cached;
  `zfin cache clear` wipes it (everything re-fetches next run).

See [Offline use and refreshing data](../guides/offline-and-refresh.md).

For the low-level implementation -- on-disk layout, the freshness model,
negative caching, and the fetch-decision flowcharts -- see
[Cache implementation](../dev/caching-implementation.md).