798 lines
37 KiB
Markdown
798 lines
37 KiB
Markdown
# Future Work
|
||
|
||
No work here is blocking — we're in a good state. Items below are
|
||
ordered roughly by priority within each section. Priority labels
|
||
(`HIGH` / `MEDIUM` / `LOW`) mark items that deserve explicit
|
||
ranking; unlabeled items are "someday, if the mood strikes."
|
||
|
||
**Next up:** configurable benchmark symbols (low-effort win, exercises
|
||
the post-refactor framework on a small feature) and the manual-check
|
||
accounts mechanism (medium effort, real user value).
|
||
|
||
## Projections: future enhancements
|
||
|
||
- **Configurable benchmark symbols — priority MEDIUM.** Currently
|
||
hardcoded SPY + AGG. Route through `projections.srf` as a
|
||
`type::config,benchmark::SYMBOL` record (or similar). Low effort.
|
||
- **Configurable return cap per position — priority MEDIUM.**
|
||
Default: none; cap outliers like NVDA. Should route through
|
||
`projections.srf` cleanly.
|
||
- **Accumulation-mode SWR rate column is misleading — priority LOW.**
|
||
When `retirement_age`/`retirement_at` is configured, the "Safe
|
||
Withdrawal" table's % column divides the SWR amount by the
|
||
CURRENT portfolio value, not the post-accumulation portfolio
|
||
value. The dollar amount is correct (it's the safe spending in
|
||
retirement, given the projected accumulation), but the % rate
|
||
comes out absurdly high (e.g., 22% of today's portfolio). The
|
||
Accumulation phase block already shows the median portfolio at
|
||
retirement, so the user can compute the real rate themselves —
|
||
but the SWR table's rate column should ideally divide by the
|
||
median post-accumulation value, or be suppressed when accumulation
|
||
is active. Decide which.
|
||
- **Chart vertical line at retirement boundary — priority LOW.**
|
||
The accumulation-phase spec called this "mandatory" but it was
|
||
explicitly deferred during implementation. The chart currently
|
||
shows the full `accumulation_years + horizon` span without a
|
||
visual marker for where accumulation ends and distribution
|
||
begins. Easier to add to the kitty-graphics chart than the braille
|
||
one.
|
||
- **Goal-seek over distribution horizon for W1 — priority LOW.**
|
||
Today the W1 ("set spending, find date") workflow reports the
|
||
earliest retirement at each user-configured `(horizon, confidence)`
|
||
cell. The philosophically correct version asks "when have I
|
||
accumulated enough wealth that the projection shows a 95%
|
||
probability of success withdrawing X per year from retirement
|
||
until age-of-death?" — i.e. goal-seek across both `accumulation_years`
|
||
AND `distribution_years` simultaneously, anchored to a configured
|
||
age-of-death. NP-shaped search; not worth optimizing until
|
||
someone wants it.
|
||
- **Per-person retirement_age — priority LOW.**
|
||
V1 of the accumulation-phase spec chose Option A: a single
|
||
household retirement boundary derived from the oldest configured
|
||
birthdate. Households where one earner retires significantly
|
||
earlier than the other would benefit from per-person
|
||
`retirement_age` fields on each `type::birthdate` record, with
|
||
contributions stopped per-person.
|
||
- **Configurable max_accumulation_years — priority LOW.**
|
||
Hardcoded at 50 years. Route through `projections.srf` if anyone
|
||
hits the cap.
|
||
- Configurable MIN period selection (currently 3Y/5Y/10Y, exclude 1Y)
|
||
- Multiple spending models: flat (current), decreasing (1-2% real annual decrease,
|
||
Blanchett "spending smile"). Late-life healthcare better modeled as a life event.
|
||
- Unclassified position handling in allocation split (warn user)
|
||
- **Historical projection overlay follow-ups.** The base
|
||
`--overlay-actuals` overlay shipped (CLI tip + TUI primary surface).
|
||
Open enhancements:
|
||
- Historical `metadata.srf` / `projections.srf` for back-dated
|
||
runs. Today the overlay re-runs against current classifications
|
||
and assumptions; for historically faithful what-the-model-said-then
|
||
output we'd check out the git-tracked versions of those files
|
||
at the as-of commit and load those instead. Edge case until
|
||
classifications materially drift.
|
||
- Contribution-attribution overlay. Today's actuals line includes
|
||
contributions implicitly; the bands assume modeled contributions
|
||
that may or may not match reality. A "decompose actuals into
|
||
market return vs contributions" annotation would clarify how
|
||
much of the trajectory was the model being right vs new money
|
||
arriving on schedule.
|
||
- Mosaic mode: overlay multiple as-of starting points on one chart
|
||
("show me 1Y, 3Y, 5Y, 10Y projections all at once") so the user
|
||
can see how the projection envelope tightened as data came in.
|
||
- **Better composition basis for imported-only as-of.** Today
|
||
the imported-only path uses today's allocations scaled by
|
||
`imported_liquid / today_total_liquid`. That's the simplest
|
||
thing that could work, but it's "today's mix back-dated" —
|
||
it ignores everything we know about the historical context.
|
||
Specifically: `imported_values.srf` already carries an
|
||
`expected_return` field per row that the user captured at
|
||
that date in their source spreadsheet. We could:
|
||
- Use the imported `expected_return` as a sanity check
|
||
against the simulation's per-position weighted return
|
||
(warn or clamp if they diverge wildly — the spreadsheet's
|
||
number reflects what the user actually saw at the time).
|
||
- Use the imported `expected_return` to bias the
|
||
stock/bond split inference: a higher expected return
|
||
implies a higher historical equity weighting than today's
|
||
mix probably reflects.
|
||
- Reach further: derive a synthetic stock/bond split from
|
||
the imported `expected_return` directly, treating it as
|
||
a weighted average of SPY and AGG returns at that date
|
||
and solving for the weights. That gives a per-imported-
|
||
row composition that's locally faithful instead of
|
||
one-mix-fits-all.
|
||
None of these are urgent — the current "today's mix scaled"
|
||
approximation is documented as such and the bands still
|
||
render meaningfully — but each would tighten the historical
|
||
faithfulness one notch. Pick whichever has the highest
|
||
payoff vs. complexity when this gets revisited.
|
||
- **Chart zoom for short-history overlays.** With a 50-year
|
||
projection horizon and only ~10 years of imported actuals,
|
||
the actuals line is squashed into the first 20% of the
|
||
chart and the comparison-against-bands story is hard to
|
||
read. Two design directions:
|
||
- **Auto-zoom**: when the overlay is on, the chart's
|
||
x-axis defaults to `[as_of, today + N years]` (where
|
||
N is small, e.g. 2x the actuals span) instead of
|
||
`[as_of, as_of + horizon]`. The bands beyond `today
|
||
+ N` are still computed but clipped from view. The
|
||
tradeoff: the user loses the long-tail terminal-value
|
||
context unless they toggle back out.
|
||
- **Toggle**: a separate keybind (e.g. `z` for zoom)
|
||
flips between full-horizon and zoomed views. Default
|
||
off so the bands tell their full story; user opts in
|
||
when they want overlay legibility.
|
||
Auto-zoom is more invasive (changes the default chart
|
||
semantics for everyone running with overlay-on) but better
|
||
matches what the user actually wants when they toggle the
|
||
overlay. Toggle is safer but requires the user to know the
|
||
feature exists. Probably do auto-zoom but expose a toggle
|
||
to escape it ("show full horizon").
|
||
|
||
## Export chart as PNG (`--export-chart <path>`) — priority MEDIUM
|
||
|
||
z2d already supports PNG export natively. Today the chart-bearing
|
||
commands (`quote`, `history`, `projections`, plus the equivalent TUI
|
||
tabs) render to braille (CLI) or Kitty graphics (TUI). Adding a
|
||
`--export-chart <path>` flag would land just the chart (not the
|
||
surrounding text output) as a PNG file at the given path, at full
|
||
fidelity, regardless of which surface invoked it.
|
||
|
||
Driver: when reviewing a back-dated projection or a notable price
|
||
move, capturing the chart as an image (e.g. for a write-up, an email
|
||
to the household, or a wiki page) is currently a screenshot-and-crop
|
||
chore. PNG export makes it a one-shot CLI invocation.
|
||
|
||
Sketch:
|
||
- `zfin quote AAPL --export-chart aapl.png` → just the price+
|
||
Bollinger chart as a PNG, no other output.
|
||
- `zfin projections --as-of 1Y --overlay-actuals --export-chart projection.png`
|
||
→ the projection-bands chart plus overlay, no other output.
|
||
- The chart code already produces RGB pixel buffers via z2d; replace
|
||
the `transmitPreEncodedImage` call (TUI) or the braille text path
|
||
(CLI) with a `Surface.write_png` call when the flag is present.
|
||
|
||
Plumbing: a thin "chart-only render" entry point in each chart
|
||
module (`projection_chart.zig`, `chart.zig` for symbols), called
|
||
from the relevant command's `run()` when `--export-chart` is set.
|
||
Exits before the rest of the text output renders.
|
||
|
||
Out of scope for V1: file-format alternatives (SVG, PDF), themed
|
||
color overrides for export (always uses the active terminal theme),
|
||
non-chart command output as PNG.
|
||
|
||
## CLI architecture overhaul — priority MEDIUM
|
||
|
||
The CLI surface has grown to 21 top-level commands and the dispatch
|
||
machinery in `src/main.zig` is starting to bloat. Three related gaps,
|
||
all best addressed together because they share the same refactor
|
||
vehicle:
|
||
|
||
1. **No per-command `--help`.** `zfin help` shows a wall of every
|
||
command and every flag. There's no way to ask "what does
|
||
`projections` accept?" without grepping the top-level usage.
|
||
Every command should support `zfin <command> --help` printing
|
||
just that command's synopsis, flags, and a few examples.
|
||
2. **`usage` text is roughly chronological-by-feature.** Re-group it
|
||
into workflow sections so users can scan for what they need:
|
||
- **Per-symbol lookups:** `perf`, `quote`, `history` (symbol
|
||
mode), `divs`, `splits`, `options`, `earnings`, `etf`
|
||
- **Portfolio analysis:** `portfolio`, `analysis`, `history`
|
||
(portfolio mode), `projections`, `milestones`
|
||
- **Time-series & journaling:** `snapshot`, `compare`,
|
||
`contributions`
|
||
- **Data hygiene:** `audit`, `enrich`, `lookup`
|
||
- **Infra:** `cache`, `version`, `interactive`
|
||
3. **Per-command flag parsing lives inline in `runCli`.** It's
|
||
~500 lines now. The `projections`, `contributions`, and `compare`
|
||
branches each carry ~50 lines of inline flag parsing plus their
|
||
own conflict-detection. Each command module should own its own
|
||
`parseArgs()`; `runCli` should be pure dispatch.
|
||
|
||
The natural mechanism for all three is a **command registry table**
|
||
mirroring the `tab_modules` registry in `src/tui.zig`:
|
||
|
||
```zig
|
||
// src/main.zig
|
||
const command_registry = .{
|
||
.perf = @import("commands/perf.zig"),
|
||
.quote = @import("commands/quote.zig"),
|
||
// ...
|
||
};
|
||
|
||
// each command module exposes:
|
||
// pub const meta = .{
|
||
// .help = "perf <SYMBOL> Show trailing returns ...",
|
||
// .group = .symbol_lookup,
|
||
// .takes_symbol = true,
|
||
// .needs_portfolio_path = false,
|
||
// };
|
||
// pub fn parseArgs(allocator, args, today) !ParsedArgs
|
||
// pub fn run(io, allocator, *DataService, parsed, today, color, *Writer) !void
|
||
```
|
||
|
||
A comptime walk over the registry produces:
|
||
- The grouped `usage` text (sort by `meta.group`, render section
|
||
headers).
|
||
- Per-command help dispatch (`zfin <cmd> --help` prints
|
||
`meta.help` plus the parsed flag descriptors).
|
||
- The dispatch chain itself (replace the giant `if-else if`).
|
||
- The "does this command take a symbol?" / "does it need a
|
||
portfolio path?" decisions currently scattered across `runCli`.
|
||
|
||
**Driver:** the per-command `--help` is the loudest user-facing gap;
|
||
the registry refactor is the cleanest way to land it without
|
||
scattering help text across more files. Also unblocks `zfin doctor`
|
||
(below) and the unified time-range parser (below) by giving them
|
||
a clean module shape to plug into.
|
||
|
||
**Open question for when this is picked up:** how to handle the
|
||
`history` command's two modes (symbol vs portfolio) under a single
|
||
registry entry. Probably one entry with a parser that branches on
|
||
`args[0]`, same as today — but the registry shape should accommodate
|
||
that without ceremony.
|
||
|
||
## `zfin doctor` health-check command — priority LOW
|
||
|
||
Front-door command for the file constellation and environment.
|
||
Answers "is my zfin setup sane?" without making any changes.
|
||
|
||
The configuration surface has grown organically and only the README
|
||
explains the file layout:
|
||
|
||
- `portfolio.srf` — lots
|
||
- `metadata.srf` — classifications
|
||
- `accounts.srf` — tax types
|
||
- `projections.srf` — retirement config
|
||
- `transaction_log.srf` — internal transfers
|
||
- `imported_values.srf` — back-history
|
||
- `history/*-portfolio.srf` — snapshots
|
||
- `~/.config/zfin/keys.srf` — keybinds
|
||
- `~/.config/zfin/theme.srf` — colors
|
||
|
||
Plus 5 API keys, a cache directory, and an optional `ZFIN_SERVER`.
|
||
|
||
### v1 scope (full health check)
|
||
|
||
- **File inventory + parse check.** For each of the files above:
|
||
does it exist, where was it resolved from (cwd vs `ZFIN_HOME` vs
|
||
`~/.config/zfin`), and does it parse cleanly. No fixes.
|
||
- **Cross-reference checks.** Every account in `portfolio.srf`
|
||
has an `accounts.srf` entry. Every held symbol has a
|
||
`metadata.srf` entry (or is opted out). Every snapshot file
|
||
parses as a portfolio. `transaction_log.srf` records reference
|
||
real account names.
|
||
- **Environment audit.** Which API keys are set (presence only,
|
||
never the value). Cache size + symbol count. `ZFIN_SERVER`
|
||
reachability if set (HEAD/OPTIONS, low timeout). Staleness of
|
||
hand-maintained data files (T-bill rates, Shiller `ie_data.csv`)
|
||
— same registry as `data/staleness.zig`.
|
||
- **Output shape.** Sectioned, color-coded. Every check is
|
||
`OK` / `WARN` / `FAIL` with a one-line context. Exit code 0 on
|
||
all-OK or warnings only; non-zero only on FAIL. Suitable for
|
||
CI / cron / pre-commit.
|
||
|
||
### Driver
|
||
|
||
The file constellation has grown to nine files in three locations,
|
||
plus five API keys, plus the cache, plus the optional server.
|
||
Today only the README explains the structure. A `doctor` command
|
||
surfaces it, catches regressions, and is the obvious place to point
|
||
new users (or to point future-self after a long break).
|
||
|
||
### Open question for when this is picked up
|
||
|
||
Does this *replace* the portfolio-hygiene portion of `audit`, or
|
||
live alongside it? Probably alongside — `audit` is reconciliation
|
||
against external broker exports, `doctor` is internal-state
|
||
validation. But worth confirming the boundary before implementing
|
||
to avoid duplicated checks.
|
||
|
||
## TUI tab re-order — priority LOW
|
||
|
||
Current tab order interleaves the two natural categories:
|
||
|
||
```
|
||
Portfolio Quote Performance Options Earnings Analysis History Projections
|
||
P S S S S P P P
|
||
```
|
||
|
||
That's `P S S S S P P P` — the per-portfolio tabs (P) are split
|
||
across the bar (slot 1, then slots 6-8). Reflects history of feature
|
||
additions rather than current shape.
|
||
|
||
### Suggested re-order to consider
|
||
|
||
```
|
||
Portfolio Analysis History Projections | Quote Performance Options Earnings
|
||
```
|
||
|
||
"Your money" tabs first (1-4), "research a symbol" tabs second
|
||
(5-8). Categorical split is clean; the 4↔5 boundary is a natural
|
||
divider in the tab bar.
|
||
|
||
### Alternative orderings worth weighing
|
||
|
||
- `Portfolio Analysis Projections History | Quote Performance Earnings Options`
|
||
— within each category, ordered by frequency of use rather than
|
||
by category cohesion.
|
||
- Status quo with just Analysis moved up next to Portfolio (minimum
|
||
churn).
|
||
|
||
### Mechanics
|
||
|
||
The `tab_modules` registry in `src/tui.zig` makes the actual reorder
|
||
a one-edit change (reorder fields in the registry literal). Cost is
|
||
in retraining muscle memory and updating the README's keybinding
|
||
table + screenshots.
|
||
|
||
### Doc drift to fix while we're there
|
||
|
||
README still says "six tabs," actual count is eight (Portfolio,
|
||
Quote, Performance, Options, Earnings, Analysis, History,
|
||
Projections).
|
||
|
||
## Unified time-range flag parser — priority LOW
|
||
|
||
`compare`, `contributions`, `projections --vs`, and `projections
|
||
--as-of` all accept overlapping but slightly different time-range
|
||
inputs. Today there are two helpers in `src/commands/common.zig`
|
||
(`parseAsOfDate`, `parseCommitSpec`) and the conflict-detection
|
||
logic (`since != null and before_spec != null`, etc.) is duplicated
|
||
across each command's flag-parsing block in `main.zig`.
|
||
|
||
### Sketch
|
||
|
||
```zig
|
||
// src/commands/common.zig
|
||
pub const TimeRange = struct {
|
||
before: Endpoint,
|
||
after: Endpoint,
|
||
pub const Endpoint = union(enum) {
|
||
date: Date,
|
||
commit_spec: CommitSpec,
|
||
live,
|
||
head,
|
||
};
|
||
};
|
||
pub fn parseTimeRange(args: ...) !TimeRange { ... }
|
||
```
|
||
|
||
Each command consumes a `TimeRange` and decides which endpoint
|
||
combinations make sense:
|
||
- `compare` rejects `live`-on-both-sides.
|
||
- `contributions` rejects `live` entirely (no meaningful "live
|
||
contributions diff").
|
||
- `projections --vs` accepts `live` on either side.
|
||
|
||
### Naturally folds into the CLI architecture overhaul
|
||
|
||
Once each command has its own `parseArgs()` (per the CLI overhaul
|
||
entry above), the time-range parser becomes a shared utility those
|
||
parsers call. Could either land standalone or be pulled in as part
|
||
of the overhaul. If the overhaul lands first, this is one of its
|
||
follow-ups; if this lands first, the overhaul inherits it for free.
|
||
|
||
## Split `audit.zig` into per-broker reconcilers — priority LOW
|
||
|
||
`src/commands/audit.zig` is 3438 lines — the largest command file
|
||
by ~2x. It bundles four logically distinct responsibilities:
|
||
|
||
- Portfolio hygiene check (no-flag mode)
|
||
- Fidelity positions CSV reconciler (`--fidelity`)
|
||
- Schwab per-account positions CSV reconciler (`--schwab`)
|
||
- Schwab account-summary stdin parser (`--schwab-summary`)
|
||
|
||
### Sketch
|
||
|
||
```
|
||
src/commands/audit/
|
||
mod.zig ← thin dispatcher; current public `run()` lives here
|
||
hygiene.zig ← portfolio hygiene check (no-flag mode)
|
||
fidelity.zig ← --fidelity CSV reconciler
|
||
schwab.zig ← --schwab CSV + --schwab-summary stdin reconciler
|
||
common.zig ← shared types (Discrepancy, ReconcileResult), formatters
|
||
```
|
||
|
||
Adding a new broker (Vanguard, Robinhood, etc.) becomes a one-file
|
||
add against a documented contract. The hygiene check can be
|
||
referenced from `zfin doctor` (above) without pulling in CSV-parser
|
||
baggage.
|
||
|
||
### Driver
|
||
|
||
Maintenance friction. The next person adding a broker reconciler
|
||
— likely future-you — has to navigate 3438 lines to find the
|
||
pattern. The split also makes the audit-bug investigations already
|
||
in this TODO file (phantom discrepancy on freshly-added lots) easier
|
||
to localize.
|
||
|
||
Pure internal refactor; no user-visible change.
|
||
|
||
## Refactor: trim `src/format.zig` once Money / Date have absorbed their helpers — priority LOW
|
||
|
||
`src/format.zig` is still a ~1700-line grab-bag, but the money- and
|
||
date-shaped helpers that used to live there have been moved out:
|
||
money formatting now lives in `src/Money.zig` (with `{f}` /
|
||
`whole()` / `trim()` / `signed()` / `padRight(N)` / `padLeft(N)`),
|
||
and date formatting lives in `src/Date.zig` (with `{f}` /
|
||
`padRight(N)` / `padLeft(N)`). What's left in `format.zig` is the
|
||
genuinely-format-domain stuff: braille charts, return formatters,
|
||
allocation notes, signed-percent rendering.
|
||
|
||
If the file ever grows enough to be annoying again, consider
|
||
renaming to `src/render.zig` to better describe what's left, or
|
||
splitting the braille chart out (it's ~600 lines on its own).
|
||
Not blocking — file it as cleanup if and when it bites.
|
||
|
||
## Bug: braille charts use raw `close`, not `adj_close` — cliff at splits
|
||
|
||
**Reproduction:** `zfin quote SOXX` (or the TUI quote tab). The
|
||
braille chart drops sharply on **2024-03-07**, which is the
|
||
iShares Semiconductor ETF's 3-for-1 split date:
|
||
|
||
- 2024-03-06 close: $689.60
|
||
- 2024-03-07 close: $237.75 (≈ $689.60 / 2.9)
|
||
|
||
The `adj_close` column in `~/.cache/zfin/SOXX/candles_daily.srf`
|
||
tracks correctly through the split (~$226 → ~$234), so the
|
||
provider data is fine. The bug is purely cosmetic: the chart
|
||
renders the *unadjusted* close price.
|
||
|
||
**Root cause:** `computeBrailleChart` in `src/format.zig:888`
|
||
indexes `data[i].close` instead of `data[i].adj_close`. Lines 901,
|
||
902, 904, 905, 935 all use `.close`.
|
||
|
||
**Independent confirmation:** `zfin splits SOXX` returns
|
||
`2024-03-07 3:1` from Polygon. So the split data exists in the
|
||
provider layer (and gets cached as `splits.srf` once requested),
|
||
but the charting code path doesn't consult it.
|
||
|
||
**Fix candidates:**
|
||
|
||
A. **Switch `computeBrailleChart` to consume `adj_close` directly.**
|
||
Simplest. Affects every chart caller (quote, history, projections
|
||
median band, TUI quote/projections tabs). Cosmetic only — no
|
||
computation depends on it. The price-axis labels would render
|
||
adjusted prices, which may surprise users used to seeing the
|
||
raw last-close. Mitigate with a comment in the chart's right-edge
|
||
label region or a header note.
|
||
|
||
B. **Pass a flag to `computeBrailleChart` selecting `close` vs
|
||
`adj_close`.** Default to adjusted; let the quote tab show raw.
|
||
More flexible, marginally more code.
|
||
|
||
C. **Add a `chart_close` accessor to `Candle` that returns
|
||
`adj_close` if non-zero, else `close`.** Same effect as (A) with a
|
||
defensive fallback.
|
||
|
||
D. **Apply split adjustments at chart-data prep time using
|
||
`splits.srf`.** Walk the candle slice with the split history and
|
||
pre-multiply pre-split closes by the cumulative ratio. More
|
||
work, but produces a chart-axis dollar value the user expects
|
||
("today's last close was $X, the chart starts at $Y from N
|
||
years ago"). This is what most charting libraries do.
|
||
Requires plumbing `DataService.getSplits` into the chart-prep
|
||
path on every chart caller, OR doing the adjustment once in the
|
||
service layer alongside candle fetching. Not all callers have a
|
||
`DataService` reference today (e.g., `runProjection`'s synthetic
|
||
median-band candles).
|
||
|
||
**Recommendation:** Start with (A) or (C) — single-line fix, gets
|
||
the cliff out of all charts immediately. (D) is the "correct" fix
|
||
but a bigger refactor; file as a follow-up if (A)/(C) lands first.
|
||
|
||
**Other affected symbols:** Any held position with a split in the
|
||
last 10 years will have the same artifact. Check NVDA (10:1 split
|
||
on 2024-06-10) for a louder example.
|
||
|
||
**Priority:** LOW. Cosmetic only — analytics already use
|
||
`adj_close` correctly via the per-position trailing-returns path.
|
||
But it's confusing when scanning a chart and seeing a 50% drop
|
||
that isn't real.
|
||
|
||
## Audit: manual-check accounts mechanism — priority HIGH
|
||
|
||
Some accounts/positions can't be reconciled from broker CSVs and need a
|
||
human-in-the-loop reminder at the audit step. Two recurring shapes:
|
||
|
||
- **No-CSV-export accounts** (e.g. some insurance / annuity products)
|
||
where values only live in periodic statements. Git can't detect a
|
||
"change" because nothing changes locally; the user has to log in
|
||
to see the new value.
|
||
- **Payroll-deduction-then-purchase accounts** (e.g. ESPP) where
|
||
payroll-deducted cash doesn't appear in the broker positions CSV
|
||
until the purchase date hits (typically every 6 months). Between
|
||
purchases the cash is a real contribution that `zfin audit` can't
|
||
see.
|
||
|
||
The existing `update_cadence::weekly|monthly|quarterly|none` field already
|
||
sort-of covers this, but has two gaps:
|
||
|
||
1. It fires off the last *git-detected change*, not the last *human
|
||
review*. For statement-only accounts, the value sometimes hasn't
|
||
changed in months — so git never fires, cadence never trips.
|
||
2. Payroll-deduction accounts need weekly-ish attention while
|
||
accumulating cash between purchases, but the accrued balance is
|
||
invisible to the CSV audit.
|
||
|
||
Drift symptom seen in practice: several accounts on
|
||
`update_cadence::weekly` in `accounts.srf` weren't flagged as overdue
|
||
despite no changes in two weeks, because the cadence reads
|
||
git-detected change time rather than human-review time. The cadence
|
||
values themselves may also be wrong for these accounts — revisit
|
||
whether weekly is the right cadence vs. monthly/quarterly given how
|
||
rarely they actually change.
|
||
|
||
### Options
|
||
|
||
A. **New `update_cadence::manual` variant** — always fires every audit
|
||
run until silenced. Blunt but zero design work.
|
||
|
||
B. **`last_refreshed::YYYY-MM-DD` field on `accounts.srf`** — explicit
|
||
human-review timestamp, decoupled from git-detected changes. Audit
|
||
compares `today - last_refreshed` against the cadence. User bumps
|
||
the field when they check the statement. Probably the most
|
||
correct fit for statement-only accounts.
|
||
|
||
C. **Sticky TODO list** — a `todos.srf` or `todo::` field on accounts
|
||
that audit always surfaces until cleared. General-purpose; also
|
||
covers "remember to rebalance on 5/15".
|
||
|
||
### ESPP-style accrual follow-through
|
||
|
||
Payroll-deduction accounts are also a contribution-attribution blind
|
||
spot. If a paystub deducts $X/week but the cash lot doesn't reach
|
||
`portfolio.srf` until the purchase date, the attribution math is
|
||
under-counting contributions and over-counting the purchase-week
|
||
gain. Possible fixes are discussed in the "Contributions diff" TODO
|
||
below — option C there (per-account `cash_is_contribution`) would
|
||
make manually-entered ESPP-style cash additions count correctly.
|
||
|
||
## Investigate: small dollar-value discrepancy between consecutive `compare` runs
|
||
|
||
**Symptom:** Last week's `zfin compare 1W` reported a "now" Total
|
||
Investable Assets that didn't match this week's `zfin compare 1W`
|
||
"then" value by a small but non-zero amount (low single-digit
|
||
thousands). The two numbers should be identical — both are reading
|
||
the same week-ago history snapshot file.
|
||
|
||
Candidates to investigate:
|
||
|
||
- Snapshot file mutated after last week's run (re-snapshot? manual
|
||
edit?). Check `git log -- history/<date>-portfolio.srf` and
|
||
`git diff` against any prior version if tracked.
|
||
- Live cache prices changed between runs and the snapshot path is
|
||
somehow falling through to live prices for some symbols. The
|
||
snapshot SHOULD be self-contained (shares × snapshotted price),
|
||
but verify by re-running `zfin compare <date> <date>` (same date
|
||
both ends) and confirming the same number both ways.
|
||
- Last week's reported "now" was computed against a working-copy
|
||
portfolio that was later edited (reconciliation tweak post-run),
|
||
while this week's "then" reads the committed snapshot. Cross-check
|
||
against any archived report output from last week.
|
||
- `price_ratio` or `adj_close` semantics differing between code paths
|
||
(the REPORT.md §2 caveat about commit-side using current prices
|
||
for DRIP/rollup deltas is a known inconsistency in attribution —
|
||
may or may not apply here).
|
||
|
||
If the source is the working-copy-vs-snapshot mismatch (third bullet),
|
||
that's a workflow issue, not a bug — but worth confirming.
|
||
|
||
## Investigate: detailed 401(k) contributions data source
|
||
|
||
Found a more detailed contributions screen on at least one
|
||
employer-sponsored 401(k) provider portal — distinct from the
|
||
standard positions/holdings view we already pull from. Worth
|
||
investigating whether this unlocks better attribution than what
|
||
we get from the positions CSV alone, and whether other 401(k)
|
||
providers expose similar screens.
|
||
|
||
Open questions to answer when picking this up:
|
||
|
||
- Which screen specifically (path / URL within the portal)? Is there
|
||
an export option, or is it view-only / scrape territory?
|
||
- What fields does it expose (employee pre-tax, employer match,
|
||
after-tax / mega-backdoor, by-pay-period dates, per-fund
|
||
allocations)?
|
||
- Refresh cadence — per-paycheck, daily, on-demand?
|
||
- Can it be auto-discovered like the existing audit CSVs, or
|
||
is it manual-entry territory?
|
||
|
||
If the export is structured and recurring, this could feed a
|
||
401(k)-specific contributions classifier that bypasses the lot-diff
|
||
heuristic for that account, similar to how `cash_is_contribution`
|
||
opts ESPP/HSA accounts into cash-based attribution.
|
||
|
||
Related: ESPP-style accrual blind spot in the "Audit: manual-check
|
||
accounts mechanism" section above.
|
||
|
||
## In-kind transfer support (`type::in_kind`) — priority MEDIUM
|
||
|
||
`transaction_log.srf` parses `type::in_kind` records but the
|
||
contributions matcher always rejects them with "in-kind transfers
|
||
not yet supported in v1." In-kind movements need per-symbol
|
||
matching across accounts: an in-kind transfer of 100 VTI shares
|
||
from Acct A to Acct B shows up as `lot_removed` on A + `new_stock`
|
||
on B (or a `rollup_delta` share increase if B already had a VTI
|
||
lot), neither of which can be matched by the current
|
||
amount-based cash matcher.
|
||
|
||
Proposed: a second pass in `matchTransfers` that iterates
|
||
`type::in_kind` records and looks for same-symbol matches across
|
||
`lot_removed` on `from` + `new_stock`/`rollup_delta` on `to`
|
||
within the window. Gated on share-count and open_price sanity so
|
||
a partial transfer doesn't false-positive against an unrelated
|
||
edit.
|
||
|
||
Driver: when the user starts moving positions between accounts
|
||
directly (e.g. Roth conversion of already-held shares, 401k →
|
||
rollover IRA in-kind) rather than liquidating and re-buying.
|
||
|
||
## Torn SRF files from server sync (root cause unknown)
|
||
|
||
**Status:** Root cause still unidentified. We have mitigations and
|
||
diagnostics in place that keep torn responses from corrupting the
|
||
cache, but we don't yet know *why* responses arrive torn. Until we
|
||
have a root cause, this is not resolved — it's mitigated.
|
||
|
||
Mitigations landed so far:
|
||
|
||
- `syncFromServer` (`src/service.zig`) validates responses via
|
||
`cache.Store.looksCompleteSrf` before `writeRaw`. Torn HTTP bodies
|
||
(empty, missing `#!srfv1` header, or no trailing newline) are
|
||
rejected with a warn-level log and NOT written to cache.
|
||
- HTTP responses are checked for an `ETag` sha256 header; on mismatch
|
||
we retry the request once before giving up and falling back to the
|
||
provider.
|
||
- Read-path self-heal: on SRF parse failure during read, the cache
|
||
entry is invalidated so a subsequent refresh can repair without
|
||
user intervention.
|
||
- Diagnostics: richer error capture around the sync path. So far,
|
||
HTTP transit is the dominant source of torn responses — but that's
|
||
an observation, not a root cause.
|
||
|
||
**Remaining work:**
|
||
|
||
- Identify root cause. Candidates to investigate: proxy/load-balancer
|
||
behavior, HTTP keepalive reuse, partial reads on the server side,
|
||
client-side buffer handling. The etag retry tells us whether the
|
||
problem is per-request or persistent; dig into the diagnostics
|
||
output when the next occurrence is captured.
|
||
- Once root cause is known, decide whether the current mitigations
|
||
are sufficient or whether a targeted fix is needed. The
|
||
mitigations may end up being the whole answer, but we can't
|
||
conclude that without understanding the underlying cause.
|
||
|
||
(Content-Length validation was considered and rejected: once the
|
||
server starts compressing response bodies, Content-Length reflects
|
||
the compressed byte count, not the decoded payload, so it's not a
|
||
reliable integrity check.)
|
||
|
||
## Market-aware cache TTL for daily candles
|
||
|
||
Daily candle TTL is currently 23h45m, but candle data only becomes meaningful
|
||
after the market close. Investigate keying the cache freshness to ~4:30 PM
|
||
Eastern rather than a rolling window. This would avoid unnecessary refetches
|
||
during the trading day and ensure a fetch shortly after close gets fresh data.
|
||
Probably alleviated by the cron job approach.
|
||
|
||
## On-demand server-side fetch for new symbols
|
||
|
||
Currently the server's SRF endpoints (`/candles`, `/dividends`, etc.) are pure
|
||
cache reads — they 404 if the data isn't already on disk. New symbols only get
|
||
populated when added to the portfolio and picked up by the next cron refresh.
|
||
|
||
Consider: on a cache miss, instead of blocking the HTTP response with a
|
||
multi-second provider fetch, kick off an async background fetch (or just
|
||
auto-add the symbol to the portfolio) and return 404 as usual. The next
|
||
request — or the next cron run — would then have the data. This gives
|
||
"instant-ish gratification" for new symbols without the downsides of
|
||
synchronous fetch-on-miss (latency, rate limit contention, unbounded cache
|
||
growth from arbitrary tickers).
|
||
|
||
Note that this process doesn't do anything to eliminate all the API keys
|
||
that are necessary for a fully functioning system. A more aggressive view
|
||
would be to treat ZFIN_SERVER as a 100% source of record, but that would
|
||
introduce some opacity to the process as we wait for candles (for example) to
|
||
populate. This could be solved on the server by spawning a thread to fetch the
|
||
data, then returning 202 Accepted, which could then be polled client side. Maybe
|
||
this is a better long term approach?
|
||
|
||
## Low-priority items
|
||
|
||
The following items are acknowledged but not prioritized. Listed here
|
||
so they don't get lost; pick up opportunistically.
|
||
|
||
### UX
|
||
|
||
- **CLI options command UX.** The `options` command auto-expands only
|
||
the nearest monthly expiration and lists others collapsed. Reconsider
|
||
the interaction model — e.g. allow specifying an expiration date,
|
||
showing all monthlies expanded by default, or filtering by strategy
|
||
(covered calls, spreads).
|
||
- **TUI: toggle to last symbol keybind.** A single-key toggle that
|
||
flips between the current symbol and the previously selected one
|
||
(like `cd -` in bash or `Ctrl+^` in vim). Store `last_symbol` on
|
||
`App`; on symbol change, stash the previous. Useful for
|
||
eyeball-comparing performance/risk data between two symbols.
|
||
|
||
### Data quality
|
||
|
||
- **Fix `enrich` command for international funds.** `deriveMetadata`
|
||
in `src/commands/enrich.zig` misclassifies international ETFs:
|
||
1. `geo` uses Alpha Vantage's `Country` field, which is the *fund
|
||
issuer's* domicile (USA for all US-listed ETFs), not the fund's
|
||
investment geography. Every US-domiciled international fund gets
|
||
`geo::US`.
|
||
2. `asset_class` short-circuits to `"ETF"` when
|
||
`asset_type == "ETF"`, or falls through to a US-market-cap
|
||
heuristic that always produces `"US Large Cap"` /
|
||
`"US Mid Cap"` / `"US Small Cap"`.
|
||
|
||
Known misclassified tickers (all came back as
|
||
`geo::US, asset_class::US Large Cap`):
|
||
- **FRDM** — Freedom 100 Emerging Markets ETF → should be
|
||
`geo::Emerging Markets, asset_class::Emerging Markets`
|
||
- **HFXI** — NYLI FTSE International Equity Currency Neutral ETF
|
||
→ should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
- **IDMO** — Invesco S&P International Developed Momentum ETF →
|
||
should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
- **IVLU** — iShares MSCI International Developed Value Factor
|
||
ETF → should be
|
||
`geo::International Developed, asset_class::International Developed`
|
||
|
||
The Alpha Vantage OVERVIEW endpoint doesn't provide fund geography
|
||
data. Options: use the ETF_PROFILE holdings/country data to infer
|
||
geography, parse the fund name for keywords ("International",
|
||
"Emerging", "ex-US"), or accept that `enrich` is a scaffold and
|
||
emit a `# TODO` comment for ETFs instead of silently misclassifying.
|
||
|
||
### Options / valuation
|
||
|
||
- **Per-account covered call adjustment.** `adjustForCoveredCalls` in
|
||
`valuation.zig` operates on portfolio-wide aggregated allocations.
|
||
It matches sold calls against total underlying shares across all
|
||
accounts. This is wrong — calls in one account can only cover
|
||
shares in that same account. Fixing means restructuring
|
||
`portfolioSummary`, since `Allocation` is currently
|
||
account-agnostic. Low priority — naked calls are rare, and calls
|
||
are typically in the same account as the underlying.
|
||
- **Covered call adjustment O(N*M) loop.** `adjustForCoveredCalls`
|
||
has a nested loop — for each allocation, it iterates all lots to
|
||
find matching option contracts. Fine for personal portfolios
|
||
(<1000 lots). Pre-indexing options by underlying would help if
|
||
someone had a very large options-heavy portfolio.
|
||
|
||
### Analysis / correctness
|
||
|
||
- **Analysis account/asset-class total mismatch.** The "By Account"
|
||
and "By Tax Type" sections in the analysis command sum to slightly
|
||
more than "Asset Class" (~0.6% error). Likely a discrepancy between
|
||
how the lot-level account loop values cash, CDs, or options vs how
|
||
the asset-class section computes them via `portfolio.totalCash()` /
|
||
`totalCdFaceValue()`. Per-account values themselves are correct
|
||
after the price_ratio fix.
|
||
|
||
### Audit
|
||
|
||
- **Audit large-lot threshold tuning.** `src/commands/audit.zig` uses
|
||
`audit_large_lot_threshold: f64 = 10_000.0` as the cutoff for
|
||
"surface this new lot for confirmation." Revisit if $10k proves too
|
||
aggressive (ESPP accruals spam the report) or too permissive (large
|
||
DRIP confirmations slip past). If runtime tuning becomes necessary,
|
||
a `--large-lot <amount>` flag or a global
|
||
`audit_large_lot_threshold` field on `accounts.srf` would be
|
||
reasonable extensions.
|
||
|
||
### Infra / performance
|
||
|
||
- **HTTP connection pooling.** Parallel server sync in `loadAllPrices`
|
||
spawns up to 8 threads, each with its own HTTP connection. Could
|
||
reuse connections to reduce TCP handshake overhead. Only matters
|
||
with very large portfolios (100+ symbols) hitting ZFIN_SERVER.
|
||
- **Streaming cache deserialization.** Cache store reads entire files
|
||
into memory (`readFileAlloc` with 50MB limit). For portfolios with
|
||
10+ years of daily candles, this could use significant memory.
|
||
Keep current approach unless memory becomes a real problem.
|