lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions
zfin/docs/guides/audit-against-brokerage.md
Emil Lerch 40869cb3a4
lint srf files in audit / doctor commands 
This also provides ensurance through zig build test that all
fields are documented in markdown so they do not get out of sync
2026-09-24 06:39:23 -07:00

361 lines
18 KiB
Markdown
Raw Blame History

# Audit against your brokerage
**Goal:** catch drift between what zfin thinks you hold and what your
brokerage actually reports -- wrong share counts, sales you forgot to
record, missing lots, stale manual prices -- by reconciling
`portfolio.srf` against a positions export.
**You'll need:** a portfolio whose [`accounts.srf`](set-up-accounts.md)
entries carry `institution::` and `account_number::` (that's how zfin
ties an export back to your accounts -- see
[How accounts are matched](#how-accounts-are-matched)), plus an export
from a supported broker.
> Heads up: this is the most heuristic corner of zfin. The brokerage
> parsers are format-specific, account matching depends on metadata you
> maintain, and the comparison uses deliberate tolerances. It's the best
> way to keep your records honest, but expect a little setup and the
> occasional "why didn't that match?" -- this guide covers the gotchas,
> not just the happy path.
## Supported brokers and how to export
`zfin audit` reconciles against **Fidelity** and **Schwab**. (Wells
Fargo is handled by [`import`](#what-about-wells-fargo), not audit.)
| Broker | How to export | Flag |
|--------------------------|-------------------------------------------------------------------------------------------------------------|--------------------|
| **Fidelity** | *Positions* tab -> the three-dot (**⋮**) menu -> **Download** (a CSV) | `--fidelity <CSV>` |
| **Schwab** (per-account) | *Accounts -> Positions* -> **Export** (one CSV per account) | `--schwab <CSV>` |
| **Schwab** (summary) | *Accounts -> Summary*: select the accounts table and copy it ([what to copy](#schwab-summary-what-to-copy)) | `--schwab-summary` |
The two Schwab inputs differ in detail: the **per-account CSV** has full
per-position data (shares, price, value); the **summary paste** carries
only each account's cash and total value, so it reconciles *totals*, not
individual holdings. Use the summary for a quick "are my account totals
right?", the CSV for position-level checks. (Fidelity money-market rows
and Schwab "Cash & Cash Investments" rows are recognized as cash.)
The Fidelity **Download** isn't a top-level button -- it's behind the
three-dot (**⋮**) menu at the top-right of the positions panel:
![Fidelity Positions tab with the three-dot menu open, showing the Download item](../images/fidelity-positions-download.png)
*(The account list down the left side is blanked out above.)*
### Schwab summary: what to copy
The summary paste comes from Schwab's **Accounts -> Summary** page.
Scroll to the **Accounts** table, drag-select from the first account's
name through the last row, and copy. Then either save it as a `.txt`
file in your `audit/` folder (where auto-discovery will find it) or pipe
it straight in:
```bash
zfin audit --schwab-summary # paste, then Ctrl-D
```
A good paste is repeating three-line blocks -- the account name, the
"ending in" line, then a values line -- and looks about like this
(figures fictional):
```
Sample Roth IRA
Account number ending in 1234 ...1234
Type IRA $46.44 $227,058.15 +$1,072.88 +0.47%
Sample Brokerage
Account number ending in 5678 ...5678
Type Brokerage $12,500.00 $980,000.00 +$3,200.00 +0.33%
Sample Trust
Account number ending in 9012 ...9012
Type $2,000.00 $415,300.00 +$1,150.00 +0.28%
```
zfin anchors on each **"Account number ending in"** line (its trailing
digits are the account number) and reads the **first two dollar figures**
on the line below as **cash** then **total value** -- everything else on
that line is ignored, and the account-type word is optional. If your copy
looks nothing like this -- no "ending in" lines, or no dollar figures --
you grabbed the wrong region. Because it carries only cash and totals,
the summary reconciles **account totals**, not individual positions.
## Run it
**Point it at a file:**
```bash
ZFIN_HOME=~/finance zfin audit --fidelity ~/Downloads/Portfolio_Positions.csv
ZFIN_HOME=~/finance zfin audit --schwab ~/Downloads/Positions-Individual.csv
ZFIN_HOME=~/finance zfin audit --schwab-summary # then paste the page, Ctrl-D
```
**Or run it with no flags** -- `zfin audit` does a portfolio hygiene
check *and* auto-discovers and reconciles any recent exports it finds
(next two sections).
## The hygiene check
With no flags, `zfin audit` first prints a health report:
- **Stale manual prices** -- lots with a manual `price::` older than
`--stale-days` (default 3).
- **Accounts overdue for update** -- accounts past their
`update_cadence` (see [accounts.srf](set-up-accounts.md#3-tune-the-maintenance-cadence)).
- **Brokerage files** it discovered, which it then reconciles.
- **Config file problems** -- field-name typos in your `.srf` files, impossible
lot dates, and `accounts.srf` values zfin had to ignore.
```
Portfolio hygiene
Stale manual prices (>3 days - --stale-days to configure)
(none)
Accounts overdue for update (weekly default - set update_cadence in accounts.srf)
Sample IRA weekly no update history found
Sample Brokerage weekly no update history found
```
"No update history found" is a nudge, not an error -- silence accounts
you don't actively track with `update_cadence::none`.
### Config file problems
Your `.srf` files can be wrong in ways that don't stop them loading.
zfin reads what it understands and quietly ignores or works around the
rest, so a mistake can change your numbers with no error at all.
`zfin audit` checks `portfolio*.srf`, `accounts.srf`, `metadata.srf`,
`transaction_log.srf`, `projections.srf`, and `watchlist.srf` for three
kinds of problem, and prints nothing when there aren't any:
```
Config file problems
portfolio.srf
line 3: unrecognized field 'price_dat' - did you mean 'price_date'?
line 4: BND close_date 2062-03-14 is in the future - the lot is still counted as held until then
line 5: AAPL has close_date but no close_price - the sale's realized gain is recorded as 0
valid fields:
symbol shares open_date open_price close_date close_price note
label account security_type maturity_date rate drip ticker price
price_date price_ratio split_factor underlying strike multiplier
option_type
accounts.srf
line 2: account 'Sample Brokerage': audit_large_lot_threshold must be > 0 (got 0); ignored, so the audit uses its default
line 4: account 'Sample Roth': tax_mix_* must not name the account's own tax_type; the tax mix is ignored and the account counts wholly as roth
```
Each finding names the line of the record it came from, and the date
and account findings also say what zfin is doing instead of what you
wrote.
#### 1. Field names (every file)
SRF matches record keys against field names exactly. A key that matches
nothing is **discarded without complaint** -- a misspelled `price_dat::`
is indistinguishable from having left the price date out. Most fields
have defaults, so most typos change behavior with no output at all.
| Report | Meaning |
|-----------------------------|----------------------------------------------------------------------------------------|
| `did you mean '...'?` | A dropped or doubled character. The suggested name is a real field. |
| `differs only by case` | `Account` is not `account`. Matching is case-sensitive, so this silently does nothing. |
| `ignored for security_type` | A real field that this kind of lot never reads -- e.g. `rate` outside a `cd`. |
| `appears twice` | SRF keeps the **first** occurrence. Editing the second one has no effect. |
| `derived by zfin` | zfin computes this field; a hand-written value is overwritten. Delete it. |
| `unrecognized field` | No match and no close relative. Check it against the `valid fields` list. |
The suggestions are deliberately cautious -- they only fire when a real
field name is a prefix of what you typed, or differs from it only by
case -- so a suggestion is never a guess. That means a typo in the
*middle* of a name (`update_cadance`) gets no suggestion, which is why
the **`valid fields` list** is printed whenever a field name is in
question. It comes straight from the code, so unlike any document it
cannot be out of date.
#### 2. Lot dates (`portfolio*.srf`)
These fields parse fine but describe a lot that can't exist. They matter
more than they look: most of zfin decides whether a lot is held by
comparing `close_date` to today, but `zfin contributions` treats a lot
as sold the moment it has any `close_date` at all -- so a bad date makes
the two disagree.
| Report | What zfin does with it |
|--------------------------------------|--------------------------------------------------------------------------------------------|
| `close_date ... is in the future` | Counts the lot as held until that date. Usually a mistyped year. |
| `close_date ... is before open_date` | The lot is never counted as held on any date. |
| `has close_date but no close_price` | Stock lots only. The sale's realized gain is recorded as 0. |
| `has close_price but no close_date` | The lot stays held. On a stock lot, its row shows the close price instead of the live one. |
| `open_date ... is in the future` | The lot is left out of your positions until that date. |
| `price_date ... is in the future` | Stock lots only. The manual price counts as fresh, so it is never flagged as stale. |
A close dated today is already closed, and a same-day buy-and-sell
(`close_date` equal to `open_date`) is fine. A future `maturity_date` is
normal -- that is the field for a CD or option that ends on a known
date. Watch lots are exempt, since zfin reads only their `symbol`. See
[Closed lots](../reference/config/portfolio-srf.md#closed-lots).
#### 3. Account values (`accounts.srf`)
Values zfin rejects and replaces with a default, so the account loads --
just not the way you wrote it.
| Report | What zfin does instead |
|-----------------------------------------------|-------------------------------------------------------------------------|
| `audit_large_lot_threshold must be > 0` | Uses the audit's built-in threshold. |
| `harvested must be a finite number` | Drops the harvested figure (only reachable by typing `inf` or `nan`). |
| `tax_mix_* ...` (any of the four rules below) | Ignores the whole tax mix; the account counts wholly as its `tax_type`. |
The four tax-mix rules: every `tax_mix_*` carve-out must be above 0 and
a finite number, none may name the account's own `tax_type`, and
together they must sum to less than 100. See
[`accounts.srf`](../reference/config/accounts-srf.md).
These three are also logged to stderr whenever `accounts.srf` loads;
the audit is where they get a line number and can't scroll past.
#### Also in `zfin doctor`
`zfin doctor` reports the same findings one file at a time, with a link
to each file's reference page, and also covers `keys.srf`, `theme.srf`,
and `acknowledgments.srf`. Neither command changes its exit code over
any of these -- they're nudges, not failures.
## Auto-discovery (and your download folder)
With no `--fidelity`/`--schwab` flag, zfin looks for exports in two
places:
1. **`$ZFIN_AUDIT_FILES`** -- a directory *you* set. Point it at wherever
your browser saves downloads (e.g. `~/Downloads`) so a
just-downloaded export is found with no copying or renaming. (zfin
does **not** scan it on its own -- you opt in by setting this.)
2. **`<portfolio-dir>/audit/`** -- a dedicated subfolder next to your
`portfolio.srf`, for exports you want to keep around.
What it considers:
- **Only files modified in the last 24 hours.** Your browser saves the
export to its download folder with names like
`Portfolio_Positions_Jun-19.csv`; the
recency window keeps zfin reconciling *the one you just pulled*, not
last quarter's.
- **Detected by content, not filename.** zfin sniffs the first lines --
Fidelity begins `Account Number`/`Account Name`, a Schwab CSV begins
`"Positions for ...`, a Schwab summary contains `Account number ending
in`. A renamed file still works; an unrelated CSV is skipped.
So with `ZFIN_AUDIT_FILES=~/Downloads`, the workflow collapses to
"download from your broker, run `zfin audit`, done."
## How accounts are matched
This is the part that trips people up. An export covers one or more
**accounts**, and zfin has to tie each one to an account in your
portfolio. It does that through
[`accounts.srf`](set-up-accounts.md#2-add-institution-and-account-number-for-auditing):
- The export carries an account number -- Schwab's from the "Positions
for account ...1234" title, Fidelity's from the Account Number column,
the summary's from "...ending in 1234".
- zfin finds the `accounts.srf` entry whose `institution::` (`fidelity`,
`schwab`) **and** `account_number::` match, and compares against that
account's lots.
- **No match -> the account is shown as `unmapped`** and flagged as a
discrepancy. Fix it by adding `institution::` and `account_number::`
to that account in `accounts.srf` (a placeholder number you recognize
is fine -- it just has to match what the export shows).
## Reading the report
zfin treats the **brokerage as the source of truth** and shows, per
account, your portfolio (PF) against the broker (BR). Figures below are
illustrative and fictional:
```
Portfolio Audit (brokerage is source of truth)
========================================
Sample Brokerage *1234
Symbol PF Shares BR Shares PF Price BR Price
VTI 100.000 100.000 373.38 373.38 ok
SCHD 200.000 210.000 31.86 31.86 brokerage +10.000
AGG 50.000 0.000 portfolio only
```
- **Portfolio-only** rows are lots the broker no longer shows -- a sale
you forgot to remove, or a mistyped symbol.
- **Brokerage-only** rows are holdings missing from your portfolio.
- A share or value **delta** flags a count or price mismatch.
`--verbose` prints the full comparison even when everything reconciles.
### Discrepancies, and which ones are muted
- **Cash matches to the penny.** It's an exact figure on both sides, so
any gap is real (e.g. money-market dividend accrual between updates)
and is surfaced as a warning.
- **Securities get ~$1 of slack** for sub-cent NAV rounding on large
positions. Beyond that, a value delta is a **warning by default** --
your records should reconcile to the dollar.
- **Two cases are muted instead** -- still shown (and still in the
totals), just greyed out as "expected, you probably don't care." Muting
never hides a discrepancy:
- **CDs.** zfin carries a CD at face value while the broker marks it to
the secondary market. That gap is muted up to a band computed from
the CD's own `rate` and time to maturity (capped at one year's
coupon); anything larger still warns. We can't reproduce the broker's
exact mark without a live yield, so we bound it instead.
- **Options.** zfin tracks options at cost while the broker marks to
market, and that gap is unbounded without a live quote. So an account
holding open options mutes its account-level value delta -- drill into
the per-position export (`--schwab` / `--fidelity`) to check share
counts, which are never muted.
- **Share-count mismatches are never muted.**
### Institutional share classes
If a lot is priced through a retail-ticker `ticker::` alias while the
account actually holds an institutional class (a different NAV), audit
compares against the broker's NAV and can **suggest a `price_ratio`** to
bridge the gap. Accounts flagged `direct_indexing::true` get the same
treatment to track drift. See
[price resolution](../reference/config/portfolio-srf.md#advanced-and-option-fields).
## Why it's finicky
- The parsers are **broker-specific and hardcode each export's column
layout** -- if Fidelity or Schwab changes their format, parsing can
break (Fidelity's header is validated to catch this; Schwab's is not).
They are not full RFC-4180 CSV parsers (no escaped quotes or
multi-line fields) -- fine for the real exports, not for arbitrary
CSVs.
- Matching is only as good as the `institution::` / `account_number::`
entries you keep in `accounts.srf`.
- Options, CDs, and cash are reconciled separately from share counts.
None of this is a reason to skip it -- it's the single best way to keep
your records honest -- just know it expects some setup and an occasional
manual nudge.
### What about Wells Fargo?
Wells Fargo's portal has no clean positions export, so it isn't an
`audit` target. Instead, [`zfin import --wells-fargo`](../reference/cli/import.md)
rebuilds a portfolio file from a paste of the WF positions table (copy
the rendered table from the brokerage portal and save it to a file).
Fidelity and Schwab exports can be imported the same way.
> **Keep brokerage exports private.** They contain real account numbers
> and holdings. Store them outside any git repository and delete them
> when you're done reconciling.
## Next steps
- [`zfin audit` reference](../reference/cli/audit.md) -- every flag.
- [Map your accounts](set-up-accounts.md) -- the `institution` / `account_number` matching keys.
- [`zfin import`](../reference/cli/import.md) -- build a portfolio file *from* an export (including Wells Fargo).
---
[Previous: Plan for retirement](plan-retirement.md) | [Next: A periodic review](periodic-review.md) | [Documentation home](../README.md)
Reference in a new issue View git blame Copy permalink