Commit Graph

1265 Commits

Author SHA1 Message Date
Oswald Buddenhagen
0089f49c4a fix expiration when syncing only new messages
this was broken by commit de6dc699 - we now iterated only over far-side
messages, which we don't necessarily load, unlike the near-side ones
(which we need to do to know their current importance).

fix by iterating over sync entries instead of messages, which basically
restores the pre-19128f15 state in that regard. the minor catch here is
that we now need an auxiliary array to sort the sync entries by UIDs. on
the upside, we can also use it to avoid repeated calculation of the
flags.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
4ddacef2c1 fix expiration completion during --noop run
make sure that an expiration calculation run is performed if ongoing
expiration transactions have been loaded from the journal. this has the
nice side effect of centralizing the regular trigger condition as well.

flags_set_p2() is also adjusted to query S_NEXPIRE only if an expiration
is actually ongoing, like the flag propagation submission loop already
did.

this went unnoticed, because expiration upon arrival of new messages
wasn't autotested at all - despite being the common case.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
ef43021f26 don't pointlessly load near-side new messages when expiring
we don't count them towards the total anyway (as they are unpaired).
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
fe4e478e95 fix expiration completion after external expunge
when the expiration is interrupted, but an external expunge removes the
near-side message before we resume, we would just drop the transaction,
resulting in an "unmarked" orphan.

a corner case is an expiration that finishes, but initially isn't
expunged (probably due to an interruption), followed by an unexpiration
that gets interrupted, and the message being expunged externally
(because it's still marked as deleted). we obviously can't complete
that transaction without re-propagating the message, so effectively
cancel it instead.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
1ca278ad0d simplify journaling of expiring unborn messages
don't use a separate journal command anymore, but handle them like
regular expirations, and let the entry purge loop at the end cleanup
them.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
eab3874918 orphan/prune sync entries also if messages were expunged externally
deletions we propagated ourselves are implicitly covered by that as
well, so we don't need to record them separately anymore.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
f2f519e20b fix bogus orphaning due to expunging
we cannot orphan all messages whose opposite we expunge, as that would
prevent subsequent propagation of the deletion. we can do that only if
the message is already known to be marked as deleted.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
3c0ad89a13 don't propagate about-to-be-expunged messages
while we already refrained from propagating messages that would be
expunged from the target, we still propagated ones that would be
expunged from the source. this would lead to the weird situation of
creating orphans, and would pose journal replay idempotence problems.

such messages will now never have a sync record, so it becomes
pointless to test for S_PENDING in the trashing loop. note that the
behavior was previously bogus: these messages would have been paired by
the end of the run, so we shouldn't have treated them as solo for the
purposes of TrashOnlyNew/TrashRemoteNew.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
fbc563e4cb autotest skipping doomed messages more thoroughly
ensure that the messages are actually skipped, not subsequently
expunged.
2022-06-19 16:09:55 +02:00
Oswald Buddenhagen
eab4a12a63 make sure that entries with deleted placeholders are pruned
we need to copy the S_DEL status bits when upgrading sync entries, so
that when a placeholder is flagged at the same time for both deletion
and upgrade, the entry pruning loop sees that the dummy message got
expunged.

note that we don't mask any bits, so the post-split entries may end up
with S_DEL for a zero UID - which doesn't matter.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0da273686f rework flag propagation during placeholder upgrade
don't implicitly propagate flags with upgrades. the user asked for
replacing the body, so do just that. if they also asked for flag
propagation, handle it like the case without upgrade as far as possible.
this makes async parallel flag propagation in the opposite direction
robust, while still being reasonably simple.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
3d90507a75 fix resuming propagation of flags on new messages
log the flags as soon as we know them, and commit them when the uid is
assigned (including during tuid matching).

amends 25b1c2b9.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a2880d740c de-noise msg_fetched() somewhat (prospectively)
assign temporary srec object instead of always spelling out the
indirection.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
be9625725c rework maxuid tracking yet again
re-introduce newmaxuid, but now it's not used at all until the state
is committed. this simplifies the new-message loop, esp. in view of a
soon significantly increased number of branches in it.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
2f4b71c56e knock out ReNew if there are no dummies
the operation requires loading the target side, which makes it somewhat
expensive in otherwise unidirectional syncs.

one could also knock out Flags and Delete if there are no living pairs,
but that wouldn't actually save much.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
1d433b4773 revamp post-journal-replay OPEN_* flags computation
instead of doing it instantly for every message, make a tally and
process it along with the OP_* flags. this allows us to print the
counts, and makes the handling more uniform.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
03d0ab0fbf fix dropping of pending message propagations when resuming
started message propagations will now complete upon resumption even if
the run was requested to be a no-op - whether that's a good thing can
be argued, but it's the least effort way to avoid that committing
discards transaction state.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
e6c6840651 make sure we fetch source flags when resuming --renew
... and the target is expunged.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a652043934 fix updating cached message flags in imap_set_msg_flags()
while this (currently) doesn't really matter (as all flag changes are
calculated before any are actually submitted), msg's flags should not
be updated before set_msg_flags() has actually succeeded.

as a side effect, this does away with the redundancy elimination and
pulling uid from msg (which were both unused since 19128f158).
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
698f9ff173 don't log right before saving state unless requested
only the journal replay autotest really needs it. in other cases it's
just wasting time and ssd life.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
d74af51fa1 rework interrupt-resume autotesting
instead of doing two runs for each journal entry, do one run for each
"write" operation, be it a journal entry or a writing driver call. this
saves runs between which no visible change occurred, which yields a 33%
improvement in runtime.

we now also exclude the final entry purge from the test, as it's really
kinda pointless, and we'd have to jump through additional hoops
(simulate an atomic commit of the state) to make it reliable in all
cases.

note that this also adds a few steps, which actually uncovered a bug in
the expunge sequencing.

amends efd72b85.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
58a69a5b63 split off -Ts from -Tj
there is actually no use case overlap between between the two (though
limiting the step count does imply keeping the journal, as we exit
before we could commit anyway).
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
13764a94b9 don't expunge until all flag propagation is complete
so far, we ensured that propagation _into_ a store completes before
expunging it, but not that propagation _from_ it completes. this way we
could end up expunging the source messages before the changes reached
the target, which could mess up resuming after an interruption.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
fa8186c8d4 heed M_DEAD more thoroughly
this doesn't really fix anything under current conditions, as so far
only Maildir driver functions that modify known messages can cause
concurrent expunges to be detected, and we don't call any of these
before the so far unchecked loops.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
4e25fd59c1 fix possible attempts to set flags of M_DEAD messages
so far, we checked for M_DEAD only in loops over messages. but we should
have checked srec->msg uses as well. this would make the code a mess, so
instead call back from the drivers when messages are expunged, so we can
reset the pointers.

the only case where this really matters so far is the flag setting loop,
which may cause the concurrent expunge of not yet handled messages to be
detected by the maildir driver.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
87d1a4edde fix invalid data accesses in proxy driver callbacks
all printing of auxiliary callback arguments must be conditional on the
command having actually succeeded. this affected fetch_msg() most, which
outright crashed due to a null pointer deref.

to fix this for good, we automate the generation of the status printing
and checking. as a side effect, this fixes the fetch_msg() callback not
printing the status at all.

amends 4cc5ad5a.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
30a6015624 test also one-sided async operation
this tests only the common case of the far side being async - adding
100% instead of 50% to the runtime of the test to cover a corner case
didn't seem worth it.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
3a8f8a8391 fake async drivers more convincingly
instead of delaying the callback, delay the actual driver call. this is
in line with how the IMAP driver would behave, as since commit 6c08f568
it queues the socket writes (the network upstream latency goes on top,
but that doesn't alter the result).

amends 4423a932.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
16238909d3 print some more debug info during state reading and journal replay 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
6e7b3d24c1 improve some debug messages
- print on which side we're upgrading a placeholder. as a side effect,
  this moves some magic out of upgrade_srec().
- don't use past tense for something that is only about to happen
- don't print status of every sync entry affected by journal replay -
  the entry load already prints it, and relevant operations print the
  new status. also, it was inconsistent with not printing the entry's
  old flags.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
950ebe833d add debug messages about effective requested operations 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
3091e2fe5a add debug pretty-printing for OPEN_* flags as well 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
17db5de0ca add debug pretty-printing for sync record status flags as well 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
c902f69c6f format message flags in the sync debug output
now that we can do it cheaply, make copious use of it.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a49017f481 streamline pretty-printing of message flags
wrap make_flags() into fmt_flags() which returns a (struct-wrapped)
string, so the calls can be inlined into the printf statements, without
reserving buffers.

we locally force optimization, so copy elision is always done, as debug
builds would otherwise suffer a somewhat unreasonable performance hit.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a5dc1baedf denoise usages of INIT_IMAP_CMD()
move the variable declarations into the macro, taking advantage of the
fact that mixing them with code is ok since C99.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
f4ed8b27f6 fold superfluous INIT_IMAP_CMD_X() macro
it's been identical to INIT_IMAP_CMD() since c3d91ae1.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
f5d234ffa1 simplify an initialization in proxy driver code generator
the 'checked' hash entry doesn't exist yet at that point, so it's always
undefined.

amends 4423a932.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
5c44732fd9 support C99 comments in proxy driver code generator 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
7f1c667910 cache drivers' ability to handle crlf in mails
it's a cheap call, but it clutters up log files.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0f1b2b646b remove questionable optimization from case-insensitive string comparison
we optimized the case where the string would be equal even without
upper-casing, but for the much more common case where the strings differ
even after upper-casing, this was just an additional conditional.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
61b08880c8 improve tracking of maxxfuid
do it when the corresponding expiration events are logged/replayed.
that makes it unnecessary to log it separately just to make the
autotest happy.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0f6362f2e2 fix tracking of sync record count through placeholder upgrades
for now, this is just pedantry, as it merely affects the size of the
already oversized { uid => srec } flathash after journal replay.

amends 70bad661.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
69118d25ec optimize { uid => srec } flathash
the sync records contain the uid, so storing it explicitly doubles the
hash's size for no good reason.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
8f4af5f78f make use of finduid[] after journal replay
we used to pass all initially loaded messages to match_tuids(), which
could be quite some when syncing old messages. as lost TUIDs result in
O(n^2) behavior, this could have a serious performance impact.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a1a3313ed4 print human-readable summary at exit
this is meant primarily for use with a redirected output, where no
progress counters are shown. but it would be weird not to have it with
tty output as well, so it replaces the counters after completion.

REFMAIL: 87bl2cgt6u.fsf@curie.anarc.at
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
0f7c231cc2 fix bogus complaints about missing Store/Account references
... when the reference is simply invalid (we already get a complaint
about that).
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
bf59636f0f don't stop validating Channel config after first error 2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
c986f80bb0 append even broken Channel configs to list
otherwise perfectly fine Group defs will complain about invalid refs.

we already do that with Store configs.
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
2cbf8a68cf abort if any invalid Channels/Groups have been specified
this is more in line with expectations, and avoids a silly "No channel
specified." error message.
2022-06-19 16:03:38 +02:00