we cannot orphan all messages whose opposite we expunge, as that would
prevent subsequent propagation of the deletion. we can do that only if
the message is already known to be marked as deleted.
while we already refrained from propagating messages that would be
expunged from the target, we still propagated ones that would be
expunged from the source. this would lead to the weird situation of
creating orphans, and would pose journal replay idempotence problems.
such messages will now never have a sync record, so it becomes
pointless to test for S_PENDING in the trashing loop. note that the
behavior was previously bogus: these messages would have been paired by
the end of the run, so we shouldn't have treated them as solo for the
purposes of TrashOnlyNew/TrashRemoteNew.
don't implicitly propagate flags with upgrades. the user asked for
replacing the body, so do just that. if they also asked for flag
propagation, handle it like the case without upgrade as far as possible.
this makes async parallel flag propagation in the opposite direction
robust, while still being reasonably simple.
re-introduce newmaxuid, but now it's not used at all until the state
is committed. this simplifies the new-message loop, esp. in view of a
soon significantly increased number of branches in it.
the operation requires loading the target side, which makes it somewhat
expensive in otherwise unidirectional syncs.
one could also knock out Flags and Delete if there are no living pairs,
but that wouldn't actually save much.
instead of doing it instantly for every message, make a tally and
process it along with the OP_* flags. this allows us to print the
counts, and makes the handling more uniform.
started message propagations will now complete upon resumption even if
the run was requested to be a no-op - whether that's a good thing can
be argued, but it's the least effort way to avoid that committing
discards transaction state.
instead of doing two runs for each journal entry, do one run for each
"write" operation, be it a journal entry or a writing driver call. this
saves runs between which no visible change occurred, which yields a 33%
improvement in runtime.
we now also exclude the final entry purge from the test, as it's really
kinda pointless, and we'd have to jump through additional hoops
(simulate an atomic commit of the state) to make it reliable in all
cases.
note that this also adds a few steps, which actually uncovered a bug in
the expunge sequencing.
amends efd72b85.
so far, we ensured that propagation _into_ a store completes before
expunging it, but not that propagation _from_ it completes. this way we
could end up expunging the source messages before the changes reached
the target, which could mess up resuming after an interruption.
this doesn't really fix anything under current conditions, as so far
only Maildir driver functions that modify known messages can cause
concurrent expunges to be detected, and we don't call any of these
before the so far unchecked loops.
so far, we checked for M_DEAD only in loops over messages. but we should
have checked srec->msg uses as well. this would make the code a mess, so
instead call back from the drivers when messages are expunged, so we can
reset the pointers.
the only case where this really matters so far is the flag setting loop,
which may cause the concurrent expunge of not yet handled messages to be
detected by the maildir driver.
- print on which side we're upgrading a placeholder. as a side effect,
this moves some magic out of upgrade_srec().
- don't use past tense for something that is only about to happen
- don't print status of every sync entry affected by journal replay -
the entry load already prints it, and relevant operations print the
new status. also, it was inconsistent with not printing the entry's
old flags.
wrap make_flags() into fmt_flags() which returns a (struct-wrapped)
string, so the calls can be inlined into the printf statements, without
reserving buffers.
we locally force optimization, so copy elision is always done, as debug
builds would otherwise suffer a somewhat unreasonable performance hit.
we used to pass all initially loaded messages to match_tuids(), which
could be quite some when syncing old messages. as lost TUIDs result in
O(n^2) behavior, this could have a serious performance impact.
it makes no sense to trash the placeholders, and in fact the common
case is that they are deleted due to being replaced by the full
message.
a separate S_PURGED state needed to be added, as S_PURGE needs to be
reset after setting F_DELETED (so the operation doesn't count as still
pending after journal replay), yet trashing needs an indicator. logging
is now done via a separate command, as piggy-backing it on flag updates
just makes things less legible for no benefit.
this is mostly academical, as trashing being done on the side where
placeholders reside is rather unlikely.
... as otherwise these messages would be just lost.
the assumption is that opposite-side trashing is used only for locally
generated messages whose size we control. it's also more consistent with
same-side trashing, where even oversized messages would be trashed.
the exclusion was broken anyway, as we failed to query the size of old
messages, particularly after 70bad661.
messages for which only a placeholder was propagated should be treated
as not propagated, as otherwise the actual contents will be lost when
only not propagated messages are trashed.
amends 70bad661.
this is a lot more legible, and makes it possible to insert values in
the middle without churn.
i didn't find a way to do this with the pre-processor, so we now have
another code generator.
we now use the $< make variable, which requires gmake on netbsd < 9.0,
and possibly other systems with an ancient make.
- wrap flow-controlled statements that contain blocks into blocks
themselves
- wrap bodies of do-while()s into blocks
- use braces on 'else' symmetrically (this obviously has a cascading
effect, so this patch touches lots of lines)
- attach braces
unavoidably, the rules are sometimes broken around #ifdef-ery.
it doesn't really add any value when resuming after an interruption, as
incomplete propagations are perfectly expected, and are recovered from.
but it does make sense in the non-UIDPLUS path, where not finding just
stored messages indicates a problem.
so move the reporting from match_tuids() to msgs_found_new(), and remove
a redundant warning from the latter.
amends 6577bf3e.
while at it, add/fix some licenses/copyrights/comments:
- it makes no sense to have a GPL exception in scripts
- ted did not contribute to the man page
- tst_timers is not part of the mbsync executable
- explicitly put the build system under GPL and add copyrights
the messages are trashed in mailbox (and thus UID) order, and in
practice we expect the operations to complete in order. however, if
older messages need to be trashed after a journal replay, and we get
interrupted again, the next replay would produce an unsorted array,
and thus break the binary search.
amends 2bba9b9.
the input isn't necessarily null-terminated (it currently is for imap,
but not for maildir), so if the message ended somewhere within the
header field name, we'd read beyond its end, which theoretically could
cause a crash. no other adverse effects could result, as we'd stop
processing such a broken message right afterwards.
amends 70bad661.
that shouldn't really be a problem, as we have 2GB of headroom, and most
growth would happen when sending an all-newlines message from maildir to
imap (due to CR additions), which is mostly non-critical. but better
safe than sorry.
