this is essentially the same as 'New', but for previously seen messages,
such as those that would have been instantly expunged (because they were
marked as deleted), those that we failed to store for some reason, and
already expired ones that are now flagged.
REFMAIL: CAOgBZNonT0s0b_yPs2vx81Ru3cQp5M93xpZ3syWBW-2CNoX_ow@mail.gmail.com
this is more symmetrical with New, and results in some less dodgy
grammar. it also avoids confusion with the \Deleted flag.
fwiw, the pedantically correct name would be Expunges, but that's
confusingly close to the target-side expunge options. also, it's longer.
when propagation of too big messages was entirely suppressed, the only
way to force it was flagging the source message. however, now that we
have placeholders that can be flagged to trigger full propagation, it's
rather pointless to keep the old method working, and still doing it
does in fact confuse users, see for example
REFMAIL: CAOgBZNq_a9yKcq8Jw5y9VS6p2Se8mD7gkf6vPr_KU0taAWuGZQ@mail.gmail.com
to avoid this, we now almost completely shadow the regular meaning of
flagging - it basically becomes a non-synchronizable flag until the
placeholder is upgraded.
this was broken by commit de6dc699 - we now iterated only over far-side
messages, which we don't necessarily load, unlike the near-side ones
(which we need to do to know their current importance).
fix by iterating over sync entries instead of messages, which basically
restores the pre-19128f15 state in that regard. the minor catch here is
that we now need an auxiliary array to sort the sync entries by UIDs. on
the upside, we can also use it to avoid repeated calculation of the
flags.
make sure that an expiration calculation run is performed if ongoing
expiration transactions have been loaded from the journal. this has the
nice side effect of centralizing the regular trigger condition as well.
flags_set_p2() is also adjusted to query S_NEXPIRE only if an expiration
is actually ongoing, like the flag propagation submission loop already
did.
this went unnoticed, because expiration upon arrival of new messages
wasn't autotested at all - despite being the common case.
we cannot orphan all messages whose opposite we expunge, as that would
prevent subsequent propagation of the deletion. we can do that only if
the message is already known to be marked as deleted.
while we already refrained from propagating messages that would be
expunged from the target, we still propagated ones that would be
expunged from the source. this would lead to the weird situation of
creating orphans, and would pose journal replay idempotence problems.
such messages will now never have a sync record, so it becomes
pointless to test for S_PENDING in the trashing loop. note that the
behavior was previously bogus: these messages would have been paired by
the end of the run, so we shouldn't have treated them as solo for the
purposes of TrashOnlyNew/TrashRemoteNew.
don't implicitly propagate flags with upgrades. the user asked for
replacing the body, so do just that. if they also asked for flag
propagation, handle it like the case without upgrade as far as possible.
this makes async parallel flag propagation in the opposite direction
robust, while still being reasonably simple.
re-introduce newmaxuid, but now it's not used at all until the state
is committed. this simplifies the new-message loop, esp. in view of a
soon significantly increased number of branches in it.
instead of doing two runs for each journal entry, do one run for each
"write" operation, be it a journal entry or a writing driver call. this
saves runs between which no visible change occurred, which yields a 33%
improvement in runtime.
we now also exclude the final entry purge from the test, as it's really
kinda pointless, and we'd have to jump through additional hoops
(simulate an atomic commit of the state) to make it reliable in all
cases.
note that this also adds a few steps, which actually uncovered a bug in
the expunge sequencing.
amends efd72b85.
there is actually no use case overlap between between the two (though
limiting the step count does imply keeping the journal, as we exit
before we could commit anyway).
this tests only the common case of the far side being async - adding
100% instead of 50% to the runtime of the test to cover a corner case
didn't seem worth it.
it makes no sense to trash the placeholders, and in fact the common
case is that they are deleted due to being replaced by the full
message.
a separate S_PURGED state needed to be added, as S_PURGE needs to be
reset after setting F_DELETED (so the operation doesn't count as still
pending after journal replay), yet trashing needs an indicator. logging
is now done via a separate command, as piggy-backing it on flag updates
just makes things less legible for no benefit.
this is mostly academical, as trashing being done on the side where
placeholders reside is rather unlikely.
this makes config+data file "sets" relocatable, which is useful for
testing.
this is technically a gratuitous backwards incompatible behavior
change, but to the degree that anyone uses relative paths at all, they
almost certainly rely on PWD being set up such that they won't see a
difference.
instead of specifying the mailboxes and sync state verbatim, use a
format which deals only with "subjects" (but no UIDs), and specifies the
whole state for each subject on a single line (exceptions prove the
rule).
the dumpers don't try to re-create the abstraction, as that's deemed
to be an unreasonable effort.
while rewriting most of the test data anyway, move it to the bottom of
the file, which is a more natural location for it.
don't abort the comparison if continuing makes sense, and try to be more
specific about the problems.
we give up if messages are excessive/missing or the subject is wrong,
as that touches upon the rather complex problem of diff optimization.
this avoids the ugly and error-prone repeated reading of the state
after a failure.
cmpbox() had to be made non-destructive on the box state.
readchan() had to be created.
parse the test data into hierarchical structures instead of using it in
its raw form. this is semantically cleaner and allows us to change the
input format more easily.
while at it, add/fix some licenses/copyrights/comments:
- it makes no sense to have a GPL exception in scripts
- ted did not contribute to the man page
- tst_timers is not part of the mbsync executable
- explicitly put the build system under GPL and add copyrights
to test async operation of the syncing core while using the synchronous
maildir driver, we add a mode to the proxy driver where it queues
callback invocations to the next main loop iteration.
delay the creation of the new state and journal until there is actually
something interesting to write. this saves some cpu cycles and prolongs
ssd life a whee bit.
now that expiration order is determined by a single loop ordered by
far-side UIDs, it is no longer necessary to accurately track the highest
seen UID.
as a side effect, this fixes a problem reported (way too long ago) by
Yuri D'Elia: we failed to up newmaxuid for messages we produced
ourselves, so we would keep enumerating the same messages until we also
propagated externally generated messages from that mailbox - which might
have been never for the server side of archive/trash mailboxes.
