Commit Graph

1292 Commits

Author SHA1 Message Date
Oswald Buddenhagen
c5e967f94d add support for LITERAL- extension
it's the same as LITERAL+, only with a strongly limited payload size.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5048521d79 make sure that all IMAP lists are parsed
... even if we ignore their contents, as they may (hypothetically)
contain literals, which are relevant for the stream syntax.

amends 6fd4e8de2.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
a07be5f175 improve error reporting from IMAP list parsing
so far, we'd print only a generic message - except in two cases, where
the generic error would be preceded by a specific one. now we always
print a single reasonably specific message.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
c7f50a3069 simplify parsing responses with multiple IMAP lists
the callbacks took the current parsing position only to pass it on to
the next list parse. so instead, store it in the state, and have a
separate function for continuation.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
ad8520b741 revamp sending of IMAP commands with literals
include the literal's length into the command itself, so error messages
become clearer. the commands are initially built for LITERAL+, and are
"edited down" right before sending if necessary.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
abd31aad61 simplify assembly of IMAP APPEND commands
treat the date string the same as the flag string.

amends eb1f1076.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
4ae0159132 handle tagged BAD [TOOBIG] server responses
some servers consider attempts to APPEND too big messages to be BAD
commands, rather than just NO-worthy ones. so just pretend that we did
in fact get a NO, and carry on.

this was tested with gmail. if we run into servers that don't send the
expected response code, we will need to relax the condition.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5e5c7fb508 free IMAP command's payload prior to invoking its completion callback
otherwise the synching may livelock due to failing to schedule follow-up
commands, thinking the buffers are still full. of course, this is
relevant only for commands that failed early and thus didn't free the
payload right after submission already - which will be possible only
after the next commit.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
1225f0b86b add ExpungeSolo option
REFMAIL: CAOgBZNonT0s0b_yPs2vx81Ru3cQp5M93xpZ3syWBW-2CNoX_ow@mail.gmail.com
2022-06-19 16:10:54 +02:00
Oswald Buddenhagen
95a22739fa don't UID EXPUNGE too many messages
we didn't check that the UIDs are adjacent, so we might have caught
not fetched deleted messages between two fetched messages below the
bulk load range.

checking adjacency of UIDs would make expunges in the bulk range (which
is likely to be full of holes) rather inefficient. so we use sequence
numbers instead.

this is admittedly a rather academical fix ...

amends 18225344.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
1631361f66 revamp handling of expunged messages
try to purge sync entries based on which messages are *actually*
expunged, rather than those that are *expected* to be expunged.

to save network bandwidth, the IMAP driver doesn't report all expunges,
so some entry purges would be delayed - potentially indefinitely, e.g.,
when only --pull-new --push is used, or Trash isn't used (nor
ExpungeSolo, prospectively). so keep a fallback path to avoid this.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
1a1ac25bc8 track IMAP message sequence numbers (and therefore expunges) 2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
df4e6383f5 track number of messages in IMAP stores
... rather than calculating it on the fly. for efficiency.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
6fe7172901 moan about trashing in non-UIDPLUS boxes
... as that's inherently racy, as the manual says.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
edbf9a35da use UID EXPUNGE also when trashing remotely
amends 2f0fbcd3.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
04c7126ce9 add autotest for syncing non-UIDPLUS stores 2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
767a318eea add new sync operation 'Old'
this is essentially the same as 'New', but for previously seen messages,
such as those that would have been instantly expunged (because they were
marked as deleted), those that we failed to store for some reason, and
already expired ones that are now flagged.

REFMAIL: CAOgBZNonT0s0b_yPs2vx81Ru3cQp5M93xpZ3syWBW-2CNoX_ow@mail.gmail.com
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
a8e145e589 split OPEN_PAIRED from OPEN_OLD, and rename OPEN_OLD_IDS accordingly
this emphasizes the non-complementarity to OPEN_NEW.
the "real" OPEN_OLD is currently used only for trashing.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
d77d67c948 concentrate calculation of minimal loaded UID in load_box()
so it's in one place with the maximal one, sans the one-sided
preparation of the bulk range + exception list.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
e98aed87f0 silently reinterpret --full
change it from --{push,pull} to --{new,gone,flags,upgrade}.
that alone doesn't change anything; combining it with any other option
would cause a complaint anyway.

this enables us to introduce --{push,pull}-full, which will matter when
--full doesn't actually include all type flags any more, as we'll then
be able to combine it with the extra type flag(s). in the same vein, we
now advertize "Sync Full" instead of "Sync All" (both continue to be
recognized).
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
58564e4f76 rename Delete to Gone
this is more symmetrical with New, and results in some less dodgy
grammar. it also avoids confusion with the \Deleted flag.

fwiw, the pedantically correct name would be Expunges, but that's
confusingly close to the target-side expunge options. also, it's longer.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
6308a7f41b rename ReNew to Upgrade
that's what it really has been for a while now.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
8f39d06015 fix mixing MaxMessages with MaxSize
this is actually a useful combination for resource-constrained devices.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
882c9825cd fix handling of 'seen' flag wrt placeholders
while we don't want to propagate seeing _from_ the placeholder, we do
want to propagate it _to_ it, and consequently also un-seeing from it.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
cb687f1bee make MaxSize ignore source-side message flagging
when propagation of too big messages was entirely suppressed, the only
way to force it was flagging the source message. however, now that we
have placeholders that can be flagged to trigger full propagation, it's
rather pointless to keep the old method working, and still doing it
does in fact confuse users, see for example
REFMAIL: CAOgBZNq_a9yKcq8Jw5y9VS6p2Se8mD7gkf6vPr_KU0taAWuGZQ@mail.gmail.com

to avoid this, we now almost completely shadow the regular meaning of
flagging - it basically becomes a non-synchronizable flag until the
placeholder is upgraded.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
e6a15bee59 make sure we fetch source flags when expiring while pulling new
otherwise important messages may be incorrectly aborted.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
3febb16fd5 fix off-by-one when loading messages outside the bulk range
this would lead to a bogus deletion being propagated to the near side.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
0089f49c4a fix expiration when syncing only new messages
this was broken by commit de6dc699 - we now iterated only over far-side
messages, which we don't necessarily load, unlike the near-side ones
(which we need to do to know their current importance).

fix by iterating over sync entries instead of messages, which basically
restores the pre-19128f15 state in that regard. the minor catch here is
that we now need an auxiliary array to sort the sync entries by UIDs. on
the upside, we can also use it to avoid repeated calculation of the
flags.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
4ddacef2c1 fix expiration completion during --noop run
make sure that an expiration calculation run is performed if ongoing
expiration transactions have been loaded from the journal. this has the
nice side effect of centralizing the regular trigger condition as well.

flags_set_p2() is also adjusted to query S_NEXPIRE only if an expiration
is actually ongoing, like the flag propagation submission loop already
did.

this went unnoticed, because expiration upon arrival of new messages
wasn't autotested at all - despite being the common case.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
ef43021f26 don't pointlessly load near-side new messages when expiring
we don't count them towards the total anyway (as they are unpaired).
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
fe4e478e95 fix expiration completion after external expunge
when the expiration is interrupted, but an external expunge removes the
near-side message before we resume, we would just drop the transaction,
resulting in an "unmarked" orphan.

a corner case is an expiration that finishes, but initially isn't
expunged (probably due to an interruption), followed by an unexpiration
that gets interrupted, and the message being expunged externally
(because it's still marked as deleted). we obviously can't complete
that transaction without re-propagating the message, so effectively
cancel it instead.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
1ca278ad0d simplify journaling of expiring unborn messages
don't use a separate journal command anymore, but handle them like
regular expirations, and let the entry purge loop at the end cleanup
them.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
eab3874918 orphan/prune sync entries also if messages were expunged externally
deletions we propagated ourselves are implicitly covered by that as
well, so we don't need to record them separately anymore.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
f2f519e20b fix bogus orphaning due to expunging
we cannot orphan all messages whose opposite we expunge, as that would
prevent subsequent propagation of the deletion. we can do that only if
the message is already known to be marked as deleted.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
3c0ad89a13 don't propagate about-to-be-expunged messages
while we already refrained from propagating messages that would be
expunged from the target, we still propagated ones that would be
expunged from the source. this would lead to the weird situation of
creating orphans, and would pose journal replay idempotence problems.

such messages will now never have a sync record, so it becomes
pointless to test for S_PENDING in the trashing loop. note that the
behavior was previously bogus: these messages would have been paired by
the end of the run, so we shouldn't have treated them as solo for the
purposes of TrashOnlyNew/TrashRemoteNew.
2022-06-19 16:10:00 +02:00
Oswald Buddenhagen
fbc563e4cb autotest skipping doomed messages more thoroughly
ensure that the messages are actually skipped, not subsequently
expunged.
2022-06-19 16:09:55 +02:00
Oswald Buddenhagen
eab4a12a63 make sure that entries with deleted placeholders are pruned
we need to copy the S_DEL status bits when upgrading sync entries, so
that when a placeholder is flagged at the same time for both deletion
and upgrade, the entry pruning loop sees that the dummy message got
expunged.

note that we don't mask any bits, so the post-split entries may end up
with S_DEL for a zero UID - which doesn't matter.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0da273686f rework flag propagation during placeholder upgrade
don't implicitly propagate flags with upgrades. the user asked for
replacing the body, so do just that. if they also asked for flag
propagation, handle it like the case without upgrade as far as possible.
this makes async parallel flag propagation in the opposite direction
robust, while still being reasonably simple.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
3d90507a75 fix resuming propagation of flags on new messages
log the flags as soon as we know them, and commit them when the uid is
assigned (including during tuid matching).

amends 25b1c2b9.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a2880d740c de-noise msg_fetched() somewhat (prospectively)
assign temporary srec object instead of always spelling out the
indirection.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
be9625725c rework maxuid tracking yet again
re-introduce newmaxuid, but now it's not used at all until the state
is committed. this simplifies the new-message loop, esp. in view of a
soon significantly increased number of branches in it.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
2f4b71c56e knock out ReNew if there are no dummies
the operation requires loading the target side, which makes it somewhat
expensive in otherwise unidirectional syncs.

one could also knock out Flags and Delete if there are no living pairs,
but that wouldn't actually save much.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
1d433b4773 revamp post-journal-replay OPEN_* flags computation
instead of doing it instantly for every message, make a tally and
process it along with the OP_* flags. this allows us to print the
counts, and makes the handling more uniform.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
03d0ab0fbf fix dropping of pending message propagations when resuming
started message propagations will now complete upon resumption even if
the run was requested to be a no-op - whether that's a good thing can
be argued, but it's the least effort way to avoid that committing
discards transaction state.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
e6c6840651 make sure we fetch source flags when resuming --renew
... and the target is expunged.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a652043934 fix updating cached message flags in imap_set_msg_flags()
while this (currently) doesn't really matter (as all flag changes are
calculated before any are actually submitted), msg's flags should not
be updated before set_msg_flags() has actually succeeded.

as a side effect, this does away with the redundancy elimination and
pulling uid from msg (which were both unused since 19128f158).
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
698f9ff173 don't log right before saving state unless requested
only the journal replay autotest really needs it. in other cases it's
just wasting time and ssd life.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
d74af51fa1 rework interrupt-resume autotesting
instead of doing two runs for each journal entry, do one run for each
"write" operation, be it a journal entry or a writing driver call. this
saves runs between which no visible change occurred, which yields a 33%
improvement in runtime.

we now also exclude the final entry purge from the test, as it's really
kinda pointless, and we'd have to jump through additional hoops
(simulate an atomic commit of the state) to make it reliable in all
cases.

note that this also adds a few steps, which actually uncovered a bug in
the expunge sequencing.

amends efd72b85.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
58a69a5b63 split off -Ts from -Tj
there is actually no use case overlap between between the two (though
limiting the step count does imply keeping the journal, as we exit
before we could commit anyway).
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
13764a94b9 don't expunge until all flag propagation is complete
so far, we ensured that propagation _into_ a store completes before
expunging it, but not that propagation _from_ it completes. this way we
could end up expunging the source messages before the changes reached
the target, which could mess up resuming after an interruption.
2022-06-19 16:03:44 +02:00