Commit Graph

1269 Commits

Author SHA1 Message Date
Oswald Buddenhagen
8fbc4323f4 rate-limit progress counters
for simple local operations, they could easily dominate the cpu load.
also, over a slow remote tty, they could slow down things noticeably.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
1867a7c5ea increase timer resolution to milliseconds
seconds are too coarse for what we intend. technically, centisecs would
be sufficient (and more honest, given that we prefer coarse timers,
which have only tick precision), but that's a somewhat untypical unit.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
8566283c59 make expiration target side configurable
REFMAIL: 87k0fauw7q.fsf@wavexx.thregr.org
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
abb596709b add --dry-run mode
REFMAIL: 20211130142121.xon5oygrpdfj5s2t@fastmail.com
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5b9256f5dc add support for incomplete headers to copy_msg_convert()
in the spirit of being liberal about what we accept.

this completely re-structures the header processing loop, with the nice
side effect of eliminating the gotos.

REFMAIL: 87bkyzhoov.fsf@curie.anarc.at
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
ed92816fdb preserve stray CRs in copy_msg_convert()
strip only CRs followed by LFs. this has mostly theoretical value (but
will make a subsequent change simpler).
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
669f7dbd8f add test for copy_msg_convert() 2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
dbffebf560 factor out ADD_NL() inside copy_msg_convert() 2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
a32964c34e factor copy_msg_convert() out to own source file 2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
46d244533e return error strings from copy_msg_convert()
... instead of reporting conversion errors directly in copy_msg_convert().
this makes it easier to autotest properly.

this reverts bc15e571 in spirit, sidestepping the problem it tried to
solve instead.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
9b72e10320 make IMAP error messages less confusing
don't print the status, as the user doesn't really care whether it's
NO or BAD. more importantly, "NO LOGIN failed" is a rather misleading
thing to report.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
3aead33008 rework IMAP list parsing
replace the "DOM-like" model with a "streaming" model, i.e., the tokens
are now immediately processed by stateful callbacks as they are found.
this avoids plenty of allocations and copies (though not of the message
contents, yet).
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
f748bd45df tweak debug output for IMAP literals
- short literals are now printed even with only -Dn.
  this ensures that we see all parts of, for example, LIST responses.
  we use xprintf() for that, so we don't mess up the -Dn output in case
  the literal contains something unexpected (we don't care with -DN).
- omitted bytes are now indicated.
- missing trailing newlines are now "pretty-printed".
- long literals are now printed in chunks as we receive them. this
  makes the output messier, but it's necessary for an upcoming change,
  and, on the upside, we'll get better indication when the transfer
  gets stuck in the middle of a message.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
80831e50b9 add xprintf()
this introduces a "commit" callback to the xprintf kernel, to avoid
pointlessly assembling a temporary output string prior to printing it.

one could lift the buffer limitations by introducing a "segment"
callback instead, but that would slow down xvasprintf() due to the
higher callback rate, for no good reason.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
0079ec774a add escaping of non-printable & non-ASCII characters to xvasprintf() 2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
57173bd289 restructure xvasprintf() for uniformity
%\\s now supports length limitations, and \\ on other format specifiers
is now ignored (like .* already was on non-strings).
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
1a89f8a178 move imap_vprintf() to util.c and rename it to xvasprintf()
it's currently used only by IMAP, but it's logically low-level.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
859b7dd7f2 try to avoid extra syscalls when reading sockets
so far we shifted down the buffered data only when we ran out of space.
however, that may cause chopping up the actual socket read, which is a
bad trade-off with avoiding a moderate-sized memmove. so try to keep
enough space for the anticipated read size.

note that this ignores the zlib path, as that always does full-size
socket reads into z_buf, and saving inflate() calls seems pointless.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
ac3b5186b0 don't notify about socket reads that fall short of expectations
this will prospectively make the debug output less messy.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
96b1e52802 make socket_read()'s interface more like socket_read_line()'s
return a pointer into the internal buffer rather than copying into a
user-supplied one. this permits zero-copy in future use cases.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
6f15980cd9 socket_read_line(): reset buffer offset when no bytes are left
socket_read() already did that.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
69653aafeb beautify socket_read_line() somewhat
- use more appropriate name for socket object
- localize some variable declarations
- denoise the code by using more local variables
- don't pointlessly do stuff when we're failing anyway
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
bc3145617a add input length argument to map_name()
... and use it when parsing IMAP LIST responses.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5243c69863 require IMAP4rev1 capability
technically, we should have been doing that since the beginning, but as
there is IMAP4rev2 now, it might actually matter (in about a decade,
when servers start dropping backwards compat ...).
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
4a5c79993c optimize IMAP flag parsing
uppercase the reference strings and utilize already known string
lengths.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
6b9d4311d2 make IMAP parsing case insensitive
it's easy to overlook, but the spec demands this. we got away with it so
far, as everyone just uses UPPERCASE ... for the things we use so far.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
8d9c68f73a optimize string operations in IMAP parser
the string length is knowable in advance everywhere, so we can use that
for strdups and short-circuiting comparisons.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
c5e967f94d add support for LITERAL- extension
it's the same as LITERAL+, only with a strongly limited payload size.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5048521d79 make sure that all IMAP lists are parsed
... even if we ignore their contents, as they may (hypothetically)
contain literals, which are relevant for the stream syntax.

amends 6fd4e8de2.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
a07be5f175 improve error reporting from IMAP list parsing
so far, we'd print only a generic message - except in two cases, where
the generic error would be preceded by a specific one. now we always
print a single reasonably specific message.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
c7f50a3069 simplify parsing responses with multiple IMAP lists
the callbacks took the current parsing position only to pass it on to
the next list parse. so instead, store it in the state, and have a
separate function for continuation.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
ad8520b741 revamp sending of IMAP commands with literals
include the literal's length into the command itself, so error messages
become clearer. the commands are initially built for LITERAL+, and are
"edited down" right before sending if necessary.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
abd31aad61 simplify assembly of IMAP APPEND commands
treat the date string the same as the flag string.

amends eb1f1076.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
4ae0159132 handle tagged BAD [TOOBIG] server responses
some servers consider attempts to APPEND too big messages to be BAD
commands, rather than just NO-worthy ones. so just pretend that we did
in fact get a NO, and carry on.

this was tested with gmail. if we run into servers that don't send the
expected response code, we will need to relax the condition.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
5e5c7fb508 free IMAP command's payload prior to invoking its completion callback
otherwise the synching may livelock due to failing to schedule follow-up
commands, thinking the buffers are still full. of course, this is
relevant only for commands that failed early and thus didn't free the
payload right after submission already - which will be possible only
after the next commit.
2022-06-19 16:10:57 +02:00
Oswald Buddenhagen
1225f0b86b add ExpungeSolo option
REFMAIL: CAOgBZNonT0s0b_yPs2vx81Ru3cQp5M93xpZ3syWBW-2CNoX_ow@mail.gmail.com
2022-06-19 16:10:54 +02:00
Oswald Buddenhagen
95a22739fa don't UID EXPUNGE too many messages
we didn't check that the UIDs are adjacent, so we might have caught
not fetched deleted messages between two fetched messages below the
bulk load range.

checking adjacency of UIDs would make expunges in the bulk range (which
is likely to be full of holes) rather inefficient. so we use sequence
numbers instead.

this is admittedly a rather academical fix ...

amends 18225344.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
1631361f66 revamp handling of expunged messages
try to purge sync entries based on which messages are *actually*
expunged, rather than those that are *expected* to be expunged.

to save network bandwidth, the IMAP driver doesn't report all expunges,
so some entry purges would be delayed - potentially indefinitely, e.g.,
when only --pull-new --push is used, or Trash isn't used (nor
ExpungeSolo, prospectively). so keep a fallback path to avoid this.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
1a1ac25bc8 track IMAP message sequence numbers (and therefore expunges) 2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
df4e6383f5 track number of messages in IMAP stores
... rather than calculating it on the fly. for efficiency.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
6fe7172901 moan about trashing in non-UIDPLUS boxes
... as that's inherently racy, as the manual says.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
edbf9a35da use UID EXPUNGE also when trashing remotely
amends 2f0fbcd3.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
04c7126ce9 add autotest for syncing non-UIDPLUS stores 2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
767a318eea add new sync operation 'Old'
this is essentially the same as 'New', but for previously seen messages,
such as those that would have been instantly expunged (because they were
marked as deleted), those that we failed to store for some reason, and
already expired ones that are now flagged.

REFMAIL: CAOgBZNonT0s0b_yPs2vx81Ru3cQp5M93xpZ3syWBW-2CNoX_ow@mail.gmail.com
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
a8e145e589 split OPEN_PAIRED from OPEN_OLD, and rename OPEN_OLD_IDS accordingly
this emphasizes the non-complementarity to OPEN_NEW.
the "real" OPEN_OLD is currently used only for trashing.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
d77d67c948 concentrate calculation of minimal loaded UID in load_box()
so it's in one place with the maximal one, sans the one-sided
preparation of the bulk range + exception list.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
e98aed87f0 silently reinterpret --full
change it from --{push,pull} to --{new,gone,flags,upgrade}.
that alone doesn't change anything; combining it with any other option
would cause a complaint anyway.

this enables us to introduce --{push,pull}-full, which will matter when
--full doesn't actually include all type flags any more, as we'll then
be able to combine it with the extra type flag(s). in the same vein, we
now advertize "Sync Full" instead of "Sync All" (both continue to be
recognized).
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
58564e4f76 rename Delete to Gone
this is more symmetrical with New, and results in some less dodgy
grammar. it also avoids confusion with the \Deleted flag.

fwiw, the pedantically correct name would be Expunges, but that's
confusingly close to the target-side expunge options. also, it's longer.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
6308a7f41b rename ReNew to Upgrade
that's what it really has been for a while now.
2022-06-19 16:10:01 +02:00
Oswald Buddenhagen
8f39d06015 fix mixing MaxMessages with MaxSize
this is actually a useful combination for resource-constrained devices.
2022-06-19 16:10:01 +02:00