Commit Graph

1044 Commits

Author SHA1 Message Date
Oswald Buddenhagen
950ebe833d add debug messages about effective requested operations 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
3091e2fe5a add debug pretty-printing for OPEN_* flags as well 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
17db5de0ca add debug pretty-printing for sync record status flags as well 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
c902f69c6f format message flags in the sync debug output
now that we can do it cheaply, make copious use of it.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a49017f481 streamline pretty-printing of message flags
wrap make_flags() into fmt_flags() which returns a (struct-wrapped)
string, so the calls can be inlined into the printf statements, without
reserving buffers.

we locally force optimization, so copy elision is always done, as debug
builds would otherwise suffer a somewhat unreasonable performance hit.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a5dc1baedf denoise usages of INIT_IMAP_CMD()
move the variable declarations into the macro, taking advantage of the
fact that mixing them with code is ok since C99.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
f4ed8b27f6 fold superfluous INIT_IMAP_CMD_X() macro
it's been identical to INIT_IMAP_CMD() since c3d91ae1.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
f5d234ffa1 simplify an initialization in proxy driver code generator
the 'checked' hash entry doesn't exist yet at that point, so it's always
undefined.

amends 4423a932.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
5c44732fd9 support C99 comments in proxy driver code generator 2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
7f1c667910 cache drivers' ability to handle crlf in mails
it's a cheap call, but it clutters up log files.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0f1b2b646b remove questionable optimization from case-insensitive string comparison
we optimized the case where the string would be equal even without
upper-casing, but for the much more common case where the strings differ
even after upper-casing, this was just an additional conditional.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
61b08880c8 improve tracking of maxxfuid
do it when the corresponding expiration events are logged/replayed.
that makes it unnecessary to log it separately just to make the
autotest happy.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
0f6362f2e2 fix tracking of sync record count through placeholder upgrades
for now, this is just pedantry, as it merely affects the size of the
already oversized { uid => srec } flathash after journal replay.

amends 70bad661.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
69118d25ec optimize { uid => srec } flathash
the sync records contain the uid, so storing it explicitly doubles the
hash's size for no good reason.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
8f4af5f78f make use of finduid[] after journal replay
we used to pass all initially loaded messages to match_tuids(), which
could be quite some when syncing old messages. as lost TUIDs result in
O(n^2) behavior, this could have a serious performance impact.
2022-06-19 16:03:44 +02:00
Oswald Buddenhagen
a1a3313ed4 print human-readable summary at exit
this is meant primarily for use with a redirected output, where no
progress counters are shown. but it would be weird not to have it with
tty output as well, so it replaces the counters after completion.

REFMAIL: 87bl2cgt6u.fsf@curie.anarc.at
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
0f7c231cc2 fix bogus complaints about missing Store/Account references
... when the reference is simply invalid (we already get a complaint
about that).
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
bf59636f0f don't stop validating Channel config after first error 2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
c986f80bb0 append even broken Channel configs to list
otherwise perfectly fine Group defs will complain about invalid refs.

we already do that with Store configs.
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
2cbf8a68cf abort if any invalid Channels/Groups have been specified
this is more in line with expectations, and avoids a silly "No channel
specified." error message.
2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
44ad8f0361 handle mixing simple and compound sync options more explicitly 2022-06-19 16:03:38 +02:00
Oswald Buddenhagen
e70a20477c complain about --noop/--no-* conflicts
REFMAIL: 20211130124527.t3u7s4fyy57gmfzc@fastmail.com
2022-06-19 16:03:34 +02:00
Oswald Buddenhagen
be6e07c5c9 tone down complaints in merge_ops()
speak more accurately of redundancy, not conflict. these are fatal
errors at all only because the user likely meant something else than
they typed, so we force them to think again.
2022-06-19 16:03:34 +02:00
Oswald Buddenhagen
d7e3ae4b74 report location of overlapping operations 2022-06-19 16:03:34 +02:00
Oswald Buddenhagen
09f08e4974 stop parsing config lines after invalid keyword
otherwise we may pointlessly complain about excess tokens.
2022-06-19 16:03:34 +02:00
Oswald Buddenhagen
5d5e07eb63 add --list-stores mode
this is useful for verifying the store configuration, and finding the
right mailbox names.

REFMAIL: YaZC3XUTWjyfjgn+@ugly
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
d5a5da9475 factor out main_sync.c & main_p.h from main.c
this moves the channel iteration & synchronization code from main(),
with all its dependencies.

then it is "re-threaded" to be more directly driven by the driver
callbacks (like sync_boxes() is), rather than being a weird state
machine.

while the code is moved, localize many variables, and use an enum
instead of #defines for the states.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
01329bdf82 exclude placeholders from trashing
it makes no sense to trash the placeholders, and in fact the common
case is that they are deleted due to being replaced by the full
message.

a separate S_PURGED state needed to be added, as S_PURGE needs to be
reset after setting F_DELETED (so the operation doesn't count as still
pending after journal replay), yet trashing needs an indicator. logging
is now done via a separate command, as piggy-backing it on flag updates
just makes things less legible for no benefit.

this is mostly academical, as trashing being done on the side where
placeholders reside is rather unlikely.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
8363dbf2d1 add autotest for message trashing 2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
4b0c5a0cd5 do not exclude oversized messages from remote trashing
... as otherwise these messages would be just lost.

the assumption is that opposite-side trashing is used only for locally
generated messages whose size we control. it's also more consistent with
same-side trashing, where even oversized messages would be trashed.

the exclusion was broken anyway, as we failed to query the size of old
messages, particularly after 70bad661.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
d92c62022a treat not-really-propagated messages as trashable
messages for which only a placeholder was propagated should be treated
as not propagated, as otherwise the actual contents will be lost when
only not propagated messages are trashed.

amends 70bad661.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
f7650993b7 re-nest conditions for trashing messages
this makes the logic easier to follow and document in place.

also add some debug statements.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
91d19cceac make journal entry argument processing saner
the mega-if was unreadable and caused lots of churn.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
608c724add assert sizes of smaller-than-int bit fields in structures 2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
b3155a8bcb merge wstate back into status
this optimizes space usage, prospectively (we'd have to extend wstate
soon otherwise).

this partially reverts 4ffe1496.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
6a78e2c5f6 automate enumeration of power-of-two #defines
this is a lot more legible, and makes it possible to insert values in
the middle without churn.

i didn't find a way to do this with the pre-processor, so we now have
another code generator.

we now use the $< make variable, which requires gmake on netbsd < 9.0,
and possibly other systems with an ancient make.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
05122b678d print version and command line args in debug mode
that makes it unnecessary to ask help-seeking users for it separately.
2022-06-19 16:03:25 +02:00
Oswald Buddenhagen
c1eb3566b1 split Verbosity off from DFlags
this clearly documents the permitted states.
2022-06-19 16:03:21 +02:00
Oswald Buddenhagen
d3f118be79 re-interpret relative local paths in config file
this makes config+data file "sets" relocatable, which is useful for
testing.

this is technically a gratuitous backwards incompatible behavior
change, but to the degree that anyone uses relative paths at all, they
almost certainly rely on PWD being set up such that they won't see a
difference.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
cf13630a00 make default config & state locations follow the XDG basedir spec
there are fallbacks to the old locations for compatibility.

the new locations use "isync" instead of "mbsync", which is preparation
for renaming the executable back in v1.6.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
8bb679ea06 create sync state parent directories recursively
this may matter with SyncState being for example ~/.local/state/isync/.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
1ba0cd7b96 factor out sync_state.c & sync_p.h from sync.c
while moving the code, localize some variables, and use C99 comments.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
4b49848288 rearrange Makefile somewhat for consistency 2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
f2450cc4b8 centralize message flag pretty-printing somewhat
the flag values themselves are central, so we can centralize their most
common representation (Maildir's) just as well.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
d789f0c1ce make some string buffers non-static
this was clearly bogus; these are short-lived local variables.

amends 4cc5ad5a.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
4eff48c54e replace 1-t => t^1
this is consistent with what we already did for in-place operations.
arguably, it's also a bit more intuitive.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
08a375ea07 rename nfcalloc() => nfzalloc()
the signature is like that of zalloc() (as found in the linux kernel;
not to be confused with zone allocators, etc.), not like that of
calloc().
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
603e740b63 move expand_strdup() to config.c
it's not really a generic function.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
7d02d6c1fe move checked FILE functions to util.c
while they are used only in sync.c, they are conceptually low-level.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
6f023376a1 turn debug() functions into macros
this makes calling them with more expensive arguments efficient without
wrapping them into additional conditionals.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
1a0255c566 centralize some #includes
these are used (almost) everywhere, so put them in common.h.
2022-06-19 16:02:03 +02:00
Oswald Buddenhagen
98f4fd4586 shuffle around global variables
the new organization clarifies the ownership, making things more
self-contained. this will potentially help with unit testing.
2022-06-19 16:01:58 +02:00
Oswald Buddenhagen
0f2220634d coding style: add "superfluous" braces
- wrap flow-controlled statements that contain blocks into blocks
  themselves
- wrap bodies of do-while()s into blocks
- use braces on 'else' symmetrically (this obviously has a cascading
  effect, so this patch touches lots of lines)
- attach braces

unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
156e9c5058 comment updates 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
6061de0ba6 autotest: use more legible format for test data
instead of specifying the mailboxes and sync state verbatim, use a
format which deals only with "subjects" (but no UIDs), and specifies the
whole state for each subject on a single line (exceptions prove the
rule).

the dumpers don't try to re-create the abstraction, as that's deemed
to be an unreasonable effort.

while rewriting most of the test data anyway, move it to the bottom of
the file, which is a more natural location for it.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
db66c4d746 autotest: remove show()
this somewhat crude way of generating test data will be obsolete soon.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
3040625a62 autotest: make state comparison more elaborate
don't abort the comparison if continuing makes sense, and try to be more
specific about the problems.

we give up if messages are excessive/missing or the subject is wrong,
as that touches upon the rather complex problem of diff optimization.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
7ce8c09145 autotest: support tests that don't modify the state file
a test run may (legitimately or not) modify only the mailboxes, or even
nothing at all. do something sensible in this case.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
96ee50d6ba autotest: transform ck*() => cmp*()
this avoids the ugly and error-prone repeated reading of the state
after a failure.

cmpbox() had to be made non-destructive on the box state.

readchan() had to be created.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
325551ce79 autotest: use ints where we mean ints
... instead of using strings. perl allows being sloppy, but it doesn't
really help understanding the code.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
fc0ad9eb65 autotest: use more appropriate data formats
parse the test data into hierarchical structures instead of using it in
its raw form. this is semantically cleaner and allows us to change the
input format more easily.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
697f35fd97 autotest: factor out readstate() from showstate() and ckstate() 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
e0c1a83fc1 autotest: make more use of readfile() 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
640b2a6649 autotest: factor out mkstate()
for clarity and structural consistency.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
9f9a2af959 autotest: let mkchan() take an atomic channel state
this is consistent with ckchan() and printchan().
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
7f38c5dc53 autotest: add support for strace-ing
works just like valgrind-ing.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
22a1df73e4 autotest: add support for starting at a specific test
... which is often useful for continuation after test data fixes.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
03a38e48d3 autotest: use sub-string matching for test selection
typing the full names is rather tedious when they get long.
also, using sub-strings, we can match multiple tests with one argument.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
27f0c47010 autotest: be more verbose about internal errors & warnings 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
929aa3281b autotest: don't die on failure to read state file
this can happen as part of a "regular" test failure.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
5d7f2c7461 autotest: remove excessive close()s from showstate()'s error path 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
254d2be9f4 remove pointless "lost track of ... messages" warnings
it doesn't really add any value when resuming after an interruption, as
incomplete propagations are perfectly expected, and are recovered from.
but it does make sense in the non-UIDPLUS path, where not finding just
stored messages indicates a problem.
so move the reporting from match_tuids() to msgs_found_new(), and remove
a redundant warning from the latter.

amends 6577bf3e.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
2b797fac61 delay TUID assignment less
we can't delay TUID assignment until after starting propagation if we
want to provide both safety and performance.

amends a0961d65.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
04e225c7ce make some maildir (error) messages more helpful
include the affected path.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
8e83649c33 slightly improve maildir rescan debugging 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
b9f0162642 make free_*_messages() loops less obfuscated
notably, free_maildir_messages() had a dead assignment.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
35375df63f don't put IMAP stores in SST_BAD state into the unowned list
nothing would ever recycle them, so they'd just waste space. so cancel
them right away.

amends 9d22641.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
ae3a61b668 clarify / micro-optimize cancel_sync() 2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
75113ef796 get rid of some redundant casts
amends c3d91ae1, 1b67c499, 9c86ec34, and 83ebe902+1039ee25.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
11352708b8 fix roff abuse in mdconvert man page
this one was missed in a33e4475.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
9356300952 convert licensing to SPDX
while at it, add/fix some licenses/copyrights/comments:
- it makes no sense to have a GPL exception in scripts
- ted did not contribute to the man page
- tst_timers is not part of the mbsync executable
- explicitly put the build system under GPL and add copyrights
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
72ba7ef125 bump copyrights
it's legally irrelevant, but whatever.
2022-06-19 16:01:29 +02:00
Oswald Buddenhagen
043a8b5835 documentation tweaks
manual:
- explain what "rename on move" really means
- reword "remote" to "opposite" to make it less confusing
  (possibly renaming TrashRemoteNew left as an exercise for later)
- mention example mbsyncrc
- consistently capitalize Store/Channel/Group where they refer to the
  respective configuration entities
- emphasize that SyncState may need a trailing slash (as we do for Path)
- fix missing mention of global usage/default for some options
example mbsyncrc:
- add big fat note that empty lines matter
- stop demoing deprecated options
- point out that CertificateFile is optional

REFMAIL: 877dd11jb3.fsf@angela.anarc.at
2022-06-19 16:01:22 +02:00
Oswald Buddenhagen
16db3498b3 fix .gitignores
- src/tmp/ is actually a symlink (and thus not a dir)
- autoconf now generates configure~
- the coverity result archive was missed so far
2022-06-19 16:01:22 +02:00
Oswald Buddenhagen
7a4a887b3c sort lists of trashed messages after journal replay
the messages are trashed in mailbox (and thus UID) order, and in
practice we expect the operations to complete in order. however, if
older messages need to be trashed after a journal replay, and we get
interrupted again, the next replay would produce an unsorted array,
and thus break the binary search.

amends 2bba9b9.
2022-06-19 16:01:22 +02:00
Oswald Buddenhagen
c1feba585a don't clobber message status when upgrading placeholders
we'd reset the possibly set M_RECENT flag, which would lead to
pointless maildir rescans.

amends 70bad661.
2022-06-19 16:01:22 +02:00
Oswald Buddenhagen
2e17f427a9 fix severing of msg -> srec links upon maildir rescan
amends 9c86ec34 (the relevant line was arguably misplaced there, and
should have been in 2277ecef or whereabouts already).
2022-06-19 16:01:22 +02:00
Oswald Buddenhagen
f74b4e0d11 fix --debug-crash suppressing the progress display
there isn't really a reason for that; DEBUG_CRASH is quite unlike the
other DEBUG_ flags.

note that the DEBUG_*_ALL flags are not checked, because they always
come with their corresponding less verbose flag anyway.
2022-06-19 16:01:19 +02:00
Oswald Buddenhagen
c9b52f5aec fix maildir driver's debug flag
copy-pasto; it certainly wasn't meant to respond to --debug-sync.
the problem was barely noticeable, as the maildir driver's only debugs
are in the rarely triggered rescan path, apart from the flags usually
being used en bloc anyway.

amends 0e1f8f9a.
2022-06-19 16:00:15 +02:00
Oswald Buddenhagen
9c2cd0abd8 plug memory leaks when OPEN_OLD_IDS is used with Maildir
amends 77acc268.
2022-06-19 16:00:15 +02:00
Oswald Buddenhagen
259132b7e7 plug memory leaks in imap_{store,trash}_msg() error paths 2022-06-19 16:00:15 +02:00
Oswald Buddenhagen
4c2fb74207 fix storing messages on non-UIDPLUS servers
the fetch mode needs to be set for messages.

amends 42f165ec.
2022-06-19 16:00:15 +02:00
Oswald Buddenhagen
ee9fd2f5c7 workaround iCloud IMAP bug
thanks to Sabahattin Gucukoglu <listsebby@me.com> for the thorough
investigation.

REFMAIL: 29C5E84D-5FE7-47BB-9A14-2EC34D3921C5@me.com
2022-06-19 16:00:09 +02:00
Oswald Buddenhagen
d6b9a139e4 re-issue IMAP CAPABILITY after authentication
... if the server didn't include a corresponding response code by
itself. required for the sorry excuse of an imap server that ms
exchange is.
2022-06-19 16:00:09 +02:00
Oswald Buddenhagen
b6c36624f0 work around "unexpected EOF" error messages at end of SSL connections
gmail apparently doesn't send a close notification (SSL_shutdown())
before closing the TCP socket.
2022-06-19 16:00:09 +02:00
Oswald Buddenhagen
6b22c837f6 overflow-check ranges[] in imap_set_range()
amends 3d64f167.
2022-06-19 16:00:09 +02:00
Oswald Buddenhagen
87c2ac1cc9 reserve enough UID ranges in imap_load_box()
in certain configurations, under very unlikely conditions (which are
practically impossible to control remotely), we'd overflow ranges[].
in a typical gcc build, the values (which are also practically
impossible to control remotely) would be written at the end of buf[],
which would be rather harmless, as only a tiny part of buf is used
subsequently. so i'm not classifying this as a security issue.

amends 77acc268.
2022-06-19 16:00:09 +02:00
Oswald Buddenhagen
e686f88318 don't complain about concurrent flagging as deleted
the result of propagating a deletion is flagging as deleted, so shut up
if the only remote change is exactly that.
2021-12-03 11:56:16 +01:00
Oswald Buddenhagen
51673214ab fix read beyond end of input in copy_msg_convert()
the input isn't necessarily null-terminated (it currently is for imap,
but not for maildir), so if the message ended somewhere within the
header field name, we'd read beyond its end, which theoretically could
cause a crash. no other adverse effects could result, as we'd stop
processing such a broken message right afterwards.

amends 70bad661.
2021-12-03 11:46:33 +01:00
Oswald Buddenhagen
127003ee37 reject unreasonably long mailbox names from IMAP LIST
this wasn't really a security problem, as the name mapping we actually
do does not change the string length, and the iteration was already
safe after the literal length fix, but it's still better to catch weird
input.
2021-12-01 10:07:40 +01:00
Oswald Buddenhagen
92921b1d3b reject messages that grow too large due to conversion
that shouldn't really be a problem, as we have 2GB of headroom, and most
growth would happen when sending an all-newlines message from maildir to
imap (due to CR additions), which is mostly non-critical. but better
safe than sorry.
2021-12-01 10:07:40 +01:00
Oswald Buddenhagen
bc15e571b6 report conversion errors directly in copy_msg_convert()
that makes it easier to report various conditions without introducing
separate error codes.
2021-12-01 10:07:40 +01:00
Oswald Buddenhagen
ba13362a52 deal with oversized messages in maildirs
don't try to read messages > 2G, as that will only lead to trouble down
the line.

this wouldn't have worked on linux anyway (we read in one chunk, and
that is limited to (2^31 - 2^12) on all architectures), but on
platforms were big reads work, this was a security problem if one
synchronized other users' maildirs.

as a minor fix on the side, we now also clip the reported message size,
so MaxSize works for excessively big messages.
2021-12-01 10:07:40 +01:00
Oswald Buddenhagen
463272eab8 CVE-2021-3657: reject excessively large IMAP literals
we didn't limit the 32-bit size of literals so far, which, given that we
use int-sized lengths & offsets, permitted all kinds of buffer
overflows. malicious/compromised servers may have been able to exploit
this. actual email senders would be constrained by size limits for
delivered mails, and to cause more than a crash they'd have to predict
the exact size of the final message.

we now limit to 2GB, which, given that we use unsigned ints since
e2d3b4d55 (v1.4.0), gives the handlers downstream plenty of headroom.

an alternative would have been using 64-bit offsets, but this seems like
major overkill, even if IMAP4rev2 recently mandated it (we talk only
IMAP4rev1, so we can ignore it).
2021-12-01 10:07:24 +01:00
Oswald Buddenhagen
87065c12b4 CVE-2021-44143: don't overflow heap on messages without headers
when a broken/compromised/malicious server gives us a message that
starts with an empty line, we'd enter the path for inserting a pristine
placeholder subject, for which we unfortunately didn't actually allocate
space (unless MaxSize is in use and the message exceeds it).

note that this cannot be triggered by merely receiving a crafted mail
with no headers (yes, it's actually possible to send such a thing), as
the delivery of mails adds plenty of headers.

amends 70bad661.
2021-11-25 16:14:32 +01:00
Oswald Buddenhagen
7979782676 limit maildir nesting depth
this is a cheap way to catch symlink loops. 10 seems like a reasonable
limit, as it's unlikely that anyone would be able to actually work with
such a deeply nested mailbox tree.

fixes debian bug #990117.
2021-07-29 13:14:18 +02:00
Oswald Buddenhagen
a846ab054d enable embedding arbitrarily long strings into IMAP commands
the AUTHENTICATE command may get insanely long for GSSAPI when SASL-IR
is available. instead of growing the buffers each time someone hits the
limit (as done in f7cec306), remove the limitation altogether.

imap_vprintf() still contains a fixed-size buffer which could overflow
when really long strings (e.g., mailbox names) need to be quoted. this
seems very unlikely, so we'll deal with it if someone actually hits it.

REFMAIL: 87sg1qxdye.fsf@cern.ch
2021-06-11 18:24:00 +02:00
Oswald Buddenhagen
444601a1e0 Merge branch '1.3' into 1.4
Conflicts:
	configure.ac
	src/drv_imap.c
2021-06-03 11:04:56 +02:00
Oswald Buddenhagen
589d2ed428 CVE-2021-3578: fix handling of unexpected APPENDUID response code
if the code was sent in response to anything but a STORE, we'd overwrite
a data pointer in one of our imap_cmd subclasses, an allocator data
structure, or the start of the next allocation, with an int that was
completely under the server's control. it's plausible that this could be
exploited for remote code execution.

to avoid this, we could ensure that the object is of the right type
prior to casting, by using a new flag in the parameter block. but it's
easier to just dispose of the out_uid field altogether and reuse the uid
field that is present in the parameter block anyway, but was used only
for FETCH commands so far.

this problem was found by Lukas Braun <koomi@moshbit.net> using a
fuzzer.
2021-06-03 11:02:23 +02:00
Oswald Buddenhagen
a86e6f8c7c don't crash on malformed CAPABILITY responses
amends 95a83c822.

this problem was found by Lukas Braun <koomi@moshbit.net> using a
fuzzer.
2021-06-02 15:51:23 +02:00
Oswald Buddenhagen
d8feb67dae tolerate INBOX mis-casing in Path
while it's technically reasonable to expect the user to match the
server's casing of INBOX if they set Path, this might come as a
surprise to those who know that the IMAP INBOX is case-insensitive.
so tolerate any casing instead. as a minor side effect, we'd now even be
able to deal with a server using different casing in NAMESPACE and LIST.
2021-03-19 18:21:34 +01:00
Oswald Buddenhagen
4b185e35fe Merge branch '1.3' into 1.4
Conflicts:
	configure.ac
	src/drv_imap.c
2021-02-21 21:26:54 +01:00
Oswald Buddenhagen
594e60bd74 make UIDVALIDITY recovery more strict about vanished messages
in particular, this covers the case of a mailbox being replaced with an
empty new one, which would subsequently lead to the opposite end being
emptied as well, which would typically be undesired.

also add plenty of comments.
2021-02-21 21:11:58 +01:00
Oswald Buddenhagen
6796e041ae improve error messages about irrecoverably changed UIDVALIDITY
don't print the actual values, which are meaningless technicalities
to the average user, and can be obtained separately for debugging if
really necessary.
also, fix the omission of the affected mailboxes from one of the
messages.
2021-02-21 21:11:58 +01:00
Oswald Buddenhagen
fe5d59f8e3 CVE-2021-20247: reject funny mailbox names from IMAP LIST/LSUB
in particular, '..' in the name could be used to escape the Path/Inbox
of a Maildir Store, which could be exploited for stealing or deleting
data, or staging a (mild) DoS attack.
2021-02-21 20:40:22 +01:00
Oswald Buddenhagen
95a83c8220 be more tolerant of formally malformed response codes
fastmail sends flags containing ']' in PERMANENTFLAGS, which is formally
illegal. however, if we parse the embedded list before looking for the
response code's closing ']', things work out fine.

as a side effect we won't complain about similarly or completely
malformed response codes we don't recognize at all, which may or may not
be considered an improvement ...
2021-02-14 23:47:14 +01:00
Oswald Buddenhagen
8c86f34bf0 fix bogus continuation of IMAP list parsing
on error, parse_imap_list() needs to reset the nesting level in the
state, as imap_socket_read() uses that as an indicator whether list
parsing is ongoing.
2021-02-14 23:47:14 +01:00
Oswald Buddenhagen
32392adbe3 accept unsolicited FETCH responses (without payload) after all
while the spec says that the server SHOULD not send FETCH responses
about STORE FLAGS when .SILENT is used, at least gmail and fastmail seem
to do it nonetheless. also, in case of concurrent flag updates on the
affected messages such responses can be legitimately sent.

in earlier versions of mbsync this would lead to duplicate messages
piling up in the store, though that would pose no problem at that point.
2021-02-14 23:47:14 +01:00
Nihal Jere
7a0ea1f15c use correct <poll.h> header
In POSIX, poll() should be accessible using <poll.h>, although most
implementations keep <sys/poll.h> to avoid breakage. This fixes some
warnings when building on musl.
2021-02-08 17:26:11 +01:00
Oswald Buddenhagen
062706fcbf Merge branch '1.3'
Conflicts:
	configure.ac
	src/drv_imap.c
2021-02-03 15:53:05 +01:00
Oswald Buddenhagen
c8b73acad2 unbreak handling of 'INBOX.' NAMESPACE again
INBOX matching must not prevent prefix (namespace) stripping, as INBOX
may be the namespace.

amends 04fc586e7.

REFMAIL: 186391612191752@vla1-ea7e194e8506.qloud-c.yandex.net
2021-02-03 14:43:11 +01:00
Anton Khirnov
fc300fd811 Set authentication id for the SASL EXTERNAL mechanism
The SASL library will refuse to use the EXTERNAL module when no auth id
is set a priori.

Tested to work with Dovecot, using TLS client certificates for
authentication.
2021-01-05 19:50:21 +01:00
Oswald Buddenhagen
e67cf01eb8 improve SASL error messages
provide context, and remove the redundant numeric codes.
2021-01-05 19:46:29 +01:00
Oswald Buddenhagen
c2e6e962b5 tune SASL-related comments
- add explanations to the callbacks
- remove bogus comment - EXTERNAL can be in fact missing (when no
  authentication id is set)
2021-01-05 19:46:29 +01:00
Oswald Buddenhagen
e295f483d9 save errno in sys_error()
the print functions prior to perror() might otherwise clobber it.
2021-01-01 14:46:31 +01:00
Oswald Buddenhagen
6e56f39fa9 autotest: remove unused boxname parameter from printbox() 2020-12-19 13:22:29 +01:00
Oswald Buddenhagen
9fbf5c2f6c autotest: pass containers by reference
this makes function prototypes a lot more useful for parameter checking.
2020-12-19 13:22:29 +01:00
Oswald Buddenhagen
4423a932f3 add forced async mode to proxy driver
to test async operation of the syncing core while using the synchronous
maildir driver, we add a mode to the proxy driver where it queues
callback invocations to the next main loop iteration.
2020-12-19 13:22:29 +01:00
Oswald Buddenhagen
be657530ee localize a variable more appropriately 2020-12-17 22:18:10 +01:00
Oswald Buddenhagen
30af61fb24 consolidate testing options behind common switch
don't pollute the namespace with random uppercase switches. instead,
have a new -T switch with suboptions, just like -D.
2020-12-17 22:18:10 +01:00
Oswald Buddenhagen
c3d91ae1e8 introduce new inheritance model based on C11 anonymous structs
the struct declarations got uglier, but their usage requires a lot fewer
explicit references to the parent struct (though some are added where
using the derived struct is more practical now).

we also use something i'd term "covariant members": derivatives of
store_t also reference derivatives of store_conf_t, etc., which
drastically cuts down the number of casts.
fwiw, to achieve this with "proper" inheritance in C++, we'd use
covariant getter functions which hide the still existing casts.

C11 is almost a decade old now, and compilers supported that feature
even longer than that, so i don't expect this to be a problem.
2020-12-17 22:18:10 +01:00
Oswald Buddenhagen
bf66f210bd add some error checking to proxy template processor
debugging is a lot easier when the unconsumed (and therefore likely
mistyped) replacements are complained about.
2020-12-17 22:18:10 +01:00
Oswald Buddenhagen
cd6f18fd2b handle indentation in proxy driver template code more flexibly
use the indentation of the placeholder, not the replacement.
this doesn't matter right now, as all placeholders are indented by one
step, but that will change soon.

the indent function cannot be inlined into the substitution, as for some
reason ^ then matches the end of the string, not the embedded line
starts (with perl v5.32). also, $1 needs to go into a temporary anyway.
2020-12-17 22:17:11 +01:00
Oswald Buddenhagen
ba7b634186 make FALLTHROUGH work with qtcreator's code model
the code model inspector claims that __GNUC__ is 10, but the #if works
only with >= 4, which is plain wrong. so just handle clang explicitly.
2020-12-16 13:42:40 +01:00
Oswald Buddenhagen
5b4766fbe4 improve docu for {Pass,User}Cmd 2020-12-12 14:56:27 +01:00
Oswald Buddenhagen
dec4b36595 improve wording in man page 2020-10-05 13:50:23 +02:00
Oswald Buddenhagen
a9ce7be962 streamline init of type & name in imap_parse_store() 2020-10-05 13:15:28 +02:00
Oswald Buddenhagen
09341c10c5 make complaints about unrecognized keywords more verbose
tell the user in what section the keyword appeared, as that may help
spotting mistakes like stray empty lines.
2020-10-05 13:14:48 +02:00
Oswald Buddenhagen
217764bd35 complain about global options following sections
while harmless for most options, such usage is counter to the
documentation, and actually breaks CopyArrivalDate, MaxMessages, and
ExpireUnread.
2020-10-05 12:41:35 +02:00
Oswald Buddenhagen
ee39e684aa make exit from parsing Group sections less convoluted
this is a de-optimization, but it makes the code consistent with the
other sections (which do not use the shortcut due to having to
post-process the data or being encapsulated by a function call).
2020-10-05 12:31:14 +02:00
Oswald Buddenhagen
6463a72f12 remove the -cT option
it was another vestige from the compat wrapper.

amends cbac8aa75.
2020-10-05 11:56:30 +02:00
Oswald Buddenhagen
23513564df improve error handling in post-STORE UIDNEXT fallback
that's mostly hypothetical, but let's not make assumptions.

this also adds EXPUNGE response handling to make total_msgs reliable. in
principle, this affects the post-SELECT UIDNEXT fallback as well, but
there the racing window is so short that this barely improves anything.

amends 94022a67.
2020-08-24 12:51:47 +02:00
Oswald Buddenhagen
42f165ecf7 fix UIDNEXT query vs. concurrent imap_fetch_msg()
the uidnext query following message stores can be interleaved with
message fetches. that means that we cannot rely on the 1st command in
flight being that query. but instead of iterating over all commands in
flight, move the uidnext query flag to imap_store (and make sure to
check for the presence of a message body before testing it) - this
avoids the loop and an extra byte in every command.

this also makes it clear that the query is mutually exclusive with
loading messages (the untagged responses are not distinguishable).
2020-08-24 12:51:47 +02:00
Oswald Buddenhagen
f099141e42 make item tracking in parse_fetch_rsp() more uniform
amends 67ea5bea7 & a5a8783ea.
2020-08-24 12:51:42 +02:00
Oswald Buddenhagen
ec47c90554 delay allocation of msgdata.msgid field
this allows us to simplify the exit path of parse_fetch_rsp().
2020-08-05 17:59:28 +02:00
Oswald Buddenhagen
b37d6b1c00 fix invalid free() in error path
the tuid isn't actually allocated - it's a pointer into the raw data.

amends a5a8783e.
2020-08-05 17:36:35 +02:00
Oswald Buddenhagen
c69718baab remove redundant zero initializations
we already use calloc().

amends 130664b6.
2020-08-05 17:29:58 +02:00
Oswald Buddenhagen
b148fd9e44 de-duplicate exit paths of imap_alloc_store() 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
c83330ffe8 don't unnecessarily re-initialize some members of imap_store
... when recycling server connections.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
8457225a50 use more appropriate return value in driver_t::select_box()
don't say DRV_CANCELED when it's really DRV_STORE_BAD, as apart from
being just wrong, it lead to the confusing effect of canceling a store
as the result of a supposed cancellation of the same store.
2020-08-04 17:16:03 +02:00