Commit Graph

848 Commits

Author SHA1 Message Date
Oswald Buddenhagen
c69718baab remove redundant zero initializations
we already use calloc().

amends 130664b6.
2020-08-05 17:29:58 +02:00
Oswald Buddenhagen
b148fd9e44 de-duplicate exit paths of imap_alloc_store() 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
c83330ffe8 don't unnecessarily re-initialize some members of imap_store
... when recycling server connections.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
8457225a50 use more appropriate return value in driver_t::select_box()
don't say DRV_CANCELED when it's really DRV_STORE_BAD, as apart from
being just wrong, it lead to the confusing effect of canceling a store
as the result of a supposed cancellation of the same store.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
481c12a8b3 handle CertificateFile more cleanly
properly distribute the certificates between the SSL context's trust
store and our host cert list.

as a drive-by, clean up some nasty type casts at the cost of including
a second OpenSSL header into socket.h.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
cfaa4848dd actually implement imap_commit_cmds()
delay reporting success of STORE FLAGS until a subsequent CHECK
succeeds.

this fixes (inverse flag change propagation) and (deletes not being
propagated) after an interruption due to prematurely logged flag
updates.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
70bad66129 create placeholders for messages over MaxSize
this is vastly more useful than just omitting the messages with no
indication at all.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
68a412115a don't rewrite state gratuitously
delay the creation of the new state and journal until there is actually
something interesting to write. this saves some cpu cycles and prolongs
ssd life a whee bit.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
0e5046e14a add/fix/de-duplicate comments 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
fd7b5659ab de-duplicate updating of uid in sync records 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
e9efc49b6c do away with newmaxuid
now that expiration order is determined by a single loop ordered by
far-side UIDs, it is no longer necessary to accurately track the highest
seen UID.

as a side effect, this fixes a problem reported (way too long ago) by
Yuri D'Elia: we failed to up newmaxuid for messages we produced
ourselves, so we would keep enumerating the same messages until we also
propagated externally generated messages from that mailbox - which might
have been never for the server side of archive/trash mailboxes.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
de6dc699c9 make expiration loops solely far-side-driven
we can do that, as unpaired near-side messages are ignored anyway.

this mildly changes expiration order, as near-side messages that
existed for a long time but were propagated much later will be expired
later. however, that has no practical relevance.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
ca72383fe9 don't forget to skip dead messages on far side during expiration
this is mostly theoretical, as at this point no updates to the message
list can have actually happened. but it's future-proof and consistent
with the near-side loop.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
5d6741a9a8 streamline counting of currently pushed messages wrt. expiration
don't count them as alive just to ignore them in the next step.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
8df1f5dd64 re-nest conditions for syncing new messages
this makes the logic easier to follow and document in place.
also, make the comments actually match reality.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
ceb09fcd44 handle messages which are newly doomed after an interruption
we already didn't propagate messages which would be instantly expunged
from the target, but failed to cancel propagations that were already
scheduled before we got interrupted. this matters a bit when the
resumption happens significantly later than the initial attempt, giving
the user time to mark messages on the source as deleted.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
4aaada18e9 remove redundant condition
the 'pending' and 'skipped' sync record states are mutually exclusive
with having a complementary message, so there is no point in testing it
explicitly.

amends bd5fb6ff.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
395f802500 fix loading of some messages' sizes in some partial sync scenarios
we need to pass a different "boundary" UID to driver_t::load_box() for
every OPEN_* flag that queries a partial range:
- OPEN_FIND refers to messages newer than all we know about
- OPEN_OLD_IDS refers to messages which are paired
- OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed
  boundary of already propagated messages

we treated the 3rd like the 2nd, which was just wrong - the actual
boundary may be lower or higher, so we'd produce wrong results when
MaxSize was set and only one of New and ReNew was requested.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
c8f402e43f deprecate master/slave terminology
the underlying metaphor refers to an inhumane practice, so using it
casually is rightfully offensive to many people. it isn't even a
particularly apt metaphor, as it suggests a strict hierarchy that is
counter to mbsync's highly symmetrical mode of operation.

the far/near terminology has been chosen as the replacement, as it is a
natural fit for the push/pull terminology. on the downside, due to these
not being nouns, a few uses are a bit awkward, and several others had to
be amended to include 'side'. also, it's conceptually quite close to
remote/local, which matches the typical use case, but is maybe a bit too
suggestive of actually non-existing limitations.

the new f/n suffixes of the -C/-R/-X options clash with pre-existing
options, so direct concatenation of short options is even less practical
than before (some suffixes of -D already clashed), but doing that leads
to unreadable command lines anyway.

as with previous deprecations, all pre-existing command line and config
options keep working, but yield a warning. the state files are silently
upgraded.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
b514d9ddbc purge handling of pending sync entries from state file
these cannot actually end up in the committed state.

amends bd5fb6ff.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
d93726067d wrap jFprintf()+debug() into a macro
this ensures that everything that is logged to the journal also appears
in the debug output, and it makes the code less noisy.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
198ca65b6e add option to get password from macOS Keychain
this is better than using PassCmd, as it allows the keychain manager to
identify the calling process and therefore use a selective whitelist.

unlike in the now removed example, we use an "internet password" for the
imap protocol, rather than a "generic password" - this seems more
appropriate.

based on a patch by Oliver Runge <oliver.runge@gmail.com>
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
03b15dbdd3 add ability to script IMAP user query
It was already possible to retrieve passwords from arbitrary commands.
But this goes only half the way to allowing automated derivation of
login credentials, as some environments may also have different user
names based on the system. Therefore, add the UserCmd option to
complement PassCmd.

Based on a patch series by Patrick Steinhardt <ps@pks.im>
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
503478533c de-duplicate FETCH response data item traversal somewhat 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
8acf56b311 complain about malformed item names in FETCH responses 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
47b477b3fb re-nest parse_fetch_rsp()
prefer early exits over else branches, which is easier to follow.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
81c4bfeefa extract parse_fetched_flags() from parse_fetch_rsp() 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
d4ead05a02 extract parse_fetched_header() from parse_fetch_rsp() 2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
67ea5bea7f handle bogus IMAP FETCH responses more robustly
don't use assert()s when the error condition can stem not only from
errors in mbsync's logic, but also from the IMAP stream being corrupted.

amends 72be55b0e.

REFMAIL: 20191021233411.55ctuvslkfqf2pna@koblih.localdomain
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
a5a8783ea3 sanitize error handling in IMAP FETCH response processing
abort on actual error conditions (protocol errors) and downgrade the
rest to warnings.

REFMAIL: 20191102164509.dxayakg3hrmozjnm@carbon
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
b91dd5b3bc centralize disposal of parsed IMAP lists
makes the code less cluttered, and it's harder to introduce leaks.

this has the hypothetical disadvantage that due to freeing being
delayed, the peak memory usage would rise significantly if we chained to
another parse_list() call which produces a big list while already
holding a big list, but that isn't the case anywhere.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
7af7354dbc fully decompose NAMESPACE response early on
that way the code becomes clearer, and we don't keep useless nodes in
memory.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
6fd4e8de24 don't store 'shared' and 'other' namespaces
they are never used anyway, and aren't going to be (because configuring
that would be more annoying than just specifying Path manually).
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
c391b06b07 drop redundant conn->writing member
this information is already encoded in the socket notifier's config.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
6010fe104e rewrite nonsensical struct packing magic
this couldn't have possibly worked - the alignment also determines the
sizeof, thus defeating the intent of the packing.
2020-08-04 17:16:03 +02:00
Oswald Buddenhagen
64e5f07ad3 consistently use NULL for null pointers
makes the code noisier, but also somewhat more expressive.
2020-08-04 17:16:01 +02:00
Oswald Buddenhagen
e2d3b4d55b fix lots of sign conversion warnings
... by making a lot of objects unsigned, and some signed.
casts which lose precision and change the sign in one go (ssize_t and
time_t to uint on LP64) are made explicit as well.
2020-08-04 17:15:39 +02:00
Oswald Buddenhagen
cc176df2c3 make some narrowing of integers explicit
this does specifically *not* cover about a bazillion warnings about
size_t being shrunk to uint - these make no sense given the expected
data set size.
2020-08-04 17:14:55 +02:00
Oswald Buddenhagen
4d7e169e57 shrink some data at the source to avoid subsequent narrowing 2020-08-04 17:14:55 +02:00
Oswald Buddenhagen
def22db096 constness fixes
add missing const qualifications, and add "const cast" suppressions
where unavoidable.
2020-08-04 17:14:55 +02:00
Oswald Buddenhagen
5c2e8d3e14 make more objects static 2020-08-04 17:14:55 +02:00
Oswald Buddenhagen
71d7d3e6df add some ATTR_* (mostly)
mostly ATTR_PRINTFLIKE(*, 0) for functions with a va_list argument.

also, one ATTR_NORETURN and one ATTR_UNUSED, both on functions.

also, an explicit suppression for a format string stored in a variable.
2020-08-04 17:13:56 +02:00
Oswald Buddenhagen
df22514ced turn maildir_again() into a proper varargs function
this is mostly to work around the fact that both gcc and clang won't
accept the format string declaration (i.e., will complain with
-Wformat-nonliteral) if the *called* function does not actually take a
va_list.

on the upside, it makes one caller cleaner. yay ...
2020-08-04 16:54:28 +02:00
Oswald Buddenhagen
234becf530 remove support for SSLv3
it's insecure and default builds of openssl don't include it any more.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
d09f988c70 add support for (disabling) TLS v1.3
this is actually potentially counterproductive, as people who have set
SSLVersions and fail to adjust it will _lose_ tls 1.3 support. however,
without the option being there, people (incorrectly) believe that tls
1.3 is not supported.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
990cc112f1 Add option to use IMAP LSUB instead of LIST
Based on patch by Cedric Ware <cedric.ware__bml@normalesup.org>
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
e9407cc1f7 IMAP: reject unqualified non-uppercased INBOX spellings
otherwise the server would interpret it as INBOX contrary to our
expectations, which might lead to moderately surprising effects.

if you really want to sync your ~/maildir/inbox to the IMAP INBOX,
specify it as the Maildir Store's Inbox.
2020-08-04 14:49:58 +02:00
Jaroslav Suchanek
07cb422cbb Add support for specifying cipher string used for ssl connection
Some distributions (e.g. Fedora) added support for system wide crypto
policies. This is supported in most common crypto libraries including
OpenSSL. Applications can override this policy using their own cipher
string. This commit adds support for specifying the cipher string in
the mbsync configuration.

For example, to exclude Diffie-Hellman, the user can specify
  CipherString "DEFAULT:!DH"
in the IMAP Account's configuration.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
25b1c2b9e7 set sync record's flags only after propagating new message
this is semantically cleaner, and fixes storing the flags in the rare
case that flags are not being synced and the target is not being
expunged, as in this case flags are queried only during the actual
propagation.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
abdca388f6 atomize & document conditions in load() exception list construction 2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
b677bfe7e5 de-noise msg_copied() and flags_set() somewhat
assign temporary srec object instead of always spelling out the
indirection.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
841f07efd0 de-noise initialization of sync records
use calloc() instead of malloc().
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
e7bc402d09 rename socket_expect_read() => socket_expect_activity()
... to better reflect its (mostly new) function.
2020-08-04 14:49:58 +02:00
Oswald Buddenhagen
8a03651dd8 re-nest conditions in socket_fd_cb()
conn->state == SCK_STARTTLS implies conn->ssl != NULL.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
2117945838 move state assignment to a more natural place
... so it's right next to the related callback assignment.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
c5d3565db5 de-noise -Dd output somewhat
drop commas and left-align fields in message lists.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
2f3cb5f481 fix signedness issues surrounding UIDs
amends bb632d1c.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
e334eb3580 make find_uint_array() never create negative indices 2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
01348f6f7c centralize "const cast" in make_key() 2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
27a1935361 fix strftime() format string warning properly
the workaround for -Wformat triggered -Wformat-nonliteral in turn.
so instead go back to using pragmas and add a proper gcc version check.

this also works with clang - mostly for qt-creator's code model, which
is clang-based.

amends/reverts 55e65147.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
b885e0e03a don't use reserved identifier pattern in stringify() 2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
85688d1c1a don't leak the channel list after all
unlike the actual configuration data, it's not kept in global variables,
so it shows up in memcheck.

amends 1de3ecd88.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
09d11245cd don't leak box list from the command line
we also free the box list obtained from IMAP, so there isn't a real
reason not to do that for one from the command line.

amends 1de3ecd88.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
83adb9a39b actually implement maildir_get_uidnext()
the assumption was that this wouldn't be needed, as maildir_store_msg()
reliably delivers a UID. however, if we crash right before the callback
can record that UID, we'd still use OPEN_FIND in the next run, which
requires the saved next UID.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
c84235b068 get rid of maildir_find_new_msgs() stub
a failed assertion isn't any better than a clean segfault with an
obvious backtrace.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
38e7b4db22 Maildir: fix setting flags on messages without ":2," part
this is mostly hypothetical, as all messages i've encountered actually
have it even if no flags are set on them.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
406931bc45 autotest: use symbolic message subjects
... instead of numbers. otherwise there is too much confusion with UIDs.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
6734b9ce7d autotest: re-order mailbox contents according to UIDs
... instead of subjects, because that's way more natural and thus less
confusing.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
f3629c69e1 autotest: re-organize mailbox storage by UID
an effect of 7ce658d is that we can index messages by UID rather than
content (or more specifically, subject). apart from being cleaner, it
allows duplicated subjects.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
b59ee239a4 autotest: assume that each message has a UID
followup to 7ce658d14.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
50eab08509 autotest: show the right state file after idempotence test failure
amends efd72b85.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
d59392e901 autotest: fix prototype of ckbox() 2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
ef2caa074e autotest: create the temp dir in $TEMP
on modern systems, this makes it likely to end up on tmpfs, which is a
lot faster and ssd-friendlier.

the symlink is not deleted at the end, to minimize fs churn. that means
it will be dangling after a reboot, which gets fixed in the next run.
2020-08-04 14:49:57 +02:00
Oswald Buddenhagen
5fee222f84 Merge branch '1.3' 2020-08-04 14:49:27 +02:00
Oswald Buddenhagen
30261fe6f1 fix version comparison in LibreSSL conditional
the operator was exactly inverted. that means that it actually wouldn't
compile with both older versions (that needed the aliases) and
potentially new versions (that will hide the data members - still not
the case as of 3.2).

amends 8a40554f0.
2020-08-04 14:42:42 +02:00
Oswald Buddenhagen
542e38dd49 fix re-using server connections for new stores
we failed to reset the box list pointer after freeing it, which would
lead to a crash.
we also failed to reset the listing status, which would lead to
malfunction if we hadn't already crashed.

this inlines imap_cleanup_store(), as there isn't much value in keeping
it. the message list is already freed when disowning the store anyway.
2020-08-04 14:42:42 +02:00
Oswald Buddenhagen
c82397cf6e don't crash in proxy_invoke_bad_callback()
we need to hold a ref to the proxy store, as after the bad_callback()
it's otherwise likely gone.
2020-08-04 14:40:19 +02:00
Oswald Buddenhagen
813ad67c56 fix simultaneously connecting to multiple hosts in non-IPv6 builds
we need to deep-copy the struct hostent data, as otherwise the
concurrent connects will overwrite each other's lookup results.

this is a rather hypothetical fix, as the bug currently affects only
channels connecting two IMAP accounts, and only if the first host's
first address asynchronously fails to connect.
2020-08-04 14:39:34 +02:00
Oswald Buddenhagen
3651c30296 increase PassCmd output buffer even more
apparently, some XOAUTH2 tokens are at 2.4K already, so make it 8K to be
*really* safe for a while.

REFMAIL: <20200716000515.GA2111668@lysator.liu.se>
2020-08-04 14:33:00 +02:00
Oswald Buddenhagen
09540b5648 unbreak CertificateFile documentation
the file may in fact contain CA certs.

amends 7d9d3e15.
2020-08-04 14:28:37 +02:00
Oswald Buddenhagen
cab14608ca Merge branch '1.3' 2020-07-08 12:51:20 +02:00
Oswald Buddenhagen
96afe8d0c2 fix propagation of flagged oversized messages
... when not syncing flags and the target is not being expunged, as in
that case flags were not queried in time.
2020-07-08 11:14:02 +02:00
Oswald Buddenhagen
aff0c88a38 fix printf length arguments on lp64
found by coverity.
2020-07-08 11:14:02 +02:00
Oswald Buddenhagen
04fc586e75 handle case-insensitivity of IMAP's INBOX
this is relevant only when listing an IMAP Store's contents, as that's
the only place where we aren't imposing the spelling ourselves.

we need to be careful not to treat our own canonical (prefix-stripped
and always slash-delimited) box names like that; codify that in
comments.

this reveals that commit 6f2160f1 may be deemed to have been incorrect -
the TODO item was ambiguous, and could quite possibly have meant this
fix. unsurprisingly, 380ccdd4 re-introduced it with more explicit
wording.
2020-07-08 11:14:02 +02:00
Oswald Buddenhagen
94022a6752 catch server's failure to FETCH *
the query is untypical enough to have caused problems with davmail (when
we still used *:*) and mailo.com (until it got fixed), so better check
that the result (not) returned by the server makes sense.
2020-07-08 11:14:02 +02:00
Oswald Buddenhagen
dfa8c16f27 don't timeout while uploading big messages
we did already set up the timeout when starting to send commands, but so
far we did not reset it when succeeding to send out data. rectify that.

REFFAIL: 87sgy92we3.fsf@jnanam.net
2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
121448ceb9 make -DC work with yama ptrace protection 2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
bee4fc54e7 fix overflows in uint comparisons 2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
a33e44758b fix roff abuse in man pages
".." is not valid. use "." instead, as recommended by groff_man(7).
this also necessitated adjusting the markup of the license blurbs.
2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
7d9d3e15f5 improve documentation of the server certificate related options 2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
48038fede1 mention --remove in --help output
amends d9a983a.
2020-07-08 11:12:59 +02:00
Oswald Buddenhagen
93fb3c29c6 fix UIDNEXT error message 2020-07-08 10:50:36 +02:00
Oswald Buddenhagen
e565d08246 don't try to propagate flags the target store does not support
$Forwarded is not standard, so it will most likely fail with mailboxes
that do not support keywords.

amends c4d7f018.
2020-01-08 18:22:48 +01:00
Oswald Buddenhagen
e8caaaaf65 rename F_PASSED => F_FORWARDED
the flag names are supposed to reflect the IMAP names (that's why
their #defines are followed by comments with the Maildir names).

amends c4d7f018.
2020-01-08 18:11:55 +01:00
Oswald Buddenhagen
462fed556a Merge branch '1.3' 2019-10-03 20:17:54 +02:00
Oswald Buddenhagen
d0a8551703 fix error handling of SSL_set_tlsext_host_name()
it is not one of the functions to be checked with SSL_get_error().

amends 17babc169.
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
6a874b5877 error-check more openssl function calls 2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
5f908b7672 attempt to improve ssl error handling (again)
the error queue may contain multiple errors, so make sure to drain it
completely. also, don't try to fall back to errno in case of
SSL_ERROR_SSL.
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
91abf2b830 modernize ssl context init
with openssl 1.1+, use TLS_client_method() instead of the deprecated
SSLv23_client_method().
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
f63e4338e8 fix leak of openssl X509 objects
SSL_get_peer_certificate() increments the refcount of the object.
2019-09-10 13:26:42 +02:00