Commit Graph

957 Commits

Author SHA1 Message Date
Oswald Buddenhagen
93fb3c29c6 fix UIDNEXT error message 2020-07-08 10:50:36 +02:00
Oswald Buddenhagen
e565d08246 don't try to propagate flags the target store does not support
$Forwarded is not standard, so it will most likely fail with mailboxes
that do not support keywords.

amends c4d7f018.
2020-01-08 18:22:48 +01:00
Oswald Buddenhagen
e8caaaaf65 rename F_PASSED => F_FORWARDED
the flag names are supposed to reflect the IMAP names (that's why
their #defines are followed by comments with the Maildir names).

amends c4d7f018.
2020-01-08 18:11:55 +01:00
Oswald Buddenhagen
462fed556a Merge branch '1.3' 2019-10-03 20:17:54 +02:00
Oswald Buddenhagen
d0a8551703 fix error handling of SSL_set_tlsext_host_name()
it is not one of the functions to be checked with SSL_get_error().

amends 17babc169.
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
6a874b5877 error-check more openssl function calls 2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
5f908b7672 attempt to improve ssl error handling (again)
the error queue may contain multiple errors, so make sure to drain it
completely. also, don't try to fall back to errno in case of
SSL_ERROR_SSL.
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
91abf2b830 modernize ssl context init
with openssl 1.1+, use TLS_client_method() instead of the deprecated
SSLv23_client_method().
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
f63e4338e8 fix leak of openssl X509 objects
SSL_get_peer_certificate() increments the refcount of the object.
2019-09-10 13:26:42 +02:00
Oswald Buddenhagen
36062c5220 wording fixes in mbsync.1
inspired by Ulrich Ölmann <u.oelmann@pengutronix.de>.
2019-09-10 13:18:49 +02:00
Caspar Schutijser
7607e53d56 Do not crash when using Tunnel in an IPv6-enabled build
socket_connected() is also called on the tunnel pipe.

amends 3ceb55310.
2019-08-19 13:23:03 +02:00
Dmitry Torokhov
bf14798700 Bump up PassCmd buffer size to 2KiB
While ordinary passwords are rarely longer than 80 bytes, XOAUTH2 tokens
easily exceed this limit. Let's bump it up to 2K to be really safe.
2019-07-19 12:58:04 +02:00
Oswald Buddenhagen
fbc432aace fix parsing of NIL hierarchy delimiters in IMAP LIST responses
a server which does not support hierarchical mailboxes (e.g., seznam.cz
as of oct 2018) can legitimately send NIL (rather than an empty string).
2019-05-28 17:27:09 +02:00
Oswald Buddenhagen
702b6ec4a1 fix spacing in "SASL mechanism(s) not available" error message
amends fdb03b91.
2019-03-10 11:30:05 +01:00
Klemens Nanni
d61f462039 Fix CopyArrivalDate on platforms without glibc
strptime(3)'s "%d" day of the month conversion specifier does not accept
leading blanks in case of single digit numbers.  "%e" does that.

While implementation details and differences between the two
day-of-month conversion specifiers vary, none of the major libcs
(incl. OpenBSD, FreeBSD, Illumos, musl) consume a leading blank for "%d"
except glibc, which consumes any number of spaces like in the "%e" case.

Using "%e" ensures that date strings like " 4-Mar-2018 16:49:25 -0500"
are successfully parsed by all major implementations in compliance to
X/Open Portability Guide Issue 4, Version 2 ("XPG4.2").  musl is now the
only one that still treats "%d" and "%e" without stripping any space.

Issue analysed and reported by Evan Silberman <evan@jklol.net> who found
mbsync 1.3.0 on OpenBSD 6.4 to fail with `CopyArrivalDate' set when
syncing mails with the above mentioned timestamp.

See https://marc.info/?l=openbsd-tech&m=155044284526535 for details.
2019-02-20 23:02:02 +01:00
Gergely Risko
a8f6eebdd9 Work around useless SASL warnings
Ater sasl_client_step() is called and the Cyrus SASL library forwards
it to the client plugin, if the result value is OK (authentication
succeeded), the clientout is filled out to be an empty string, even if
the client plugin wanted to return NULL.

To avoid that mbsync complains at this point, check the returned length
instead of the pointer.
2019-02-05 11:29:13 +01:00
Oswald Buddenhagen
b72800944c fix formatting of uint in callback debug stubs
amends bb632d1c.
2019-01-05 00:00:10 +01:00
Oswald Buddenhagen
95d18e2778 Merge branch '1.3' 2018-11-27 00:51:03 +01:00
Oswald Buddenhagen
acfa3a2bbc sort messages from UID FETCH request
turns out that some IMAP servers (e.g., poczta.o2.pl) do not return
messages in ascending UID order in response to a UID FETCH request,
which makes the driver violate the API contract.

counter this by sorting the messages. this also addresses the
long-standing (but hypothetical) issue that parallel UID FETCH requests
could be handled out-of-order and thus also lead to mixed up results.

based on patch by Marcin Niestroj <macius1990w@gmail.com>.
2018-11-24 14:36:31 +01:00
Oswald Buddenhagen
f7cec3064d bump IMAP command buffer size to 4KiB
while only 1KiB is required by the IMAP spec, AUTHENTICATE GSSAPI with
Kerberos requires about 1700 bytes.
accomodate that, plus some reserve.

fix suggested by Tollef Fog Heen <tfheen@err.no> via Debian BTS.
2018-09-09 13:43:53 +02:00
Oswald Buddenhagen
17babc1695 use SNI when connecting with SSL
based on patch by Vincent Bernat <vincent@bernat.ch>.
2018-09-08 18:37:15 +02:00
Oswald Buddenhagen
37feeddbfb Merge branch '1.3' 2018-07-01 13:26:10 +02:00
Oswald Buddenhagen
f698f16967 fix type of 'port' and check its range in config reader 2018-07-01 13:25:16 +02:00
Klemens Nanni
470210fa86 Fix time_t format strings
For time_t, long long handles dates after Y2038 and should be safe on
32-bit architectures.

From Jeremie Courreges-Anglas <jca@openbsd.org>.
2018-07-01 13:24:59 +02:00
Klemens Nanni
8a40554f07 User functions provided by recent LibreSSL versions
At least on OpenBSD, this enables new APIs out of the box provided by
LibreSSL 2.7.1 and higher.

From Jeremie Courreges-Anglas <jca@openbsd.org>.
2018-07-01 12:44:19 +02:00
Michael J Gruber
c4d7f0189c implement Forwarded flag
maildir supports a 'P' flag which denotes the fact that a message has
been 'passed' on (forwarded, bounced). notmuch syncs this to the
'passed' tag.

Per https://tools.ietf.org/html/rfc5788, IMAP has a user-defined flag
(keyword) '$Forwarded' that is supported by many servers and clients
these days. (Technically, one should check for '$Forwarded' in the
server response.)

Restructure mbsync's flag parser to accept keywords (flags starting with
'$') but still bail out on unknown system flags (flags starting with '\').
Support '$Forwarded' as a first keyword since it maps to maildir's 'P'
and needs to be sorted in between the system flags.

Signed-off-by: Michael J Gruber <github@grubix.eu>
2018-07-01 12:36:28 +02:00
Michael J Gruber
e71f0ccc2a mark MAILBOX_DRIVER_FLAG locations in code
Mailbox driver flags are defined in several places. It is essential that
they are kept in sync, so mark them with the same string for easy
grepping with an alerting boiler plate.

Signed-off-by: Michael J Gruber <github@grubix.eu>
2018-07-01 12:30:59 +02:00
Oswald Buddenhagen
f82c172d2b fix IMAP UID sequence also in imap_find_new_msgs()
use just * instead of the rather nonsensical *:* (which davmail happens
to actually barf at).

amends 72be55b0 (and 0a5a8479).
2018-07-01 11:05:21 +02:00
Oswald Buddenhagen
904858365d Merge branch '1.3'
Conflicts:
	configure.ac
2018-04-08 18:17:10 +02:00
Oswald Buddenhagen
5072032939 fix uidvalidity recovery with really long message-id headers
re-using the file name buffer for the headers wasn't such a great idea,
as _POSIX_PATH_MAX is only 256, while RFC2822 permits lines up to 1000
chars. and sure enough, i have a message with a whopping 470-char
message-id header ...
2018-04-08 18:10:21 +02:00
Oswald Buddenhagen
0a5a847932 fix IMAP UID sequence in UIDNEXT determination fallback
use just * instead of the rather nonsensical *:* (which davmail happens
to actually barf at).

amends 72be55b0.
2017-11-18 09:59:34 +01:00
Oswald Buddenhagen
af1acdac97 make more use of equals() 2017-10-15 16:55:23 +02:00
Oswald Buddenhagen
c29eceaeed make map_name() interpret empty strings as "no separator"
empty strings were previously meaningless, and starting with 72c2d695a,
failure to handle them lead to bogus results when the IMAP hierarchy
separator is legitimately empty (when the server genuinely supports none
and none is manually configured). non-null can be asserted more cleanly
than null-or-non-empty, so change the api like that.
incidentally, this also removes the need to work around gcc's bogus
warning in -Os mode.

problem found by "Casper Ti. Vector" <caspervector@gmail.com>
2017-10-15 16:53:27 +02:00
Oswald Buddenhagen
53e8e79488 remove pointless conditional in assignment of ctx->delimiter
amends 72c2d695a.
2017-10-15 16:52:59 +02:00
Oswald Buddenhagen
094af8720c limit -Wmaybe-uninitialized suppression to gcc >= 4.3
apple gcc 4.2 complains about the use of the pragma inside a function.
clang also complains, but because the pragma is entirely unknown to it.

as neither compiler emits the bogus warning in the first place, there is
no point in suppressing it anyway.
2017-10-07 16:03:50 +02:00
Oswald Buddenhagen
5aab050198 don't forward-declare SSL types any more
our current project structure precludes the clash between some indirect
include of ssl.h and our definition of 'S' (or 'M', i don't remember)
that happened on some system, so there is no need to avoid including it
any more.

this avoids complaints from some more picky compilers, as re-defining
typedefs (even to the same thing) is illegal before C11.
2017-10-07 16:03:50 +02:00
Oswald Buddenhagen
cbac8aa75c delete the compat wrapper
it was deprecated in 1.2. until 1.4 gets released, enough time will have
passed for sure.
2017-10-01 15:30:07 +02:00
Oswald Buddenhagen
a5d4a0fe60 make sync records with stray TUID non-fatal
while the situation indicates an internal error, it is harmless in
itself. also, printing some more information may help identify the
problem.
2017-10-01 10:42:00 +02:00
Oswald Buddenhagen
3a2e6b3793 prune SSL 2 support
OpenSSL actually did that a while ago already, so this was dead code.
2017-10-01 10:42:00 +02:00
Oswald Buddenhagen
d2e5134ebd another fix for -Wimplicit-fallthrough (new on master) 2017-10-01 10:40:55 +02:00
Oswald Buddenhagen
1b354fa61a Merge branch '1.2'
Conflicts:
	Makefile.am
	configure.ac
	debian/.gitignore
2017-10-01 10:38:43 +02:00
Alessandro Ghedini
a9feea71fe Fix spelling of error messages 2017-10-01 10:04:31 +02:00
Oswald Buddenhagen
33ee4a4ffe fixes for -Wimplicit-fallthrough 2017-10-01 10:04:31 +02:00
Oswald Buddenhagen
47bdbb4aab enable TLS 1.1 and 1.2 by default
there is no reason not to, and debian even disabled 1.0 globally,
because it's (theoretically) too insecure in some contexts (BEAST
attack).

in the compat wrapper, the UseTLSv1 option has been re-interpreted as
v1.x, to avoid adding new options.
2017-08-19 13:15:19 +02:00
Oswald Buddenhagen
118fdc4f18 Merge branch '1.2'
Conflicts:
	configure.ac
	src/mbsync.1
2017-08-05 21:20:48 +02:00
Oswald Buddenhagen
906dc989e4 mention the need for renaming Maildir files upon move
mu4e config line offered by Ben Maughan <benmaughan@gmail.com>.
2017-08-05 20:24:11 +02:00
Oswald Buddenhagen
ada0ae4b8e fix spurious decompression errors
while that's just bad api, inflate() can return Z_BUF_ERROR during
normal operation.

contrary to the zpipe example and what the documentation implies,
deflate() actually isn't that braindead. add respective comments.

REFMAIL: CALA3aExMjtRL0tAmgUANpDTnn-_HJ0sYkOEXWzoO6DVaiNFUHQ@mail.gmail.com
2017-07-30 18:47:30 +02:00
Oswald Buddenhagen
366ed7d762 improve zlib error reporting
zlib is not exactly thorough about always populating z_stream->msg, so
fall back to the error code if necessary.
2017-07-30 13:47:51 +02:00
Patrick Steinhardt
56515abe94 socket: use next addrinfo if opening socket fails
The `socket_connect_one` function previously did an `exit(1)` when
encountering any errors with opening the socket. This would break
connecting to a host where multiple possible addrinfos are returned,
where the leading addrinfos are in fact impossible to connect to. E.g.
with a kernel configured without support for IPv6, the `getaddrinfo`
call may still return a hint containing an IPv6 address alongside
another hint with an IPv4 address. Creating the socket with the IPv6
address, which will cause an error, lead us to exiting early without
even trying remaining hints.

While one can argue that the user should have compiled without HAVE_IPV6
or used an appropriate DNS configuration, we can do better by simply
skipping over the current addrinfo causing an error. To do so, we split
out a new function `socket_connect_next`, which selects the next
available address info and subsequently calls `socket_connect_one`
again. When no hints remain, `sock_connect_one` will error out at that
point.
2017-06-22 09:23:56 +02:00
Oswald Buddenhagen
1039ee25f7 fix build without BDB
amends 83ebe902.

REFMAIL: CAHyO48z0DcoFPC8rCNAL38oxVQtZNKifVd-NEF3sp1EfR-GgxQ@mail.gmail.com
2017-05-14 09:44:43 +02:00
Oswald Buddenhagen
bb632d1cd0 make UIDs unsigned
complies with the IMAP spec, thus removing the (not really) arbitrary
limitation to INT_MAX for UIDs.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
a0961d6505 delay assignment of TUID when propagating messages
go back to assigning TUIDs only right before actually propagating them.
this avoids spurious "TUID lost" warnings.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
bd5fb6fff3 move away from magic UIDs in the sync state
the only legitimate "deviant" UID is zero, meaning "no message". this
can be futher qualified by additional flags in the sync record, rather
than using magic values for the UID. in fact, the zero UID (so far
meaning only "expunged") was already optionally qualifed with "expired".

as a side effect, driver->store_msg() now returns 0 instead of -2 for
unknown UIDs. this was a hack to avoid translating the value later
on, but it made the api horrible, and now it's superflous in the first
place.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
4ffe149666 split off ephemeral sync record state to a separate member
this allows us to simplify logging of expiration, as we now can just log
the entire persistent state instead of fiddling with bits.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
efd72b85cc autotest: implement much more thorough resumption verification
the test will now make a test run for every journaled step, both right
before and right after the logging.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
7ce658d14c autotest: pre-assign all UIDs of the test messages
this ensures stable results when the boxes are used with different
OPEN_FLAGS (which will happen in a subsequent commit), at the negligible
cost of removing the implicit test of the maildir driver's ability to
enumerate new messages.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
4cc5ad5a1a introduce driver call debugging
do that by wrapping the actual stores into proxies.

the proxy driver's code is auto-generated from function templates, some
parameters, and the declarations of the driver functions themselves.
attempts to do it with CPP macros turned out to be a nightmare.
2017-04-22 11:26:11 +02:00
Oswald Buddenhagen
bbe4567bce let driver_t::openbox_box() return the UID validity
... and make 'uidvalidity' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
48ad58b9a3 use a #define for invalid UIDVALIDITY 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
8d4918affd introduce get_uidnext() driver callback
... and make 'uidnext' private to the imap driver.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
683e581340 let driver_t::find_new_msgs() return the list of messages
consistently with driver_t::load_box().
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
74e9368121 let driver_t::load_box() return the list of messages
... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
6e32b88f3d let driver_t::list_store() return the list of boxes
... and make 'boxes' and 'listed' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
8b7d3792e4 factor out transform_refcounted_msg_response()
the missing cross of transform_refcounted_box_response() and
transform_msg_response().
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
39247197f9 make struct imap_cmd_refcounted_state "abstract"
take the callback out of it, so it can be individualized.
so far, this only increases code size ...
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
96b5ae8360 don't mess with the driver's mailbox list from outside
the api specifies that the list remains owned by the driver, so poking
around in it is ugly and risky.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
c886f71054 make driver_t::prepare_load_box() return the final options
... and make 'opts' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
83ebe9022d introduce get_box_path() driver callback
... and make 'path' private to the maildir driver.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
d624c9af5d make set_bad_callback() a proper driver_t entry
... and make the pointers private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
f46cf8c887 provide a proper getter callback for driver capabilities
that way driver_t contains only callbacks.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
d54809e268 prepend "get_" to getters in driver_t
this makes it callbacks consistently start with a verb.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
36666f7e52 rewrite tracking of highest expired UID
so far, we tracked the slave side, and calculated the master side on the
fly. that complicated things unnecessarily.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
677accfd84 streamline syncing of old entries
the order of the conditionals was purely historical (pre 4ec56f8cf, anno
2005) and hard to follow, as were the comments.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
990c8a1404 sort uid exception list in a smarter place
do it closer to where it is populated. that way the debug output is
sorted, and we don't sort the list if it's known to be empty.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
72be55b0e3 add fallbacks for determining UIDNEXT
if the server sends no UIDNEXT, do an initial FETCH to query the UID of
the last message.

same if the server sends no APPENDUID.

this allows us to remove the arbitrary limitation of the UID range to
INT_MAX, at the cost of additional round-trips.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
416ced25dd fix repeated listing of same Store with different flags
multiple Channels can call driver_t::list_store() with different LIST_*
flags. assuming the flags are actually taken into consideration, using a
single boolean 'listed' flag to track whether the Store still needs to
be listed obviously wouldn't cut it - if INBOX does not live right under
Path and the Channels used entirely disjoint Patterns (say, * and
INBOX*), the second Channel in a single run (probably a Group) would
fail to match anything.

to fix this, make store_t::listed more granular. this also requires
moving its handling back into the drivers (thus reverting c66afdc0),
because the actually performed queries and their possible implicit
results are driver-specific.

note that this slightly pessimizes some cases - e.g., an IMAP Store with
Path "" will now list the entire namespace even if there is only one
Channel with Pattern "INBOX*" (because a hypothetical Pattern "*" would
also include INBOX*, and the queries are kept disjoint to avoid the need
for de-duplication). this isn't expected to be a problem, as listing
mailboxes is generally cheap.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
b9505301cc factor out listing Maildir++ Stores
Maildir++ is sufficiently different from the other SubFolder styles to
justify a separate function; the resulting code duplication is minimal,
but the separated functions are a lot clearer.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
9eaa97923a fix exclusion of bogus "INBOX.*" folders in Maildir++
this also adds code which avoids that the message about excluding the
mailbox is printed multiple times - this could happen with Maildir++, as
the hierarchy is flattened.

amends 0f24ca31b.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
2d1cfc2c7f make "Patterns *" match INBOX* in Maildir++ Stores
this also has the side effect that we won't fail to include INBOX itself
when Inbox is nested under Path when using other SubFolder styles
(regression introduced with Maildir++ support in 0f24ca31b).

REFMAIL: 1489492069.2148187.910409864.7727F9FC@webmail.messagingengine.com
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
763cb8341f make help screen print some more compile time options 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
28d52b2b18 de-obfuscate cmd_sendable()
split the monster conditional and add comments.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
0aa4c628df add comments 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
34993fbca6 fix sync resumption with aborted entries
we need a separate log entry type which does proper mmaxxuid tracking.

while moving code around, this also removes a redundant debug statement.

amends b1842617.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
887b2205ff remove nonsensical statement from journal replay of aborted entries
at this stage, entries cannot possibly have messages assigned to them,
so trying to unlink them makes no sense.

amends b1842617.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
5c2ce59217 fix sync resumption with re-newed messages
the UID of the entries needs to be bumped from -1 to -2, as otherwise
the resumed run would see a TUID in a sync entry which may not have one.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
7c466fc3e7 don't emit redundant flag updates for re-newed messages 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
1ea2e69aa7 fix maxuid tracking
newmaxuid represents the highest UID for which a sync entry was created,
while maxuid represents the end of the range which is guaranteed to have
been propagated. that means that the former needs to be instantly
incremented (and logged), while the latter must not be touched until the
entire new message sync completes. this matters particularly in the case
of resuming an interrupted run, where sync entry creation must resume
exactly where it left off, while loading the box must use the old limit
to ensure that all messages are available for actual propagation.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
6705604c4a de-duplicate journal replay somewhat
we've been using indices to separate master/slave state for a long time,
so there is no point in using pairs of matching brackets to signify the
side in the journal. instead, use somewhat descriptive letters (S[een],
F[ind], T[rashed]) and the index itself.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
af4b8896f0 use typedefs for structs more
makes the code more compact (and consistent, as typedefs were already
used in some places).
2017-04-02 15:56:30 +02:00
Oswald Buddenhagen
c3350753b0 factor out jFprintf() 2017-04-02 15:24:03 +02:00
Oswald Buddenhagen
1fdf793a3f fix signedness of 'nex' variables
they are derived from srec->status, which is unsigned. for not
understood reasons, the compiler complains only after extending status
to a full unsigned int.

on the way, localize the declarations.
2017-04-02 12:16:57 +02:00
Oswald Buddenhagen
1e939bafd8 don't use strncpy() to copy TUIDs
latest since 77acc268, the code prior to these statements ensures that
the full length is available, so just use memcpy(). the code for
comparing TUIDs uses memcmp() anyway.
2017-04-02 12:16:57 +02:00
Oswald Buddenhagen
d754608f55 autotest: improve valgrind integration
introduce recognition of $USE_VALGRIND to run all mbsync invocations
through valgrind.

this also removes the seemingly purposeless --log-fd=3 indirection.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
f29bed526b autotest: write logs to files
they can get long, and having actual files also helps with diffing.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
991e809c38 autotest: factor out readfile() function 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
2da7951fe0 autotest: de-duplicate error reporting paths 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
373abcef02 autotest: print consistent information for journal replay failures 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
ca43c57e85 autotest: don't print expected result if the mbsync run itself fails
there isn't an actual result to compare with anyway.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
b4d1a05365 autotest: de-duplicate: use print*() in show*()
this actually adds some (minimal) transformation overhead, but it makes
the relationship clearer.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
8aed94420f autotest: determine path of mbsync prior to chdir()
that allows tmp/ to be a symlink to a ramdisk.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
71ced65fcc Merge remote-tracking branch 'origin/1.2'
Conflicts:
	src/sync.c
2017-04-01 20:31:51 +02:00
Oswald Buddenhagen
62808c9003 autotest: use warnings 2017-04-01 20:25:58 +02:00
Oswald Buddenhagen
b45e711da5 autotest: remove stray close() call from printstate() 2017-04-01 20:25:53 +02:00
Oswald Buddenhagen
f934e995d6 don't populate sync record map with invalid UIDs
this would obviously just bloat the hash with nonsense, slowing down the
actual lookup later.
2017-03-14 11:36:25 +01:00
Oswald Buddenhagen
f62b3c7be9 fix mislabeling of test 2017-02-15 17:44:35 +01:00
Oswald Buddenhagen
3ebb066aba make -DN print also the sent data 2017-02-15 17:30:15 +01:00
Oswald Buddenhagen
2457b2baa3 don't arbitrarily limit UIDs to a billion, part 2
imap_find_new_msgs() had the same fixed limit as imap_load_box().

amends 815822d8.
2017-02-15 17:25:59 +01:00
Oswald Buddenhagen
77acc26812 implement Message-Id based UIDVALIDITY recovery 2017-01-21 12:09:01 +01:00
Oswald Buddenhagen
f9fe75602e don't fetch message size unless necessary
when syncing flags but not re-newing non-fetched messages, there is no
need to query the message size for all messages, as the old ones are
queried only for their flags.
2017-01-21 11:41:12 +01:00
Oswald Buddenhagen
ec4b21535f some reshuffling in maildir_scan() for clarity 2017-01-21 11:41:12 +01:00
Oswald Buddenhagen
3d64f16702 make fetching of partial UID ranges more flexible
instead of a single hard-coded branch, use a generic method to split
ranges as needed.

this is of course entirely over-engineered as of now, but subsequent
commits will make good use of it.
2017-01-21 11:41:12 +01:00
Oswald Buddenhagen
1d3b36f89e factor out app_cr 2017-01-17 22:14:07 +01:00
Oswald Buddenhagen
3dffd68825 factor out copy_msg_convert() 2017-01-17 22:08:49 +01:00
Oswald Buddenhagen
951b7e77f8 factor out copy_msg_bytes() 2017-01-15 13:25:46 +01:00
Oswald Buddenhagen
509a191474 make more use of strnlen()
includes adding the so far superfluous prototype for the replacemnt to
common.h.
2016-12-29 15:18:39 +01:00
Oswald Buddenhagen
3eea668052 make use of memchr() in strnlen() replacement after all
turns out the comment advising against it was bogus - unlike for
memcmp(), the standard does indeed prescribe that the memchr()
implementation may not read past the first occurrence of the searched
char.
2016-12-29 15:17:04 +01:00
Oswald Buddenhagen
357dd51405 adjust return type of strnlen() replacement
it doesn't really matter, but it's nicer to stay consistent with the
official prototype.
2016-12-29 15:17:03 +01:00
Oswald Buddenhagen
67f4aeff1f standardize on 'int' for message sizes
that's what the sources already assumed anyway. size_t is total
overkill, as No Email Ever (TM) will exceed 2GiB.

this also fixes a harmless format string warning in 32 bit builds.
2016-12-29 14:10:35 +01:00
Oswald Buddenhagen
0c36655201 print actually read TUID in debug message 2016-12-26 16:20:27 +01:00
Oswald Buddenhagen
1330f43034 null-terminate lines read from state file & journal
makes the subsequent code less convoluted.
2016-12-26 16:20:27 +01:00
Oswald Buddenhagen
879eb623be mark string_list_t as packed
otherwise we'll regularly over-allocate due to the struct's stride.
2016-12-18 22:03:51 +01:00
Oswald Buddenhagen
4db64967c9 make more use of shifted_bit()
technically, this introduces a redundant AND, but the compiler is smart
enough to prove that (((A & M) ^ B) & M) == ((A ^ B) & M).
2016-12-18 22:03:51 +01:00
Oswald Buddenhagen
3b615bba3c make more use of nfstrndup() 2016-12-18 22:03:51 +01:00
Oswald Buddenhagen
22145f6674 document some additional uses for PipelineDepth 2016-12-18 22:03:51 +01:00
Oswald Buddenhagen
ab955ffe6b Merge branch '1.2'
Conflicts:
	src/drv_imap.c
2016-12-11 12:52:46 +01:00
Oswald Buddenhagen
743968737c silence bogus [-Wmaybe-uninitialized] with -O0/-O1/-Os 2016-12-11 12:51:47 +01:00
Oswald Buddenhagen
ef0e7fdd3e accept NAMESPACE responses without hierarchy delimiter
RFC2342 states that the delimiter may be NIL, which some servers
apparently actually make use of.

REFMAIL: CAM0xXk_FQ83CPrd37iQCMKtc1B2P8=u-r5jX0n2WE5Y+3483nQ@mail.gmail.com
2016-12-04 11:26:06 +01:00
Oswald Buddenhagen
03e25db3b8 validate NAMESPACE response earlier
... and don't silently fail later on.
2016-12-04 11:14:34 +01:00
Oswald Buddenhagen
1a707ab156 inform user if LOGIN was skipped because of missing SSL
'AuthMechs *' technically includes LOGIN, so it is a bit unintuitive
when it's still not used.
2016-12-03 21:17:25 +01:00
Oswald Buddenhagen
1b235d3d46 make * not match LOGIN even in non-SSL builds
this is consistent with the plain text transmission warning below.
2016-12-03 21:13:49 +01:00
Oswald Buddenhagen
fdb03b91f2 be more helpful when no SASL mechanisms are available 2016-12-03 21:13:44 +01:00
Oswald Buddenhagen
2f91e22371 fix LOGIN in SASL builds
if AuthMechs includes more than just LOGIN and the server announces any
AUTH= mechanism, we try SASL. but that can still fail to find any
suitable authentication mechanism, and we must not error out in that
case if we are supposed to fall back to LOGIN.
2016-12-03 20:58:16 +01:00
Oswald Buddenhagen
bc51d0206a fix LOGIN in non-SASL builds
specifically, if AuthMechs included more than just LOGIN (which would be
the case for '*') and the server announced any AUTH= mechanism, we'd
immediately error out upon seeing it, thus failing to actually try
LOGIN.
2016-12-03 14:32:51 +01:00
Oswald Buddenhagen
815822d81c don't arbitrarily limit UIDs to a billion
the number was chosen to make queries more comprehensible when the
server sends no UIDNEXT, but it appears that such insanely large UIDs
actually show up in the wild. so send 32-bit INT_MAX instead.

note that this is again making an assumption: that no server uses
unsigned ints for UIDs. but we can't sent UINT_MAX, as that would break
with servers which use signed ints. also, *we* use signed ints (which is
actually a clear violation of the spec).

it would be possible to special-case the range [1,inf] to 1:*, thus
entirely removing arbitrary limits. however, when the range doesn't
start at 1, we may actually get a single message instead of none due to
the imap uid range limits being unordered. this gets really nasty when
we need to issue multiple queries, as we may list the same message
twice.

a reliable way around this would be issuing a separate query to find the
actual value of UID '*', to make up for the server not sending UIDNEXT
in the first place. this would obviously imply an additional round-trip
per mailbox ...
2016-11-20 13:05:08 +01:00
Oswald Buddenhagen
2bba9b903c wrap message trashing into simple transactions
trashing many messages at once inevitably overtaxes m$ exchange, and the
connection breaks. without any progress tracking, it would restart from
scratch each time, which would lead to a) it never finishing and b) many
copies of the messages in the trash.

full transactions as we do for "proper" syncing would be over the top,
as it's not *that* bad if some messages get duplicated in the trash. so
we record only the messages for which trashing completed, thus allowing
some overlap between the attempts.
2016-11-06 09:26:16 +01:00
Oswald Buddenhagen
5b0c8cfa60 use a temporary for sanity 2016-11-05 18:16:43 +01:00
Oswald Buddenhagen
ae95490d52 pre-sort exception list passed to driver->load_box()
... and use that to optimize the maildir driver somewhat.
2016-11-05 17:32:34 +01:00
Oswald Buddenhagen
7b567164ff abstract growable arrays somewhat
... and sneak in a C99 requirement on the way. just because.
2016-11-05 17:32:34 +01:00
Oswald Buddenhagen
0f24ca31b5 fix SubFolders style Maildir++
turns out i misread the spec in a subtle way: while all other folders
are physically nested under INBOX, the IMAP view puts them at the same
(root) level. to get them shown as subfolders of INBOX, they need to
have _two_ leading dots.

this also implies that the Maildir++ mode has no use for a Path, so
reject attempts to specify one.
2016-11-05 17:32:34 +01:00
Oswald Buddenhagen
b2f6ef391b Merge branch 'isync_1_2_branch' 2016-11-05 13:16:32 +01:00
Oswald Buddenhagen
41308e4814 fix build with openssl 1.1
they finally made their structs opaque, and provided proper getters.
2016-07-24 11:58:57 +02:00
Oswald Buddenhagen
719d4a2437 prune obsolete #include
hmac.h was needed only for the cram-md5 implementation.
2016-07-24 11:58:09 +02:00
Oswald Buddenhagen
2648ef578f fix server certificate validation error reporting
use the right function to decode the error code.

found by Andrés Ramírez <sunshavi@fastmail.fm>.
2016-05-21 13:09:56 +02:00
Oswald Buddenhagen
46e792c3df improve .gitignore files
- add missing entries
- remove redundant entries which are inherited from parent dirs
- mark dirctories as such
- anchor specific files
2015-11-08 12:31:20 +01:00
Oswald Buddenhagen
7ddd8d1737 Merge branch 'isync_1_2_branch' 2015-11-08 12:04:44 +01:00
Oswald Buddenhagen
e054c575ea fix CertificateFile docs & samples
the mbsync manual says explicitly that the system's default certificate
store should *not* be specified.
however, the isync manual talked about CA certificates, which is (and
always was) exactly wrong.
also adjust both .sample rc files.
2015-11-06 22:37:58 +01:00
Oswald Buddenhagen
89dc7592ee don't crash when dns lookup fails (ipv6 path)
we call socket_connect_bail() when getaddrinfo() failed, so it must deal
with no addrinfo being there yet.
2015-09-27 12:13:34 +02:00
Oswald Buddenhagen
cda596d530 remove legacy (bsd-style) locking
flock() may be implemented via fcntl(), which may cause the process to
deadlock itself when trying to apply both types of locks. this is the
case even on linux when the file lives on NFS.

it's unlikely that anything except mbsync would try to access the
.uidvalidity files anyway, so there is no point in trying to be
compatible with anything else ...

REFMAIL: uddy4g589ym.fsf@eismej-u14.spgear.lab.emc.com
2015-09-27 11:47:45 +02:00
Oswald Buddenhagen
8bd6eb433f don't attempt to issue LOGOUT on bad stores
amends 9d22641b.
2015-09-12 11:14:46 +02:00
Oswald Buddenhagen
9a0403f446 de-duplicate cleanup of name-related data in error paths
"name" being both the ipv6 dns info and our own socket label.

sort-of amends 9d22641b.
2015-09-12 11:13:59 +02:00
Oswald Buddenhagen
682a05a676 mention safety of concurrent access; wording improvements 2015-09-07 12:35:12 +02:00
Oswald Buddenhagen
8979ebbdf2 tolerate case changes in X-TUID header name
it is legal for an email system to simply change the case of rfc2822
headers, and at least one imap server apparently does just that.
this would lead to us not finding our own header, which is obviously not
helpful.

REFMAIL: CA+fD2U3hJEszmvwBsXEpTsaWgJ2Dh373mCESM3M0kg3ZwAYjaw@mail.gmail.com
2015-09-01 15:40:54 +02:00
Anton Khirnov
167964933f add support for sending a TLS client certificate 2015-08-09 09:44:55 +02:00
Oswald Buddenhagen
17f3348ff1 make it possible to specifiy Pattern INBOX* with no Path defined
that pattern may very well expand to INBOXNOT, which would naturally
live under Path, so we need to look into the Path. of course, this
actually makes sense only if there *is* a Path, and complaining about
it being absent is backwards.
2015-05-24 14:51:31 +02:00
Oswald Buddenhagen
72c2d695ac remove support for multi-char imap path delimiters again
the idea that this is even possible was based on an incomplete reading
of the imap spec.

however, the infrastructure for supporting multi-char delimiters as such
is retained, as the Flatten option can be used with them.
2015-05-24 14:51:31 +02:00
Oswald Buddenhagen
9d22641b62 make server connection a cancellable operation
this entails splitting drv->open_store() into alloc_store() and
connect_store().
2015-05-24 14:51:31 +02:00
Oswald Buddenhagen
246c417874 validate Path earlier
we cannot know whether it will be needed later, but we can validate it
if it's set.
2015-05-24 14:51:31 +02:00
Oswald Buddenhagen
eb190d2bd5 prune dead SYNC_FAIL_ALL define 2015-05-24 14:51:31 +02:00
Oswald Buddenhagen
1aaf713ffe Merge branch 'isync_1_2_branch' 2015-05-24 14:50:01 +02:00
Oswald Buddenhagen
a3b131b6e8 don't make bogus attempts at enabling compression
recycling server connections skips everything up to setting up the
prefix (Path/NAMESPACE). "everything" should obviously include enabling
compression, as that must be done at most once per connection.
2015-05-24 14:45:50 +02:00
Oswald Buddenhagen
bcd43e2c66 Merge branch 'isync_1_2_branch'
Conflicts:
	configure.ac
	src/drv_imap.c
2015-05-09 19:31:55 +02:00
Oswald Buddenhagen
2013e50b1c rename misnamed functions concerning sending imap commands
cmd_submittable() => cmd_sendable()
cancel_submitted_imap_cmds() => cancel_sent_imap_cmds()

the sequence is exec -> submit -> send.
2015-05-09 19:25:51 +02:00
Oswald Buddenhagen
02af3f4c73 ensure direct exit after calling back
any structures may be invalid after callback invocation.

this has the side effect that the socket write callback now returns
void, like all other callbacks do.
2015-05-09 19:18:40 +02:00
Oswald Buddenhagen
6c08f568d0 fix socket_write() recursion
the synchronous writing to the socket would have typically invoked the
write callback, which would flush further commands, thus recursing.

we take the easy way out and make it fully asynchronous, i.e., no data
is sent before (re-)entering the event loop.

this also has the effect that socket_write() cannot fail any more, and
any errors will be reported asynchronously. this is consistent with
socket_read(), and produces cleaner code.

this introduces a marginal performance regression: the maildir driver is
synchronous, so all messages (which fit into memory) will be read before
any data is sent. this is not considered relevant.
2015-05-09 19:17:41 +02:00
Oswald Buddenhagen
2f7e60a3ed fix #ifdefs around AuthMech & RequireCRAM
these options don't depend on HAVE_LIBSSL.
2015-05-09 18:57:30 +02:00
Oswald Buddenhagen
16aa17053d mask AUTHENTICATE PLAIN commands in error output as well
amends bd0f3af5.
2015-05-09 18:57:30 +02:00
Oswald Buddenhagen
b8d6d833c6 add DisableExtension option to work around (server) bugs 2015-05-08 10:20:09 +02:00
Oswald Buddenhagen
549e6739e8 support verbatim and real Maildir++ subfolder naming styles
the legacy style is a poorly executed attempt at Maildir++, so introduce
the latter for the sake of completeness. but most users will probably
just want to use subfolders without any additional dots.
2015-05-01 20:53:23 +02:00
Oswald Buddenhagen
064f579a92 make maildir_list_recurse() recursion less convoluted
move the unconditional addition of INBOX out ouf the function.
this makes it possible to move the folder check and addition to the
listing before the recursion, which seems clearer.
2015-05-01 20:51:32 +02:00
Oswald Buddenhagen
da9adcc4bd pass a maildir_store_conf_t to maildir_validate_path()
casting early on makes the code clearer.
2015-05-01 20:51:32 +02:00
Oswald Buddenhagen
3de60c8f5c make flags in pattern debugging non-cumulative 2015-05-01 20:51:32 +02:00
Oswald Buddenhagen
ea9f4f0b96 use \fB and \fI consistently, take 2
\fB means literal, while \fI means placeholder, value for placeholder,
or emphasis.
2015-05-01 18:53:08 +02:00
Oswald Buddenhagen
a041766140 Merge branch 'isync_1_2_branch' 2015-04-26 20:59:11 +02:00
Oswald Buddenhagen
b85153f8eb make skipping of failed stores more thorough
in the case of imap stores, the failure is bound to the server config,
not just the store config.

that means that the storage of the failure state needs to be private to
the driver, accessible only through a function.
2015-04-26 20:58:43 +02:00
Oswald Buddenhagen
1eb88d4fea add socket timeout handling 2015-04-26 20:58:22 +02:00
Oswald Buddenhagen
5c4015aee5 remove caching of current time
it's too hard to reliably predict when invalidation will be necessary.
2015-04-26 18:41:32 +02:00
Oswald Buddenhagen
e0171b71e7 don't get system time when dealing with null timers
they fire immediately regardless of wall time, so we can save some
pointless syscalls.
2015-04-26 18:41:32 +02:00
Oswald Buddenhagen
ac7cd86c73 fix -DN not implying -Dn 2015-04-26 18:39:09 +02:00
Oswald Buddenhagen
98bd2b115d make it possible to nest maildir Path under Inbox
simply make the code symmetrical to the inverse case.

note that the result will be sort of awkward, as the folders under Path
(and thus the subfolders of Inbox) don't start with a dot, while the
subfolders of these folders do. this needs to be addressed separately.
2015-04-18 12:01:35 +02:00
Oswald Buddenhagen
2d4ce72a8b make handling of Inbox-in-Path nesting less obfuscated
when we run into Inbox while listing Path, check whether Inbox is being
listed anyway, and just skip it if so, instead of listing it right away
and resetting LIST_INBOX (and thus having a calling order dependency).
2015-04-18 11:58:05 +02:00
Oswald Buddenhagen
41ed101224 don't list IMAP Path under INBOX twice
if NAMESPACE is "INBOX.", listing INBOX recursively will already include
it.

REFMAIL: 1890363108.1020695.1428757117731.JavaMail.yahoo@mail.yahoo.com
2015-04-18 10:54:00 +02:00
Oswald Buddenhagen
83eaac8787 fix uninitialized variable warning
the returned UID is not used when trashing messages, but we still
shouldn't just return an undefined value.
2015-04-13 09:20:09 +02:00
Reimar Döffinger
08dab9465b Make Berkley DB an optional dependency.
It doesn't seem necessary for any of the basic functionality.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
2015-04-13 09:19:02 +02:00
Oswald Buddenhagen
8dc776c528 fix SASL, take 2
USER (the authorization identity) specifies whom to act for.
AUTHNAME (the authentication identity) specifies who is acting (and
thus whose PASS is being used).
USER is derived from AUTHNAME if omitted, but apparently the
GSS-API module automatically adds the REALM, which is not helpful.

it appears to be common to set both USER and AUTHNAME to the same value,
so let's just do it as well.

REFMAIL: 20150407194807.GA1714@leeloo.kyriasis.com
2015-04-09 10:33:51 +02:00
Dmitrij D. Czarkoff
0840026a4b fix crash in maildir_set_msg_flags()
memcpy(3)'s behavior is undefined when source and destination addresses
overlap, and it actually crashed on OpenBSD. use memmove() instead.
2015-04-04 10:46:14 +02:00
Oswald Buddenhagen
95276cd967 don't insert unnecessary linebreaks upon PassCmd invocation
the PassCmd will be typically non-interactive (or it will use a gui
password agent), so starting a new line just makes the progress counter
uglier. so make it configurable and default to no line break.
2015-03-30 14:52:02 +02:00
Oswald Buddenhagen
ba2b42ec9b fix crash when no mailboxes match Patterns
filter_boxes() just returns a null pointer in this case.

REFMAIL: 20150330121211.GA10315@venus.fritz.box
2015-03-30 14:27:23 +02:00
Oswald Buddenhagen
138983c91e fix SASL PLAIN for User != $USER
REFMAIL: 87d29mysnx.fsf@ericabrahamsen.net
2015-03-30 12:59:40 +02:00
Oswald Buddenhagen
03a124051f add debugging for main()
so far, that's mostly mailbox listing
2015-03-30 10:31:59 +02:00
Oswald Buddenhagen
0e1f8f9a3f revamp console output options
- the old meaning of -V[V] was moved to -D{n|N}, as these are really
  debugging options.
- don't print the info messages by default; this can be re-enabled with
  the -V switch, and is implied by most debug options (it was really
  kind of stupid that verbose/debug operation disabled these).
- the sync algo/state debugging can be separately enabled with -Ds now.
2015-03-30 10:31:26 +02:00
Oswald Buddenhagen
8aa22a62e7 make progress counters global
which means they are now cumulative, and include channels and boxes.
2015-03-30 10:30:35 +02:00
Oswald Buddenhagen
1de3ecd883 pre-calculate channel and box lists (as far as possible)
... instead of determining them on the fly, because
- it enables early display of totals (to be used soon)
- it enables re-use of the data (to be used at some point)
- the code is less cryptic

note that we leak the data created in main(), consistently with other
configuration-related data.
2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
f361738ad2 don't claim that INBOX is absent even if it was not listed
IMAP defines that INBOX is always present.
2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
05deb008db rework Pattern application
instead of creating three lists of mailboxes (common, master, slave)
and deriving the mailbox presence from the list being processed, create
a single joined list which contains the presence information.
additionally, the list is sorted alphabetically (with INBOX first),
which is neater.
2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
e00d0f1ac3 static my_strndup() => extern nfstrndup() 2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
4d638c3cf2 make sure that INBOX always exists
makes the Maildir driver consistent with IMAP.
2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
d8225390fc don't refuse to strip Path just because it's INBOX/
no ambiguity can result from this, so there is no reason to treat
sub-folders of INBOX differently from any other namespace.
2015-03-28 17:56:10 +01:00
Oswald Buddenhagen
bd0f3af578 mask AUTHENTICATE PLAIN commands in debug output
they are almost as bad as LOGIN.
2015-03-28 17:56:08 +01:00
Oswald Buddenhagen
06c1a43aa2 remove double "Logging in ..." when using legacy LOGIN 2015-03-28 17:56:08 +01:00
Oswald Buddenhagen
c333a36aee Merge branch 'isync_1_1_branch' 2015-03-28 17:54:23 +01:00
Oswald Buddenhagen
ee8b835c55 fix out-of-Path INBOX never being matched by Patterns
"(null)I" really doesn't cut it.

amends cf0f32f8.
2015-03-23 19:03:16 +01:00
Oswald Buddenhagen
c0ba6f0395 escape backslashes in PassCmd examples
the config parser strips one level of backslash escapes.
2015-03-22 11:44:33 +01:00
Oswald Buddenhagen
4842f5148d fix bogus "unexpected command continuation request"
it helps if the code actually does what the comment above it claims.
clarify it a bit, so i don't get stupid ideas again.

This reverts commit cf6a7b4d18.
2015-03-21 12:30:42 +01:00
Oswald Buddenhagen
1701e3d84f fix chaining of COMPRESS invocation
it was bound to the use of NAMESPACE, which made no sense at all.
2015-03-07 17:46:41 +01:00
Oswald Buddenhagen
a8b26dc4ac soft-limit peak memory usage
propagating many messages from a fast store (typically maildir or a
local IMAP server) to a slow asynchronous store could cause gigabytes of
data being buffered. avoid this by throttling fetches if the target
context reports memory usage above a configurable limit.

REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 18:13:05 +01:00
Oswald Buddenhagen
4b31522fdf complain about excess values supplied to options 2015-02-15 12:48:46 +01:00
Oswald Buddenhagen
d4392c9220 handle clean SSL connection shutdowns
some servers actually bother to close down the SSL connection before
closing the socket.

this fixes the spurious "unhandled SSL error 6" messages.

REFMAIL: 20150120114805.GA17586@leeloo.kyriasis.com
2015-02-15 12:19:47 +01:00
Oswald Buddenhagen
9e15ab4a5a refactor socket EOF handling
handling EOF already at the socket level isn't a very good idea - it
breaks the abstraction, and makes implementing sane semantics hard.
2015-02-15 12:15:46 +01:00
Oswald Buddenhagen
13c742529c fix crash on shutdown of compressed connection
the callback may destroy the socket, so it is very unwise to use it for
buffering the return value. use a temporary instead.
2015-02-15 11:49:58 +01:00
Oswald Buddenhagen
c9b8cefc29 handle clean shutdown of zlib stream
the server can actually close the zlib stream before closing the socket,
so we need to accept it.

we don't do anything beyond that - the actual EOF will be signaled by
the socket, and if the server (erroneously) sends more data, zlib will
tell us about it.

REFMAIL: 1423048708-975-1-git-send-email-alex.bennee@linaro.org
2015-02-15 11:49:58 +01:00
Oswald Buddenhagen
ef70bd4a40 don't try to flush if there is nothing to flush
zlib reports Z_BUF_ERROR when a flush is attempted without any activity
since the previous flush (if any). while this is harmless as such,
discerning the condition from genuine errors would be much harder than
avoiding the pointless flush in the first place.

REFMAIL: eb5681612f17be777bc8d138d31dd6d6@mpcjanssen.nl
2015-02-15 11:49:58 +01:00
Oswald Buddenhagen
f4240761f1 introduce and use pending_wakeup()
so we don't need to peek into internal data structures.
2015-02-15 11:49:52 +01:00
Oswald Buddenhagen
74c78c70b9 deal sensibly with permanent errors
don't retry dead Stores for every Channel.

this also introduces a state for transient errors (specifically, connect
failures), but this is currently unused.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
5f265ad7da unify .isyncuidmap.db handling with that of .uidvalidity
that is, open the database on demand when locking and close it again
when unlocking.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
4da89af7be refactor maildir_set_uid() and maildir_store_uidval()
the latter now handles both the db and the plaintext file, so the former
can make use of it.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
f61efdbb9d rename some *uid*() => *uidval*() to better reflect their function 2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
2eece82276 lock .uidvalidity on demand
a maildir re-scan doesn't need to lock it if it doesn't need to allocate
any new uids.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
4aad8c9e04 delay unlocking of .uidvalidity
it's wasteful to re-lock all the time. instead, unlock in the background
after 1-2 seconds of inactivity.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
d9a983add6 add support for propagating folder deletions 2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
a7eddc6ede don't make intermediate directories proper maildirs
"phantom" mailboxes waste time on syncing. furthermore, mutt's mailbox
navigator provides no means to enter subfolders of maildirs.
2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
7489ff8613 deal sensibly with incomplete maildir directories
a directory is no mailbox unless it contains a cur/ subdir.
but if that one is present, create new/ and tmp/ if they are missing.

this makes it possible to resume interrupted maildir creations.
2015-01-17 17:51:19 +01:00
Oswald Buddenhagen
5f4e3b285e factor out maildir_clear_tmp() 2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
926788f3ae supplement open_box() with box existence information from list_store()
there is no point in trying to open a non-existing box before trying to
create it.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
7b7304b625 split create_box() off from open_box()
this allows us to do something else than creating missing boxes
depending on circumstances. hypothetically, that is.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
f1809ddd2b open the mailboxes after loading the sync state
this allows us to react differently to a box'es absence depending on the
state. hypothetically, so far.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
f43617cd94 lock sync state lazily
don't try to lock it until we actually read or write it.
the idea is to not fail with SyncState * if we tried to load the state
before selecting a non-existing mailbox. this is ok, because if the
mailbox is missing, we obviously have no sync state pertaining to it,
either.
as a side effect, this allows simplifying an error path.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
fb19d644f7 split off open_box() from select_box()
aka prepare_paths() reloaded. we'll need it in a moment.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
97a42cd825 factor out {prepare,lock,save,load}_state() 2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
5af1796777 de-duplicate handling of box operation (create & expunge) options
loops work just fine ...
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
9982e7bf08 make some driver function names more descriptive 2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
00ebf45be2 rename driver::prepare_opts() => prepare_load()
... and move it to the right place in the structure and fix the doc to
not claim that it is called before select().
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
bac2b00f1b disable use of LITERAL+ for payloads > 100k
when LITERAL+ is used, the server has no chance for early rejection of
messages. this means that the client can upload megabytes for nothing.
so simply don't use LITERAL+ for big messages. of course this adds
server roundtrips, but that's tough luck.

the limit could be arguably higher than 100k (or even configurable).
i set it to ~2 sec with my fairly average DSL line.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
6c959c3ee4 fix handling of unsolicited BYE responses
they can come in at any time, after which we must expect the connection
to be closed (and not complain about it).
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
4f3ef54f3a fix treatment of untagged NO and BAD responses
they aren't possible greeting responses. however, they are warning resp.
error reports from the server, so print them accordingly.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
139b90be29 added support for IMAP DEFLATE
initial patch by Jesse Weaver <pianohacker@gmail.com>, but mostly
rewritten by me.
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
f0b80e7d35 make socket writing buffered
the primary objective is reducing the number of small SSL packets (which
are always padded), but fewer syscalls in the non-SSL case should be
good as well.
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
efb23ab96a vectorize socket_write()
the objective is making the buffer code aware of the total size of a
write in advance. this commit doesn't take advantage of that yet.
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
3f629af07e remove support for faking notifications
with the existence of timers, this is now superfluous.
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
a4e2f1a60d use null timer instead of faking a socket notification 2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
7b76d9ff7e add timers to mainloop
they are called "wakeups", though, as timer_t already exists in time.h.
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
1fd66195d8 change socket notifier design
instead of keeping the structures in an opaque array (which was a shadow
of the struct pollfd array if poll() was supported), make them directly
addressable.

this has the advantage that notifier-altering operations (mostly
en-/disabling) don't need to look up the structure by file handle each
time.
on the downside, data locality in the main loop is worse.
neither of these have any real effect on performance.

note that the structures are not allocated separately, but embedded into
the the parent structure (like sockets already were).
2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
f68e021b90 factor out socket_open_internal() 2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
42cedc8f81 introduce uchar, ushort & uint typedefs 2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
b730f66f7d Merge branch 'isync_1_1_branch' into HEAD
Conflicts:
	src/socket.c
2015-01-11 14:32:15 +01:00
Oswald Buddenhagen
2fa75cf159 fix UID assignment with some non-UIDPLUS servers
the seznam.cz IMAP server seems very eager to send UIDNEXT responses
despite not supporting UIDPLUS. this doesn't appear to be a particularly
sensible combination, but it's valid nonetheless.

however, that means that we need to save the UIDNEXT value before we
start storing messages, lest imap_find_new_msgs() will simply overlook
them. we do that outside the driver, in an already present field - this
actually makes the main path more consistent with the journal recovery
path.

analysis by Tomas Tintera <trosos@seznam.cz>.

REFMAIL: 20141220215032.GA10115@kyvadlo.trosos.seznam.cz
2015-01-11 14:29:19 +01:00
Oswald Buddenhagen
958af473a0 fix conditional for early failure in cancel_done() 2015-01-02 12:38:48 +01:00
Oswald Buddenhagen
9eba3d8cd9 don't leave 2nd store in limbo if opening 1st store fails synchronously
we can't leave the store FRESH, as otherwise the error handling code
will assume it is still being opened and will return to the main loop.
depending on the config this would cause an immediate termination or an
indefinite wait.
2015-01-02 11:29:51 +01:00
Oswald Buddenhagen
3db3f4718e remove stray CRLF from AUTHENTICATE continuations
this re-introduces 6741bc94 (just a bit differently), thus effectively
reverting fbfcfea5. i suppose this extra CRLF is needed by a broken
CRAM-MD5 implementation of some server, which is why it was there in the
original implementation as well. however, it breaks more pedantic
non-broken servers. if somebody complains, we'll need to add a much
more sophisticated hack.
2014-12-29 02:30:33 +01:00
Oswald Buddenhagen
518b5630dc cleanup dead cram() prototype 2014-12-29 02:30:33 +01:00
Oswald Buddenhagen
5dfca41422 fix more memcmp() abuse
amends 1217193fb and 4f383a807.
2014-12-29 02:30:01 +01:00
Oswald Buddenhagen
6f7d416bb8 fix acceptance of trusted SSL certs
we should make no assumptions about the layout of OpenSSL's certificate
store, in particular when they are wrong. so copy the interesting part
instead of "deep-linking" into it.

this code is uglier than it should be, but OpenSSL's extensive use of
macros to manage data types would force us to include the ssl headers
into our headers otherwise, which would be even uglier.

REFMAIL: <545442CC.9020400@nodivisions.com>
2014-11-08 14:00:28 +01:00
Oswald Buddenhagen
f377e7b696 introduce FieldDelimiter and InfoDelimiter options
... for windows fs compatibility.

the maildir-specific InfoDelimiter inherits the global FieldDelimiter
(which affects SyncState), based on the assumption that if the sync
state is on a windows FS, the mailboxes certainly will be as well, while
the inverse is not necessarily true (when running on unix, anyway).

REFMAIL: <CA+m_8J1ynqAjHRJagvKt9sb31yz047Q7NH-ODRmHOKyfru8vtA@mail.gmail.com>
2014-10-25 17:42:48 +02:00
Oswald Buddenhagen
85fd5ceb54 move orig_name out of store_t
it's state specific to the synchronizer.
2014-10-25 15:06:50 +02:00
Oswald Buddenhagen
7ee0483436 nuke home-grown CRAM-MD5 support
it was obsoleted by SASL support.
i deem the additional dependency acceptable when one wants the feature.
2014-10-20 10:10:55 +02:00
Oswald Buddenhagen
eb1005151c add SASL support
patch initially by Jack Stone <jwjstone@fastmail.fm>,
cleaned up by Jan Synacek <jsynacek@redhat.com>,
... and then almost completely rewritten by me. ^^
2014-10-20 10:10:55 +02:00
Oswald Buddenhagen
360600b98d factor out ensure_user() and ensure_password() 2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
1217193fbb rework authentication mechanism configuration
RequireCRAM (another fairly stupid "use if available" option) is now
deprecated. instead, the AuthMech option can be used to give a precise
list of acceptable authentication mechanisms (which is currently "a bit"
short). in particular, this allows *not* using CRAM-MD5 even if it's
available.
2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
aba3524d9b make it possible to disable usage of system cert store 2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
7822bd8a91 require Host if SSL is used despite Tunnel 2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
7ce57b9c00 make SSL certificate verification less arcane
instead of using a callback which messes with the certificate chain
verification, simply let OpenSSL ignore errors during that phase and
check the result only afterwards.
2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
2745813367 re-design SSL/TLS configuration
the combinations of the various options made quite a mess. additionally,
'RequireSSL no' is inherently insecure - "use SSL if available" is plain
stupid.

the old options are still accepted, but will elicit a warning.
2014-10-18 16:18:48 +02:00
Oswald Buddenhagen
aa4f7a7d00 move use_imaps out of server_conf_t
it doesn't belong there - it's a property of imap_server_conf_t.
the port setup is now done while reading the config.

this makes socket.[hc] imap-agnostic.
2014-10-18 16:15:02 +02:00
Oswald Buddenhagen
3742fc475b deprecate the compat wrapper
after a decade, it's about time to phase it out.
2014-10-18 16:15:02 +02:00
Oswald Buddenhagen
47897d2403 fix memory management of current mailbox name
it was a stupid idea to store the pointer to a variable we need to
dispose in a structure which has its own lifetime.
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
4f383a8074 stop abusing memcmp()
memcmp() is unfortunately not guaranteed to read forward byte-by-byte,
which means that the clever use as a strncmp() without the pointless
strlen()s is not permitted, and can actually misbehave with
SSE-optimized string functions.

so implement proper equals() and starts_with() functions. as a bonus,
the calls are less cryptic.
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
003ddb2199 permit IMAP Stores with explicitly empty Path
this is useful if the server sends an unhelpful NAMESPACE like
"INBOX." (which precludes clean use of Patterns with the real INBOX).
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
f385355bdb don't attempt to filter LIST response if there is no Path
we won't filter anything in that case anyway.
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
608834c6f1 permit Maildir Stores without a Path
it is perfectly reasonable to have a Store which has only an Inbox.
2014-10-04 18:37:33 +02:00
Oswald Buddenhagen
6ad7371f46 use resolved Path for initial filtering of LIST response
otherwise we'd ignore NAMESPACE, and funny things could happen.
2014-10-04 18:37:33 +02:00
Oswald Buddenhagen
de82023427 consider unexpected structure of NAMESPACE fatal 2014-10-04 18:37:33 +02:00
Oswald Buddenhagen
2a2c53ae43 ignore INBOX only in Path itself, not its subfolders
a box foo/INBOX cannot be mistaken for the INBOX, so there is no point
in filtering these out.
2014-10-04 11:45:30 +02:00
Oswald Buddenhagen
313c9193f8 use \fB and \fI consistently
notably, the docu for the Sync option had it mixed up in the
description.
2014-07-12 14:08:34 +02:00
Oswald Buddenhagen
9a463768ea complain about RequireSSL with no SSL versions enabled 2014-07-06 10:10:12 +02:00
Oswald Buddenhagen
06c731cbf8 actually use STARTTLS if only TLSv1.1 or TLSv1.2 is enabled 2014-07-06 10:09:38 +02:00
Oswald Buddenhagen
95db373e54 clarify error message about missing connection details 2014-07-06 09:09:54 +02:00
Oswald Buddenhagen
834a65d85c make store/account error messages less redundant
this will become more relevant when more are added.
2014-07-06 09:09:54 +02:00
Oswald Buddenhagen
d9c78b7787 clarify effect of Tunnel on Host and Port 2014-07-06 09:09:54 +02:00
Oswald Buddenhagen
639c84ea28 don't ignore RequireSSL for PREAUTHenticated connections
such connections don't support STARTTLS. that is reasonable, as whatever
makes the connection preauthenticated (typically a Tunnel used to launch
imapd via a shell login) must already rely on the connection's security.
consequently, we would not try to use STARTTLS with such connections.
unfortunately, we'd also skip the RequireSSL check as a side effect.

this means that a rogue server (via a MITM attack) could simply offer a
preauthenticated connection to make us not use SSL, and thus bypass
server authentication. as a result, we could send potentially sensitive
data to the attacker:
- with Patterns used, we would send a LIST command which reveals the
  remote Path setting. this isn't very useful to an attacker. also, IMAP
  Accounts usually rely on the server-provided NAMESPACE to start with.
- with Create enabled for the remote Store, we would upload messages
  from newly appeared local folders. this isn't a very likely situation,
  unless the attacker manages to convince the victim to move/copy
  interesting mails to a new folder right before the attack.
- with Expunge enabled for the local Store, previously synchronized
  folders would be wiped. however, this would require the attacker to
  know the correct UIDVALIDITY of each remote folder, which would
  require incredible luck or convincing the victim to disclose them.
  the first mismatch would likely tip off the victim.

in practice, someone with the level of technical and social engineering
skills required for this attack would very likely find more attractive
attack vectors. therefore, i don't consider this a particularly serious
issue.

configurations with UseIMAPS enabled or using a secure Tunnel were not
affected to start with.

a side effect of this fix is that most users of Tunnel will now need to
explicitly set RequireSSL to false.
an alternative approach would be defaulting all SSL-related settings to
off when Tunnel is used. this would be too invasive for a patch release,
but i'll consider it for 1.2.

see also CVE-2014-2567 for the Trojita MUA.
2014-07-06 09:09:54 +02:00
Oswald Buddenhagen
526231bc22 initialize store_t::name
the field is marked foreign (for the drivers), so a recycled store may
contain an old pointer in it. that would make our error path crash.

REFMAIL: CAF_KswU7aBS7unnK+rdZy1PG_8SZUAW=tcg75HixDLLE0w3Lhw@mail.gmail.com
2014-07-02 08:50:22 +02:00
Oswald Buddenhagen
29b07ca7a6 actually print the faulty mailbox name, not some garbage
REFMAIL: CAF_KswU7aBS7unnK+rdZy1PG_8SZUAW=tcg75HixDLLE0w3Lhw@mail.gmail.com
2014-07-02 08:49:47 +02:00
Oswald Buddenhagen
3d5539bb63 detect inconsistent state of highest assigned UID
the highest assigned UID must always be at least as high as the highest
actually found UID, as otherwise we'd hand out duplicate UIDs at some
point. also, getting into such a state in the first place indicates some
potentially serious trouble, or at least external interference (e.g.,
moving/copying a message from another folder without giving it a
pristine filename).

REFMAIL: 20140626211831.GA11590@sie.protva.ru
2014-06-28 12:06:12 +02:00
Oswald Buddenhagen
8513358e0a zero-terminate imap literals
now that we properly support literals for strings, we must expect that
the consumer code will use them as strings.

amends fc77feacc.

discovered by Armands Liepins <armandsl@gmail.com>

REFMAIL: CAF_KswXoxdm7KXnWW4b_1odf=XsE4qRqRN4AsecwcPF1d+dSTA@mail.gmail.com
2014-06-28 11:04:41 +02:00
Oswald Buddenhagen
4ab12ae76e don't lie about the default of User
unlike the isync wrapper, mbsync does not have a default for the IMAP
user. the remote user seldomly matches the local one, so "forwarding" it
is more confusing than helpful.

CCMAIL: 744389@bugs.debian.org
2014-04-13 17:07:53 +02:00
Oswald Buddenhagen
4d8575100d don't forget to reset message counts when skipping scan
amends b6949c64d2.

CCMAIL: 744259@bugs.debian.org
2014-04-12 19:05:08 +02:00
Oswald Buddenhagen
8844ff3063 remove apparently pointless resetting of recent message count
past this point, it's not used for anything anyway.
2014-04-12 19:00:33 +02:00
Oswald Buddenhagen
532d964aea error-check renaming of uid mapping database
for pedantry.

found by coverity.
2014-04-12 18:59:45 +02:00
Oswald Buddenhagen
09db83809a error-check reading of old uidvalidity and maxuid files
found by coverity.
2014-04-12 18:34:26 +02:00
Oswald Buddenhagen
2d4bc1e613 error-check committing of sync state
a failure here is rather unlikely, but let's be pedantic.
a failure is not fatal (we'll just enter the journal replay path next
time), so only print warnings.

found by coverity.
2014-04-12 18:31:18 +02:00
Oswald Buddenhagen
aa0118d047 better error messages for sync state and journal related errors
we can make perfectly good use of errno here.
2014-04-12 18:30:09 +02:00
Oswald Buddenhagen
c6ddad6ac4 remove pointless/counterproductive "Disk full?" error message suffixes
the affected functions will set errno to ENOSPC when necessary.
2014-04-12 18:28:21 +02:00
Oswald Buddenhagen
d7d5fd20bc fix "inverse copy&pasto" in account labeling code
the code was copied and the original adjusted ... but not quite
completely.
this means that clashing server names never really worked since - not
that i would have expected this to be a particularly common
configuration to start with. :D

also added comments explaining why there are two implementations of the
same thing.

amends aea4be19e3 (anno 2006).

found by coverity.
2014-04-12 16:56:00 +02:00
Oswald Buddenhagen
9932352df0 assert !where implying !pseudo
to help poor coverity.
2014-04-12 16:06:33 +02:00
Oswald Buddenhagen
c5f2943ff6 don't crash in message expiration debug print
we would try to print the uids from the non-existing srec of unpaired
messages while preparing expiration.
this would happen only if a) MaxMessages was configured and b) new
messages appeared on the slave but we were not pushing, so it's a bit of
a corner case.

found by coverity.
2014-04-12 15:28:28 +02:00
Oswald Buddenhagen
31ba8375b0 fix segfault on passing --{create,expunge}-{master,slave}
stupid copy&pasto.

found by coverity.
2014-04-12 15:16:22 +02:00
Oswald Buddenhagen
ae49a37a3e don't crash on malformed response code
this would happen in the absurd corner case that the response code is
properly terminated with a closing bracket, but the atom itself is an
unterminated double-quoted string.

NOT found by coverity.
2014-04-12 15:02:40 +02:00
Oswald Buddenhagen
fd872a7ff7 don't crash on truncated LIST response
found by coverity.
2014-04-12 14:58:18 +02:00
Oswald Buddenhagen
0dfbf6f6fb remove pointless pointer assignment 2014-04-12 13:08:10 +02:00
Oswald Buddenhagen
d34baeb886 fix hypothetical buffer overflows
if something managed to make the maildir .uidvalidity files big enough
(possible only by appending garbage or scrambling them alltogether), we
would overflow the read buffer by one when appending the terminating
null.
this is not expected to have any real-world impact.

found by coverity.
2014-04-12 13:03:46 +02:00
Oswald Buddenhagen
df29c592d1 close a bunch of fd leaks in error paths
found by coverity.
2014-04-12 12:46:36 +02:00
Oswald Buddenhagen
dec5f73f57 actually use prime numbers for all hash bucket sizes
for some reason lost in history, the prime_deltas were actually wrong,
leading to using composite numbers.
the right sequence is available at http://oeis.org/A092131.
2014-03-19 10:27:06 +01:00