Commit Graph

325 Commits

Author SHA1 Message Date
Oswald Buddenhagen
66895f9cce try harder to list all necessary boxes
the pattern "INB*" may or may not refer to something in the INBOX. even
just "*" may. so list both the INBOX and the Path in case of
uncertainty.
2012-09-16 12:34:07 +02:00
Oswald Buddenhagen
35851f133b add option to control amount of fsync()ing 2012-09-15 15:28:15 +02:00
Oswald Buddenhagen
49223b2df2 avoid that a system crash can cause messages to be propagated twice
fdatasync() the journal after creating the pair record and recording
the TUID, but before the message propagation actually starts.

all other writes to the journal are not flushed, as they will at worst
cause some unnecessary network traffic without visible effect.
2012-09-15 15:28:15 +02:00
Oswald Buddenhagen
a326bf2f58 avoid that a system crash can lose mails
this fixes two possible failure scenarios:
- if the journal is committed but the mails are not, the missing files
  would be erroneously interpreted as deletions which would be
  propagated
- less seriously, if the mail files' meta data was committed but the
  file contents were not, we would end up with empty files, which would
  have to be re-fetched "behind mbsync's back" (just deleting the files
  would not work - see above)
2012-09-15 15:27:23 +02:00
Oswald Buddenhagen
df6c3b64b7 avoid that a system crash can clobber the sync state file
make sure that the new state is committed to disk before overwriting the
old version - by default meta data is committed first, so we may end up
with no valid state at all otherwise.
2012-09-15 13:25:50 +02:00
Oswald Buddenhagen
bbf98bb165 quote mailbox names written to config file 2012-09-15 11:57:14 +02:00
Oswald Buddenhagen
18936f6696 make more config file errors fatal
we really shouldn't just synchronize despite config parsing errors.
2012-09-15 11:49:24 +02:00
Oswald Buddenhagen
16e5aade3f store config error status in conffile_t object
this makes passing it around more straight-forward
2012-09-15 11:46:42 +02:00
Oswald Buddenhagen
725a122e91 make config parser a bit more careful about quotes
the parsing is more shell-like now:
- quoted and unquoted parts can be mixed in one argument
- the hashmark can be meaningfully quoted
2012-09-15 11:24:57 +02:00
Oswald Buddenhagen
2e07e68630 call fdatasync() after updating .uidvalidity files
they must be flushed before the file system meta data, as otherwise we
may end up with duplicate UIDs after a system crash.
2012-09-09 12:18:14 +02:00
Ben Kibbey
47fe4b7998 Fix certificate verification.
The connection state wasn't getting updated.
2012-09-08 14:26:22 +02:00
Oswald Buddenhagen
0a8f19294c pre-release doc updates 2012-09-01 21:15:53 +02:00
Oswald Buddenhagen
f11504aa07 update copyrights
make the wrapper's help string also mention copyrights pertaining only
to the actual syncer, as this is the only string many people will ever
see.
2012-09-01 21:15:53 +02:00
Oswald Buddenhagen
d4c786823d replace FSF address with something more ... contemporary 2012-09-01 21:15:53 +02:00
Oswald Buddenhagen
4e849196b8 install the config examples to docdir 2012-09-01 21:15:53 +02:00
Oswald Buddenhagen
6d49c343fc use a hash table for message => sync record lookup
this removes the pathological O(<number of sync records> * <number of
new messages>) case at the cost of being a bit more cpu-intensive (but
O(<number of all messages>)) for old messages.
2012-09-01 21:15:53 +02:00
Oswald Buddenhagen
18225344c6 make use of UID EXPUNGE 2012-09-01 21:15:52 +02:00
Oswald Buddenhagen
dfd7516b9a introduce ability to flatten the hierarchy of Stores 2012-09-01 21:15:52 +02:00
Oswald Buddenhagen
2585dd3324 add support for hierarchical mailboxes 2012-09-01 21:15:18 +02:00
Oswald Buddenhagen
4f94197e41 calculate trash box path already in maildir_open_store()
this gives us some cleaner code paths later on, as we can treat the
trash box like a regular mailbox.
2012-09-01 21:15:08 +02:00
Oswald Buddenhagen
8121224744 ensure that mailbox creation in maildir_store() is limited to trashing
other mailboxes would have been maildir_select()ed already.
2012-09-01 21:15:08 +02:00
Oswald Buddenhagen
a3f66f8f1d refactor: imap_select2_p2 => imap_refcounted_done_box
soon, we'll use it for something different, too
2012-09-01 21:15:08 +02:00
Oswald Buddenhagen
343f16771a don't crash when select() on master fails synchronously
svars->drv[S] would not be initialized yet, so cancel_sync() would
crash.
2012-09-01 21:15:08 +02:00
Oswald Buddenhagen
28cccf4b35 fix error handling of invalid SyncState *
when we find that the store is incompatible with in-store sync state,
we want to fail the whole channel. however, we must not claim that the
store died, otherwise it won't be disposed of properly.
2012-09-01 21:15:08 +02:00
Oswald Buddenhagen
1bc9c6d9cf reject qualified mailboxes with the magic name INBOX
otherwise we couldn't tell them apart from the real INBOX after
stripping away the Path.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
40f2812a41 suppress bdb complaints about unknown file format
pass DB_TRUNCATE when creating databases. otherwise bdb will complain
about the empty file we pass it (we have to create it upfront to
implement our locking).
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
9bbb02b8fd Revert "fix UIDNEXT handling"
in fact, UIDNEXT (and UIDVALIDITY) null is *not* allowed (see RFC3501
section 9). them popping up nonetheless was a dovecot bug (which would
also confuse dovecot itself).
having it in as a workaround was no good either, as quite some other
code in mbsync assumes that UIDs are not null.

This reverts commit e1fa867 and most of 39006d7.

-REFMAIL: 4CA62BA1.4020104@lemma.co.uk
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
233f563569 deal with concurrent maildir modifications during listing
files may be renamed (due to new -> cur transition or flag changes),
which may lead to two effects if ignored:
- we see both the old and the new name, so we report a spurious
  duplicate UID
- we see neither name, so we report a spurious deletion

as countermeasure, record and compare directory modification times. upon
mismatch, we just start over - as usual.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
3386285205 make maildir uidvalidity change fatal
it's best to give the user a chance to fix it rather than completely
messing up the syncstate by re-enumerating the UIDs.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
a3bd10c04d cleanup maildir error paths
don't try to unlock and close databases and files - this will happen a
moment later anyway, through cancelation or re-selection.
ironically, this plugs a memory leak, because an open main database is
used as a signal to close a temporary db in maildir_scan().
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
04ca97920d fix potential double free
the store may be discarded before we reach maildir_select() again, which
will leave us with a dangling pointer.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
e71ad53b7f plug memory leak in maildir_store_msg() upon failure to acquire UID 2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
9c86ec3442 employ alternative scheme to finding messages by TUID
instead of SEARCHing every single message (which is slow and happens to
be unreliabe with M$ Exchange 2010), just FETCH the new messages from
the mailbox - the ones we just appended will be amongst them.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
b4cef554fc clearer debug msg 2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
06b303da88 use ATTR_PRINTFALIKE 2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
7c815538ab fix line wrapping before info messages
unless an info message is explictly marked as a continuation, it must
terminate any pending line (typically the progress information) first.

debug output is not affected, as it is mutually exclusive with info
output, and no debug lines are left unterminated outside clear scopes.
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
2aae866e80 remove Ontty flag
i can't figure out why i added it in the first place. it doesn't seem to
make any sense ...
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
6b3b6f12bb centralize flushing of unfinished debug lines 2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
d2bed4990d unify error reporting
- introduce sys_error() and use it instead of perror() and
  error(strerror()) in all expected error conditions
- perror() is used only for "something's really wrong with the system"
  kind of errors
- file names, etc. are quoted if they are not validated yet, so e.g. an
  empty string becomes immediately obvious
- improve and unify language
- add missing newlines
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
dee9f51096 don't complain about disappearing temp files
some other process might be cleaning up concurrently ...
2012-09-01 21:15:07 +02:00
Oswald Buddenhagen
ae85e455d3 enlarge receive buffer considerably
the tiny buffer makes no sense in the face of huge payloads and now
additionally masses of replies from pipelined commands.
2012-09-01 21:15:06 +02:00
Oswald Buddenhagen
256a147945 compile with -ansi -pedantic on gcc
greatly helps portability ...
2012-09-01 21:15:06 +02:00
Oswald Buddenhagen
96eaeb428d define _GNU_SOURCE on the command line
that way it is already set in configure and can thus be used by tests.
2012-09-01 21:14:53 +02:00
Oswald Buddenhagen
faeb9b5bf7 centralize imap_cmd_refcounted_state refcount decrementing
no else branches remain, so the if() can be put into
imap_refcounted_done()
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
f5086f735c get rid of redundant literal_pending state flag 2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
bd93d689db fully asynchronous IMAP operation
- asynchronous sockets using an event loop
  - connect & starttls have completion callback parameters
  - callbacks for notification about filled input buffer and emptied
    output buffer
- unsent imap command queue
  - used when
    - socket output buffer is non-empty
    - number of commands in flight exceeds limit
    - last sent command requires round-trip
    - command has a dependency on completion of previous command
  - trashnc is tri-state so only a single "scout" trash APPEND/COPY is
    sent at first. a possibly resulting CREATE is injected in front of
    the remaining trash commands, so they can succeed (or be cancel()d
    if it fails).
  - queue's presence necessitates imap_cancel implementation
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
7867eb9009 add simple mainloop implementation
not used so far
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
a55354516b move responsibility for closing sockets on error to user
the only user being imap and the first thing in imap_cancel_store()
being a call to socket_close(), this code was pretty pointless anyway.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
802c99edcf make socket read/write error reporting callback-based
the functions still have synchronous return codes as well - this enables
early error returns without having to resort to refcounting.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
f1df2f40d1 decouple the filling of the read buffer from consuming it
this prepares the code for being called from a callback.

notably, this makes the imap list parser have a "soft stack", so the
recursion can be suspended at any time.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
886cd03e37 centralize imap_cmd disposal 2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
f8d73ac346 make socket_write() capable of taking ownership of the buffer
that way the user code doesn't have to free it any more.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
8a72d204c9 change socket_write() return code semantics
instead of returning a write()-like result, return only a binary status
code - write errors are handled internally anyway, so user code doesn't
have to check the write length.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
a85013d6ff make IMAP pipeline depth configurable
currently, this affects only "clustered" message listings and
flag stores.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
a266f28f1c cancel submitted commands when canceling store
we already have some minimal asynchronicity, so there might be commands
in flight when a fatal error comes in.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
171f7d6cd3 Socket_t + buffer_t => conn_t
remove the layering, in favor of a "buffered connection" abstraction.
2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
3447694c2b security fix: failure to load the certificate file is *not* OK ... 2012-09-01 16:03:36 +02:00
Oswald Buddenhagen
ef41349035 move socket code to a separate file
this makes the layering more obvious
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
d1ee94f02c move greeting response handling into get_cmd_result()
the primary purpose of this is getting rid of the "free-standing"
buffer_gets() call.
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
584e51ed7d docs
- insert "separator comments" between driver entry points
- document driver API
- document sync_vars_t parts that are stored in the sync state header
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
2ab689b3df make imap_exec() result reporting callback-based
this makes the IMAP command submission interface asynchronous.

the functions still have synchronous return codes as well - this enables
clean error return paths. only when we invoke callbacks we resort to
refcounting.

as a "side effect", properly sequence commands after CREATE resulting
from [TRYCREATE].
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
e5d323cc47 rely on the maildir's existence with "SyncState *"
now that we open the box first, we know that it will exist at this
point.
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
3169c59e10 validate maildirs more strictly
now that "SyncState *" won't create fake mailboxes any more, we can make
a full validity check again.
2012-09-01 16:03:35 +02:00
Oswald Buddenhagen
05fd0b9970 split out drv->load() from drv->select() 2012-09-01 16:02:50 +02:00
Oswald Buddenhagen
c741d5ffb5 make creation of trash folder independent from -C option
the trash is not a box which is synced, but a "byproduct" of
manipulating synced boxes, so it makes no sense to bind it to the same
option.
2012-07-30 01:21:32 +02:00
Oswald Buddenhagen
7addc3bea8 minor cleanup: use ctx->gen instead of gctx for consistency 2012-07-30 01:21:32 +02:00
Oswald Buddenhagen
424e0e7221 make callbacks return early when canceling
even after driver->cancel() the store may complete commands successfully.
return early in this case, so we don't attempt to continue syncing.
2012-07-30 01:21:32 +02:00
Oswald Buddenhagen
ea951a697f fix error paths wrt sync drivers, take 2
synchronous error codes which are passed through callbacks aren't a
particularly good idea, after all: latest when the callback does stuff
which does not concern the caller, the return code becomes ambiguous.
instead, protect the sync_vars object with a refcount when invoking
driver functions from loops, as the callbacks they call could invalidate
the object and we would have no way of knowing that the loop should be
aborted prematurely. the upcoming async imap driver will also need a
refcount to protect the cancelation marker of the imap socket dispatcher
loop.
2012-07-30 01:21:32 +02:00
Oswald Buddenhagen
ec8f440383 don't call cancel() repeatedly on a store
erroring command replies will trickle in even after canceling
2012-07-30 01:21:32 +02:00
Oswald Buddenhagen
b0bbd23512 replace DRV_STORE_BAD with a separate bad_callback()
that way we don't have to piggy-back (possibly asynchronous) fatal
errors to particular commands.

internally, the drivers still use synchronous return values as well,
so they don't try to access the invalidated store after calling back.
2012-07-30 01:21:31 +02:00
Oswald Buddenhagen
6d86e5347e don't access free'd memory in cancel_sync()
as it happens, the 1st round *may* trash svars - if we get the
cancelation request after the slave store has already died.
2012-07-29 12:26:38 +02:00
Oswald Buddenhagen
9554026443 make drv->cancel()'s callback have no status code
this function is not going to actually execute any commands, so it
makes no sense for the callback to have a status code.
2012-07-29 12:25:24 +02:00
Oswald Buddenhagen
57444e9df9 don't decode aux pointer on DRV_CANCELED
the aux data was already free()d by the error callback by the time we
get this status code.
2012-07-22 20:19:20 +02:00
Oswald Buddenhagen
06ccac1fdd always use return value from get_cmd_result()
once we have callback-based error reporting, this will ensure that we
don't operate on invalidated data structures.
2012-07-22 17:27:42 +02:00
Oswald Buddenhagen
121ce76e46 make response code parse failure of untagged OK/NO/BYE/BAD non-fatal
as such, it does not disrupt the data stream
2012-07-22 17:27:42 +02:00
Oswald Buddenhagen
d2e13f147c de-duplicate code a bit 2012-07-22 17:27:40 +02:00
Oswald Buddenhagen
4d4de6e275 remove redundant use_ssl variables
just use the presence of an SSL object as an indicator. if something
goes wrong during the ssl handshake or certificate validation, the
socket must be immediately closed anyway.
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
fd229040d8 DRV_SERVER_BAD is and will probably stay unused => trash 2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
17dc64b414 after [TRYCREATE], just resend the same command instead of cloning it 2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
83efbe327d use return values from correct set in get_cmd_result()
DRV_OK == RESP_OK, so this worked by accident
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
99cc328f17 do away with the dreaded rcaps hack
don't pretend that the server has no literal+ for the time of the
first relevant command's synchronous execution. instead, enable the
lower layer to do the processing by telling it for which commands
trashnc ("trash's existence not confirmed") is relevant.
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
1545ed90a0 purge imap_store_t::currentnc vestiges
we always actually open the mailbox before appending to it, so we
obviously know that it exists - that's why the code was already
commented out. changing this assumption would significantly complicate
matters for little gain, so let's just assume it won't happen.

consequently, also don't set param.create when appending to regular
mailboxes.
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
c66afdc0a8 move setting of ctx->listed outside the drivers
it's essentially an external state flag
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
7bab2d6d94 de-duplicate error paths
makes the code more compact. yay for gotos.
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
f6a25b331f check return value from close() after write()
otherwise we may lose data when quota is exceeded
or nfs is in a bad mood.
2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
61d98c5a1d fix a bunch of warnings 2012-07-07 18:00:47 +02:00
Oswald Buddenhagen
4afd31a457 avoid preprocessor warnings on missing features: #if => #ifdef 2011-03-27 12:06:41 +02:00
Oswald Buddenhagen
cf6a7b4d18 less bizarre code
we know that there is only one command in progress, so there
is no need to employ tricks to access the last command.
2011-03-27 11:54:48 +02:00
Oswald Buddenhagen
7e1c16ae02 make cram() sane
- don't silently fail in release mode (expression with side effects
  inside assert())
- save some redundand strlen()s by not throwing away known lengths
- reorganize the code for legibility
2011-03-27 11:54:48 +02:00
Oswald Buddenhagen
058d01f179 don't compare find_old_done with find_new_total
this didn't have any effect as no async drivers currently exist.
2011-03-27 11:54:48 +02:00
Oswald Buddenhagen
9e10e871fd don't hang if store cannot be opened asynchronously 2011-03-27 11:54:48 +02:00
Oswald Buddenhagen
2c729bf9e6 don't leak SSL objects 2011-03-27 11:54:45 +02:00
Oswald Buddenhagen
296ac0364c add CR after TUID during LF => CRLF conversion 2011-03-27 11:54:02 +02:00
Oswald Buddenhagen
8df1ebaf40 fix (another) out-of-bounds access in CRLF conversion
if the header contained no CRs but the body (or the post-TUID part of
the header) did, the TUID insertion would add an excess CR, thus
overflowing the buffer by one byte.
2011-03-27 11:54:02 +02:00
Oswald Buddenhagen
39006d7f24 document some breakage 2010-11-14 17:23:59 +01:00
Oswald Buddenhagen
d637772339 turns out, free(NULL) is just fine ... 2010-11-14 16:44:50 +01:00
Oswald Buddenhagen
e1fa867423 fix UIDNEXT handling
UIDNEXT *can* be legally zero, so deal with it.

-REFMAIL: 4CA62BA1.4020104@lemma.co.uk
2010-10-03 14:33:24 +02:00
Oswald Buddenhagen
5ade279839 don't hang after failed start_tls()
we'd send a LOGOUT command in plain text while the server was already
expecting an encrypted command, which would typically lead to waiting
for more data and thus an indefinite hang.
so close the socket immediately instead of letting the normal shutdown
path take care of it.
inspired by a patch by Steven Flintham.

-REFMAIL: 4C9AB98E.3000400@lemma.co.uk
2010-10-03 12:37:59 +02:00
Oswald Buddenhagen
70e87eb99e remove useless message
don't complain about missing greeting response - we already complained
about an unexpected EOF anyway.
2010-10-03 12:37:59 +02:00
Oswald Buddenhagen
90a38ea810 assert valid file handles
i've seen error logs of the type
  SSL_write: Bad file descriptor
and i simply can't nail it, so go for some more drastic measures.
2010-10-03 12:37:59 +02:00