Commit Graph

167 Commits

Author SHA1 Message Date
Oswald Buddenhagen
f934e995d6 don't populate sync record map with invalid UIDs
this would obviously just bloat the hash with nonsense, slowing down the
actual lookup later.
2017-03-14 11:36:25 +01:00
Oswald Buddenhagen
67f4aeff1f standardize on 'int' for message sizes
that's what the sources already assumed anyway. size_t is total
overkill, as No Email Ever (TM) will exceed 2GiB.

this also fixes a harmless format string warning in 32 bit builds.
2016-12-29 14:10:35 +01:00
Oswald Buddenhagen
8979ebbdf2 tolerate case changes in X-TUID header name
it is legal for an email system to simply change the case of rfc2822
headers, and at least one imap server apparently does just that.
this would lead to us not finding our own header, which is obviously not
helpful.

REFMAIL: CA+fD2U3hJEszmvwBsXEpTsaWgJ2Dh373mCESM3M0kg3ZwAYjaw@mail.gmail.com
2015-09-01 15:40:54 +02:00
Oswald Buddenhagen
0e1f8f9a3f revamp console output options
- the old meaning of -V[V] was moved to -D{n|N}, as these are really
  debugging options.
- don't print the info messages by default; this can be re-enabled with
  the -V switch, and is implied by most debug options (it was really
  kind of stupid that verbose/debug operation disabled these).
- the sync algo/state debugging can be separately enabled with -Ds now.
2015-03-30 10:31:26 +02:00
Oswald Buddenhagen
8aa22a62e7 make progress counters global
which means they are now cumulative, and include channels and boxes.
2015-03-30 10:30:35 +02:00
Oswald Buddenhagen
a8b26dc4ac soft-limit peak memory usage
propagating many messages from a fast store (typically maildir or a
local IMAP server) to a slow asynchronous store could cause gigabytes of
data being buffered. avoid this by throttling fetches if the target
context reports memory usage above a configurable limit.

REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 18:13:05 +01:00
Oswald Buddenhagen
d9a983add6 add support for propagating folder deletions 2015-01-17 17:51:20 +01:00
Oswald Buddenhagen
926788f3ae supplement open_box() with box existence information from list_store()
there is no point in trying to open a non-existing box before trying to
create it.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
7b7304b625 split create_box() off from open_box()
this allows us to do something else than creating missing boxes
depending on circumstances. hypothetically, that is.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
f1809ddd2b open the mailboxes after loading the sync state
this allows us to react differently to a box'es absence depending on the
state. hypothetically, so far.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
f43617cd94 lock sync state lazily
don't try to lock it until we actually read or write it.
the idea is to not fail with SyncState * if we tried to load the state
before selecting a non-existing mailbox. this is ok, because if the
mailbox is missing, we obviously have no sync state pertaining to it,
either.
as a side effect, this allows simplifying an error path.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
fb19d644f7 split off open_box() from select_box()
aka prepare_paths() reloaded. we'll need it in a moment.
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
97a42cd825 factor out {prepare,lock,save,load}_state() 2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
9982e7bf08 make some driver function names more descriptive 2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
00ebf45be2 rename driver::prepare_opts() => prepare_load()
... and move it to the right place in the structure and fix the doc to
not claim that it is called before select().
2015-01-11 15:05:29 +01:00
Oswald Buddenhagen
42cedc8f81 introduce uchar, ushort & uint typedefs 2015-01-11 15:05:28 +01:00
Oswald Buddenhagen
b730f66f7d Merge branch 'isync_1_1_branch' into HEAD
Conflicts:
	src/socket.c
2015-01-11 14:32:15 +01:00
Oswald Buddenhagen
2fa75cf159 fix UID assignment with some non-UIDPLUS servers
the seznam.cz IMAP server seems very eager to send UIDNEXT responses
despite not supporting UIDPLUS. this doesn't appear to be a particularly
sensible combination, but it's valid nonetheless.

however, that means that we need to save the UIDNEXT value before we
start storing messages, lest imap_find_new_msgs() will simply overlook
them. we do that outside the driver, in an already present field - this
actually makes the main path more consistent with the journal recovery
path.

analysis by Tomas Tintera <trosos@seznam.cz>.

REFMAIL: 20141220215032.GA10115@kyvadlo.trosos.seznam.cz
2015-01-11 14:29:19 +01:00
Oswald Buddenhagen
958af473a0 fix conditional for early failure in cancel_done() 2015-01-02 12:38:48 +01:00
Oswald Buddenhagen
f377e7b696 introduce FieldDelimiter and InfoDelimiter options
... for windows fs compatibility.

the maildir-specific InfoDelimiter inherits the global FieldDelimiter
(which affects SyncState), based on the assumption that if the sync
state is on a windows FS, the mailboxes certainly will be as well, while
the inverse is not necessarily true (when running on unix, anyway).

REFMAIL: <CA+m_8J1ynqAjHRJagvKt9sb31yz047Q7NH-ODRmHOKyfru8vtA@mail.gmail.com>
2014-10-25 17:42:48 +02:00
Oswald Buddenhagen
85fd5ceb54 move orig_name out of store_t
it's state specific to the synchronizer.
2014-10-25 15:06:50 +02:00
Oswald Buddenhagen
47897d2403 fix memory management of current mailbox name
it was a stupid idea to store the pointer to a variable we need to
dispose in a structure which has its own lifetime.
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
4f383a8074 stop abusing memcmp()
memcmp() is unfortunately not guaranteed to read forward byte-by-byte,
which means that the clever use as a strncmp() without the pointless
strlen()s is not permitted, and can actually misbehave with
SSE-optimized string functions.

so implement proper equals() and starts_with() functions. as a bonus,
the calls are less cryptic.
2014-10-04 18:37:34 +02:00
Oswald Buddenhagen
526231bc22 initialize store_t::name
the field is marked foreign (for the drivers), so a recycled store may
contain an old pointer in it. that would make our error path crash.

REFMAIL: CAF_KswU7aBS7unnK+rdZy1PG_8SZUAW=tcg75HixDLLE0w3Lhw@mail.gmail.com
2014-07-02 08:50:22 +02:00
Oswald Buddenhagen
29b07ca7a6 actually print the faulty mailbox name, not some garbage
REFMAIL: CAF_KswU7aBS7unnK+rdZy1PG_8SZUAW=tcg75HixDLLE0w3Lhw@mail.gmail.com
2014-07-02 08:49:47 +02:00
Oswald Buddenhagen
2d4bc1e613 error-check committing of sync state
a failure here is rather unlikely, but let's be pedantic.
a failure is not fatal (we'll just enter the journal replay path next
time), so only print warnings.

found by coverity.
2014-04-12 18:31:18 +02:00
Oswald Buddenhagen
aa0118d047 better error messages for sync state and journal related errors
we can make perfectly good use of errno here.
2014-04-12 18:30:09 +02:00
Oswald Buddenhagen
c6ddad6ac4 remove pointless/counterproductive "Disk full?" error message suffixes
the affected functions will set errno to ENOSPC when necessary.
2014-04-12 18:28:21 +02:00
Oswald Buddenhagen
c5f2943ff6 don't crash in message expiration debug print
we would try to print the uids from the non-existing srec of unpaired
messages while preparing expiration.
this would happen only if a) MaxMessages was configured and b) new
messages appeared on the slave but we were not pushing, so it's a bit of
a corner case.

found by coverity.
2014-04-12 15:28:28 +02:00
Oswald Buddenhagen
6d2fd370a6 fix _POSIX_SYNCHRONIZED_IO usage
it can be -1 for unsupported, or 0 for runtime detection (which we don't
do).
2014-01-02 21:09:09 +01:00
Oswald Buddenhagen
359091625d MaxMessages: ignore entries with no master while calculating bulk fetch 2013-12-13 15:38:50 +01:00
Oswald Buddenhagen
2bbd07ec87 adjust comments to new reality 2013-12-11 16:29:34 +01:00
Oswald Buddenhagen
5a21042e98 ensure sequencing of message propagation and store closing
by putting the message propagation last, d3f634702 uncovered a
long-standing problem: we might have closed the source store before all
messages were propagated from it.
2013-12-11 16:29:33 +01:00
Oswald Buddenhagen
c47ee1c8c4 fix error paths wrt sync drivers, take 3
msgs_copied() was not checked at all, and msgs_flags_set() was doing it
wrong (sync_close() was not checked).

instead of trying to fix/extend the msgs_flags_set() model (ref-counting
and cancelation checking in lower-level functions, and return values to
propagate the status), place the refs/derefs around higher-level scopes
and do the checking only there. this is effectively simpler, and does
away with some obscure macros.
2013-12-11 16:29:33 +01:00
Oswald Buddenhagen
03b3b566f1 reshuffle sources a bit
split header and move some code to more logical places.
2013-12-08 23:19:12 +01:00
Oswald Buddenhagen
71524cb6b0 reduce FSync option to a boolean
there is no use for Thorough mode any more, so simplify the
configuration.
2013-12-08 11:12:09 +01:00
Oswald Buddenhagen
29a56e2dc4 don't fsync after logging every TUID
as we now don't actually start propagating new messages until all TUIDs
have been generated, it's sufficient to sync just once. this makes it
a cheap operation, so we can do it at SYNC_NORMAL level already.
2013-12-08 11:12:09 +01:00
Oswald Buddenhagen
8d5bd62537 add ExpireUnread option 2013-12-08 11:12:09 +01:00
Oswald Buddenhagen
c0ba0c7ecf replace global_* with a channel_conf_t instance
this makes the (growing) list of getopt_helper()'s parameters
manageable. the few wasted bytes are worth it.
2013-12-08 11:12:09 +01:00
Oswald Buddenhagen
49a32910a7 move handling of new messages after that of old ones
i.e., move it back. whatever the original reason was, it's now gone.

this order is way more natural, which allows us to remove the osrecadd
and S_DONE hacks.
2013-12-01 13:36:28 +01:00
Oswald Buddenhagen
b1842617f7 make MaxMessages work for new mails as well
this helps enormously on the first sync of a 100k message box with a
limit of 1k messages. it also happens to make the syncing idempotent.

in a few conditionals we now explicitly test for max_messages being
enabled, not smaxxuid != 0, as after the initial fetch with no important
messages smaxxuid is zero, but we still have to skip over 99k messages
in the above case.
2013-12-01 13:36:28 +01:00
Oswald Buddenhagen
d3f6347021 delay propagation of new messages
previous sequence:
  examine & propagate new => examine old => propagate old
new sequence:
  examine new => examine old => propagate new => propagate old

this alone does not buy us much ...
2013-12-01 13:36:28 +01:00
Oswald Buddenhagen
391ec01f28 make message propagation recording less magic
assign the sync record to the source message asap, and later on rely
on a more explicit condition than not doing so.
2013-12-01 13:36:28 +01:00
Oswald Buddenhagen
7f784fd235 log maxuid bumping less aggressively
we can bump the internal variable whereever convenient, but we cannot
log it until we know that all messages were copied, as otherwise we
could miss some new messages after an interruption. with the new
approach, interruption would merely cause some additonal traffic.
2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
8b76412b0d document message expiration transactions 2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
ecb4c7ab07 propagate deletions with other flag changes
less code duplication, more logical order of issued driver commands
(especially after the next commit), and the "side effect" of letting the
message expiration code see those deletions if they are asynchronous.
2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
273ac899f3 don't delay loading master even if messages were expired
the delay optimized the corner case of previously important but now
expired messages on the slave disappearing, either through an external
expunge or after a journal replay. no point in pessimizing the common
case.
2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
12676f28da remove cleanup of expired entries during setup of master load
the removed code would only ever trigger if a) we were after a journal
replay or b) something external expunged the expired messages - both are
corner cases not worth the extra code.
however, this means that the syncing code further down now needs to take
care of these zombies.
in the end, the normal cleanup will take care of all expired entries,
new and old.
2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
9a62521cff micro-optimization/-clarification: swap condition order 2013-12-01 13:36:27 +01:00
Oswald Buddenhagen
014d9b9081 make message counting in expiration code less confusing 2013-12-01 13:36:27 +01:00