Commit Graph

1216 Commits

Author SHA1 Message Date
Oswald Buddenhagen
bb632d1cd0 make UIDs unsigned
complies with the IMAP spec, thus removing the (not really) arbitrary
limitation to INT_MAX for UIDs.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
a0961d6505 delay assignment of TUID when propagating messages
go back to assigning TUIDs only right before actually propagating them.
this avoids spurious "TUID lost" warnings.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
bd5fb6fff3 move away from magic UIDs in the sync state
the only legitimate "deviant" UID is zero, meaning "no message". this
can be futher qualified by additional flags in the sync record, rather
than using magic values for the UID. in fact, the zero UID (so far
meaning only "expunged") was already optionally qualifed with "expired".

as a side effect, driver->store_msg() now returns 0 instead of -2 for
unknown UIDs. this was a hack to avoid translating the value later
on, but it made the api horrible, and now it's superflous in the first
place.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
4ffe149666 split off ephemeral sync record state to a separate member
this allows us to simplify logging of expiration, as we now can just log
the entire persistent state instead of fiddling with bits.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
efd72b85cc autotest: implement much more thorough resumption verification
the test will now make a test run for every journaled step, both right
before and right after the logging.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
7ce658d14c autotest: pre-assign all UIDs of the test messages
this ensures stable results when the boxes are used with different
OPEN_FLAGS (which will happen in a subsequent commit), at the negligible
cost of removing the implicit test of the maildir driver's ability to
enumerate new messages.
2017-04-22 11:26:12 +02:00
Oswald Buddenhagen
4cc5ad5a1a introduce driver call debugging
do that by wrapping the actual stores into proxies.

the proxy driver's code is auto-generated from function templates, some
parameters, and the declarations of the driver functions themselves.
attempts to do it with CPP macros turned out to be a nightmare.
2017-04-22 11:26:11 +02:00
Oswald Buddenhagen
bbe4567bce let driver_t::openbox_box() return the UID validity
... and make 'uidvalidity' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
48ad58b9a3 use a #define for invalid UIDVALIDITY 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
8d4918affd introduce get_uidnext() driver callback
... and make 'uidnext' private to the imap driver.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
683e581340 let driver_t::find_new_msgs() return the list of messages
consistently with driver_t::load_box().
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
74e9368121 let driver_t::load_box() return the list of messages
... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
6e32b88f3d let driver_t::list_store() return the list of boxes
... and make 'boxes' and 'listed' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
8b7d3792e4 factor out transform_refcounted_msg_response()
the missing cross of transform_refcounted_box_response() and
transform_msg_response().
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
39247197f9 make struct imap_cmd_refcounted_state "abstract"
take the callback out of it, so it can be individualized.
so far, this only increases code size ...
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
96b5ae8360 don't mess with the driver's mailbox list from outside
the api specifies that the list remains owned by the driver, so poking
around in it is ugly and risky.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
c886f71054 make driver_t::prepare_load_box() return the final options
... and make 'opts' private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
83ebe9022d introduce get_box_path() driver callback
... and make 'path' private to the maildir driver.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
d624c9af5d make set_bad_callback() a proper driver_t entry
... and make the pointers private to the drivers.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
f46cf8c887 provide a proper getter callback for driver capabilities
that way driver_t contains only callbacks.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
d54809e268 prepend "get_" to getters in driver_t
this makes it callbacks consistently start with a verb.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
36666f7e52 rewrite tracking of highest expired UID
so far, we tracked the slave side, and calculated the master side on the
fly. that complicated things unnecessarily.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
677accfd84 streamline syncing of old entries
the order of the conditionals was purely historical (pre 4ec56f8cf, anno
2005) and hard to follow, as were the comments.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
990c8a1404 sort uid exception list in a smarter place
do it closer to where it is populated. that way the debug output is
sorted, and we don't sort the list if it's known to be empty.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
72be55b0e3 add fallbacks for determining UIDNEXT
if the server sends no UIDNEXT, do an initial FETCH to query the UID of
the last message.

same if the server sends no APPENDUID.

this allows us to remove the arbitrary limitation of the UID range to
INT_MAX, at the cost of additional round-trips.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
416ced25dd fix repeated listing of same Store with different flags
multiple Channels can call driver_t::list_store() with different LIST_*
flags. assuming the flags are actually taken into consideration, using a
single boolean 'listed' flag to track whether the Store still needs to
be listed obviously wouldn't cut it - if INBOX does not live right under
Path and the Channels used entirely disjoint Patterns (say, * and
INBOX*), the second Channel in a single run (probably a Group) would
fail to match anything.

to fix this, make store_t::listed more granular. this also requires
moving its handling back into the drivers (thus reverting c66afdc0),
because the actually performed queries and their possible implicit
results are driver-specific.

note that this slightly pessimizes some cases - e.g., an IMAP Store with
Path "" will now list the entire namespace even if there is only one
Channel with Pattern "INBOX*" (because a hypothetical Pattern "*" would
also include INBOX*, and the queries are kept disjoint to avoid the need
for de-duplication). this isn't expected to be a problem, as listing
mailboxes is generally cheap.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
b9505301cc factor out listing Maildir++ Stores
Maildir++ is sufficiently different from the other SubFolder styles to
justify a separate function; the resulting code duplication is minimal,
but the separated functions are a lot clearer.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
9eaa97923a fix exclusion of bogus "INBOX.*" folders in Maildir++
this also adds code which avoids that the message about excluding the
mailbox is printed multiple times - this could happen with Maildir++, as
the hierarchy is flattened.

amends 0f24ca31b.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
2d1cfc2c7f make "Patterns *" match INBOX* in Maildir++ Stores
this also has the side effect that we won't fail to include INBOX itself
when Inbox is nested under Path when using other SubFolder styles
(regression introduced with Maildir++ support in 0f24ca31b).

REFMAIL: 1489492069.2148187.910409864.7727F9FC@webmail.messagingengine.com
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
763cb8341f make help screen print some more compile time options 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
28d52b2b18 de-obfuscate cmd_sendable()
split the monster conditional and add comments.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
0aa4c628df add comments 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
34993fbca6 fix sync resumption with aborted entries
we need a separate log entry type which does proper mmaxxuid tracking.

while moving code around, this also removes a redundant debug statement.

amends b1842617.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
887b2205ff remove nonsensical statement from journal replay of aborted entries
at this stage, entries cannot possibly have messages assigned to them,
so trying to unlink them makes no sense.

amends b1842617.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
5c2ce59217 fix sync resumption with re-newed messages
the UID of the entries needs to be bumped from -1 to -2, as otherwise
the resumed run would see a TUID in a sync entry which may not have one.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
7c466fc3e7 don't emit redundant flag updates for re-newed messages 2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
1ea2e69aa7 fix maxuid tracking
newmaxuid represents the highest UID for which a sync entry was created,
while maxuid represents the end of the range which is guaranteed to have
been propagated. that means that the former needs to be instantly
incremented (and logged), while the latter must not be touched until the
entire new message sync completes. this matters particularly in the case
of resuming an interrupted run, where sync entry creation must resume
exactly where it left off, while loading the box must use the old limit
to ensure that all messages are available for actual propagation.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
6705604c4a de-duplicate journal replay somewhat
we've been using indices to separate master/slave state for a long time,
so there is no point in using pairs of matching brackets to signify the
side in the journal. instead, use somewhat descriptive letters (S[een],
F[ind], T[rashed]) and the index itself.
2017-04-02 17:12:50 +02:00
Oswald Buddenhagen
af4b8896f0 use typedefs for structs more
makes the code more compact (and consistent, as typedefs were already
used in some places).
2017-04-02 15:56:30 +02:00
Oswald Buddenhagen
c3350753b0 factor out jFprintf() 2017-04-02 15:24:03 +02:00
Oswald Buddenhagen
1fdf793a3f fix signedness of 'nex' variables
they are derived from srec->status, which is unsigned. for not
understood reasons, the compiler complains only after extending status
to a full unsigned int.

on the way, localize the declarations.
2017-04-02 12:16:57 +02:00
Oswald Buddenhagen
1e939bafd8 don't use strncpy() to copy TUIDs
latest since 77acc268, the code prior to these statements ensures that
the full length is available, so just use memcpy(). the code for
comparing TUIDs uses memcmp() anyway.
2017-04-02 12:16:57 +02:00
Oswald Buddenhagen
d754608f55 autotest: improve valgrind integration
introduce recognition of $USE_VALGRIND to run all mbsync invocations
through valgrind.

this also removes the seemingly purposeless --log-fd=3 indirection.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
f29bed526b autotest: write logs to files
they can get long, and having actual files also helps with diffing.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
991e809c38 autotest: factor out readfile() function 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
2da7951fe0 autotest: de-duplicate error reporting paths 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
373abcef02 autotest: print consistent information for journal replay failures 2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
ca43c57e85 autotest: don't print expected result if the mbsync run itself fails
there isn't an actual result to compare with anyway.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
b4d1a05365 autotest: de-duplicate: use print*() in show*()
this actually adds some (minimal) transformation overhead, but it makes
the relationship clearer.
2017-04-01 20:32:36 +02:00
Oswald Buddenhagen
8aed94420f autotest: determine path of mbsync prior to chdir()
that allows tmp/ to be a symlink to a ramdisk.
2017-04-01 20:32:36 +02:00