lobo/isync
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7d02d6c1fe
isync/src/sync.c

2424 lines
73 KiB
C
Raw Normal View History

convert licensing to SPDX while at it, add/fix some licenses/copyrights/comments: - it makes no sense to have a GPL exception in scripts - ted did not contribute to the man page - tst_timers is not part of the mbsync executable - explicitly put the build system under GPL and add copyrights
2022-06-16 10:44:35 +00:00
// SPDX-FileCopyrightText: 2000-2002 Michael R. Elkins <me@mutt.org>
// SPDX-FileCopyrightText: 2002-2022 Oswald Buddenhagen <ossi@users.sf.net>
// SPDX-License-Identifier: GPL-2.0-or-later WITH LicenseRef-isync-GPL-exception
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
/*
* mbsync - mailbox synchronizer
initial import
2000-12-20 21:41:21 +00:00
*/
turn debug() functions into macros this makes calling them with more expensive arguments efficient without wrapping them into additional conditionals.
2021-12-21 17:44:39 +00:00
#define DEBUG_FLAG DEBUG_SYNC
reshuffle sources a bit split header and move some code to more logical places.
2013-12-08 19:46:40 +00:00
#include "sync.h"
- make it work without SSL - switch from -Ds in Makefile to config.h - small header cleaup
2003-05-07 00:06:37 +00:00
initial import
2000-12-20 21:41:21 +00:00
#include <fcntl.h>
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
#include <ctype.h>
added BUGS section to manpage detailing the fact that we break the maildir(5) spec by parsing the filename change message delivery to use the method described in maildir(5)
2000-12-21 18:16:44 +00:00
#include <errno.h>
broke config code into config.c added support for uploading local messages with no UID to the IMAP server added Expunge configuration option added CopyDeletedTo configuration option
2001-01-11 10:13:47 +00:00
#include <sys/stat.h>
initial import
2000-12-20 21:41:21 +00:00
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
#define JOURNAL_VERSION "4"
reshuffle sources a bit split header and move some code to more logical places.
2013-12-08 19:46:40 +00:00
channel_conf_t global_conf;
channel_conf_t *channels;
group_conf_t *groups;
shuffle around global variables the new organization clarifies the ownership, making things more self-contained. this will potentially help with unit testing.
2022-04-24 12:30:10 +00:00
uint BufferLimit = 10 * 1024 * 1024;
int new_total[2], new_done[2];
int flags_total[2], flags_done[2];
int trash_total[2], trash_done[2];
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
const char *str_fn[] = { "far side", "near side" }, *str_hl[] = { "push", "pull" };
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
cope with out-of-disk-space halfways gracefully (that is, don't clobber the sync state including the journal, but exit immediately).
2004-11-13 09:19:36 +00:00
mark MAILBOX_DRIVER_FLAG locations in code Mailbox driver flags are defined in several places. It is essential that they are kept in sync, so mark them with the same string for easy grepping with an alerting boiler plate. Signed-off-by: Michael J Gruber <github@grubix.eu>
2018-06-21 14:52:00 +00:00
/* Keep the mailbox driver flag definitions in sync: */
/* grep for MAILBOX_DRIVER_FLAG */
/* The order is according to alphabetical maildir flag sort */
implement Forwarded flag maildir supports a 'P' flag which denotes the fact that a message has been 'passed' on (forwarded, bounced). notmuch syncs this to the 'passed' tag. Per https://tools.ietf.org/html/rfc5788, IMAP has a user-defined flag (keyword) '$Forwarded' that is supported by many servers and clients these days. (Technically, one should check for '$Forwarded' in the server response.) Restructure mbsync's flag parser to accept keywords (flags starting with '$') but still bail out on unknown system flags (flags starting with '\'). Support '$Forwarded' as a first keyword since it maps to maildir's 'P' and needs to be sorted in between the system flags. Signed-off-by: Michael J Gruber <github@grubix.eu>
2018-06-21 14:52:01 +00:00
static const char Flags[] = { 'D', 'F', 'P', 'R', 'S', 'T' };
initial import
2000-12-20 21:41:21 +00:00
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
static uchar
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
parse_flags( const char *buf )
initial import
2000-12-20 21:41:21 +00:00
{
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint i, d;
uchar flags;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
for (flags = i = d = 0; i < as(Flags); i++) {
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
if (buf[d] == Flags[i]) {
flags |= (1 << i);
d++;
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
return flags;
initial import
2000-12-20 21:41:21 +00:00
}
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
static uint
make_flags( uchar flags, char *buf )
updated year in copyright notice the uid for each message in the maildir is now stored in a dbm database rather than the filename. this change was necessary because isync became confused if you copied a message to another folder, in which case the uid was invalid. as a result of the above change, isync now acquires a mutex on the mailbox to protect the dbm database from concurrent access. main() was reworked to continue gracefully when an error is encountered, and to always call maildir_close() so that the lock can be disabled, and the database closed.
2002-01-16 19:47:28 +00:00
{
introduce uchar, ushort & uint typedefs
2014-12-07 12:19:30 +00:00
uint i, d;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
for (i = d = 0; i < as(Flags); i++)
if (flags & (1 << i))
buf[d++] = Flags[i];
buf[d] = 0;
return d;
updated year in copyright notice the uid for each message in the maildir is now stored in a dbm database rather than the filename. this change was necessary because isync became confused if you copied a message to another folder, in which case the uid was invalid. as a result of the above change, isync now acquires a mutex on the mailbox to protect the dbm database from concurrent access. main() was reworked to continue gracefully when an error is encountered, and to always call maildir_close() so that the lock can be disabled, and the database closed.
2002-01-16 19:47:28 +00:00
}
add/fix/de-duplicate comments
2019-11-17 18:45:00 +00:00
// This is the (mostly) persistent status of the sync record.
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
// Most of these bits are actually mutually exclusive. It is a
// bitfield to allow for easy testing for multiple states.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
#define S_EXPIRE (1<<0) // the entry is being expired (near side message removal scheduled)
#define S_EXPIRED (1<<1) // the entry is expired (near side message removal confirmed)
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
#define S_PENDING (1<<2) // the entry is new and awaits propagation (possibly a retry)
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
#define S_DUMMY(fn) (1<<(3+(fn))) // f/n message is only a placeholder
#define S_SKIPPED (1<<5) // pre-1.4 legacy: the entry was not propagated (message is too big)
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
#define S_DEAD (1<<7) // ephemeral: the entry was killed and should be ignored
// Ephemeral working set.
#define W_NEXPIRE (1<<0) // temporary: new expiration state
#define W_DELETE (1<<1) // ephemeral: flags propagation is a deletion
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
#define W_DEL(fn) (1<<(2+(fn))) // ephemeral: f/n message would be subject to expunge
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
#define W_UPGRADE (1<<4) // ephemeral: upgrading placeholder, do not apply MaxSize
#define W_PURGE (1<<5) // ephemeral: placeholder is being nuked
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
typedef struct sync_rec {
struct sync_rec *next;
/* string_list_t *keywords; */
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uid[2];
put message references into the sync records. match up the uids after opening the boxes instead of "sort-of-on-demand" - this is much simpler. match from messages to sync records, not the other way round - makes the debug output shorter, as the separate dump_box() is gone now.
2005-12-28 19:17:40 +00:00
message_t *msg[2];
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
uchar status, wstate, flags, pflags, aflags[2], dflags[2];
wrap message storing into transactions. nice side effect: drivers don't need to deal with line end conversion any move.
2006-02-03 21:33:43 +00:00
char tuid[TUIDL];
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
} sync_rec_t;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
typedef struct {
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
int t[2];
void (*cb)( int sts, void *aux ), *aux;
fix memory management of current mailbox name it was a stupid idea to store the pointer to a variable we need to dispose in a structure which has its own lifetime.
2014-10-04 16:26:10 +00:00
char *dname, *jname, *nname, *lname, *box_name[2];
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
FILE *jfp, *nfp;
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
sync_rec_t *srecs, **srecadd;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
channel_conf_t *chan;
store_t *ctx[2];
driver_t *drv[2];
move orig_name out of store_t it's state specific to the synchronizer.
2014-10-25 13:06:50 +00:00
const char *orig_name[2];
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
message_t *msgs[2], *new_msgs[2];
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint_array_alloc_t trashed_msgs[2];
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
int state[2], lfd, ret, existing, replayed;
uint ref_count, nsrecs, opts[2];
uint new_pending[2], flags_pending[2], trash_pending[2];
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint maxuid[2]; // highest UID that was already propagated
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
uint oldmaxuid[2]; // highest UID that was already propagated before this run
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uidval[2]; // UID validity value
uint newuidval[2]; // UID validity obtained from driver
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
uint finduid[2]; // TUID lookup makes sense only for UIDs >= this
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
uint maxxfuid; // highest expired UID on far side
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
uint oldmaxxfuid; // highest expired UID on far side before this run
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
uchar good_flags[2], bad_flags[2];
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
} sync_vars_t;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void sync_ref( sync_vars_t *svars ) { ++svars->ref_count; }
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void sync_deref( sync_vars_t *svars );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static int check_cancel( sync_vars_t *svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
#define AUX &svars->t[t]
fix TrashRemoteNew copy direction
2012-09-22 15:35:39 +00:00
#define INV_AUX &svars->t[1-t]
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
#define DECL_SVARS \
int t; \
sync_vars_t *svars
#define INIT_SVARS(aux) \
t = *(int *)aux; \
svars = (sync_vars_t *)(((char *)(&((int *)aux)[-t])) - offsetof(sync_vars_t, t))
#define DECL_INIT_SVARS(aux) \
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
int t = *(int *)aux; \
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
sync_vars_t *svars = (sync_vars_t *)(((char *)(&((int *)aux)[-t])) - offsetof(sync_vars_t, t))
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
/* operation dependencies:
adjust comments to new reality
2013-12-11 15:29:02 +00:00
select(x): -
load(x): select(x)
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
new(F), new(N), flags(F), flags(N): load(F) & load(N)
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
find_new(x): new(x)
trash(x): flags(x)
adjust comments to new reality
2013-12-11 15:29:02 +00:00
close(x): trash(x) & find_new(x) & new(!x) // with expunge
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
cleanup: close(F) & close(N)
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
*/
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
#define ST_LOADED (1<<0)
fix enum abuse amends 9c86ec344. S_FIND was for the sync record status field. it has no business in the sync vars status fields. its value coincided with ST_SELECTED, which luckily only means that we always tried to match up TUIDs even if there was nothing to do. the need for TUID matching arises in two mostly independent circumstances, so add two separate flags ST_FIND_{OLD,NEW}.
2013-11-02 11:57:39 +00:00
#define ST_FIND_OLD (1<<1)
#define ST_SENT_NEW (1<<2)
#define ST_FIND_NEW (1<<3)
#define ST_FOUND_NEW (1<<4)
#define ST_SENT_FLAGS (1<<5)
#define ST_SENT_TRASH (1<<6)
#define ST_CLOSED (1<<7)
#define ST_SENT_CANCEL (1<<8)
#define ST_CANCELED (1<<9)
#define ST_SELECTED (1<<10)
#define ST_DID_EXPUNGE (1<<11)
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
#define ST_CLOSING (1<<12)
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
#define ST_CONFIRMED (1<<13)
#define ST_PRESENT (1<<14)
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
#define ST_SENDING_NEW (1<<15)
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
static void
create_state( sync_vars_t *svars )
{
if (!(svars->nfp = fopen( svars->nname, "w" ))) {
sys_error( "Error: cannot create new sync state %s", svars->nname );
exit( 1 );
}
}
make more objects static
2019-07-28 18:54:01 +00:00
static void ATTR_PRINTFLIKE(2, 3)
factor out jFprintf()
2017-04-02 13:24:03 +00:00
jFprintf( sync_vars_t *svars, const char *msg, ... )
{
va_list va;
autotest: implement much more thorough resumption verification the test will now make a test run for every journaled step, both right before and right after the logging.
2017-03-19 10:53:16 +00:00
if (JLimit && !--JLimit)
exit( 101 );
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
if (!svars->jfp) {
create_state( svars );
if (!(svars->jfp = fopen( svars->jname, svars->replayed ? "a" : "w" ))) {
sys_error( "Error: cannot create journal %s", svars->jname );
exit( 1 );
}
setlinebuf( svars->jfp );
if (!svars->replayed)
Fprintf( svars->jfp, JOURNAL_VERSION "\n" );
}
factor out jFprintf()
2017-04-02 13:24:03 +00:00
va_start( va, msg );
vFprintf( svars->jfp, msg, va );
va_end( va );
autotest: implement much more thorough resumption verification the test will now make a test run for every journaled step, both right before and right after the logging.
2017-03-19 10:53:16 +00:00
if (JLimit && !--JLimit)
exit( 100 );
factor out jFprintf()
2017-04-02 13:24:03 +00:00
}
wrap jFprintf()+debug() into a macro this ensures that everything that is logged to the journal also appears in the debug output, and it makes the code less noisy.
2020-07-08 20:16:47 +00:00
#define JLOG_(log_fmt, log_args, dbg_fmt, ...) \
do { \
debug( "-> log: " log_fmt " (" dbg_fmt ")\n", __VA_ARGS__ ); \
jFprintf( svars, log_fmt "\n", deparen(log_args) ); \
} while (0)
#define JLOG3(log_fmt, log_args, dbg_fmt) \
JLOG_(log_fmt, log_args, dbg_fmt, deparen(log_args))
#define JLOG4(log_fmt, log_args, dbg_fmt, dbg_args) \
JLOG_(log_fmt, log_args, dbg_fmt, deparen(log_args), deparen(dbg_args))
#define JLOG_SEL(_1, _2, _3, _4, x, ...) x
#define JLOG(...) JLOG_SEL(__VA_ARGS__, JLOG4, JLOG3, NO_JLOG2, NO_JLOG1)(__VA_ARGS__)
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
static void
assign_uid( sync_vars_t *svars, sync_rec_t *srec, int t, uint uid )
{
srec->uid[t] = uid;
if (uid == svars->maxuid[t] + 1)
svars->maxuid[t] = uid;
srec->status &= ~S_PENDING;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
srec->wstate &= ~W_UPGRADE;
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
srec->tuid[0] = 0;
}
#define ASSIGN_UID(srec, t, nuid, ...) \
do { \
JLOG( "%c %u %u %u", ("<>"[t], srec->uid[F], srec->uid[N], nuid), __VA_ARGS__ ); \
assign_uid( svars, srec, t, nuid ); \
} while (0)
delay TUID assignment less we can't delay TUID assignment until after starting propagation if we want to provide both safety and performance. amends a0961d65.
2022-01-08 11:53:02 +00:00
static void
assign_tuid( sync_vars_t *svars, sync_rec_t *srec )
{
for (uint i = 0; i < TUIDL; i++) {
uchar c = arc4_getbyte() & 0x3f;
srec->tuid[i] = (char)(c < 26 ? c + 'A' : c < 52 ? c + 'a' - 26 :
c < 62 ? c + '0' - 52 : c == 62 ? '+' : '/');
}
JLOG( "# %u %u %." stringify(TUIDL) "s", (srec->uid[F], srec->uid[N], srec->tuid), "new TUID" );
}
remove pointless "lost track of ... messages" warnings it doesn't really add any value when resuming after an interruption, as incomplete propagations are perfectly expected, and are recovered from. but it does make sense in the non-UIDPLUS path, where not finding just stored messages indicates a problem. so move the reporting from match_tuids() to msgs_found_new(), and remove a redundant warning from the latter. amends 6577bf3e.
2022-02-02 12:06:43 +00:00
static int
let driver_t::find_new_msgs() return the list of messages consistently with driver_t::load_box().
2017-03-26 16:44:43 +00:00
match_tuids( sync_vars_t *svars, int t, message_t *msgs )
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
{
sync_rec_t *srec;
consistently use NULL for null pointers makes the code noisier, but also somewhat more expressive.
2019-07-28 18:50:31 +00:00
message_t *tmsg, *ntmsg = NULL;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
const char *diag;
warn if we cannot find some messages by TUID
2013-07-27 18:17:07 +00:00
int num_lost = 0;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
continue;
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
if (!srec->uid[t] && srec->tuid[0]) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "pair(%u,%u) TUID %." stringify(TUIDL) "s\n", srec->uid[F], srec->uid[N], srec->tuid );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
for (tmsg = ntmsg; tmsg; tmsg = tmsg->next) {
if (tmsg->status & M_DEAD)
continue;
if (tmsg->tuid[0] && !memcmp( tmsg->tuid, srec->tuid, TUIDL )) {
diag = (tmsg == ntmsg) ? "adjacently" : "after gap";
goto mfound;
}
}
let driver_t::find_new_msgs() return the list of messages consistently with driver_t::load_box().
2017-03-26 16:44:43 +00:00
for (tmsg = msgs; tmsg != ntmsg; tmsg = tmsg->next) {
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
if (tmsg->status & M_DEAD)
continue;
if (tmsg->tuid[0] && !memcmp( tmsg->tuid, srec->tuid, TUIDL )) {
diag = "after reset";
goto mfound;
}
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "& %u %u", (srec->uid[F], srec->uid[N]), "TUID lost" );
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
// Note: status remains S_PENDING.
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
srec->tuid[0] = 0;
warn if we cannot find some messages by TUID
2013-07-27 18:17:07 +00:00
num_lost++;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
continue;
mfound:
tmsg->srec = srec;
set srec->msg[] when finding messages by tuid otherwise we would propagate phantom deletions. this affected only sync runs after an interruption while storing messages, so it went (mostly?) unnoticed.
2013-11-09 10:42:09 +00:00
srec->msg[t] = tmsg;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
ntmsg = tmsg->next;
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
ASSIGN_UID( srec, t, tmsg->uid, "TUID matched %s", diag );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
}
}
remove pointless "lost track of ... messages" warnings it doesn't really add any value when resuming after an interruption, as incomplete propagations are perfectly expected, and are recovered from. but it does make sense in the non-UIDPLUS path, where not finding just stored messages indicates a problem. so move the reporting from match_tuids() to msgs_found_new(), and remove a redundant warning from the latter. amends 6577bf3e.
2022-02-02 12:06:43 +00:00
return num_lost;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
}
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
static uchar
sanitize_flags( uchar tflags, sync_vars_t *svars, int t )
{
if (!(DFlags & QUIET)) {
// We complain only once per flag per store - even though _theoretically_
// each mailbox can support different flags according to the IMAP spec.
uchar bflags = tflags & ~(svars->good_flags[t] | svars->bad_flags[t]);
if (bflags) {
char bfbuf[16];
make_flags( bflags, bfbuf );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
notice( "Notice: %s store does not support flag(s) '%s'; not propagating.\n", str_fn[t], bfbuf );
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
svars->bad_flags[t] |= bflags;
}
}
return tflags & svars->good_flags[t];
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
typedef struct copy_vars {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
void (*cb)( int sts, uint uid, struct copy_vars *vars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
void *aux;
sync_rec_t *srec; /* also ->tuid */
message_t *msg;
msg_data_t data;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
int minimal;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
} copy_vars_t;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void msg_fetched( int sts, void *aux );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
copy_msg( copy_vars_t *vars )
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_INIT_SVARS(vars->aux);
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
t ^= 1;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
vars->data.flags = vars->msg->flags;
added sync support for the arrival date of messages initial patch by Marc Hoersken <info@marc-hoersken.de>
2013-07-28 13:55:13 +00:00
vars->data.date = svars->chan->use_internal_date ? -1 : 0;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
svars->drv[t]->fetch_msg( svars->ctx[t], vars->msg, &vars->data, vars->minimal, msg_fetched, vars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
static void msg_stored( int sts, uint uid, void *aux );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
factor out copy_msg_bytes()
2016-11-05 17:32:49 +00:00
static void
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
copy_msg_bytes( char **out_ptr, const char *in_buf, uint *in_idx, uint in_len, int in_cr, int out_cr )
factor out copy_msg_bytes()
2016-11-05 17:32:49 +00:00
{
char *out = *out_ptr;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint idx = *in_idx;
factor out copy_msg_bytes()
2016-11-05 17:32:49 +00:00
if (out_cr != in_cr) {
char c;
if (out_cr) {
for (; idx < in_len; idx++) {
if ((c = in_buf[idx]) != '\r') {
if (c == '\n')
*out++ = '\r';
*out++ = c;
}
}
} else {
for (; idx < in_len; idx++) {
if ((c = in_buf[idx]) != '\r')
*out++ = c;
}
}
} else {
memcpy( out, in_buf + idx, in_len - idx );
out += in_len - idx;
idx = in_len;
}
*out_ptr = out;
*in_idx = idx;
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
static int
report conversion errors directly in copy_msg_convert() that makes it easier to report various conditions without introducing separate error codes.
2021-11-24 16:46:43 +00:00
copy_msg_convert( int in_cr, int out_cr, copy_vars_t *vars, int t )
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
{
char *in_buf = vars->data.data;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint in_len = vars->data.len;
CVE-2021-44143: don't overflow heap on messages without headers when a broken/compromised/malicious server gives us a message that starts with an empty line, we'd enter the path for inserting a pristine placeholder subject, for which we unfortunately didn't actually allocate space (unless MaxSize is in use and the message exceeds it). note that this cannot be triggered by merely receiving a crafted mail with no headers (yes, it's actually possible to send such a thing), as the delivery of mails adds plenty of headers. amends 70bad661.
2021-11-22 19:57:24 +00:00
uint idx = 0, sbreak = 0, ebreak = 0, break2 = UINT_MAX;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint lines = 0, hdr_crs = 0, bdy_crs = 0, app_cr = 0, extra = 0;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
uint add_subj = 0;
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
if (vars->srec) {
nloop: ;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint start = idx;
uint line_crs = 0;
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
while (idx < in_len) {
char c = in_buf[idx++];
if (c == '\r') {
line_crs++;
} else if (c == '\n') {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!ebreak && starts_with_upper( in_buf + start, (int)(in_len - start), "X-TUID: ", 8 )) {
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
extra = (sbreak = start) - (ebreak = idx);
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!vars->minimal)
goto oke;
} else {
fix read beyond end of input in copy_msg_convert() the input isn't necessarily null-terminated (it currently is for imap, but not for maildir), so if the message ended somewhere within the header field name, we'd read beyond its end, which theoretically could cause a crash. no other adverse effects could result, as we'd stop processing such a broken message right afterwards. amends 70bad661.
2021-11-26 22:05:30 +00:00
if (break2 == UINT_MAX && vars->minimal &&
starts_with_upper( in_buf + start, (int)(in_len - start), "SUBJECT:", 8 )) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
break2 = start + 8;
fix read beyond end of input in copy_msg_convert() the input isn't necessarily null-terminated (it currently is for imap, but not for maildir), so if the message ended somewhere within the header field name, we'd read beyond its end, which theoretically could cause a crash. no other adverse effects could result, as we'd stop processing such a broken message right afterwards. amends 70bad661.
2021-11-26 22:05:30 +00:00
if (break2 < in_len && in_buf[break2] == ' ')
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
break2++;
}
lines++;
hdr_crs += line_crs;
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
}
if (idx - line_crs - 1 == start) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!ebreak)
sbreak = ebreak = start;
if (vars->minimal) {
in_len = idx;
CVE-2021-44143: don't overflow heap on messages without headers when a broken/compromised/malicious server gives us a message that starts with an empty line, we'd enter the path for inserting a pristine placeholder subject, for which we unfortunately didn't actually allocate space (unless MaxSize is in use and the message exceeds it). note that this cannot be triggered by merely receiving a crafted mail with no headers (yes, it's actually possible to send such a thing), as the delivery of mails adds plenty of headers. amends 70bad661.
2021-11-22 19:57:24 +00:00
if (break2 == UINT_MAX) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
break2 = start;
add_subj = 1;
}
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
goto oke;
}
goto nloop;
}
}
report conversion errors directly in copy_msg_convert() that makes it easier to report various conditions without introducing separate error codes.
2021-11-24 16:46:43 +00:00
warn( "Warning: message %u from %s has incomplete header; skipping.\n",
vars->msg->uid, str_fn[1-t] );
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
free( in_buf );
return 0;
oke:
factor out app_cr
2016-11-05 17:33:47 +00:00
app_cr = out_cr && (!in_cr || hdr_crs);
extra += 8 + TUIDL + app_cr + 1;
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
}
if (out_cr != in_cr) {
for (; idx < in_len; idx++) {
char c = in_buf[idx];
if (c == '\r')
bdy_crs++;
else if (c == '\n')
lines++;
}
extra -= hdr_crs + bdy_crs;
if (out_cr)
extra += lines;
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
uint dummy_msg_len = 0;
char dummy_msg_buf[180];
static const char dummy_pfx[] = "[placeholder] ";
static const char dummy_subj[] = "Subject: [placeholder] (No Subject)";
static const char dummy_msg[] =
"Having a size of %s, this message is over the MaxSize limit.%s"
"Flag it and sync again (Sync mode ReNew) to fetch its real contents.%s";
if (vars->minimal) {
char sz[32];
if (vars->msg->size < 1024000)
sprintf( sz, "%dKiB", (int)(vars->msg->size >> 10) );
else
sprintf( sz, "%.1fMiB", vars->msg->size / 1048576. );
const char *nl = app_cr ? "\r\n" : "\n";
dummy_msg_len = (uint)sprintf( dummy_msg_buf, dummy_msg, sz, nl, nl );
extra += dummy_msg_len;
extra += add_subj ? strlen(dummy_subj) + app_cr + 1 : strlen(dummy_pfx);
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
vars->data.len = in_len + extra;
reject messages that grow too large due to conversion that shouldn't really be a problem, as we have 2GB of headroom, and most growth would happen when sending an all-newlines message from maildir to imap (due to CR additions), which is mostly non-critical. but better safe than sorry.
2021-11-24 16:51:06 +00:00
if (vars->data.len > INT_MAX) {
warn( "Warning: message %u from %s is too big after conversion; skipping.\n",
vars->msg->uid, str_fn[1-t] );
free( in_buf );
return 0;
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
char *out_buf = vars->data.data = nfmalloc( vars->data.len );
idx = 0;
if (vars->srec) {
CVE-2021-44143: don't overflow heap on messages without headers when a broken/compromised/malicious server gives us a message that starts with an empty line, we'd enter the path for inserting a pristine placeholder subject, for which we unfortunately didn't actually allocate space (unless MaxSize is in use and the message exceeds it). note that this cannot be triggered by merely receiving a crafted mail with no headers (yes, it's actually possible to send such a thing), as the delivery of mails adds plenty of headers. amends 70bad661.
2021-11-22 19:57:24 +00:00
if (break2 < sbreak) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
copy_msg_bytes( &out_buf, in_buf, &idx, break2, in_cr, out_cr );
memcpy( out_buf, dummy_pfx, strlen(dummy_pfx) );
out_buf += strlen(dummy_pfx);
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
copy_msg_bytes( &out_buf, in_buf, &idx, sbreak, in_cr, out_cr );
memcpy( out_buf, "X-TUID: ", 8 );
out_buf += 8;
memcpy( out_buf, vars->srec->tuid, TUIDL );
out_buf += TUIDL;
factor out app_cr
2016-11-05 17:33:47 +00:00
if (app_cr)
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
*out_buf++ = '\r';
*out_buf++ = '\n';
idx = ebreak;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
CVE-2021-44143: don't overflow heap on messages without headers when a broken/compromised/malicious server gives us a message that starts with an empty line, we'd enter the path for inserting a pristine placeholder subject, for which we unfortunately didn't actually allocate space (unless MaxSize is in use and the message exceeds it). note that this cannot be triggered by merely receiving a crafted mail with no headers (yes, it's actually possible to send such a thing), as the delivery of mails adds plenty of headers. amends 70bad661.
2021-11-22 19:57:24 +00:00
if (break2 != UINT_MAX && break2 >= sbreak) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
copy_msg_bytes( &out_buf, in_buf, &idx, break2, in_cr, out_cr );
if (!add_subj) {
memcpy( out_buf, dummy_pfx, strlen(dummy_pfx) );
out_buf += strlen(dummy_pfx);
} else {
memcpy( out_buf, dummy_subj, strlen(dummy_subj) );
out_buf += strlen(dummy_subj);
if (app_cr)
*out_buf++ = '\r';
*out_buf++ = '\n';
}
}
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
}
copy_msg_bytes( &out_buf, in_buf, &idx, in_len, in_cr, out_cr );
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (vars->minimal)
memcpy( out_buf, dummy_msg_buf, dummy_msg_len );
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
free( in_buf );
return 1;
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
msg_fetched( int sts, void *aux )
{
copy_vars_t *vars = (copy_vars_t *)aux;
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_SVARS;
factor out copy_msg_convert()
2016-11-05 17:33:16 +00:00
int scr, tcr;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
switch (sts) {
case DRV_OK:
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
INIT_SVARS(vars->aux);
make callbacks return early when canceling even after driver->cancel() the store may complete commands successfully. return early in this case, so we don't attempt to continue syncing.
2012-07-29 21:15:12 +00:00
if (check_cancel( svars )) {
free( vars->data.data );
vars->cb( SYNC_CANCELED, 0, vars );
return;
}
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
vars->msg->flags = vars->data.flags = sanitize_flags( vars->data.flags, svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
provide a proper getter callback for driver capabilities that way driver_t contains only callbacks.
2017-03-19 12:50:41 +00:00
scr = (svars->drv[1-t]->get_caps( svars->ctx[1-t] ) / DRV_CRLF) & 1;
tcr = (svars->drv[t]->get_caps( svars->ctx[t] ) / DRV_CRLF) & 1;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if (vars->srec || scr != tcr) {
report conversion errors directly in copy_msg_convert() that makes it easier to report various conditions without introducing separate error codes.
2021-11-24 16:46:43 +00:00
if (!copy_msg_convert( scr, tcr, vars, t )) {
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_NOGOOD, 0, vars );
return;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
svars->drv[t]->store_msg( svars->ctx[t], &vars->data, !vars->srec, msg_stored, vars );
break;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
case DRV_CANCELED:
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_CANCELED, 0, vars );
break;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
case DRV_MSG_BAD:
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_NOGOOD, 0, vars );
break;
comment updates
2021-12-08 12:55:33 +00:00
default: // DRV_BOX_BAD
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_FAIL, 0, vars );
break;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
msg_stored( int sts, uint uid, void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
copy_vars_t *vars = (copy_vars_t *)aux;
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_SVARS;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
switch (sts) {
case DRV_OK:
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_OK, uid, vars );
break;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
case DRV_CANCELED:
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_CANCELED, 0, vars );
break;
some more error reporting relating malformed messages
2010-02-06 09:32:10 +00:00
case DRV_MSG_BAD:
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
INIT_SVARS(vars->aux);
(void)svars;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
warn( "Warning: %s refuses to store message %u from %s.\n",
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
str_fn[t], vars->msg->uid, str_fn[1-t] );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_NOGOOD, 0, vars );
break;
comment updates
2021-12-08 12:55:33 +00:00
default: // DRV_BOX_BAD
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
vars->cb( SYNC_FAIL, 0, vars );
break;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
static void sync_bail( sync_vars_t *svars );
static void sync_bail2( sync_vars_t *svars );
introduce ability to flatten the hierarchy of Stores
2012-08-18 11:58:14 +00:00
static void sync_bail3( sync_vars_t *svars );
make drv->cancel()'s callback have no status code this function is not going to actually execute any commands, so it makes no sense for the callback to have a status code.
2012-07-22 15:32:32 +00:00
static void cancel_done( void *aux );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
static void
cancel_sync( sync_vars_t *svars )
{
clarify / micro-optimize cancel_sync()
2022-05-20 10:33:51 +00:00
int state1 = svars->state[1];
for (int t = 0; ; t++) {
make drv->cancel()'s callback have no status code this function is not going to actually execute any commands, so it makes no sense for the callback to have a status code.
2012-07-22 15:32:32 +00:00
if (svars->ret & SYNC_BAD(t)) {
cancel_done( AUX );
don't call cancel() repeatedly on a store erroring command replies will trickle in even after canceling
2011-04-03 14:29:18 +00:00
} else if (!(svars->state[t] & ST_SENT_CANCEL)) {
/* ignore subsequent failures from in-flight commands */
svars->state[t] |= ST_SENT_CANCEL;
make some driver function names more descriptive
2014-12-27 21:13:24 +00:00
svars->drv[t]->cancel_cmds( svars->ctx[t], cancel_done, AUX );
make drv->cancel()'s callback have no status code this function is not going to actually execute any commands, so it makes no sense for the callback to have a status code.
2012-07-22 15:32:32 +00:00
}
clarify / micro-optimize cancel_sync()
2022-05-20 10:33:51 +00:00
if (t || (state1 & ST_CANCELED))
don't access free'd memory in cancel_sync() as it happens, the 1st round *may* trash svars - if we get the cancelation request after the slave store has already died.
2012-07-22 15:46:54 +00:00
break;
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
static void
make drv->cancel()'s callback have no status code this function is not going to actually execute any commands, so it makes no sense for the callback to have a status code.
2012-07-22 15:32:32 +00:00
cancel_done( void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_INIT_SVARS(aux);
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
svars->state[t] |= ST_CANCELED;
if (svars->state[1-t] & ST_CANCELED) {
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (svars->nfp) {
avoid that a system crash can clobber the sync state file make sure that the new state is committed to disk before overwriting the old version - by default meta data is committed first, so we may end up with no valid state at all otherwise.
2012-09-15 11:25:50 +00:00
Fclose( svars->nfp, 0 );
Fclose( svars->jfp, 0 );
split out drv->load() from drv->select()
2011-07-23 14:06:32 +00:00
}
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
sync_bail( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
replace DRV_STORE_BAD with a separate bad_callback() that way we don't have to piggy-back (possibly asynchronous) fatal errors to particular commands. internally, the drivers still use synchronous return values as well, so they don't try to access the invalidated store after calling back.
2012-07-15 10:55:04 +00:00
static void
store_bad( void *aux )
{
DECL_INIT_SVARS(aux);
svars->drv[t]->cancel_store( svars->ctx[t] );
svars->ret |= SYNC_BAD(t);
cancel_sync( svars );
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static int
check_cancel( sync_vars_t *svars )
{
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
return (svars->state[F] | svars->state[N]) & (ST_SENT_CANCEL | ST_CANCELED);
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
static int
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
check_ret( int sts, void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_SVARS;
make callbacks return early when canceling even after driver->cancel() the store may complete commands successfully. return early in this case, so we don't attempt to continue syncing.
2012-07-29 21:15:12 +00:00
if (sts == DRV_CANCELED)
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
return 1;
make callbacks return early when canceling even after driver->cancel() the store may complete commands successfully. return early in this case, so we don't attempt to continue syncing.
2012-07-29 21:15:12 +00:00
INIT_SVARS(aux);
if (sts == DRV_BOX_BAD) {
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
svars->ret |= SYNC_FAIL;
cancel_sync( svars );
return 1;
}
make callbacks return early when canceling even after driver->cancel() the store may complete commands successfully. return early in this case, so we don't attempt to continue syncing.
2012-07-29 21:15:12 +00:00
return check_cancel( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
#define SVARS_CHECK_RET \
DECL_SVARS; \
if (check_ret( sts, aux )) \
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return; \
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
INIT_SVARS(aux)
#define SVARS_CHECK_RET_VARS(type) \
type *vars = (type *)aux; \
DECL_SVARS; \
if (check_ret( sts, vars->aux )) { \
free( vars ); \
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return; \
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
} \
INIT_SVARS(vars->aux)
#define SVARS_CHECK_CANCEL_RET \
DECL_SVARS; \
if (sts == SYNC_CANCELED) { \
free( vars ); \
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return; \
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
} \
INIT_SVARS(vars->aux)
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
static char *
clean_strdup( const char *s )
{
char *cs;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint i;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
cs = nfstrdup( s );
for (i = 0; cs[i]; i++)
if (cs[i] == '/')
cs[i] = '!';
return cs;
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
static sync_rec_t *
upgrade_srec( sync_vars_t *svars, sync_rec_t *srec )
{
// Create an entry and append it to the current one.
sync_rec_t *nsrec = nfcalloc( sizeof(*nsrec) );
nsrec->next = srec->next;
srec->next = nsrec;
if (svars->srecadd == &srec->next)
svars->srecadd = &nsrec->next;
// Move the placeholder to the new entry.
int t = (srec->status & S_DUMMY(F)) ? F : N;
nsrec->uid[t] = srec->uid[t];
srec->uid[t] = 0;
if (srec->msg[t]) { // NULL during journal replay; is assigned later.
nsrec->msg[t] = srec->msg[t];
nsrec->msg[t]->srec = nsrec;
srec->msg[t] = NULL;
}
// Mark the original entry for upgrade.
srec->status = (srec->status & ~(S_DUMMY(F)|S_DUMMY(N))) | S_PENDING;
srec->wstate |= W_UPGRADE;
// Mark the placeholder for nuking.
nsrec->wstate = W_PURGE;
nsrec->aflags[t] = F_DELETED;
return nsrec;
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
static int
prepare_state( sync_vars_t *svars )
split out drv->load() from drv->select()
2011-07-23 14:06:32 +00:00
{
char *s, *cmname, *csname;
channel_conf_t *chan;
chan = svars->chan;
replace global_* with a channel_conf_t instance this makes the (growing) list of getopt_helper()'s parameters manageable. the few wasted bytes are worth it.
2013-11-24 18:32:42 +00:00
if (!strcmp( chan->sync_state ? chan->sync_state : global_conf.sync_state, "*" )) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
const char *path = svars->drv[N]->get_box_path( svars->ctx[N] );
introduce get_box_path() driver callback ... and make 'path' private to the maildir driver.
2017-03-24 13:06:19 +00:00
if (!path) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
error( "Error: store '%s' does not support in-box sync state\n", chan->stores[N]->name );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
introduce get_box_path() driver callback ... and make 'path' private to the maildir driver.
2017-03-24 13:06:19 +00:00
nfasprintf( &svars->dname, "%s/." EXE "state", path );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
csname = clean_strdup( svars->box_name[N] );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (chan->sync_state) {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
nfasprintf( &svars->dname, "%s%s", chan->sync_state, csname );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else {
introduce FieldDelimiter and InfoDelimiter options ... for windows fs compatibility. the maildir-specific InfoDelimiter inherits the global FieldDelimiter (which affects SyncState), based on the assumption that if the sync state is on a windows FS, the mailboxes certainly will be as well, while the inverse is not necessarily true (when running on unix, anyway). REFMAIL: <CA+m_8J1ynqAjHRJagvKt9sb31yz047Q7NH-ODRmHOKyfru8vtA@mail.gmail.com>
2014-10-25 15:30:57 +00:00
char c = FieldDelimiter;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
cmname = clean_strdup( svars->box_name[F] );
introduce FieldDelimiter and InfoDelimiter options ... for windows fs compatibility. the maildir-specific InfoDelimiter inherits the global FieldDelimiter (which affects SyncState), based on the assumption that if the sync state is on a windows FS, the mailboxes certainly will be as well, while the inverse is not necessarily true (when running on unix, anyway). REFMAIL: <CA+m_8J1ynqAjHRJagvKt9sb31yz047Q7NH-ODRmHOKyfru8vtA@mail.gmail.com>
2014-10-25 15:30:57 +00:00
nfasprintf( &svars->dname, "%s%c%s%c%s_%c%s%c%s", global_conf.sync_state,
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
c, chan->stores[F]->name, c, cmname, c, chan->stores[N]->name, c, csname );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
free( cmname );
}
free( csname );
rely on the maildir's existence with "SyncState *" now that we open the box first, we know that it will exist at this point.
2011-07-24 18:27:09 +00:00
if (!(s = strrchr( svars->dname, '/' ))) {
unify error reporting - introduce sys_error() and use it instead of perror() and error(strerror()) in all expected error conditions - perror() is used only for "something's really wrong with the system" kind of errors - file names, etc. are quoted if they are not validated yet, so e.g. an empty string becomes immediately obvious - improve and unify language - add missing newlines
2011-04-10 13:32:25 +00:00
error( "Error: invalid SyncState location '%s'\n", svars->dname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
rely on the maildir's existence with "SyncState *" now that we open the box first, we know that it will exist at this point.
2011-07-24 18:27:09 +00:00
}
*s = 0;
if (mkdir( svars->dname, 0700 ) && errno != EEXIST) {
unify error reporting - introduce sys_error() and use it instead of perror() and error(strerror()) in all expected error conditions - perror() is used only for "something's really wrong with the system" kind of errors - file names, etc. are quoted if they are not validated yet, so e.g. an empty string becomes immediately obvious - improve and unify language - add missing newlines
2011-04-10 13:32:25 +00:00
sys_error( "Error: cannot create SyncState directory '%s'", svars->dname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
rely on the maildir's existence with "SyncState *" now that we open the box first, we know that it will exist at this point.
2011-07-24 18:27:09 +00:00
}
*s = '/';
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
nfasprintf( &svars->jname, "%s.journal", svars->dname );
nfasprintf( &svars->nname, "%s.new", svars->dname );
nfasprintf( &svars->lname, "%s.lock", svars->dname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 1;
}
static int
lock_state( sync_vars_t *svars )
{
struct flock lck;
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (svars->lfd >= 0)
return 1;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
memset( &lck, 0, sizeof(lck) );
#if SEEK_SET != 0
lck.l_whence = SEEK_SET;
#endif
#if F_WRLCK != 0
lck.l_type = F_WRLCK;
#endif
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if ((svars->lfd = open( svars->lname, O_WRONLY|O_CREAT, 0666 )) < 0) {
unify error reporting - introduce sys_error() and use it instead of perror() and error(strerror()) in all expected error conditions - perror() is used only for "something's really wrong with the system" kind of errors - file names, etc. are quoted if they are not validated yet, so e.g. an empty string becomes immediately obvious - improve and unify language - add missing newlines
2011-04-10 13:32:25 +00:00
sys_error( "Error: cannot create lock file %s", svars->lname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if (fcntl( svars->lfd, F_SETLK, &lck )) {
"fprintf( stderr," => "error(". new functions debugn() and infon() for messages with missing newline; warn() and error() act upon this.
2006-03-19 11:29:12 +00:00
error( "Error: channel :%s:%s-:%s:%s is locked\n",
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->chan->stores[F]->name, svars->orig_name[F], svars->chan->stores[N]->name, svars->orig_name[N] );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
close( svars->lfd );
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
svars->lfd = -1;
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 1;
}
static void
save_state( sync_vars_t *svars )
{
sync_rec_t *srec;
char fbuf[16]; /* enlarge when support for keywords is added */
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
// If no change was made, the state is also unmodified.
if (!svars->jfp && !svars->replayed)
return;
if (!svars->nfp)
create_state( svars );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
Fprintf( svars->nfp,
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
"FarUidValidity %u\nNearUidValidity %u\nMaxPulledUid %u\nMaxPushedUid %u\n",
svars->uidval[F], svars->uidval[N], svars->maxuid[F], svars->maxuid[N] );
if (svars->maxxfuid)
Fprintf( svars->nfp, "MaxExpiredFarUid %u\n", svars->maxxfuid );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
Fprintf( svars->nfp, "\n" );
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
continue;
make_flags( srec->flags, fbuf );
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
Fprintf( svars->nfp, "%u %u %s%s%s\n", srec->uid[F], srec->uid[N],
(srec->status & S_DUMMY(F)) ? "<" : (srec->status & S_DUMMY(N)) ? ">" : "",
purge handling of pending sync entries from state file these cannot actually end up in the committed state. amends bd5fb6ff.
2020-07-04 14:13:50 +00:00
(srec->status & S_SKIPPED) ? "^" : (srec->status & S_EXPIRED) ? "~" : "", fbuf );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
}
Fclose( svars->nfp, 1 );
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
if (svars->jfp)
Fclose( svars->jfp, 0 );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
if (!(DFlags & KEEPJOURNAL)) {
/* order is important! */
if (rename( svars->nname, svars->dname ))
warn( "Warning: cannot commit sync state %s\n", svars->dname );
else if (unlink( svars->jname ))
warn( "Warning: cannot delete journal %s\n", svars->jname );
}
}
static int
load_state( sync_vars_t *svars )
{
sync_rec_t *srec, *nsrec;
char *s;
FILE *jfp;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint ll;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
uint maxxnuid = 0;
use a temporary for sanity
2016-11-05 17:16:43 +00:00
char c;
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
struct stat st;
char fbuf[16]; /* enlarge when support for keywords is added */
char buf[128], buf1[64], buf2[64];
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if ((jfp = fopen( svars->dname, "r" ))) {
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (!lock_state( svars ))
goto jbail;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
debug( "reading sync state %s ...\n", svars->dname );
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
int line = 0;
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
while (fgets( buf, sizeof(buf), jfp )) {
line++;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if (!(ll = strlen( buf )) || buf[ll - 1] != '\n') {
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
error( "Error: incomplete sync state header entry at %s:%d\n", svars->dname, line );
jbail:
fclose( jfp );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if (ll == 1)
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
goto gothdr;
add/fix/de-duplicate comments
2019-11-17 18:45:00 +00:00
if (line == 1 && isdigit( buf[0] )) { // Pre-1.1 legacy
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
if (sscanf( buf, "%63s %63s", buf1, buf2 ) != 2 ||
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
sscanf( buf1, "%u:%u", &svars->uidval[F], &svars->maxuid[F] ) < 2 ||
sscanf( buf2, "%u:%u:%u", &svars->uidval[N], &maxxnuid, &svars->maxuid[N] ) < 3) {
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
error( "Error: invalid sync state header in %s\n", svars->dname );
goto jbail;
}
goto gothdr;
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uid;
if (sscanf( buf, "%63s %u", buf1, &uid ) != 2) {
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
error( "Error: malformed sync state header entry at %s:%d\n", svars->dname, line );
goto jbail;
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (!strcmp( buf1, "FarUidValidity" ) || !strcmp( buf1, "MasterUidValidity" ) /* Pre-1.4 legacy */) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->uidval[F] = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (!strcmp( buf1, "NearUidValidity" ) || !strcmp( buf1, "SlaveUidValidity" ) /* Pre-1.4 legacy */) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->uidval[N] = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (!strcmp( buf1, "MaxPulledUid" )) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->maxuid[F] = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (!strcmp( buf1, "MaxPushedUid" )) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->maxuid[N] = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (!strcmp( buf1, "MaxExpiredFarUid" ) || !strcmp( buf1, "MaxExpiredMasterUid" ) /* Pre-1.4 legacy */) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->maxxfuid = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (!strcmp( buf1, "MaxExpiredSlaveUid" )) { // Pre-1.3 legacy
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
maxxnuid = uid;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else {
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
error( "Error: unrecognized sync state header entry at %s:%d\n", svars->dname, line );
goto jbail;
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
make sync state header format less obscure the header is not space-critical, so use proper name-value pairs. this has the additional advantage that subsequent format changes can be done much easier.
2013-11-16 12:25:31 +00:00
error( "Error: unterminated sync state header in %s\n", svars->dname );
goto jbail;
gothdr:
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
while (fgets( buf, sizeof(buf), jfp )) {
line++;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if (!(ll = strlen( buf )) || buf[--ll] != '\n') {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: incomplete sync state entry at %s:%d\n", svars->dname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
cope with out-of-disk-space halfways gracefully (that is, don't clobber the sync state including the journal, but exit immediately).
2004-11-13 09:19:36 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
buf[ll] = 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
fbuf[0] = 0;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint t1, t2;
if (sscanf( buf, "%u %u %15s", &t1, &t2, fbuf ) < 2) {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: invalid sync state entry at %s:%d\n", svars->dname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
de-noise initialization of sync records use calloc() instead of malloc().
2019-12-03 12:06:12 +00:00
srec = nfcalloc( sizeof(*srec) );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
srec->uid[F] = t1;
srec->uid[N] = t2;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
s = fbuf;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (*s == '<') {
s++;
srec->status = S_DUMMY(F);
} else if (*s == '>') {
s++;
srec->status = S_DUMMY(N);
}
if (*s == '^') { // Pre-1.4 legacy
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
s++;
srec->status = S_SKIPPED;
} else if (*s == '~' || *s == 'X' /* Pre-1.3 legacy */) {
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
s++;
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
srec->status = S_EXPIRE | S_EXPIRED;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
} else if (srec->uid[F] == (uint)-1) { // Pre-1.3 legacy
srec->uid[F] = 0;
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
srec->status = S_SKIPPED;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
} else if (srec->uid[N] == (uint)-1) {
srec->uid[N] = 0;
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
srec->status = S_SKIPPED;
de-noise initialization of sync records use calloc() instead of malloc().
2019-12-03 12:06:12 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
srec->flags = parse_flags( s );
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
debug( " entry (%u,%u,%u,%s%s)\n", srec->uid[F], srec->uid[N], srec->flags,
(srec->status & S_SKIPPED) ? "SKIP" : (srec->status & S_EXPIRED) ? "XPIRE" : "",
(srec->status & S_DUMMY(F)) ? ",F-DUMMY" : (srec->status & S_DUMMY(N)) ? ",N-DUMMY" : "" );
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
*svars->srecadd = srec;
svars->srecadd = &srec->next;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
svars->nsrecs++;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
fclose( jfp );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->existing = 1;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
} else {
if (errno != ENOENT) {
better error messages for sync state and journal related errors we can make perfectly good use of errno here.
2014-04-12 16:30:09 +00:00
sys_error( "Error: cannot read sync state %s", svars->dname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->existing = 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// This is legacy support for pre-1.3 sync states.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (maxxnuid) {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint minwuid = UINT_MAX;
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((srec->status & (S_DEAD | S_SKIPPED | S_PENDING)) || !srec->uid[F])
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
continue;
if (srec->status & S_EXPIRED) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->uid[N]) {
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// The expired message was already gone.
continue;
}
// The expired message was not expunged yet, so re-examine it.
// This will happen en masse, so just extend the bulk fetch.
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (srec->uid[N] && maxxnuid >= srec->uid[N]) {
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// The non-expired message is in the generally expired range,
// so don't make it contribute to the bulk fetch.
continue;
}
// Usual non-expired message.
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (minwuid > srec->uid[F])
minwuid = srec->uid[F];
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->maxxfuid = minwuid - 1;
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
int line = 0;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if ((jfp = fopen( svars->jname, "r" ))) {
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (!lock_state( svars ))
goto jbail;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if (!stat( svars->nname, &st ) && fgets( buf, sizeof(buf), jfp )) {
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
debug( "recovering journal ...\n" );
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if (!(ll = strlen( buf )) || buf[--ll] != '\n') {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: incomplete journal header in %s\n", svars->jname );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
versioned journal. the commands and their meanings change all the time, so better handle that case. ps: yes, i think not upgrading mbsync between interrupting and resuming a run is a reasonable requirement.
2006-02-02 11:07:54 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
buf[ll] = 0;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
if (!equals( buf, (int)ll, JOURNAL_VERSION, strlen(JOURNAL_VERSION) )) {
"fprintf( stderr," => "error(". new functions debugn() and infon() for messages with missing newline; warn() and error() act upon this.
2006-03-19 11:29:12 +00:00
error( "Error: incompatible journal version "
null-terminate lines read from state file & journal makes the subsequent code less convoluted.
2016-11-13 21:19:40 +00:00
"(got %s, expected " JOURNAL_VERSION ")\n", buf );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
versioned journal. the commands and their meanings change all the time, so better handle that case. ps: yes, i think not upgrading mbsync between interrupting and resuming a run is a reasonable requirement.
2006-02-02 11:07:54 +00:00
}
consistently use NULL for null pointers makes the code noisier, but also somewhat more expressive.
2019-07-28 18:50:31 +00:00
srec = NULL;
versioned journal. the commands and their meanings change all the time, so better handle that case. ps: yes, i think not upgrading mbsync between interrupting and resuming a run is a reasonable requirement.
2006-02-02 11:07:54 +00:00
line = 1;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
while (fgets( buf, sizeof(buf), jfp )) {
line++;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if (!(ll = strlen( buf )) || buf[--ll] != '\n') {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: incomplete journal entry at %s:%d\n", svars->jname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
cope with out-of-disk-space halfways gracefully (that is, don't clobber the sync state including the journal, but exit immediately).
2004-11-13 09:19:36 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
buf[ll] = 0;
int tn;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
uint t1, t2, t3, t4;
use a temporary for sanity
2016-11-05 17:16:43 +00:00
if ((c = buf[0]) == '#' ?
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
(tn = 0, (sscanf( buf + 2, "%u %u %n", &t1, &t2, &tn ) < 2) || !tn || (ll - (uint)tn != TUIDL + 2)) :
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
c == '!' ?
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
(sscanf( buf + 2, "%u", &t1 ) != 1) :
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
c == 'N' || c == 'F' || c == 'T' || c == '+' || c == '&' || c == '-' || c == '=' || c == '_' || c == '|' ?
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
(sscanf( buf + 2, "%u %u", &t1, &t2 ) != 2) :
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
c != '^' ?
(sscanf( buf + 2, "%u %u %u", &t1, &t2, &t3 ) != 3) :
(sscanf( buf + 2, "%u %u %u %u", &t1, &t2, &t3, &t4 ) != 4))
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
{
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: malformed journal entry at %s:%d\n", svars->jname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (c == 'N') {
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
svars->maxuid[t1] = t2;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (c == 'F') {
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
svars->finduid[t1] = t2;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (c == 'T') {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
*uint_array_append( &svars->trashed_msgs[t1] ) = t2;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (c == '!') {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->maxxfuid = t1;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (c == '|') {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->uidval[F] = t1;
svars->uidval[N] = t2;
use a temporary for sanity
2016-11-05 17:16:43 +00:00
} else if (c == '+') {
de-noise initialization of sync records use calloc() instead of malloc().
2019-12-03 12:06:12 +00:00
srec = nfcalloc( sizeof(*srec) );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
srec->uid[F] = t1;
srec->uid[N] = t2;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
debug( " new entry(%u,%u)\n", t1, t2 );
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
srec->status = S_PENDING;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
*svars->srecadd = srec;
svars->srecadd = &srec->next;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
svars->nsrecs++;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
} else {
for (nsrec = srec; srec; srec = srec->next)
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (srec->uid[F] == t1 && srec->uid[N] == t2)
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
goto syncfnd;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
for (srec = svars->srecs; srec != nsrec; srec = srec->next)
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (srec->uid[F] == t1 && srec->uid[N] == t2)
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
goto syncfnd;
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: journal entry at %s:%d refers to non-existing sync state entry\n", svars->jname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
syncfnd:
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debugn( " entry(%u,%u,%u) ", srec->uid[F], srec->uid[N], srec->flags );
use a temporary for sanity
2016-11-05 17:16:43 +00:00
switch (c) {
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
case '-':
debug( "killed\n" );
srec->status = S_DEAD;
break;
fix sync resumption with aborted entries we need a separate log entry type which does proper mmaxxuid tracking. while moving code around, this also removes a redundant debug statement. amends b1842617.
2017-03-30 18:04:02 +00:00
case '=':
debug( "aborted\n" );
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
if (svars->maxxfuid < srec->uid[F])
svars->maxxfuid = srec->uid[F];
fix sync resumption with aborted entries we need a separate log entry type which does proper mmaxxuid tracking. while moving code around, this also removes a redundant debug statement. amends b1842617.
2017-03-30 18:04:02 +00:00
srec->status = S_DEAD;
break;
wrap message storing into transactions. nice side effect: drivers don't need to deal with line end conversion any move.
2006-02-03 21:33:43 +00:00
case '#':
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
memcpy( srec->tuid, buf + tn + 2, TUIDL );
print actually read TUID in debug message
2016-11-13 21:20:55 +00:00
debug( "TUID now %." stringify(TUIDL) "s\n", srec->tuid );
wrap message storing into transactions. nice side effect: drivers don't need to deal with line end conversion any move.
2006-02-03 21:33:43 +00:00
break;
case '&':
debug( "TUID %." stringify(TUIDL) "s lost\n", srec->tuid );
srec->tuid[0] = 0;
break;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
case '<':
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "far side now %u\n", t3 );
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
assign_uid( svars, srec, F, t3 );
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
break;
case '>':
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "near side now %u\n", t3 );
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
assign_uid( svars, srec, N, t3 );
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
break;
case '*':
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
debug( "flags now %u\n", t3 );
make some narrowing of integers explicit this does specifically *not* cover about a bazillion warnings about size_t being shrunk to uint - these make no sense given the expected data set size.
2019-07-28 20:10:21 +00:00
srec->flags = (uchar)t3;
comment updates
2021-12-08 12:55:33 +00:00
srec->aflags[F] = srec->aflags[N] = 0; // Clear F_DELETED from purge
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
srec->wstate &= ~W_PURGE;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
break;
case '~':
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
debug( "status now %#x\n", t3 );
make some narrowing of integers explicit this does specifically *not* cover about a bazillion warnings about size_t being shrunk to uint - these make no sense given the expected data set size.
2019-07-28 20:10:21 +00:00
srec->status = (uchar)t3;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
break;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
case '_':
debug( "has placeholder now\n" );
srec->status = S_PENDING; // Pre-1.4 legacy only
srec->status |= !srec->uid[F] ? S_DUMMY(F) : S_DUMMY(N);
break;
case '^':
debug( "is being upgraded, flags %u, srec flags %u\n", t3, t4 );
srec->pflags = (uchar)t3;
srec->flags = (uchar)t4;
srec = upgrade_srec( svars, srec );
break;
make the sync entry search in the journal replay wrap around at the end of the list. the "always forward" assumption is violated in some cases.
2006-02-02 10:44:19 +00:00
default:
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
error( "Error: unrecognized journal entry at %s:%d\n", svars->jname, line );
de-duplicate error paths makes the code more compact. yay for gotos.
2011-04-10 07:58:41 +00:00
goto jbail;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
}
}
}
fclose( jfp );
sort lists of trashed messages after journal replay the messages are trashed in mailbox (and thus UID) order, and in practice we expect the operations to complete in order. however, if older messages need to be trashed after a journal replay, and we get interrupted again, the next replay would produce an unsorted array, and thus break the binary search. amends 2bba9b9.
2021-12-22 19:13:53 +00:00
sort_uint_array( svars->trashed_msgs[F].array );
sort_uint_array( svars->trashed_msgs[N].array );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
} else {
if (errno != ENOENT) {
better error messages for sync state and journal related errors we can make perfectly good use of errno here.
2014-04-12 16:30:09 +00:00
sys_error( "Error: cannot read journal %s", svars->jname );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->replayed = line;
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
return 1;
}
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
static void
delete_state( sync_vars_t *svars )
{
unlink( svars->nname );
unlink( svars->jname );
if (unlink( svars->dname ) || unlink( svars->lname )) {
sys_error( "Error: channel %s: sync state cannot be deleted", svars->chan->name );
svars->ret = SYNC_FAIL;
}
}
fix signedness issues surrounding UIDs amends bb632d1c.
2020-07-08 11:45:06 +00:00
static void box_confirmed( int sts, uint uidvalidity, void *aux );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
static void box_confirmed2( sync_vars_t *svars, int t );
static void box_deleted( int sts, void *aux );
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
static void box_created( int sts, void *aux );
fix signedness issues surrounding UIDs amends bb632d1c.
2020-07-08 11:45:06 +00:00
static void box_opened( int sts, uint uidvalidity, void *aux );
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
static void box_opened2( sync_vars_t *svars, int t );
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
static void load_box( sync_vars_t *svars, int t, uint minwuid, uint_array_t mexcs );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
void
constness fixes add missing const qualifications, and add "const cast" suppressions where unavoidable.
2019-07-28 19:13:28 +00:00
sync_boxes( store_t *ctx[], const char * const names[], int present[], channel_conf_t *chan,
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
void (*cb)( int sts, void *aux ), void *aux )
{
sync_vars_t *svars;
int t;
svars = nfcalloc( sizeof(*svars) );
svars->t[1] = 1;
svars->ref_count = 1;
svars->cb = cb;
svars->aux = aux;
svars->ctx[0] = ctx[0];
svars->ctx[1] = ctx[1];
svars->chan = chan;
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
svars->lfd = -1;
use a #define for invalid UIDVALIDITY
2017-01-29 14:39:36 +00:00
svars->uidval[0] = svars->uidval[1] = UIDVAL_BAD;
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->srecadd = &svars->srecs;
for (t = 0; t < 2; t++) {
svars->orig_name[t] =
(!names[t] || (ctx[t]->conf->map_inbox && !strcmp( ctx[t]->conf->map_inbox, names[t] ))) ?
"INBOX" : names[t];
make map_name() interpret empty strings as "no separator" empty strings were previously meaningless, and starting with 72c2d695a, failure to handle them lead to bogus results when the IMAP hierarchy separator is legitimately empty (when the server genuinely supports none and none is manually configured). non-null can be asserted more cleanly than null-or-non-empty, so change the api like that. incidentally, this also removes the need to work around gcc's bogus warning in -Os mode. problem found by "Casper Ti. Vector" <caspervector@gmail.com>
2017-10-15 14:14:20 +00:00
if (!ctx[t]->conf->flat_delim[0]) {
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->box_name[t] = nfstrdup( svars->orig_name[t] );
} else if (map_name( svars->orig_name[t], &svars->box_name[t], 0, "/", ctx[t]->conf->flat_delim ) < 0) {
error( "Error: canonical mailbox name '%s' contains flattened hierarchy delimiter\n", svars->orig_name[t] );
support verbatim and real Maildir++ subfolder naming styles the legacy style is a poorly executed attempt at Maildir++, so introduce the latter for the sake of completeness. but most users will probably just want to use subfolders without any additional dots.
2015-05-01 17:16:23 +00:00
bail3:
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->ret = SYNC_FAIL;
sync_bail3( svars );
return;
}
introduce driver call debugging do that by wrapping the actual stores into proxies. the proxy driver's code is auto-generated from function templates, some parameters, and the declarations of the driver functions themselves. attempts to do it with CPP macros turned out to be a nightmare.
2017-04-02 12:57:17 +00:00
svars->drv[t] = ctx[t]->driver;
make set_bad_callback() a proper driver_t entry ... and make the pointers private to the drivers.
2017-03-21 18:27:04 +00:00
svars->drv[t]->set_bad_callback( ctx[t], store_bad, AUX );
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
}
/* Both boxes must be fully set up at this point, so that error exit paths
* don't run into uninitialized variables. */
split off open_box() from select_box() aka prepare_paths() reloaded. we'll need it in a moment.
2014-12-27 22:39:55 +00:00
for (t = 0; t < 2; t++) {
support verbatim and real Maildir++ subfolder naming styles the legacy style is a poorly executed attempt at Maildir++, so introduce the latter for the sake of completeness. but most users will probably just want to use subfolders without any additional dots.
2015-05-01 17:16:23 +00:00
switch (svars->drv[t]->select_box( ctx[t], svars->box_name[t] )) {
use more appropriate return value in driver_t::select_box() don't say DRV_CANCELED when it's really DRV_STORE_BAD, as apart from being just wrong, it lead to the confusing effect of canceling a store as the result of a supposed cancellation of the same store.
2020-08-03 22:23:33 +00:00
case DRV_STORE_BAD:
split off open_box() from select_box() aka prepare_paths() reloaded. we'll need it in a moment.
2014-12-27 22:39:55 +00:00
store_bad( AUX );
return;
support verbatim and real Maildir++ subfolder naming styles the legacy style is a poorly executed attempt at Maildir++, so introduce the latter for the sake of completeness. but most users will probably just want to use subfolders without any additional dots.
2015-05-01 17:16:23 +00:00
case DRV_BOX_BAD:
goto bail3;
split off open_box() from select_box() aka prepare_paths() reloaded. we'll need it in a moment.
2014-12-27 22:39:55 +00:00
}
}
open the mailboxes after loading the sync state this allows us to react differently to a box'es absence depending on the state. hypothetically, so far.
2015-01-03 22:16:20 +00:00
if (!prepare_state( svars )) {
svars->ret = SYNC_FAIL;
sync_bail2( svars );
return;
}
if (!load_state( svars )) {
svars->ret = SYNC_FAIL;
sync_bail( svars );
return;
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
sync_ref( svars );
open the mailboxes after loading the sync state this allows us to react differently to a box'es absence depending on the state. hypothetically, so far.
2015-01-03 22:16:20 +00:00
for (t = 0; ; t++) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
info( "Opening %s box %s...\n", str_fn[t], svars->orig_name[t] );
supplement open_box() with box existence information from list_store() there is no point in trying to open a non-existing box before trying to create it.
2014-12-30 14:16:38 +00:00
if (present[t] == BOX_ABSENT)
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
box_confirmed2( svars, t );
supplement open_box() with box existence information from list_store() there is no point in trying to open a non-existing box before trying to create it.
2014-12-30 14:16:38 +00:00
else
svars->drv[t]->open_box( ctx[t], box_confirmed, AUX );
open the mailboxes after loading the sync state this allows us to react differently to a box'es absence depending on the state. hypothetically, so far.
2015-01-03 22:16:20 +00:00
if (t || check_cancel( svars ))
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
break;
}
sync_deref( svars );
}
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
static void
fix signedness issues surrounding UIDs amends bb632d1c.
2020-07-08 11:45:06 +00:00
box_confirmed( int sts, uint uidvalidity, void *aux )
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
{
DECL_SVARS;
if (sts == DRV_CANCELED)
return;
INIT_SVARS(aux);
if (check_cancel( svars ))
return;
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
if (sts == DRV_OK) {
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
svars->state[t] |= ST_PRESENT;
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
svars->newuidval[t] = uidvalidity;
}
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
box_confirmed2( svars, t );
}
static void
box_confirmed2( sync_vars_t *svars, int t )
{
svars->state[t] |= ST_CONFIRMED;
if (!(svars->state[1-t] & ST_CONFIRMED))
return;
sync_ref( svars );
for (t = 0; ; t++) {
if (!(svars->state[t] & ST_PRESENT)) {
if (!(svars->state[1-t] & ST_PRESENT)) {
if (!svars->existing) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
error( "Error: channel %s: both far side %s and near side %s cannot be opened.\n",
svars->chan->name, svars->orig_name[F], svars->orig_name[N] );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
bail:
svars->ret = SYNC_FAIL;
} else {
/* This can legitimately happen if a deletion propagation was interrupted.
* We have no place to record this transaction, so we just assume it.
* Of course this bears the danger of clearing the state if both mailboxes
* temorarily cannot be opened for some weird reason (while the stores can). */
delete_state( svars );
}
done:
sync_bail( svars );
break;
}
if (svars->existing) {
if (!(svars->chan->ops[1-t] & OP_REMOVE)) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
error( "Error: channel %s: %s box %s cannot be opened.\n",
svars->chan->name, str_fn[t], svars->orig_name[t] );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
goto bail;
}
if (svars->drv[1-t]->confirm_box_empty( svars->ctx[1-t] ) != DRV_OK) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
warn( "Warning: channel %s: %s box %s cannot be opened and %s box %s is not empty.\n",
svars->chan->name, str_fn[t], svars->orig_name[t], str_fn[1-t], svars->orig_name[1-t] );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
goto done;
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
info( "Deleting %s box %s...\n", str_fn[1-t], svars->orig_name[1-t] );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
svars->drv[1-t]->delete_box( svars->ctx[1-t], box_deleted, INV_AUX );
} else {
if (!(svars->chan->ops[t] & OP_CREATE)) {
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
box_opened( DRV_BOX_BAD, UIDVAL_BAD, AUX );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
info( "Creating %s box %s...\n", str_fn[t], svars->orig_name[t] );
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
svars->drv[t]->create_box( svars->ctx[t], box_created, AUX );
}
}
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
} else {
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
box_opened2( svars, t );
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
}
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
if (t || check_cancel( svars ))
break;
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
}
add support for propagating folder deletions
2014-12-29 01:08:48 +00:00
sync_deref( svars );
}
static void
box_deleted( int sts, void *aux )
{
DECL_SVARS;
if (check_ret( sts, aux ))
return;
INIT_SVARS(aux);
delete_state( svars );
svars->drv[t]->finish_delete_box( svars->ctx[t] );
sync_bail( svars );
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
}
static void
box_created( int sts, void *aux )
{
DECL_SVARS;
if (check_ret( sts, aux ))
return;
INIT_SVARS(aux);
svars->drv[t]->open_box( svars->ctx[t], box_opened, AUX );
}
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
static void
fix signedness issues surrounding UIDs amends bb632d1c.
2020-07-08 11:45:06 +00:00
box_opened( int sts, uint uidvalidity, void *aux )
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
{
DECL_SVARS;
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
if (sts == DRV_CANCELED)
return;
INIT_SVARS(aux);
if (check_cancel( svars ))
return;
if (sts == DRV_BOX_BAD) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
error( "Error: channel %s: %s box %s cannot be opened.\n",
svars->chan->name, str_fn[t], svars->orig_name[t] );
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
svars->ret = SYNC_FAIL;
sync_bail( svars );
} else {
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
svars->newuidval[t] = uidvalidity;
split create_box() off from open_box() this allows us to do something else than creating missing boxes depending on circumstances. hypothetically, that is.
2014-12-29 00:42:17 +00:00
box_opened2( svars, t );
}
}
static void
box_opened2( sync_vars_t *svars, int t )
{
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
store_t *ctx[2];
channel_conf_t *chan;
open the mailboxes after loading the sync state this allows us to react differently to a box'es absence depending on the state. hypothetically, so far.
2015-01-03 22:16:20 +00:00
sync_rec_t *srec;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint_array_alloc_t mexcs;
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
uint opts[2], fails, minwuid;
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
svars->state[t] |= ST_SELECTED;
if (!(svars->state[1-t] & ST_SELECTED))
return;
open the mailboxes after loading the sync state this allows us to react differently to a box'es absence depending on the state. hypothetically, so far.
2015-01-03 22:16:20 +00:00
ctx[0] = svars->ctx[0];
ctx[1] = svars->ctx[1];
chan = svars->chan;
split out drv->load() from drv->select()
2011-07-23 14:06:32 +00:00
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
fails = 0;
split out drv->load() from drv->select()
2011-07-23 14:06:32 +00:00
for (t = 0; t < 2; t++)
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
if (svars->uidval[t] != UIDVAL_BAD && svars->uidval[t] != svars->newuidval[t])
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
fails++;
make UIDVALIDITY recovery more strict about vanished messages in particular, this covers the case of a mailbox being replaced with an empty new one, which would subsequently lead to the opposite end being emptied as well, which would typically be undesired. also add plenty of comments.
2021-02-20 21:52:49 +00:00
// If only one side changed UIDVALIDITY, we will try to re-approve it further down.
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
if (fails == 2) {
improve error messages about irrecoverably changed UIDVALIDITY don't print the actual values, which are meaningless technicalities to the average user, and can be obtained separately for debugging if really necessary. also, fix the omission of the affected mailboxes from one of the messages.
2021-02-20 21:52:01 +00:00
error( "Error: channel %s: UIDVALIDITY of both far side %s and near side %s changed.\n",
svars->chan->name, svars->orig_name[F], svars->orig_name[N]);
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
bail:
svars->ret = SYNC_FAIL;
sync_bail( svars );
return;
}
split out drv->load() from drv->select()
2011-07-23 14:06:32 +00:00
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (!lock_state( svars ))
goto bail;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
opts[F] = opts[N] = 0;
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
if (fails)
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
opts[F] = opts[N] = OPEN_OLD|OPEN_OLD_IDS;
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
for (t = 0; t < 2; t++) {
if (chan->ops[t] & (OP_DELETE|OP_FLAGS)) {
opts[t] |= OPEN_SETFLAGS;
opts[1-t] |= OPEN_OLD;
if (chan->ops[t] & OP_FLAGS)
opts[1-t] |= OPEN_FLAGS;
}
if (chan->ops[t] & (OP_NEW|OP_RENEW)) {
opts[t] |= OPEN_APPEND;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (chan->ops[t] & OP_NEW) {
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
opts[1-t] |= OPEN_NEW;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (chan->stores[t]->max_size != UINT_MAX)
fix propagation of flagged oversized messages ... when not syncing flags and the target is not being expunged, as in that case flags were not queried in time.
2019-12-29 11:59:40 +00:00
opts[1-t] |= OPEN_FLAGS|OPEN_NEW_SIZE;
don't fetch message size unless necessary when syncing flags but not re-newing non-fetched messages, there is no need to query the message size for all messages, as the old ones are queried only for their flags.
2016-12-18 20:22:52 +00:00
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (chan->ops[t] & OP_RENEW) {
opts[t] |= OPEN_OLD|OPEN_FLAGS|OPEN_SETFLAGS;
opts[1-t] |= OPEN_OLD|OPEN_FLAGS;
}
if (chan->ops[t] & OP_EXPUNGE) // Don't propagate doomed msgs
opts[1-t] |= OPEN_FLAGS;
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
}
if (chan->ops[t] & OP_EXPUNGE) {
opts[t] |= OPEN_EXPUNGE;
if (chan->stores[t]->trash) {
if (!chan->stores[t]->trash_only_new)
opts[t] |= OPEN_OLD;
opts[t] |= OPEN_NEW|OPEN_FLAGS;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else if (chan->stores[1-t]->trash && chan->stores[1-t]->trash_remote_new) {
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
opts[t] |= OPEN_NEW|OPEN_FLAGS;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
}
}
comment updates
2021-12-08 12:55:33 +00:00
// While only new messages can cause expiration due to displacement,
// updating flags can cause expiration of already overdue messages.
// The latter would also apply when the expired box is the source,
// but it's more natural to treat it as read-only in that case.
// OP_RENEW makes sense only for legacy S_SKIPPED entries.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((chan->ops[N] & (OP_NEW|OP_RENEW|OP_FLAGS)) && chan->max_messages)
opts[N] |= OPEN_OLD|OPEN_NEW|OPEN_FLAGS;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (svars->replayed) {
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
wrap message storing into transactions. nice side effect: drivers don't need to deal with line end conversion any move.
2006-02-03 21:33:43 +00:00
if (srec->status & S_DEAD)
continue;
if (srec->tuid[0]) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->uid[F])
opts[F] |= OPEN_NEW|OPEN_FIND, svars->state[F] |= ST_FIND_OLD;
else if (!srec->uid[N])
opts[N] |= OPEN_NEW|OPEN_FIND, svars->state[N] |= ST_FIND_OLD;
assert no stray TUIDs
2013-11-02 22:32:42 +00:00
else
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
warn( "Warning: sync record (%u,%u) has stray TUID. Ignoring.\n", srec->uid[F], srec->uid[N] );
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (srec->wstate & W_PURGE) {
t = srec->uid[F] ? F : N;
opts[t] |= OPEN_SETFLAGS;
}
if (srec->wstate & W_UPGRADE) {
t = !srec->uid[F] ? F : N;
opts[t] |= OPEN_APPEND;
opts[1-t] |= OPEN_OLD;
}
wrap message storing into transactions. nice side effect: drivers don't need to deal with line end conversion any move.
2006-02-03 21:33:43 +00:00
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->opts[F] = svars->drv[F]->prepare_load_box( ctx[F], opts[F] );
svars->opts[N] = svars->drv[N]->prepare_load_box( ctx[N], opts[N] );
move driver options composition below journal replay - it might make additional actions necessary (it doesn't, yet).
2006-01-29 11:35:22 +00:00
abstract growable arrays somewhat ... and sneak in a C99 requirement on the way. just because.
2016-11-04 20:48:58 +00:00
ARRAY_INIT( &mexcs );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (svars->opts[F] & OPEN_OLD) {
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
if (chan->max_messages) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
/* When messages have been expired on the near side, the far side fetch is split into
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
* two ranges: The bulk fetch which corresponds with the most recent messages, and an
* exception list of messages which would have been expired if they weren't important. */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "preparing far side selection - max expired far uid is %u\n", svars->maxxfuid );
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
/* First, find out the lower bound for the bulk fetch. */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
minwuid = svars->maxxfuid + 1;
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
/* Next, calculate the exception fetch. */
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
continue;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->uid[F])
atomize & document conditions in load() exception list construction
2020-07-07 19:14:59 +00:00
continue; // No message; other state is irrelevant
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (srec->uid[F] >= minwuid)
atomize & document conditions in load() exception list construction
2020-07-07 19:14:59 +00:00
continue; // Message is in non-expired range
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((svars->opts[F] & OPEN_NEW) && srec->uid[F] >= svars->maxuid[F])
atomize & document conditions in load() exception list construction
2020-07-07 19:14:59 +00:00
continue; // Message is in expired range, but new range overlaps that
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->uid[N] && !(srec->status & S_PENDING))
atomize & document conditions in load() exception list construction
2020-07-07 19:14:59 +00:00
continue; // Only actually paired up messages matter
// The pair is alive, but outside the bulk range
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
*uint_array_append( &mexcs ) = srec->uid[F];
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
}
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
sort_uint_array( mexcs.array );
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
} else {
minwuid = 1;
}
} else {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
minwuid = UINT_MAX;
don't delay loading master even if messages were expired the delay optimized the corner case of previously important but now expired messages on the slave disappearing, either through an external expunge or after a journal replay. no point in pessimizing the common case.
2013-11-17 10:23:44 +00:00
}
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
sync_ref( svars );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
load_box( svars, F, minwuid, mexcs.array );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
if (!check_cancel( svars ))
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
load_box( svars, N, (svars->opts[N] & OPEN_OLD) ? 1 : UINT_MAX, (uint_array_t){ NULL, 0 } );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
sync_deref( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix signedness issues surrounding UIDs amends bb632d1c.
2020-07-08 11:45:06 +00:00
static uint
don't fetch message size unless necessary when syncing flags but not re-newing non-fetched messages, there is no need to query the message size for all messages, as the old ones are queried only for their flags.
2016-12-18 20:22:52 +00:00
get_seenuid( sync_vars_t *svars, int t )
{
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint seenuid = 0;
don't fetch message size unless necessary when syncing flags but not re-newing non-fetched messages, there is no need to query the message size for all messages, as the old ones are queried only for their flags.
2016-12-18 20:22:52 +00:00
for (sync_rec_t *srec = svars->srecs; srec; srec = srec->next)
if (!(srec->status & S_DEAD) && seenuid < srec->uid[t])
seenuid = srec->uid[t];
return seenuid;
}
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
static void box_loaded( int sts, message_t *msgs, int total_msgs, int recent_msgs, void *aux );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
load_box( sync_vars_t *svars, int t, uint minwuid, uint_array_t mexcs )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
uint maxwuid = 0, pairuid = UINT_MAX;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
make driver_t::prepare_load_box() return the final options ... and make 'opts' private to the drivers.
2017-03-24 13:18:41 +00:00
if (svars->opts[t] & OPEN_NEW) {
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if (minwuid > svars->maxuid[t] + 1)
minwuid = svars->maxuid[t] + 1;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
maxwuid = UINT_MAX;
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
if (svars->opts[t] & OPEN_OLD_IDS) // Implies OPEN_OLD
pairuid = get_seenuid( svars, t );
make driver_t::prepare_load_box() return the final options ... and make 'opts' private to the drivers.
2017-03-24 13:18:41 +00:00
} else if (svars->opts[t] & OPEN_OLD) {
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
maxwuid = get_seenuid( svars, t );
don't fetch message size unless necessary when syncing flags but not re-newing non-fetched messages, there is no need to query the message size for all messages, as the old ones are queried only for their flags.
2016-12-18 20:22:52 +00:00
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
info( "Loading %s box...\n", str_fn[t] );
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
svars->drv[t]->load_box( svars->ctx[t], minwuid, maxwuid, svars->finduid[t], pairuid, svars->maxuid[t], mexcs, box_loaded, AUX );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
typedef struct {
void *aux;
sync_rec_t *srec;
int aflags, dflags;
} flag_vars_t;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
typedef struct {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uid;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
sync_rec_t *srec;
} sync_rec_map_t;
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
static void flags_set( int sts, void *aux );
static void flags_set_p2( sync_vars_t *svars, sync_rec_t *srec, int t );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void msgs_flags_set( sync_vars_t *svars, int t );
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
static void msg_copied( int sts, uint uid, copy_vars_t *vars );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void msgs_copied( sync_vars_t *svars, int t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
box_loaded( int sts, message_t *msgs, int total_msgs, int recent_msgs, void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
DECL_SVARS;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
sync_rec_t *srec;
sync_rec_map_t *srecmap;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
message_t *tmsg;
flag_vars_t *fv;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
int no[2], del[2], alive, todel;
shrink some data at the source to avoid subsequent narrowing
2019-07-28 19:24:17 +00:00
uchar sflags, nflags, aflags, dflags;
introduce uchar, ushort & uint typedefs
2014-12-07 12:19:30 +00:00
uint hashsz, idx;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
if (check_ret( sts, aux ))
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
INIT_SVARS(aux);
svars->state[t] |= ST_LOADED;
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
svars->msgs[t] = msgs;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
info( "%s: %d messages, %d recent\n", str_fn[t], total_msgs, recent_msgs );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
fix enum abuse amends 9c86ec344. S_FIND was for the sync record status field. it has no business in the sync vars status fields. its value coincided with ST_SELECTED, which luckily only means that we always tried to match up TUIDs even if there was nothing to do. the need for TUID matching arises in two mostly independent circumstances, so add two separate flags ST_FIND_{OLD,NEW}.
2013-11-02 11:57:39 +00:00
if (svars->state[t] & ST_FIND_OLD) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "matching previously copied messages on %s\n", str_fn[t] );
let driver_t::find_new_msgs() return the list of messages consistently with driver_t::load_box().
2017-03-26 16:44:43 +00:00
match_tuids( svars, t, msgs );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "matching messages on %s against sync records\n", str_fn[t] );
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
hashsz = bucketsForSize( svars->nsrecs * 3 );
srecmap = nfcalloc( hashsz * sizeof(*srecmap) );
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
continue;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uid = srec->uid[t];
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
if (!uid)
don't populate sync record map with invalid UIDs this would obviously just bloat the hash with nonsense, slowing down the actual lookup later.
2017-03-11 12:27:53 +00:00
continue;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
idx = (uint)(uid * 1103515245U) % hashsz;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
while (srecmap[idx].uid)
if (++idx == hashsz)
idx = 0;
srecmap[idx].uid = uid;
srecmap[idx].srec = srec;
}
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
for (tmsg = svars->msgs[t]; tmsg; tmsg = tmsg->next) {
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
if (tmsg->srec) /* found by TUID */
continue;
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
uint uid = tmsg->uid;
idx = (uint)(uid * 1103515245U) % hashsz;
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
while (srecmap[idx].uid) {
if (srecmap[idx].uid == uid) {
srec = srecmap[idx].srec;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
goto found;
}
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
if (++idx == hashsz)
idx = 0;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
continue;
found:
tmsg->srec = srec;
srec->msg[t] = tmsg;
}
use a hash table for message => sync record lookup this removes the pathological O(<number of sync records> * <number of new messages>) case at the cost of being a bit more cpu-intensive (but O(<number of all messages>)) for old messages.
2012-08-26 11:36:12 +00:00
free( srecmap );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
if (!(svars->state[1-t] & ST_LOADED))
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
for (t = 0; t < 2; t++) {
let driver_t::openbox_box() return the UID validity ... and make 'uidvalidity' private to the drivers.
2017-03-24 18:24:30 +00:00
if (svars->uidval[t] != UIDVAL_BAD && svars->uidval[t] != svars->newuidval[t]) {
make UIDVALIDITY recovery more strict about vanished messages in particular, this covers the case of a mailbox being replaced with an empty new one, which would subsequently lead to the opposite end being emptied as well, which would typically be undesired. also add plenty of comments.
2021-02-20 21:52:49 +00:00
// This code checks whether the messages with known UIDs are actually the
// same messages, as recognized by their Message-IDs.
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
unsigned need = 0, got = 0;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "trying to re-approve uid validity of %s\n", str_fn[t] );
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
continue;
make UIDVALIDITY recovery more strict about vanished messages in particular, this covers the case of a mailbox being replaced with an empty new one, which would subsequently lead to the opposite end being emptied as well, which would typically be undesired. also add plenty of comments.
2021-02-20 21:52:49 +00:00
need++;
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
if (!srec->msg[t])
continue; // Message disappeared.
make UIDVALIDITY recovery more strict about vanished messages in particular, this covers the case of a mailbox being replaced with an empty new one, which would subsequently lead to the opposite end being emptied as well, which would typically be undesired. also add plenty of comments.
2021-02-20 21:52:49 +00:00
// Present paired messages require re-validation.
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
if (!srec->msg[t]->msgid)
continue; // Messages without ID are useless for re-validation.
if (!srec->msg[1-t])
continue; // Partner disappeared.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->msg[1-t]->msgid || strcmp( srec->msg[F]->msgid, srec->msg[N]->msgid )) {
error( "Error: channel %s, %s box %s: UIDVALIDITY genuinely changed (at UID %u).\n",
svars->chan->name, str_fn[t], svars->orig_name[t], srec->uid[t] );
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
uvchg:
svars->ret |= SYNC_FAIL;
cancel_sync( svars );
return;
}
got++;
}
make UIDVALIDITY recovery more strict about vanished messages in particular, this covers the case of a mailbox being replaced with an empty new one, which would subsequently lead to the opposite end being emptied as well, which would typically be undesired. also add plenty of comments.
2021-02-20 21:52:49 +00:00
// We encountered no messages that contradict the hypothesis that the
// UIDVALIDITY change was spurious.
// If we got enough messages confirming the hypothesis, we just accept it.
// If there aren't quite enough messages, we check that at least 80% of
// those previously present are still there and confirm the hypothesis;
// this also covers the case of a box that was already empty.
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
if (got < 20 && got * 5 < need * 4) {
// Too few confirmed messages. This is very likely in the drafts folder.
// A proper fallback would be fetching more headers (which potentially need
// normalization) or the message body (which should be truncated for sanity)
// and comparing.
improve error messages about irrecoverably changed UIDVALIDITY don't print the actual values, which are meaningless technicalities to the average user, and can be obtained separately for debugging if really necessary. also, fix the omission of the affected mailboxes from one of the messages.
2021-02-20 21:52:01 +00:00
error( "Error: channel %s, %s box %s: Unable to recover from UIDVALIDITY change.\n",
svars->chan->name, str_fn[t], svars->orig_name[t] );
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
goto uvchg;
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
notice( "Notice: channel %s, %s box %s: Recovered from change of UIDVALIDITY.\n",
svars->chan->name, str_fn[t], svars->orig_name[t] );
use a #define for invalid UIDVALIDITY
2017-01-29 14:39:36 +00:00
svars->uidval[t] = UIDVAL_BAD;
implement Message-Id based UIDVALIDITY recovery
2016-12-18 19:50:20 +00:00
}
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (svars->uidval[F] == UIDVAL_BAD || svars->uidval[N] == UIDVAL_BAD) {
svars->uidval[F] = svars->newuidval[F];
svars->uidval[N] = svars->newuidval[N];
JLOG( "| %u %u", (svars->uidval[F], svars->uidval[N]), "new UIDVALIDITYs" );
Added MaxMessages patch from Eivind Eklund <eivind@FreeBSD.org>. config_defaults() can just use memcpy() instead of assigning each struct member individually. config_defaults() can be declared static
2001-11-15 23:59:27 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
svars->oldmaxuid[F] = svars->maxuid[F];
svars->oldmaxuid[N] = svars->maxuid[N];
svars->oldmaxxfuid = svars->maxxfuid;
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
info( "Synchronizing...\n" );
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
for (t = 0; t < 2; t++)
svars->good_flags[t] = (uchar)svars->drv[t]->get_supported_flags( svars->ctx[t] );
move fetching new messages in front of syncing old entries. this alone does not buy us a whole lot ...
2006-01-29 15:52:49 +00:00
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
int any_new[2] = { 0, 0 };
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
debug( "synchronizing old entries\n" );
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
if (srec->status & S_DEAD)
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
continue;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "pair (%u,%u)\n", srec->uid[F], srec->uid[N] );
delay assignment of TUID when propagating messages go back to assigning TUIDs only right before actually propagating them. this avoids spurious "TUID lost" warnings.
2017-04-02 15:21:39 +00:00
assert( !srec->tuid[0] );
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
// no[] means that a message is known to be not there.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
no[F] = !srec->msg[F] && (svars->opts[F] & OPEN_OLD);
no[N] = !srec->msg[N] && (svars->opts[N] & OPEN_OLD);
if (no[F] && no[N]) {
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
// It does not matter whether one side was already known to be missing
// (never stored [skipped or failed] or expunged [possibly expired]) -
// now both are missing, so the entry is superfluous.
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
srec->status = S_DEAD;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "- %u %u", (srec->uid[F], srec->uid[N]), "both missing" );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
} else {
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
// del[] means that a message becomes known to have been expunged.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
del[F] = no[F] && srec->uid[F];
del[N] = no[N] && srec->uid[N];
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
for (t = 0; t < 2; t++) {
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
if (srec->msg[t] && (srec->msg[t]->flags & F_DELETED))
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
srec->wstate |= W_DEL(t);
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
if (del[t]) {
// The target was newly expunged, so there is nothing to update.
// The deletion is propagated in the opposite iteration.
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
} else if (!srec->uid[t]) {
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
// The target was never stored, or was previously expunged, so there
// is nothing to update.
// Note: the opposite UID must be valid, as otherwise the entry would
// have been pruned already.
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
} else if (del[1-t]) {
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
// The source was newly expunged, so possibly propagate the deletion.
// The target may be in an unknown state (not fetched).
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((t == F) && (srec->status & (S_EXPIRE|S_EXPIRED))) {
remove cleanup of expired entries during setup of master load the removed code would only ever trigger if a) we were after a journal replay or b) something external expunged the expired messages - both are corner cases not worth the extra code. however, this means that the syncing code further down now needs to take care of these zombies. in the end, the normal cleanup will take care of all expired entries, new and old.
2013-11-17 08:06:20 +00:00
/* Don't propagate deletion resulting from expiration. */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "> %u %u 0", (srec->uid[F], srec->uid[N]), "near side expired, orphaning far side" );
srec->uid[N] = 0;
remove cleanup of expired entries during setup of master load the removed code would only ever trigger if a) we were after a journal replay or b) something external expunged the expired messages - both are corner cases not worth the extra code. however, this means that the syncing code further down now needs to take care of these zombies. in the end, the normal cleanup will take care of all expired entries, new and old.
2013-11-17 08:06:20 +00:00
} else {
don't complain about concurrent flagging as deleted the result of propagating a deletion is flagging as deleted, so shut up if the only remote change is exactly that.
2021-12-01 10:25:06 +00:00
if (srec->msg[t] && (srec->msg[t]->status & M_FLAGS) &&
// Ignore deleted flag, as that's what we'll change ourselves ...
(((srec->msg[t]->flags & ~F_DELETED) != (srec->flags & ~F_DELETED)) ||
// ... except for undeletion, as that's the opposite.
(!(srec->msg[t]->flags & F_DELETED) && (srec->flags & F_DELETED))))
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
notice( "Notice: conflicting changes in (%u,%u)\n", srec->uid[F], srec->uid[N] );
remove cleanup of expired entries during setup of master load the removed code would only ever trigger if a) we were after a journal replay or b) something external expunged the expired messages - both are corner cases not worth the extra code. however, this means that the syncing code further down now needs to take care of these zombies. in the end, the normal cleanup will take care of all expired entries, new and old.
2013-11-17 08:06:20 +00:00
if (svars->chan->ops[t] & OP_DELETE) {
debug( " %sing delete\n", str_hl[t] );
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
srec->aflags[t] = F_DELETED;
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
srec->wstate |= W_DELETE;
remove cleanup of expired entries during setup of master load the removed code would only ever trigger if a) we were after a journal replay or b) something external expunged the expired messages - both are corner cases not worth the extra code. however, this means that the syncing code further down now needs to take care of these zombies. in the end, the normal cleanup will take care of all expired entries, new and old.
2013-11-17 08:06:20 +00:00
} else {
debug( " not %sing delete\n", str_hl[t] );
}
}
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
} else if (!srec->msg[1-t]) {
// We have no source to work with, because it was never stored,
// it was previously expunged, or we did not fetch it.
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( " no %s\n", str_fn[1-t] );
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
} else {
// We have a source. The target may be in an unknown state.
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if (svars->chan->ops[t] & OP_FLAGS) {
don't try to propagate flags the target store does not support $Forwarded is not standard, so it will most likely fail with mailboxes that do not support keywords. amends c4d7f018.
2020-01-08 17:22:48 +00:00
sflags = sanitize_flags( srec->msg[1-t]->flags, svars, t );
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((t == F) && (srec->status & (S_EXPIRE|S_EXPIRED))) {
add/fix comments and improve debug messages
2013-11-02 19:06:08 +00:00
/* Don't propagate deletion resulting from expiration. */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( " near side expiring\n" );
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
sflags &= ~F_DELETED;
add/fix comments and improve debug messages
2013-11-02 19:06:08 +00:00
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (srec->status & S_DUMMY(1-t)) {
// For placeholders, don't propagate:
// - Seen, because the real contents were obviously not seen yet
// - Flagged, because it's just a request to upgrade
sflags &= ~(F_SEEN|F_FLAGGED);
}
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
srec->aflags[t] = sflags & ~srec->flags;
srec->dflags[t] = ~sflags & srec->flags;
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
if ((DFlags & DEBUG_SYNC) && (srec->aflags[t] || srec->dflags[t])) {
less cluttered debug output
2006-02-02 11:23:57 +00:00
char afbuf[16], dfbuf[16]; /* enlarge when support for keywords is added */
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
make_flags( srec->aflags[t], afbuf );
make_flags( srec->dflags[t], dfbuf );
less cluttered debug output
2006-02-02 11:23:57 +00:00
debug( " %sing flags: +%s -%s\n", str_hl[t], afbuf, dfbuf );
}
streamline syncing of old entries the order of the conditionals was purely historical (pre 4ec56f8cf, anno 2005) and hard to follow, as were the comments.
2017-03-10 16:40:54 +00:00
}
}
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
sync_rec_t *nsrec = srec;
if (((srec->status & S_DUMMY(F)) && (svars->chan->ops[F] & OP_RENEW)) ||
((srec->status & S_DUMMY(N)) && (svars->chan->ops[N] & OP_RENEW))) {
// Flagging the message on either side causes an upgrade of the dummy.
// We ignore flag resets, because that corner case is not worth it.
ushort muflags = srec->msg[F] ? srec->msg[F]->flags : 0;
ushort suflags = srec->msg[N] ? srec->msg[N]->flags : 0;
if ((muflags | suflags) & F_FLAGGED) {
t = (srec->status & S_DUMMY(F)) ? F : N;
// We calculate the flags for the replicated message already now,
// because after an interruption the dummy may be already gone.
srec->pflags = ((srec->msg[t]->flags & ~(F_SEEN|F_FLAGGED)) | srec->aflags[t]) & ~srec->dflags[t];
// Consequently, the srec's flags are committed right away as well.
srec->flags = (srec->flags | srec->aflags[t]) & ~srec->dflags[t];
JLOG( "^ %u %u %u %u", (srec->uid[F], srec->uid[N], srec->pflags, srec->flags), "upgrading placeholder" );
nsrec = upgrade_srec( svars, srec );
}
}
// This is separated, because the upgrade can come from the journal.
if (srec->wstate & W_UPGRADE) {
t = !srec->uid[F] ? F : N;
tmsg = srec->msg[1-t];
if ((svars->chan->ops[t] & OP_EXPUNGE) && (srec->pflags & F_DELETED)) {
JLOG( "- %u %u", (srec->uid[F], srec->uid[N]), "killing upgrade - would be expunged anyway" );
tmsg->srec = NULL;
srec->status = S_DEAD;
} else {
// Pretend that the source message has the adjusted flags of the dummy.
tmsg->flags = srec->pflags;
don't clobber message status when upgrading placeholders we'd reset the possibly set M_RECENT flag, which would lead to pointless maildir rescans. amends 70bad661.
2022-04-04 19:53:02 +00:00
tmsg->status |= M_FLAGS;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
any_new[t] = 1;
}
}
comment updates
2021-12-08 12:55:33 +00:00
srec = nsrec; // Minor optimization: skip freshly created placeholder entry.
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
Added MaxMessages patch from Eivind Eklund <eivind@FreeBSD.org>. config_defaults() can just use memcpy() instead of assigning each struct member individually. config_defaults() can be declared static
2001-11-15 23:59:27 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
for (t = 0; t < 2; t++) {
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
debug( "synchronizing new messages on %s\n", str_fn[1-t] );
let driver_t::load_box() return the list of messages ... and make 'msgs', 'count', and 'recent' private to the drivers.
2017-03-24 17:09:40 +00:00
for (tmsg = svars->msgs[1-t]; tmsg; tmsg = tmsg->next) {
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
srec = tmsg->srec;
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
if (srec) {
if (srec->status & S_SKIPPED) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
// Pre-1.4 legacy only: The message was skipped due to being too big.
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
// We must have already seen the UID, but we might have been interrupted.
if (svars->maxuid[1-t] < tmsg->uid)
svars->maxuid[1-t] = tmsg->uid;
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
if (!(svars->chan->ops[t] & OP_RENEW))
continue;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
srec->status = S_PENDING;
// The message size was not queried, so this won't be dummified below.
if (!(tmsg->flags & F_FLAGGED)) {
srec->status |= S_DUMMY(t);
JLOG( "_ %u %u", (srec->uid[F], srec->uid[N]), "placeholder only - was previously skipped" );
} else {
JLOG( "~ %u %u %u", (srec->uid[F], srec->uid[N], srec->status), "was previously skipped" );
}
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
} else {
if (!(svars->chan->ops[t] & OP_NEW))
continue;
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
// This catches messages:
// - that are actually new
// - whose propagation got interrupted
// - whose propagation was completed, but not logged yet
// - that aren't actually new, but a result of syncing, and the instant
// maxuid upping was prevented by the presence of actually new messages
if (svars->maxuid[1-t] < tmsg->uid)
svars->maxuid[1-t] = tmsg->uid;
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
if (!(srec->status & S_PENDING))
continue; // Nothing to do - the message is paired or expired
// Propagation was scheduled, but we got interrupted
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
debug( "unpropagated old message %u\n", tmsg->uid );
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
}
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
if ((svars->chan->ops[t] & OP_EXPUNGE) && (tmsg->flags & F_DELETED)) {
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
JLOG( "- %u %u", (srec->uid[F], srec->uid[N]), "killing - would be expunged anyway" );
tmsg->srec = NULL;
srec->status = S_DEAD;
continue;
}
} else {
if (!(svars->chan->ops[t] & OP_NEW))
continue;
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
if (tmsg->uid <= svars->maxuid[1-t]) {
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
// The message should be already paired. It's not, so it was:
// - previously paired, but the entry was expired and pruned => ignore
// - attempted, but failed => ignore (the wisdom of this is debatable)
// - ignored, as it would have been expunged anyway => ignore (even if undeleted)
continue;
}
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
svars->maxuid[1-t] = tmsg->uid;
re-nest conditions for syncing new messages this makes the logic easier to follow and document in place. also, make the comments actually match reality.
2020-07-08 12:37:57 +00:00
debug( "new message %u\n", tmsg->uid );
if ((svars->chan->ops[t] & OP_EXPUNGE) && (tmsg->flags & F_DELETED)) {
debug( "-> ignoring - would be expunged anyway\n" );
continue;
}
srec = nfcalloc( sizeof(*srec) );
*svars->srecadd = srec;
svars->srecadd = &srec->next;
svars->nsrecs++;
srec->status = S_PENDING;
srec->uid[1-t] = tmsg->uid;
srec->msg[1-t] = tmsg;
tmsg->srec = srec;
JLOG( "+ %u %u", (srec->uid[F], srec->uid[N]), "fresh" );
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!(tmsg->flags & F_FLAGGED) && tmsg->size > svars->chan->stores[t]->max_size &&
!(srec->wstate & W_UPGRADE) && !(srec->status & (S_DUMMY(F)|S_DUMMY(N)))) {
srec->status |= S_DUMMY(t);
JLOG( "_ %u %u", (srec->uid[F], srec->uid[N]), "placeholder only - too big" );
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
}
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
any_new[t] = 1;
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
}
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((svars->chan->ops[N] & (OP_NEW|OP_RENEW|OP_FLAGS)) && svars->chan->max_messages) {
// Note: When this branch is entered, we have loaded all near side messages.
count unread messages like flagged messages when expiring that is, don't count them towards the total only below the cut-off point. making them extend the working set even though they are inside it is counterintuitive.
2013-11-10 18:57:08 +00:00
/* Expire excess messages. Important (flagged, unread, or unpropagated) messages
* older than the first not expired message are not counted towards the total. */
make message counting in expiration code less confusing
2013-11-23 11:01:23 +00:00
debug( "preparing message expiration\n" );
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
// Due to looping only over the far side, we completely ignore unpaired
// near-side messages. This is correct, as we cannot expire them without
// data loss anyway; consequently, we also don't count them.
// Note that we also ignore near-side messages we're currently propagating,
// which delays expiration of some messages by one cycle. Otherwise, we'd have
// to sequence flag propagation after message propagation to avoid a race
// with 3rd-party expunging, and that seems unreasonably expensive.
make message counting in expiration code less confusing
2013-11-23 11:01:23 +00:00
alive = 0;
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
for (tmsg = svars->msgs[F]; tmsg; tmsg = tmsg->next) {
rewrite condition for readability and consistency
2013-05-20 16:53:22 +00:00
if (tmsg->status & M_DEAD)
continue;
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
// We ignore unpaired far-side messages, as there is obviously nothing
// to expire in the first place.
if (!(srec = tmsg->srec))
continue;
if (!(srec->status & S_PENDING)) {
if (!srec->msg[N])
continue; // Already expired or skipped.
nflags = (srec->msg[N]->flags | srec->aflags[N]) & ~srec->dflags[N];
make message counting in expiration code less confusing
2013-11-23 11:01:23 +00:00
} else {
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
nflags = tmsg->flags;
make message counting in expiration code less confusing
2013-11-23 11:01:23 +00:00
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (!(nflags & F_DELETED) || (srec->status & (S_EXPIRE | S_EXPIRED))) {
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
// The message is not deleted, or it is, but only due to being expired.
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
alive++;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
}
make message counting in expiration code less confusing
2013-11-23 11:01:23 +00:00
todel = alive - svars->chan->max_messages;
debug( "%d alive messages, %d excess - expiring\n", alive, todel );
add ExpireUnread option
2013-11-24 19:26:33 +00:00
alive = 0;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
for (tmsg = svars->msgs[F]; tmsg; tmsg = tmsg->next) {
don't forget to skip dead messages on far side during expiration this is mostly theoretical, as at this point no updates to the message list can have actually happened. but it's future-proof and consistent with the near-side loop.
2020-07-16 11:08:29 +00:00
if (tmsg->status & M_DEAD)
continue;
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
if (!(srec = tmsg->srec))
continue;
if (!(srec->status & S_PENDING)) {
if (!srec->msg[N])
continue;
nflags = (srec->msg[N]->flags | srec->aflags[N]) & ~srec->dflags[N];
} else {
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
nflags = tmsg->flags;
make expiration loops solely far-side-driven we can do that, as unpaired near-side messages are ignored anyway. this mildly changes expiration order, as near-side messages that existed for a long time but were propagated much later will be expired later. however, that has no practical relevance.
2020-07-16 12:47:30 +00:00
}
if (!(nflags & F_DELETED) || (srec->status & (S_EXPIRE|S_EXPIRED))) {
if ((nflags & F_FLAGGED) ||
!((nflags & F_SEEN) || ((void)(todel > 0 && alive++), svars->chan->expire_unread > 0))) {
// Important messages are always fetched/kept.
debug( " pair(%u,%u) is important\n", srec->uid[F], srec->uid[N] );
todel--;
} else if (todel > 0 ||
((srec->status & (S_EXPIRE|S_EXPIRED)) == (S_EXPIRE|S_EXPIRED)) ||
((srec->status & (S_EXPIRE|S_EXPIRED)) && (srec->msg[N]->flags & F_DELETED))) {
/* The message is excess or was already (being) expired. */
srec->wstate |= W_NEXPIRE;
debug( " pair(%u,%u) expired\n", srec->uid[F], srec->uid[N] );
if (svars->maxxfuid < srec->uid[F])
svars->maxxfuid = srec->uid[F];
todel--;
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
}
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
debug( "%d excess messages remain\n", todel );
fix lots of sign conversion warnings ... by making a lot of objects unsigned, and some signed. casts which lose precision and change the sign in one go (ssize_t and time_t to uint on LP64) are made explicit as well.
2020-07-08 15:27:37 +00:00
if (svars->chan->expire_unread < 0 && alive * 2 > svars->chan->max_messages) {
add ExpireUnread option
2013-11-24 19:26:33 +00:00
error( "%s: %d unread messages in excess of MaxMessages (%d).\n"
"Please set ExpireUnread to decide outcome. Skipping mailbox.\n",
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
svars->orig_name[N], alive, svars->chan->max_messages );
add ExpireUnread option
2013-11-24 19:26:33 +00:00
svars->ret |= SYNC_FAIL;
cancel_sync( svars );
return;
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
if (srec->status & S_DEAD)
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
continue;
delay assignment of TUID when propagating messages go back to assigning TUIDs only right before actually propagating them. this avoids spurious "TUID lost" warnings.
2017-04-02 15:21:39 +00:00
if (!(srec->status & S_PENDING)) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->msg[N])
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
continue;
shrink some data at the source to avoid subsequent narrowing
2019-07-28 19:24:17 +00:00
uchar nex = (srec->wstate / W_NEXPIRE) & 1;
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
if (nex != ((srec->status / S_EXPIRED) & 1)) {
/* The record needs a state change ... */
if (nex != ((srec->status / S_EXPIRE) & 1)) {
/* ... and we need to start a transaction. */
srec->status = (srec->status & ~S_EXPIRE) | (nex * S_EXPIRE);
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "~ %u %u %u", (srec->uid[F], srec->uid[N], srec->status), "expire %u - begin", nex );
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
} else {
/* ... but the "right" transaction is already pending. */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "-> pair(%u,%u): expire %u (pending)\n", srec->uid[F], srec->uid[N], nex );
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
}
document message expiration transactions
2013-11-17 16:36:08 +00:00
} else {
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
/* Note: the "wrong" transaction may be pending here,
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
* e.g.: W_NEXPIRE = 0, S_EXPIRE = 1, S_EXPIRED = 0. */
document message expiration transactions
2013-11-17 16:36:08 +00:00
}
} else {
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
if (srec->wstate & W_NEXPIRE) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "= %u %u", (srec->uid[F], srec->uid[N]), "expire unborn" );
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// If we have so many new messages that some of them are instantly expired,
// but some are still propagated because they are important, we need to
// ensure explicitly that the bulk fetch limit is upped.
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
if (svars->maxxfuid < srec->uid[F])
svars->maxxfuid = srec->uid[F];
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
srec->msg[F]->srec = NULL;
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
srec->status = S_DEAD;
}
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
}
}
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
sync_ref( svars );
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
debug( "synchronizing flags\n" );
move handling of new messages after that of old ones i.e., move it back. whatever the original reason was, it's now gone. this order is way more natural, which allows us to remove the osrecadd and S_DONE hacks.
2013-11-24 17:26:11 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (srec->status & S_DEAD)
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
continue;
for (t = 0; t < 2; t++) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!srec->uid[t])
continue;
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
aflags = srec->aflags[t];
dflags = srec->dflags[t];
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (srec->wstate & (W_DELETE|W_PURGE)) {
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
if (!aflags) {
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
// This deletion propagation goes the other way round, or
// this deletion of a dummy happens on the other side.
continue;
}
if (!srec->msg[t] && (svars->opts[t] & OPEN_OLD)) {
// The message disappeared. This can happen, because the wstate may
// come from the journal, and things could have happened meanwhile.
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
continue;
}
} else {
document message expiration transactions
2013-11-17 16:36:08 +00:00
/* The trigger is an expiration transaction being ongoing ... */
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if ((t == N) && ((shifted_bit(srec->status, S_EXPIRE, S_EXPIRED) ^ srec->status) & S_EXPIRED)) {
comment updates
2021-12-08 12:55:33 +00:00
// ... but the actual action derives from the wanted state -
// so that canceled transactions are rolled back as well.
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
if (srec->wstate & W_NEXPIRE)
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
aflags |= F_DELETED;
else
dflags |= F_DELETED;
}
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
}
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
if ((svars->chan->ops[t] & OP_EXPUNGE) && (((srec->msg[t] ? srec->msg[t]->flags : 0) | aflags) & ~dflags & F_DELETED) &&
(!svars->ctx[t]->conf->trash || svars->ctx[t]->conf->trash_only_new))
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
{
add/fix comments and improve debug messages
2013-11-02 19:06:08 +00:00
/* If the message is going to be expunged, don't propagate anything but the deletion. */
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
srec->aflags[t] &= F_DELETED;
aflags &= F_DELETED;
srec->dflags[t] = dflags = 0;
}
if (srec->msg[t] && (srec->msg[t]->status & M_FLAGS)) {
add/fix comments and improve debug messages
2013-11-02 19:06:08 +00:00
/* If we know the target message's state, optimize away non-changes. */
major overhaul of flag change propagation and MaxMessages handling: - wrap message (un)expirations into transactions - no redundand flag propagations in conjunction with expirations - better prepared for the upcoming async operation
2006-02-02 17:03:01 +00:00
aflags &= ~srec->msg[t]->flags;
dflags &= srec->msg[t]->flags;
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if (aflags | dflags) {
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
flags_total[t]++;
stats();
svars->flags_pending[t]++;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fv = nfmalloc( sizeof(*fv) );
fv->aux = AUX;
fv->srec = srec;
fv->aflags = aflags;
fv->dflags = dflags;
make some driver function names more descriptive
2014-12-27 21:13:24 +00:00
svars->drv[t]->set_msg_flags( svars->ctx[t], srec->msg[t], srec->uid[t], aflags, dflags, flags_set, fv );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
if (check_cancel( svars ))
goto out;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else {
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
flags_set_p2( svars, srec, t );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
added BUGS section to manpage detailing the fact that we break the maildir(5) spec by parsing the filename change message delivery to use the method described in maildir(5)
2000-12-21 18:16:44 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
for (t = 0; t < 2; t++) {
make some driver function names more descriptive
2014-12-27 21:13:24 +00:00
svars->drv[t]->commit_cmds( svars->ctx[t] );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
svars->state[t] |= ST_SENT_FLAGS;
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
msgs_flags_set( svars, t );
if (check_cancel( svars ))
goto out;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
delay propagation of new messages previous sequence: examine & propagate new => examine old => propagate old new sequence: examine new => examine old => propagate new => propagate old this alone does not buy us much ...
2013-11-23 14:55:02 +00:00
debug( "propagating new messages\n" );
for (t = 0; t < 2; t++) {
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
if (any_new[t]) {
delay TUID assignment less we can't delay TUID assignment until after starting propagation if we want to provide both safety and performance. amends a0961d65.
2022-01-08 11:53:02 +00:00
// fsync'ing the UIDNEXT bump is not strictly necessary, but advantageous.
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
svars->finduid[t] = svars->drv[t]->get_uidnext( svars->ctx[t] );
JLOG( "F %d %u", (t, svars->finduid[t]), "save UIDNEXT of %s", str_fn[t] );
svars->new_msgs[t] = svars->msgs[1-t];
} else {
svars->state[t] |= ST_SENT_NEW;
}
delay TUID assignment less we can't delay TUID assignment until after starting propagation if we want to provide both safety and performance. amends a0961d65.
2022-01-08 11:53:02 +00:00
}
if (any_new[F] | any_new[N]) {
// TUID assignment needs to be fsync'd, as otherwise a system crash may
// lead to the newly propagated messages becoming duplicated.
// Of course, we could assign each TUID only after fetching the message
// and fsync it separately, but that would be horribly inefficient.
for (srec = svars->srecs; srec; srec = srec->next)
if (srec->status & S_PENDING)
assign_tuid( svars, srec );
if (UseFSync && svars->jfp)
fdatasync( fileno( svars->jfp ) );
}
for (t = 0; t < 2; t++) {
delay propagation of new messages previous sequence: examine & propagate new => examine old => propagate old new sequence: examine new => examine old => propagate new => propagate old this alone does not buy us much ...
2013-11-23 14:55:02 +00:00
msgs_copied( svars, t );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
if (check_cancel( svars ))
goto out;
delay propagation of new messages previous sequence: examine & propagate new => examine old => propagate old new sequence: examine new => examine old => propagate new => propagate old this alone does not buy us much ...
2013-11-23 14:55:02 +00:00
}
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
out:
sync_deref( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
msg_copied( int sts, uint uid, copy_vars_t *vars )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
SVARS_CHECK_CANCEL_RET;
de-noise msg_copied() and flags_set() somewhat assign temporary srec object instead of always spelling out the indirection.
2019-12-29 11:31:10 +00:00
sync_rec_t *srec = vars->srec;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
switch (sts) {
case SYNC_OK:
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
if (!(srec->wstate & W_UPGRADE) && vars->msg->flags != srec->flags) {
set sync record's flags only after propagating new message this is semantically cleaner, and fixes storing the flags in the rare case that flags are not being synced and the target is not being expunged, as in this case flags are queried only during the actual propagation.
2019-12-29 11:34:36 +00:00
srec->flags = vars->msg->flags;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "* %u %u %u", (srec->uid[F], srec->uid[N], srec->flags), "%sed with flags", str_hl[t] );
set sync record's flags only after propagating new message this is semantically cleaner, and fixes storing the flags in the rare case that flags are not being synced and the target is not being expunged, as in this case flags are queried only during the actual propagation.
2019-12-29 11:34:36 +00:00
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
if (!uid) // Stored to a non-UIDPLUS mailbox
fix enum abuse amends 9c86ec344. S_FIND was for the sync record status field. it has no business in the sync vars status fields. its value coincided with ST_SELECTED, which luckily only means that we always tried to match up TUIDs even if there was nothing to do. the need for TUID matching arises in two mostly independent circumstances, so add two separate flags ST_FIND_{OLD,NEW}.
2013-11-02 11:57:39 +00:00
svars->state[t] |= ST_FIND_NEW;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
else
de-duplicate updating of uid in sync records
2019-12-29 13:41:45 +00:00
ASSIGN_UID( srec, t, uid, "%sed message", str_hl[t] );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
break;
case SYNC_NOGOOD:
de-noise msg_copied() and flags_set() somewhat assign temporary srec object instead of always spelling out the indirection.
2019-12-29 11:31:10 +00:00
srec->status = S_DEAD;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "- %u %u", (srec->uid[F], srec->uid[N]), "%s failed", str_hl[t] );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
break;
default:
cancel_sync( svars );
free( vars );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
free( vars );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
new_done[t]++;
stats();
svars->new_pending[t]--;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
msgs_copied( svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
let driver_t::find_new_msgs() return the list of messages consistently with driver_t::load_box().
2017-03-26 16:44:43 +00:00
static void msgs_found_new( int sts, message_t *msgs, void *aux );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
static void msgs_new_done( sync_vars_t *svars, int t );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void sync_close( sync_vars_t *svars, int t );
- instead of having {m,s}foo, we have foo[2] now, so we can do everything with loops instead of symmetric function calls - added some const
2005-12-28 10:02:22 +00:00
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
msgs_copied( sync_vars_t *svars, int t )
{
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
message_t *tmsg;
sync_rec_t *srec;
copy_vars_t *cv;
if (svars->state[t] & ST_SENDING_NEW)
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
sync_ref( svars );
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
if (!(svars->state[t] & ST_SENT_NEW)) {
for (tmsg = svars->new_msgs[t]; tmsg; tmsg = tmsg->next) {
delay assignment of TUID when propagating messages go back to assigning TUIDs only right before actually propagating them. this avoids spurious "TUID lost" warnings.
2017-04-02 15:21:39 +00:00
if ((srec = tmsg->srec) && (srec->status & S_PENDING)) {
prepend "get_" to getters in driver_t this makes it callbacks consistently start with a verb.
2017-03-19 12:46:03 +00:00
if (svars->drv[t]->get_memory_usage( svars->ctx[t] ) >= BufferLimit) {
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
svars->new_msgs[t] = tmsg;
goto out;
}
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
new_total[t]++;
stats();
svars->new_pending[t]++;
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
svars->state[t] |= ST_SENDING_NEW;
cv = nfmalloc( sizeof(*cv) );
cv->cb = msg_copied;
cv->aux = AUX;
cv->srec = srec;
cv->msg = tmsg;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
cv->minimal = (srec->status & S_DUMMY(t));
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
copy_msg( cv );
svars->state[t] &= ~ST_SENDING_NEW;
if (check_cancel( svars ))
goto out;
}
}
svars->state[t] |= ST_SENT_NEW;
}
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
if (svars->new_pending[t])
soft-limit peak memory usage propagating many messages from a fast store (typically maildir or a local IMAP server) to a slow asynchronous store could cause gigabytes of data being buffered. avoid this by throttling fetches if the target context reports memory usage above a configurable limit. REFMAIL: 9737edb14457c71af4ed156c1be0ae59@mpcjanssen.nl
2015-02-15 17:13:05 +00:00
goto out;
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
sync_close( svars, 1-t );
if (check_cancel( svars ))
goto out;
log maxuid bumping less aggressively we can bump the internal variable whereever convenient, but we cannot log it until we know that all messages were copied, as otherwise we could miss some new messages after an interruption. with the new approach, interruption would merely cause some additonal traffic.
2013-11-02 19:47:20 +00:00
fix enum abuse amends 9c86ec344. S_FIND was for the sync record status field. it has no business in the sync vars status fields. its value coincided with ST_SELECTED, which luckily only means that we always tried to match up TUIDs even if there was nothing to do. the need for TUID matching arises in two mostly independent circumstances, so add two separate flags ST_FIND_{OLD,NEW}.
2013-11-02 11:57:39 +00:00
if (svars->state[t] & ST_FIND_NEW) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "finding just copied messages on %s\n", str_fn[t] );
fix loading of some messages' sizes in some partial sync scenarios we need to pass a different "boundary" UID to driver_t::load_box() for every OPEN_* flag that queries a partial range: - OPEN_FIND refers to messages newer than all we know about - OPEN_OLD_IDS refers to messages which are paired - OPEN_{OLD,NEW}_SIZE refers to messages (not) above the committed boundary of already propagated messages we treated the 3rd like the 2nd, which was just wrong - the actual boundary may be lower or higher, so we'd produce wrong results when MaxSize was set and only one of New and ReNew was requested.
2019-12-29 10:52:26 +00:00
svars->drv[t]->find_new_msgs( svars->ctx[t], svars->finduid[t], msgs_found_new, AUX );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
} else {
msgs_new_done( svars, t );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
out:
sync_deref( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
let driver_t::find_new_msgs() return the list of messages consistently with driver_t::load_box().
2017-03-26 16:44:43 +00:00
msgs_found_new( int sts, message_t *msgs, void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
SVARS_CHECK_RET;
remove pointless "lost track of ... messages" warnings it doesn't really add any value when resuming after an interruption, as incomplete propagations are perfectly expected, and are recovered from. but it does make sense in the non-UIDPLUS path, where not finding just stored messages indicates a problem. so move the reporting from match_tuids() to msgs_found_new(), and remove a redundant warning from the latter. amends 6577bf3e.
2022-02-02 12:06:43 +00:00
debug( "matching just copied messages on %s\n", str_fn[t] );
int num_lost = match_tuids( svars, t, msgs );
if (num_lost)
warn( "Warning: lost track of %d %sed message(s)\n", num_lost, str_hl[t] );
employ alternative scheme to finding messages by TUID instead of SEARCHing every single message (which is slow and happens to be unreliabe with M$ Exchange 2010), just FETCH the new messages from the mailbox - the ones we just appended will be amongst them.
2011-04-10 11:06:07 +00:00
msgs_new_done( svars, t );
}
static void
msgs_new_done( sync_vars_t *svars, int t )
{
svars->state[t] |= ST_FOUND_NEW;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
sync_close( svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
flags_set( int sts, void *aux )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
SVARS_CHECK_RET_VARS(flag_vars_t);
de-noise msg_copied() and flags_set() somewhat assign temporary srec object instead of always spelling out the indirection.
2019-12-29 11:31:10 +00:00
sync_rec_t *srec = vars->srec;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
switch (sts) {
case DRV_OK:
if (vars->aflags & F_DELETED)
de-noise msg_copied() and flags_set() somewhat assign temporary srec object instead of always spelling out the indirection.
2019-12-29 11:31:10 +00:00
srec->wstate |= W_DEL(t);
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
else if (vars->dflags & F_DELETED)
de-noise msg_copied() and flags_set() somewhat assign temporary srec object instead of always spelling out the indirection.
2019-12-29 11:31:10 +00:00
srec->wstate &= ~W_DEL(t);
flags_set_p2( svars, srec, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
break;
}
free( vars );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
flags_done[t]++;
stats();
svars->flags_pending[t]--;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
msgs_flags_set( svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
static void
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
flags_set_p2( sync_vars_t *svars, sync_rec_t *srec, int t )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
split off ephemeral sync record state to a separate member this allows us to simplify logging of expiration, as we now can just log the entire persistent state instead of fiddling with bits.
2017-04-01 15:02:22 +00:00
if (srec->wstate & W_DELETE) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "%c %u %u 0", ("><"[t], srec->uid[F], srec->uid[N]), "%sed deletion", str_hl[t] );
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
srec->uid[1-t] = 0;
} else {
shrink some data at the source to avoid subsequent narrowing
2019-07-28 19:24:17 +00:00
uchar nflags = (srec->flags | srec->aflags[t]) & ~srec->dflags[t];
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
if (srec->flags != nflags) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "* %u %u %u", (srec->uid[F], srec->uid[N], nflags), "%sed flags; were %u", (str_hl[t], srec->flags) );
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
srec->flags = nflags;
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (t == N) {
shrink some data at the source to avoid subsequent narrowing
2019-07-28 19:24:17 +00:00
uchar nex = (srec->wstate / W_NEXPIRE) & 1;
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
if (nex != ((srec->status / S_EXPIRED) & 1)) {
srec->status = (srec->status & ~S_EXPIRED) | (nex * S_EXPIRED);
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "~ %u %u %u", (srec->uid[F], srec->uid[N], srec->status), "expired %d - commit", nex );
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
} else if (nex != ((srec->status / S_EXPIRE) & 1)) {
srec->status = (srec->status & ~S_EXPIRE) | (nex * S_EXPIRE);
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "~ %u %u %u", (srec->uid[F], srec->uid[N], srec->status), "expire %d - cancel", nex );
propagate deletions with other flag changes less code duplication, more logical order of issued driver commands (especially after the next commit), and the "side effect" of letting the message expiration code see those deletions if they are asynchronous.
2013-11-24 14:58:32 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
}
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
typedef struct {
void *aux;
message_t *msg;
} trash_vars_t;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void msg_trashed( int sts, void *aux );
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
static void msg_rtrashed( int sts, uint uid, copy_vars_t *vars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
msgs_flags_set( sync_vars_t *svars, int t )
{
message_t *tmsg;
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
trash_vars_t *tv;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
copy_vars_t *cv;
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
if (!(svars->state[t] & ST_SENT_FLAGS) || svars->flags_pending[t])
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
return;
sync_ref( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if ((svars->chan->ops[t] & OP_EXPUNGE) &&
(svars->ctx[t]->conf->trash || (svars->ctx[1-t]->conf->trash && svars->ctx[1-t]->conf->trash_remote_new))) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "trashing on %s\n", str_fn[t] );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
for (tmsg = svars->msgs[t]; tmsg; tmsg = tmsg->next) {
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
if ((tmsg->flags & F_DELETED) && !find_uint_array( svars->trashed_msgs[t].array, tmsg->uid ) &&
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
(t == F || !tmsg->srec || !(tmsg->srec->status & (S_EXPIRE|S_EXPIRED)))) {
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if (svars->ctx[t]->conf->trash) {
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
if (!svars->ctx[t]->conf->trash_only_new || !tmsg->srec || (tmsg->srec->status & (S_PENDING | S_SKIPPED))) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "%s: trashing message %u\n", str_fn[t], tmsg->uid );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
trash_total[t]++;
stats();
svars->trash_pending[t]++;
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
tv = nfmalloc( sizeof(*tv) );
tv->aux = AUX;
tv->msg = tmsg;
svars->drv[t]->trash_msg( svars->ctx[t], tmsg, msg_trashed, tv );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
if (check_cancel( svars ))
goto out;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "%s: not trashing message %u - not new\n", str_fn[t], tmsg->uid );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
} else {
move away from magic UIDs in the sync state the only legitimate "deviant" UID is zero, meaning "no message". this can be futher qualified by additional flags in the sync record, rather than using magic values for the UID. in fact, the zero UID (so far meaning only "expunged") was already optionally qualifed with "expired". as a side effect, driver->store_msg() now returns 0 instead of -2 for unknown UIDs. this was a hack to avoid translating the value later on, but it made the api horrible, and now it's superflous in the first place.
2017-03-11 12:20:53 +00:00
if (!tmsg->srec || (tmsg->srec->status & (S_PENDING | S_SKIPPED))) {
use INT_MAX instead of zero for "no size limit" this simplifies the actual conditions
2013-05-11 08:12:33 +00:00
if (tmsg->size <= svars->ctx[1-t]->conf->max_size) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "%s: remote trashing message %u\n", str_fn[t], tmsg->uid );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
trash_total[t]++;
stats();
svars->trash_pending[t]++;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
cv = nfmalloc( sizeof(*cv) );
cv->cb = msg_rtrashed;
fix TrashRemoteNew copy direction
2012-09-22 15:35:39 +00:00
cv->aux = INV_AUX;
consistently use NULL for null pointers makes the code noisier, but also somewhat more expressive.
2019-07-28 18:50:31 +00:00
cv->srec = NULL;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
cv->msg = tmsg;
create placeholders for messages over MaxSize this is vastly more useful than just omitting the messages with no indication at all.
2019-12-29 13:37:53 +00:00
cv->minimal = 0;
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
copy_msg( cv );
if (check_cancel( svars ))
goto out;
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "%s: not remote trashing message %u - too big\n", str_fn[t], tmsg->uid );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
} else {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "%s: not remote trashing message %u - not new\n", str_fn[t], tmsg->uid );
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
}
coding style: add "superfluous" braces - wrap flow-controlled statements that contain blocks into blocks themselves - wrap bodies of do-while()s into blocks - use braces on 'else' symmetrically (this obviously has a cascading effect, so this patch touches lots of lines) - attach braces unavoidably, the rules are sometimes broken around #ifdef-ery.
2022-01-08 13:12:18 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
svars->state[t] |= ST_SENT_TRASH;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
sync_close( svars, t );
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
out:
sync_deref( svars );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
msg_trashed( int sts, void *aux )
{
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
trash_vars_t *vars = (trash_vars_t *)aux;
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
DECL_SVARS;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if (sts == DRV_MSG_BAD)
sts = DRV_BOX_BAD;
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
if (check_ret( sts, vars->aux ))
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
INIT_SVARS(vars->aux);
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "T %d %u", (t, vars->msg->uid), "trashed on %s", str_fn[t] );
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
free( vars );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
trash_done[t]++;
stats();
svars->trash_pending[t]--;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
sync_close( svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make UIDs unsigned complies with the IMAP spec, thus removing the (not really) arbitrary limitation to INT_MAX for UIDs.
2017-03-21 19:05:29 +00:00
msg_rtrashed( int sts, uint uid ATTR_UNUSED, copy_vars_t *vars )
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
SVARS_CHECK_CANCEL_RET;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
switch (sts) {
case SYNC_OK:
case SYNC_NOGOOD: /* the message is gone or heavily busted */
break;
default:
cancel_sync( svars );
free( vars );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix TrashRemoteNew copy direction
2012-09-22 15:35:39 +00:00
t ^= 1;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
JLOG( "T %d %u", (t, vars->msg->uid), "trashed remotely on %s", str_fn[1-t] );
wrap message trashing into simple transactions trashing many messages at once inevitably overtaxes m$ exchange, and the connection breaks. without any progress tracking, it would restart from scratch each time, which would lead to a) it never finishing and b) many copies of the messages in the trash. full transactions as we do for "proper" syncing would be over the top, as it's not *that* bad if some messages get duplicated in the trash. so we record only the messages for which trashing completed, thus allowing some overlap between the attempts.
2016-11-04 20:23:39 +00:00
free( vars );
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
trash_done[t]++;
stats();
svars->trash_pending[t]--;
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
sync_close( svars, t );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void box_closed( int sts, void *aux );
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
static void box_closed_p2( sync_vars_t *svars, int t );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
sync_close( sync_vars_t *svars, int t )
{
make progress counters global which means they are now cumulative, and include channels and boxes.
2015-03-28 16:26:08 +00:00
if ((~svars->state[t] & (ST_FOUND_NEW|ST_SENT_TRASH)) || svars->trash_pending[t] ||
!(svars->state[1-t] & ST_SENT_NEW) || svars->new_pending[1-t])
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
return;
if (svars->state[t] & ST_CLOSING)
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
return;
ensure sequencing of message propagation and store closing by putting the message propagation last, d3f634702 uncovered a long-standing problem: we might have closed the source store before all messages were propagated from it.
2013-12-11 15:25:30 +00:00
svars->state[t] |= ST_CLOSING;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
if ((svars->chan->ops[t] & OP_EXPUNGE) /*&& !(svars->state[t] & ST_TRASH_BAD)*/) {
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
debug( "expunging %s\n", str_fn[t] );
make some driver function names more descriptive
2014-12-27 21:13:24 +00:00
svars->drv[t]->close_box( svars->ctx[t], box_closed, AUX );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
} else {
box_closed_p2( svars, t );
fix error paths wrt sync drivers
2006-12-09 10:39:30 +00:00
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
}
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
static void
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
box_closed( int sts, void *aux )
{
don't decode aux pointer on DRV_CANCELED the aux data was already free()d by the error callback by the time we get this status code.
2012-06-17 12:52:46 +00:00
SVARS_CHECK_RET;
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
svars->state[t] |= ST_DID_EXPUNGE;
box_closed_p2( svars, t );
}
static void
box_closed_p2( sync_vars_t *svars, int t )
{
sync_rec_t *srec;
svars->state[t] |= ST_CLOSED;
if (!(svars->state[1-t] & ST_CLOSED))
return;
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// All the journalling done in this function is merely for the autotest -
// the operations are idempotent, and we're about to commit the new state
// right afterwards anyway.
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
for (t = 0; t < 2; t++) {
// Committing maxuid is delayed until all messages were propagated, to
// ensure that all pending messages are still loaded next time in case
comment updates
2021-12-08 12:55:33 +00:00
// of interruption - in particular skipping messages would otherwise
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
// up the limit too early.
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
if (svars->maxuid[t] != svars->oldmaxuid[t])
JLOG( "N %d %u", (t, svars->maxuid[t]), "up maxuid of %s", str_fn[t] );
do away with newmaxuid now that expiration order is determined by a single loop ordered by far-side UIDs, it is no longer necessary to accurately track the highest seen UID. as a side effect, this fixes a problem reported (way too long ago) by Yuri D'Elia: we failed to up newmaxuid for messages we produced ourselves, so we would keep enumerating the same messages until we also propagated externally generated messages from that mailbox - which might have been never for the server side of archive/trash mailboxes.
2020-07-20 18:53:21 +00:00
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (((svars->state[F] | svars->state[N]) & ST_DID_EXPUNGE) || svars->chan->max_messages) {
make MaxMessages work for new mails as well this helps enormously on the first sync of a 100k message box with a limit of 1k messages. it also happens to make the syncing idempotent. in a few conditionals we now explicitly test for max_messages being enabled, not smaxxuid != 0, as after the initial fetch with no important messages smaxxuid is zero, but we still have to skip over 99k messages in the above case.
2013-11-30 12:03:12 +00:00
debug( "purging obsolete entries\n" );
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
for (srec = svars->srecs; srec; srec = srec->next) {
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
if (srec->status & S_DEAD)
continue;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
if (!srec->uid[N] || ((srec->wstate & W_DEL(N)) && (svars->state[N] & ST_DID_EXPUNGE))) {
if (!srec->uid[F] || ((srec->wstate & W_DEL(F)) && (svars->state[F] & ST_DID_EXPUNGE)) ||
((srec->status & S_EXPIRED) && svars->maxuid[F] >= srec->uid[F] && svars->maxxfuid >= srec->uid[F])) {
JLOG( "- %u %u", (srec->uid[F], srec->uid[N]), "killing" );
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
srec->status = S_DEAD;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
} else if (srec->uid[N]) {
JLOG( "> %u %u 0", (srec->uid[F], srec->uid[N]), "orphaning" );
srec->uid[N] = 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
} else if (srec->uid[F] && ((srec->wstate & W_DEL(F)) && (svars->state[F] & ST_DID_EXPUNGE))) {
JLOG( "< %u %u 0", (srec->uid[F], srec->uid[N]), "orphaning" );
srec->uid[F] = 0;
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
}
}
initial import
2000-12-20 21:41:21 +00:00
}
set compiler warnings for gcc-3.0 as well display message with count of uploaded messages --quiet now supresses warnings in sync_mailbox() fixed compiler warnings with -Wshadow
2001-10-31 19:50:01 +00:00
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
// This is just an optimization, so it needs no journaling of intermediate states.
// However, doing it before the entry purge would require ensuring that the
// exception list includes all relevant messages.
don't rewrite state gratuitously delay the creation of the new state and journal until there is actually something interesting to write. this saves some cpu cycles and prolongs ssd life a whee bit.
2019-11-25 19:55:41 +00:00
if (svars->maxxfuid != svars->oldmaxxfuid)
JLOG( "! %u", svars->maxxfuid, "max expired UID on far side" );
rewrite tracking of highest expired UID so far, we tracked the slave side, and calculated the master side on the fly. that complicated things unnecessarily.
2017-04-02 12:04:43 +00:00
factor out {prepare,lock,save,load}_state()
2014-12-27 22:13:45 +00:00
save_state( svars );
- update isyncmaxuid properly
2002-12-28 04:12:07 +00:00
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
sync_bail( svars );
}
static void
sync_bail( sync_vars_t *svars )
{
sync_rec_t *srec, *nsrec;
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
free( svars->trashed_msgs[F].array.data );
free( svars->trashed_msgs[N].array.data );
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
for (srec = svars->srecs; srec; srec = nsrec) {
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
nsrec = srec->next;
free( srec );
}
lock sync state lazily don't try to lock it until we actually read or write it. the idea is to not fail with SyncState * if we tried to load the state before selecting a non-existing mailbox. this is ok, because if the mailbox is missing, we obviously have no sync state pertaining to it, either. as a side effect, this allows simplifying an error path.
2014-12-27 22:50:31 +00:00
if (svars->lfd >= 0) {
unlink( svars->lname );
close( svars->lfd );
}
make the driver model, sync_chans() and sync_boxes() fully async. async drivers to follow ...
2006-03-21 20:03:21 +00:00
sync_bail2( svars );
}
static void
sync_bail2( sync_vars_t *svars )
{
async merge: aggregate most variables of main() & sync_boxes() in main_vars_t resp. sync_vars_t. also some minor var renames, whitespace, comments.
2006-03-21 15:53:43 +00:00
free( svars->lname );
free( svars->nname );
free( svars->jname );
free( svars->dname );
introduce ability to flatten the hierarchy of Stores
2012-08-18 11:58:14 +00:00
sync_bail3( svars );
}
static void
sync_bail3( sync_vars_t *svars )
{
deprecate master/slave terminology the underlying metaphor refers to an inhumane practice, so using it casually is rightfully offensive to many people. it isn't even a particularly apt metaphor, as it suggests a strict hierarchy that is counter to mbsync's highly symmetrical mode of operation. the far/near terminology has been chosen as the replacement, as it is a natural fit for the push/pull terminology. on the downside, due to these not being nouns, a few uses are a bit awkward, and several others had to be amended to include 'side'. also, it's conceptually quite close to remote/local, which matches the typical use case, but is maybe a bit too suggestive of actually non-existing limitations. the new f/n suffixes of the -C/-R/-X options clash with pre-existing options, so direct concatenation of short options is even less practical than before (some suffixes of -D already clashed), but doing that leads to unreadable command lines anyway. as with previous deprecations, all pre-existing command line and config options keep working, but yield a warning. the state files are silently upgraded.
2020-07-22 17:44:26 +00:00
free( svars->box_name[F] );
free( svars->box_name[N] );
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
sync_deref( svars );
initial import
2000-12-20 21:41:21 +00:00
}
The Big Rewrite. too many change to list them all. as opposed to earlier threats, BerkDB was not entirely dropped; i suppose the isync 0.7 -> 0.8 change had a reason, so i added an alternative UID storage scheme. note that BDB 4.0 is not sufficient, as the db->open function changed in an incompatible way ... i updated the debian packaging except for a changelog entry. note that i removed the upgrade blurb, as upstream now has a smooth upgrade path down to at least isync 0.4.
2004-03-27 16:07:20 +00:00
fix error paths wrt sync drivers, take 3 msgs_copied() was not checked at all, and msgs_flags_set() was doing it wrong (sync_close() was not checked). instead of trying to fix/extend the msgs_flags_set() model (ref-counting and cancelation checking in lower-level functions, and return values to propagate the status), place the refs/derefs around higher-level scopes and do the checking only there. this is effectively simpler, and does away with some obscure macros.
2013-12-11 15:13:49 +00:00
static void
sync_deref( sync_vars_t *svars )
fix error paths wrt sync drivers, take 2 synchronous error codes which are passed through callbacks aren't a particularly good idea, after all: latest when the callback does stuff which does not concern the caller, the return code becomes ambiguous. instead, protect the sync_vars object with a refcount when invoking driver functions from loops, as the callbacks they call could invalidate the object and we would have no way of knowing that the loop should be aborted prematurely. the upcoming async imap driver will also need a refcount to protect the cancelation marker of the imap socket dispatcher loop.
2012-07-29 21:14:48 +00:00
{
if (!--svars->ref_count) {
void (*cb)( int sts, void *aux ) = svars->cb;
void *aux = svars->aux;
int ret = svars->ret;
free( svars );
cb( ret, aux );
}
}
Reference in New Issue Copy Permalink