Hi developers,

This is one I said I'd send an email about last week and then forgot to do so...

There's a long standing problem with 'ctl_mboxlist -d' (dump mailboxes.db to a 
file) and 'ctl_mboxlist -u' (undump such a file back into a mailboxes.db): they 
do not know about all the fields mailboxes.db stores, so you lose the data from 
those fields in the process.  I don't know how long ago the tool became out of 
synchronisation with the data.

Relatedly, there's a long standing problem with the "improved_mboxlist_sort" 
option being off by default, meaning there's surely a lot of deployments stuck 
with weird sorting issues around space/hyphen/etc (which in ASCII order sort 
earlier than the hierarchy delimiters).  The documentation for this setting 
says that it must not be changed on a live system, and that you must dump and 
then undump your mailboxes.db between changing it.  But we can't be 
recommending people do that if it's going to lose data in the process.

Dilyan has provided a fix for ctl_mboxlist which correctly dumps and undumps 
all the fields that are present in mailboxes.db in 3.4: 
https://github.com/cyrusimap/cyrus-imapd/pull/3384  But the thing is, in master 
and 3.6, there are even more new fields, which the fix omits.  One of the new 
fields in 3.6/master is name_history, which is itself a multi-value field I 
think, so it won't be trivial to just further extend Dilyan's work to include 
it.

I think I plan to accept Dilyan's patch for 3.4, but even so, the new fields in 
3.6/master mean we will then have an immediate regression when 3.6 is released. 
 So while it's in beta I want to get this tool fixed up, and in a way that's 
forward compatible with whatever complicated new stuff we add next.  I was 
expecting to be spending some quality time on the tools this year, and I guess 
this might be the first candidate.

To that end, I want to change the dump format to something capable of 
representing complex data (such as name_history), and which we already have 
generic parser/deparsers for, which means probably dlist or json.

dlist may be cheaper, since I think all those "fields" are already stored as 
dlist format in the database anyway.  It could also retain some pretence at 
backward compatibility, if the existing dump format was kept mostly as is, with 
the new fields added in a dlist at the end of each row.

json, on the other hand, would fit in with our general plan of having some kind 
of json output from every tool.  If I were going to use json I would want the 
dump format to be pure json, not json chunks appended at the end of structured 
lines.  Though this would not be compatible with dumping a db from an old 
version of Cyrus and undumping it onto a new one!

I don't know if dump/undump across versions is a thing we expect to work.  I 
don't think I would expect it myself, but there's almost certainly someone out 
there who does and will be put out if we change the format.  Though arguably, 
since it loses data at the moment anyway, replacing the format entirely isn't 
making it any worse.

I'm interested in your input, particularly with regards to:

* dlist or json?

* sort-of-backwards-compatible or not-at-all?

* dragons I should be aware of?

Cheers,

ellie

------------------------------------------
Cyrus: Devel
Permalink: 
https://cyrus.topicbox.com/groups/devel/Tea9b3c8c5728d1e8-M7b68e56d5c41dafa2acbba63
Delivery options: https://cyrus.topicbox.com/groups/devel/subscription

Reply via email to