Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages. The data in co

Re: Cyrus IMAP in the next Debian

2019-02-11 Thread Lars Schimmer
On 2/8/19 6:39 PM, Paul van der Vlis wrote: > Op 08-02-19 om 11:48 schreef Paul van der Vlis: >> Hello, >> >> The freeze for the next version of Debian is in the next week. >> >> Somebody has packaged Cyrus version 3.08, but there are problems with >> some of the Cassandane tests. >> >> When my inf

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
Hi Bron, --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana wrote: The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. do you know why that does not seem to happen when using the "old" sync protocol for replication?

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
That sounds like the source messages have no thread id, and hence they aren't being stored. This is an interesting question actually, should we still store G keys for messages without thread identifier (CID)? Bron. On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: > Hi Bron, > > --On

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Hi Bron, So, it would be interesting to run once a day... for instance in cyrus.conf in events section : repack_xapian cmd="squatter -F" at=0200 Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as : START { # do not delete this entry! recover cmd="c

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
So running ctl_conversationsdb -z followed by -b would assign thread ids to those messages? Because it works when I do that. Clearly this is an edge case, but IMO it should be handled somehow other than silently failing ;-) --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana wrote: That

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Now I'm noticing for instance, for moving data between Xapian databases.. you need to launch something like : sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z archive -t temp,meta,data,archive -u user/ego...@sarenet.es perhaps would be better to do : sudo -u cyrus /usr/c

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! Bron. On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: > So running ctl_conversationsdb -z followed by -b would assign thread ids to > those messages? Beca

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
It's definitely safe to have one rolling mode writing and one repacking. I wouldn't run multiple repacks in parallel, as they can wind up doing duplicate work (though the end result should always be correct and safe). Here's what we run: # Any time the disk gets over 50%, compress -o single dow

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Hi mates! Just for finishing this thread... Two squatter proccesses then... one in rolling mode and another one for info movement between Xapian databases and repacking databases as Brong said... can be running without known issues?. I say for avoid damaging something... thanks a lot mates! Che

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Many many many many thanks a lot Brong!! :) :) :) :) :) :) :) --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerl

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
Thanks! I rolled my own RPM with that patch, and I can confirm that it works. --On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana wrote: Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! Bron. On Mon, Feb 1

Re: Cyrus IMAP in the next Debian

2019-02-11 Thread Paul van der Vlis
Op 11-02-19 om 10:45 schreef Lars Schimmer: > On 2/8/19 6:39 PM, Paul van der Vlis wrote: >> In some time it will be in Debian unstable and a few days later in >> testing. https://packages.debian.org/search?keywords=cyrus-imapd > > Hip Hip Hooray! > Thanks to all involved. > Will test as soon as

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Excellent - I hope you grabbed both that commit and the one afterwards where I fixed the order of CID parsing. Actually, it might not be as big a deal on 3.0, but not calculating the CID first did break one JMAP case on future. Cheers, Bron. On Tue, Feb 12, 2019, at 02:22, Sebastian Hagedorn

Re: cyrus-imapd not starting after upgrade

2019-02-11 Thread Max Kosmach
Hi! As far as I remember You should rebuild 2.5.20-3 package with at least those patches https://github.com/cyrusimap/cyrus-imapd/commit/1e7cf05a6bcba8c639fe8222c28851b1a9802e63 https://github.com/cyrusimap/cyrus-imapd/commit/575b6b4503a5566115a6259f98c812b2b465f8b6 8 февраля 2019 г. 22:46:00 G