Re: Removing email from Xapian tier databases

2019-02-12 Thread Sebastian Hagedorn
No, I hadn't, but I have now – just in case. Thanks, Sebastian --On 11. Februar 2019 um 11:40:46 -0500 Bron Gondwana wrote: Excellent - I hope you grabbed both that commit and the one afterwards where I fixed the order of CID parsing. Actually, it might not be as big a deal on 3.0, but not

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Excellent - I hope you grabbed both that commit and the one afterwards where I fixed the order of CID parsing. Actually, it might not be as big a deal on 3.0, but not calculating the CID first did break one JMAP case on future. Cheers, Bron. On Tue, Feb 12, 2019, at 02:22, Sebastian Hagedorn

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
Thanks! I rolled my own RPM with that patch, and I can confirm that it works. --On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana wrote: Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! Bron. On Mon, Feb 1

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Hi mates! Just for finishing this thread... Two squatter proccesses then... one in rolling mode and another one for info movement between Xapian databases and repacking databases as Brong said... can be running without known issues?. I say for avoid damaging something... thanks a lot mates! Che

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Many many many many thanks a lot Brong!! :) :) :) :) :) :) :) --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerl

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
It's definitely safe to have one rolling mode writing and one repacking. I wouldn't run multiple repacks in parallel, as they can wind up doing duplicate work (though the end result should always be correct and safe). Here's what we run: # Any time the disk gets over 50%, compress -o single dow

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! Bron. On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: > So running ctl_conversationsdb -z followed by -b would assign thread ids to > those messages? Beca

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Now I'm noticing for instance, for moving data between Xapian databases.. you need to launch something like : sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z archive -t temp,meta,data,archive -u user/ego...@sarenet.es perhaps would be better to do : sudo -u cyrus /usr/c

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
So running ctl_conversationsdb -z followed by -b would assign thread ids to those messages? Because it works when I do that. Clearly this is an edge case, but IMO it should be handled somehow other than silently failing ;-) --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana wrote: That

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea
Hi Bron, So, it would be interesting to run once a day... for instance in cyrus.conf in events section : repack_xapian cmd="squatter -F" at=0200 Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as : START { # do not delete this entry! recover cmd="c

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
That sounds like the source messages have no thread id, and hence they aren't being stored. This is an interesting question actually, should we still store G keys for messages without thread identifier (CID)? Bron. On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: > Hi Bron, > > --On

Re: Removing email from Xapian tier databases

2019-02-11 Thread Sebastian Hagedorn
Hi Bron, --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana wrote: The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. do you know why that does not seem to happen when using the "old" sync protocol for replication?

Re: Removing email from Xapian tier databases

2019-02-11 Thread Bron Gondwana
Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages. The data in co