Question about squatter for Xapian
Good morning, We have mailboxes indexed with Xapian. We run for that Squatter in rolling mode (-R) so that a log can create new index records while new mail enters.. is removed and so… Nightly too move records between tiers and so… In an emergency on one mailbox server, we moved traffic to what was the slave server, and from that moment obviously the master… but we did a mistake, we forgot setting in the new slave "sync_log: true" so no squatter rolling log was generated. We noted about it 12 hours later. As we have "search_fuzzy_always: 1” set all searches go through Xapian, so no indexed mail won’t never appear in the searchs run by the users. I think we could run squatter for indexing non indexed emails, but I was wondering : - Does it regenerate all mailboxes indexes?. Just the non-indexed emails?. I assume it should be extremely slow… so could this be launched?. Could you advise me please, if another way is preferred? - I assume not, but as we move records between Xapian tiers nightly… if the Squatter launched by me, by hand (for those non indexed emails), runs at the same time as this between tiers movement of records or at the same time too as the rolling mode squatter (-R) could one squatter process interfere in the job of the other instance of squatter?. Thank you so much, Best regards, Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Database upgrade and Xapian version dependency
Hi!! Thanks a lot for your nice answer :) . Totally logical, reasonable and clear answer. Nicely explained. I’ll share in the list all the relevant situations I’ll find. Thanks again ;) Egoitz, > El 29 ene 2020, a las 0:08, ellie timoney escribió: > On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: Just for having it slightly clearer… When you upgrade the Cyrus version and the version you are upgrading to is a too close one… for instance from 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail folders, 13 in both… is it needed to launch (or recommended for some reason) the final upgrade commands : reconstruct -V max ctl_conversationsdb -b -r quota -f > > As long as you're looking at 3.x.x and higher*, then if it's a stable release > (where only the third number has changed, e.g. 3.0.8->3.0.13) you shouldn't > need to. We don't do big world-breaking/data-format-changing changes in > stable releases like this. > > That said, we might have fixed a bug between the versions, and your existing > data might be bad due to the bug that was fixed, and you might need to run > commands like these after upgrading to the fixed version to repair the bad > data. If I know this is the case, I'll say so in the release notes (so check > those, including for the intermediate releases if you're skipping over some). > But I might not know until someone upgrades and reports that they needed to > do it. > > So, if it doesn't say you need to do it, you probably don't need to do it. > But if you don't do it, and things like quota or conversations seem weird > after the upgrade, try these sort of commands. And if it fixes it, let us > know (on the mailing list or a github issue) so we can retcon the release > notes to mention it for the next person. :) > > * For people looking at 2.5 and earlier, this pattern doesn't necessarily > hold, and I don't know enough about those older versions to provide general > advice. If in doubt, ask the mailing list, and if you can run a recent > version (that's closer to what's in Fastmail's organisational memory), that's > your best bet ;) > > Cheers, > > ellie > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian index not being used for message search in Roundcube
Hi Simon,
Try selecting the non conversation view before doing the search. It seems
Roundcube has something weird there… but I could assure you it uses it when you
set the non-conversation (non thread view). I think really it does even there,
but perhaps it enter in a non-controlled loop with all the messages of the
conversations or something similar… I think this is an issue in Roundcube
rather than in Cyrus… We had this same doubt some time ago,
Cheers,
> El 23 feb 2020, a las 23:54, ellie timoney escribió:
>
> I don't really understand search in any depth, but it's interesting to
> observe that, in addition to the different command (SEARCH vs THREAD), those
> two searches are also using different search criteria ("BODY linux" vs "TEXT
> linux").
>
> It might be informative to try do the SEARCH search with "TEXT linux" instead
> of "BODY linux", to narrow down whether the difference is due to the use of
> the SEARCH vs THREAD command, or the use of the "BODY" vs "TEXT" search key?
>
> Looking at the source on master, SEARCH and THREAD both seem to be using the
> same search API, so at a glance it seems like they should both be using
> Xapian if either is. And looking at the commit dates on those functions, it
> doesn't look like it's changed substantially since 3.0, at least not at a
> level I can easily see.
>
> I had a quick look at the RFC, and "BODY" searches just the message body,
> whereas "TEXT" searches both body and headers. So I wonder if the difference
> is that TEXT needs to open all the message files to read the headers, whereas
> BODY can just return results straight from the Xapian index?
>
> I'm not sure if there's been changes to header searching (like, maybe we
> index more of the header content?) since 3.0, but this is getting beyond what
> I know off the cuff or can just casually look up.
>
> Anyway, if you could try "UID SEARCH TEXT linux" and see if that's similarly
> slow to the THREAD version, that would give us a definite pointer in the
> right direction.
>
> Cheers,
>
> ellie
>
> On Sun, Feb 23, 2020, at 10:11 PM, Frederik Himpe via Info-cyrus wrote:
>> I have configured Cyrus 3.0.13 with the Xapian search engine and
>> enabled search_fuzzy_always. This appears to work fine when I search in
>> the message body using the Evolution mail client, as I get a response
>> quickly:
>>
>> <1582453709>> 1582453713>* SEARCH 226927
>> 226929 226964 226974 226999 227215 227238 [...]
>> L03163 OK Completed (643
>> msgs in 0.970 secs)
>>
>> However when I search messages using the Roundcube webmail client,
>> Roundcube does not get a response in time and shows no results. An
>> strace of the imapd proceess indicates it is STATing, OPENing and
>> MMAPing all files in the mailbox.
>>
>> This is the log:
>> <1582455581>> 1582455723>* THREAD
>> (229566)(229570)(229574)(229599)(229618)(229639)[...]
>> A0004 OK Completed (157 msgs in 11.340 secs)
>>
>> So it appears Roundcube is using a different command to search. Is it
>> expected that this command does not use the Xapian search engine? Is
>> there a way to make it use it?
>>
>> Some relevant snippets from imapd.conf:
>> sync_log: on
>> sync_log_channels: squatter
>>
>> conversations: 1
>> search_engine: xapian
>> search_index_headers: no
>> search_batchsize: 8192
>> search_fuzzy_always: 1
>> defaultsearchtier: temp
>> tempsearchpartition-default: /var/lib/cyrus/search.temp
>> datasearchpartition-default: /var/lib/cyrus/search.data
>>
>> cyrus.conf:
>>
>> EVENTS {
>>squatter1 cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus
>> squatter -z data -t temp,data" at=0517
>>
>> }
>> DAEMON {
>> squatter cmd="squatter -R"
>> }
>>
>>
>> Regards,
>>
>> --
>> Frederik Himpe
>>
>>
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Migrating calendars to Cyrus
Hi! Yes it exists. It’s called Vdirsyncer. Cherrs! Egoitz Aurrekoetxea Dpto. de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es <mailto:ego...@sarenet.es> www.sarenet.es <http://www.sarenet.es/> Antes de imprimir este correo electrónico piense si es necesario hacerlo. > El 21 feb 2020, a las 12:09, Andrea Venturoli escribió: > > Hello. > > I've been using CyrusIMAP for a long time, since before it supported > CalDAV/CardDAV. > At a site, I had installed DAVical, but I'd like to get rid of it, now that > Cyrus can do what it does. > > I've migrated most calendars and I'm left with only one user, who, > unfortunately, has more than 2k items in his calendar. > Doing it via a client is a pain. > > Is there something like imapsync I can use? > > bye & Thanks > av. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian index not being used for message search in Roundcube
Mmm I’ll try too to take a look at these when I have a little time… We though
all could came from a Roundcube bug when dealing with mail threads… apart from
that, we didn’t really know too how these search commands (that a THREAD
command existed) worked too… we should have tried seeing it by ngreping the
IMAP conversation, thing that we didn’t… it’s true…
Thanks mates :)
Bye!
Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> El 25 feb 2020, a las 21:47, Egoitz Aurrekoetxea via Info-cyrus
> escribió:
>
> Hi,
>
> Please try disabling the threads or conversación view in Roundcube and
> repleta the search,
>
> Cheers!
>
> Egoitz,
>
> El 24 feb 2020, a las 11:04, "ego...@sarenet.es <mailto:ego...@sarenet.es>"
> mailto:ego...@sarenet.es>> escribió:
>
>> Hi Simon,
>>
>> Try selecting the non conversation view before doing the search. It seems
>> Roundcube has something weird there… but I could assure you it uses it when
>> you set the non-conversation (non thread view). I think really it does even
>> there, but perhaps it enter in a non-controlled loop with all the messages
>> of the conversations or something similar… I think this is an issue in
>> Roundcube rather than in Cyrus… We had this same doubt some time ago,
>>
>> Cheers,
>>
>>
>>> El 23 feb 2020, a las 23:54, ellie timoney >> <mailto:el...@fastmail.com>> escribió:
>>>
>>> I don't really understand search in any depth, but it's interesting to
>>> observe that, in addition to the different command (SEARCH vs THREAD),
>>> those two searches are also using different search criteria ("BODY linux"
>>> vs "TEXT linux").
>>>
>>> It might be informative to try do the SEARCH search with "TEXT linux"
>>> instead of "BODY linux", to narrow down whether the difference is due to
>>> the use of the SEARCH vs THREAD command, or the use of the "BODY" vs
>>> "TEXT" search key?
>>>
>>> Looking at the source on master, SEARCH and THREAD both seem to be using
>>> the same search API, so at a glance it seems like they should both be using
>>> Xapian if either is. And looking at the commit dates on those functions,
>>> it doesn't look like it's changed substantially since 3.0, at least not at
>>> a level I can easily see.
>>>
>>> I had a quick look at the RFC, and "BODY" searches just the message body,
>>> whereas "TEXT" searches both body and headers. So I wonder if the
>>> difference is that TEXT needs to open all the message files to read the
>>> headers, whereas BODY can just return results straight from the Xapian
>>> index?
>>>
>>> I'm not sure if there's been changes to header searching (like, maybe we
>>> index more of the header content?) since 3.0, but this is getting beyond
>>> what I know off the cuff or can just casually look up.
>>>
>>> Anyway, if you could try "UID SEARCH TEXT linux" and see if that's
>>> similarly slow to the THREAD version, that would give us a definite pointer
>>> in the right direction.
>>>
>>> Cheers,
>>>
>>> ellie
>>>
>>> On Sun, Feb 23, 2020, at 10:11 PM, Frederik Himpe via Info-cyrus wrote:
>>>> I have configured Cyrus 3.0.13 with the Xapian search engine and
>>>> enabled search_fuzzy_always. This appears to work fine when I search in
>>>> the message body using the Evolution mail client, as I get a response
>>>> quickly:
>>>>
>>>> <1582453709>>>> 1582453713>* SEARCH 226927
>>>> 226929 226964 226974 226999 227215 227238 [...]
>>>> L03163 OK Completed (643
>>>> msgs in 0.970 secs)
>>>>
>>>> However when I search messages using the Roundcube webmail client,
>>>> Roundcube does not get a response in time and shows no results. An
>>>> strace of the imapd proceess indicates it is STATing, OPENing and
>>>> MMAPing all files in the mailbox.
>>>>
>>>> This is the log:
>>>> <1582455581>>>> 1582455723>* THREAD
>>>> (229566)(229570)(229574)(229599)(229618)(229639)[...]
>>>> A0004 OK Completed (157 msgs in 11.340 secs)
>>>>
>>>> So it appears Roundcube is using a different command to search. Is it
>>>> expected that this command does not use the Xapian search engine
Mailbox rename and Cyrus murder
Hi, Sometimes, we need to redistribute mailboxes between different Cyrus partitions in the same server. We do it by blocking previously any kind of accesses to the mailbox and after stopping any kind of active connection to that mailbox. I was wondering, if this exactly, is really needed because it’s really something that could cause problems to the mailbox (we do it this way always, as a procedure for avoiding problems, but is it really needed??). I’m in the process of automating this kind of operation, but reading some info before writing code, I have seen that Cyrus murder allows “live” mailbox migration… perhaps is this another option for avoid doing mailbox rename operations for moving a mailbox to a different partition?. I have read too (perhaps have not understand properly), that a Cyrus Murder can redirect requests to another mailbox server, in case one of them fails?. Is it like a automated way of changing master/slave roles?. How is your experience with murders?. Do they allow this what I have asked?. If they do have you ever seen any kind of issue with it?, Cheers, Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Mailbox rename and Cyrus murder
Hi!, I have been reading this interesting thread https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2015-March/003165.html from 2015. As a curiosity, did finally that ended in some sort of production env?. I have seen this kind of pages https://www.cyrusimap.org/imap/concepts/features/server-aggregation.html but have not seen their publish date or similar… I’m in the state at which, operations in mail servers are being more and more frequent… they are starting not being able to be done without automation, but before that.. and writing code… I wanted to know if someone could share some experience in horizontal scaling (backend or mailbox server clustering), because live moving mailboxes without interruption or in the most possible easy way (just as examples), are important things to know before writing all the automation stuff… knowing if a Cyrus Murder could failover from backend automatically (after a crash) or similar… Reading at present https://www.cyrusimap.org/imap/concepts/features/server-aggregation.html … Any shared experience would be highly appreciated, Cheers! Egoitz Aurrekoetxea Dpto. de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es <mailto:undefined> www.sarenet.es <http://www.sarenet.es/> Antes de imprimir este correo electrónico piense si es necesario hacerlo. > El 29 abr 2020, a las 17:58, ego...@sarenet.es escribió: > > Hi, > > Sometimes, we need to redistribute mailboxes between different Cyrus > partitions in the same server. We do it by blocking previously any kind of > accesses to the mailbox and after stopping any kind of active connection to > that mailbox. I was wondering, if this exactly, is really needed because it’s > really something that could cause problems to the mailbox (we do it this way > always, as a procedure for avoiding problems, but is it really needed??). I’m > in the process of automating this kind of operation, but reading some info > before writing code, I have seen that Cyrus murder allows “live” mailbox > migration… perhaps is this another option for avoid doing mailbox rename > operations for moving a mailbox to a different partition?. > > I have read too (perhaps have not understand properly), that a Cyrus Murder > can redirect requests to another mailbox server, in case one of them fails?. > Is it like a automated way of changing master/slave roles?. > > How is your experience with murders?. Do they allow this what I have asked?. > If they do have you ever seen any kind of issue with it?, > > Cheers, > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Question about replication, split-brains and unclean failover
Hi!, I was writing some code for automating the server fail-over and was trying to see, how should or could I handle not run log files from sync_client. Well when in a clean shutdown, it’s pretty easy to know how to manage because the replication is up-to-date… so almost no problem there, it’s pretty fast... The problem comes in an unclean shutdown where some delay exists. I sometimes suffer about unclean shutdowns, the way are described here https://www.fastmail.com/help/technical/architecture.html "Unclean failover”. There sais too, some improvements where going to be done (or that were committed perhaps) to the replication in order to avoid them. But, by the way replication is handled (as I have seen in the source code about locks, modseq checks and so) and the way Cyrus writes replication logs for rolling replication later, the possibility of replying the logs from an actual slave (just failed over), for covering a split-brain I assume is not carried out at least nowadays?. Perhaps am I wrong?. I ask this just, for confirming and avoid having wrong ideas… If it’s undone or at least partially undone, I would love really doing something… although unfortunately due to our high work load... I can’t say when I could have some time for it… but I’ll try my bests... Cheers! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: CalDav CardDav webmail client ?
Hi, Although we at present, are not running Cyrus Caldav/Carddav, Davical instead (it comes from long time ago), we are running Roundcube. I adapted the Caldav Kolab plugin in order to even support Free/Busy and for fixing some bug… I should have uploaded it to Github or wherever, but I have not have time for getting it ready for that. By the way, it’s not done for Roundcube 1.4. It’s for Roundcube 1.3. We even use Caldavzap (with a lot of js work too) with Roundcube as tasks plugin too. The only important thing in the combination of all of them, is that you need to use a specific version of Sabre for the contacts plugin and there’s a packaged version of the Contacts plugin with the needed Sabredav version which allows you to run it combined with the Kolab caldav plugin, although it’s slightly limited and slightly buggy (Kolab caldav)…. I hope to have some time for uploading our patches next months… I have not tested them and don’t expect them to work in Roundcube 1.4. by the way… sadly... Cheers, > El 12 jun 2020, a las 9:35, Marco escribió: > > Hello, > > I'm interesting in Cyrus Caldav/Carddav server. I never used it before, > because most webmail solutions implement their own calendar and addressbook. > > I wonder if I can interface Cyrus CalDav/CardDav in some opensource webmail, > such as SOGo, Roundcube, or Horde Webmail. > > Could you help me or tell me about your experience about these integrations? > > Thank you very much > Greetings > Marco > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: CalDav CardDav webmail client ?
Hi!
Answering below!!
> El 17 jun 2020, a las 15:09, Niels Dettenbach escribió:
>
> Am Mittwoch, 17. Juni 2020, 14:39:36 CEST schrieb ego...@sarenet.es:
>> Although we at present, are not running Cyrus Caldav/Carddav, Davical
>> instead (it comes from long time ago), we are running Roundcube. I adapted
>> the Caldav Kolab plugin in order to even support Free/Busy and for fixing
>> some bug… I should have uploaded it to Github or wherever, but I have not
>> have time for getting it ready for that. By the way, it’s not done for
>> Roundcube 1.4. It’s for Roundcube 1.3. We even use Caldavzap (with a lot
>> of js work too) with Roundcube as tasks plugin too.
> Thanks for that tip,
>
> but it seems requiring PHP5 (which is obsolete today) and don't get any
> secuity updates anymore.
>
You could get it working with PHP7…. It has some work… but you can….
Cheers!!
> It seems that even many other famous open source projects for CalDAV/CardDAV
> servers stacks are still more maintained - including i.e. Apples Darwin
> Calendar Servers (DCS):
> https://www.calendarserver.org/
>
> We use Cyrus IMAP with Horde5 to provide CalDAV/CardDAV as "MS Exchange Sync"
> emulation ("newer ActiveSync") "out of one box" plus a "nice" (responsive)
> Groupware web GUI, but the Calendar / Adressbook server stack is still used
> from Horde (on SQL) while Cyrus serves the Emails for Horde only. Horde5 now
> works on PHP7.x and is still maintained.
>
> It would be nice to get that Horde5 stack completely run with Cyrus as this
> would offer even more compatibility / protocols as the very known "Cyrus
> robustness"...ß) ...and on the longer run Horde out of the path for
> compatible clients / users.
>
> just my .02$
>
>
>
> niels.
>
> --
> ---
> Niels Dettenbach
> Syndicat IT & Internet
> http://www.syndicat.com
> PGP: https://syndicat.com/pub_key.asc
> ---
>
>
>
>
>
>
>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
