Stuart,
Thanks again for all the explanations!
> Yes. Try www/lighttpd for a simple example of this (vim is a poor example to
> crib from as it has a combination of multipackages *and* flavours).
>
> games/cataclysm-dda is a more complex example (though still just flavours not
> multipackages) using PFRAG.no_x11 for files which are only in the no_x11
> flavour, PFRAG.no-no_x11 for files which are _not_ in the no_x11 flavour, and
> PLIST has the files common to both flavours.
OK, this part is completely clear now, thanks!
> With flavours, all of the individual choices should be built as part of a bulk
> build, to make sure that any breakage in non-default options is picked up,
> plus whatever combinations are useful for real-world use should be built too.
Do I understand it correctly that apart from the common use-case flavor
combinations, there should be built an all-on combination too to test that
everything builds as expected?
> There is also ellie's reply, which makes a lot of sense at least in
> cases where it doesn't pull in a bunch of extra dependencies:
>
> "I would simply provide one package with all the
> stable/non-experimental features enabled, and admins can
> just not use the features they don't need."
Ellie is the one who prepares cyrus-imapd releases. We worked with her a lot on
all the fixes for the past 6 months, she has a lot of experience and I tend to
agree with her on most things, we also recently updated together [1] the
"compiling cyrus-imapd" doc page [2] with all the latest options and
non-obvious dependencies, but in this case I have to disagree.
The situation with cyrus-imapd is something like this. Let's see all relevant
configure features available as returned by the port's current cyr_buildinfo
(I removed some irrelevant ones that are always on or don't apply):
{
"component": {
"gssapi": false,
"autocreate": true,
"idled": true,
"httpd": false,
"kerberos_v4": false,
"murder": true,
"nntpd": true,
"replication": false,
"sieve": true,
"calalarmd": false,
"jmap": false,
"objectstore": false,
"backup": false,
},
"dependency": {
"ldap": false,
"clamav": false,
"ucdsnmp": false,
"netsnmp": false,
"openio": false,
"nghttp2": false,
"wslay": false,
"brotli": false,
"xml2": false,
"ical": false,
"shapelib": false,
"chardet": false
},
"database": {
"mysql": false,
"pgsql": false,
"sqlite": false
},
"search": {
"squat": true,
"xapian": false
}
}
For the empty base flavor I'd turn off nntp (IMO should always be turned off,
not sure if someone is still using it at all nowadays, it's an old code base
with a previous 7.5 score CVE), murder and squat as these are not the essential
features (but save murder for later).
With this combination we have a base POP3/IMAP server that is fully functional
and is suitable for use-cases like hosting providers that offer their clients
basic POP3/IMAP email accounts without anything special, but they require the
setup to be as simple and as solid as possible.
This combination has a few direct dependencies: pcre sqlite3 cyrus-sasl icu4c
e2fsprogs jansson + 2 indirect ones: libiconv and gettext-runtime. Quite a clean
install.
Now, if we turn on all the available options, we'd have dozens of direct
dependencies and a totally unnecessarily bloated active code-base. So IMO the
starting point should be the trimmed down basic IMAP server with as few
dependencies as possible.
The next use-case is a setup for small/mid-sized organizations that, apart from
basic IMAP, also want remote/shared calendaring support, reminders and
server-side scheduling with free/busy states (CalDAV), remote/shared contacts
(CardDAV), remote/shared folders (WebDAV), etc.
All this is enabled with --enable-http option and it brings some 12 additional
direct dependencies, but IMO together with the base IMAP, this is the most
typically used deployment: solid, easy to setup and works well with most
widely-used clients.
Then the next use-case is for larger organizations that have enough resources
(both monetary and human) to run cyrus-imapd on multiple servers with
duplication, triplication, .. of mailstore. For them there are the following
options:
--enable-murder: server aggregation in an imap cluster - a separation of front
and back-end servers, sharding of the mailstore, HA, etc.,
--enable-replication: replication of the mailstore between multiple machines,
and
--enable-backup: an experimental feature similar to the replication that runs a
backup instance on another server and duplicates there the mailstore for
non-realtime backups.
I'd group these 3 features together under the "replication" combination as they
share the same concept, could be used together and all imply certain resources
and know-how to deploy these setups.
So I'd provide the 3 combinations (base, http, replication) as pre-built
binaries. And possibly the 4th one that combines the http (*DAV) features with
the 3 replication features: http + replication.
And then there are niche features like DB back-ends (mysql, pgsql), objectstores
(caringo & openio), auth/directory (kerberos, gssapi, ldap), search (squat,
xapian), JMAP protocol (semi-experimental, AFAIK not all of the related RFCs are
fully ironed out yet and I don't know of any widely-used client supporting it,
but it's an interesting protocol and I'd enable it by default when a major
client like TB implements it (it's in the plans BTW [3]); it requires
--enable-http and --enable-xapian) and a misc feature clamav integration.
I'd make all these options as flavors (except the objectstore ones as they
require paid products for integration and their support in cyrus was sponsored
by the respective vendors) with their corresponding dependencies, PLIST, etc.
correctly specified, but I wouldn't build them by default.
This way we'd offer the pkg_add users only 3-4 clearly defined combinations, but
at the same time in the ports the advanced users would be able to build their
perfect combination.
How would you recommend to proceed?
Regards,
Anatoli
[1] https://github.com/cyrusimap/cyrus-imapd/pull/3095
[2] https://www.cyrusimap.org/imap/developer/compiling.html
[3]
https://developer.thunderbird.net/planning/roadmap#jmap-support-tb-2021-tb-2022
On 23/11/20 08:39, Stuart Henderson wrote:
> On 2020/11/23 07:31, Anatoli wrote:
>> Stuart,
>>
>> Thanks a lot for the explanation! Much appreciated!
>>
>> cyrus-imapd project is mostly modular (e.g. if it's built with --enable-http,
>> there would be additional binary httpd), but there are also some elements
>> that
>> hold the compilation information, like cyr_buildinfo [1] which returns the
>> details about the components, dependencies, etc. that were activated or not
>> during the build:
>
> ok, so subpackages won't work for this - seems it also links libcyrus
> against all the various libraries used by the various components. (last
> time I ran cyrus imapd it was in 2.2.something days so I'm not familiar
> with current development :-)
>
>>> If you've built it with everything included, then there is no particular
>>> advantage to packaging up subsets of that. You might think "we can avoid
>>> bloat by excluding the features we don't need", but in this case all the
>>> APIs
>>> to support those features would still be in libcyrus, you would just be
>>> missing binaries. You could have achieved the same effect by just not
>>> running
>>> those services in cyrus.conf
>>
>> I guess it would be more appropriate to proceed with flavors.
>
> There is also ellie's reply, which makes a lot of sense at least in
> cases where it doesn't pull in a bunch of extra dependencies:
>
> "I would simply provide one package with all the
> stable/non-experimental features enabled, and admins can
> just not use the features they don't need."
>
>> I was thinking about actually defining a number of flavors (basically for
>> most
>> components and dependencies as returned by cyr_buildinfo), but to build only
>> 4
>> combinations as part of the build infrastructure (the most clear use-cases),
>> so
>> the rest of possible combinations could be activated by the users in the
>> ports.
>>
>> Does that sound good?
>
> With flavours, all of the individual choices should be built as part of
> a bulk build, to make sure that any breakage in non-default options is
> picked up, plus whatever combinations are useful for real-world use
> should be built too.
>
> When the port is updated later, the set of different "standard build"
> flavours should be tested. When deciding what flavours to support, this
> extra work for every update needs factoring in :-)
>
> Additionally consider the case where new users install the software;
> pkg_add will present the big list of options which can be a bit
> overwhelming if a flavour is provided for every build option.
>
> Generally my approach would be to add flavours for things which are
> fairly popular but that also some users will really not want to use.
> This could either be "positive" flavours to enable those features,
> or "negative" flavours i.e. "no_whatever" to disable them (for example
> many ports have a no_x11 flavour). I would tend towards "negative"
> for core features that some people may want to avoid (considering
> cyrus then http might be a candidate for this) - and "positive" for
> things pulling in heavier-weight extra dependencies.
>
>> And if I understood it correctly, I'd have to use PFRAGs and %%flavor%% in
>> PLIST
>> to define the build differences for each flavor, right?
>
> Yes. Try www/lighttpd for a simple example of this (vim is a poor
> example to crib from as it has a combination of multipackages *and*
> flavours).
>
> games/cataclysm-dda is a more complex example (though still just
> flavours not multipackages) using PFRAG.no_x11 for files which are only
> in the no_x11 flavour, PFRAG.no-no_x11 for files which are _not_ in
> the no_x11 flavour, and PLIST has the files common to both flavours.
>
>> Anatoli
>>
>> [1]
>> https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_buildinfo.html
>> [2]
>> https://cyrus.topicbox.com/groups/devel/T5acbcb9536dc47c3/cyrus-imapd-packaging-flavors-variations
>>
>>
>> On 22/11/20 10:53, Stuart Henderson wrote:
>>> On 2020/11/22 00:27, Anatoli wrote:
>>>> Then, I'm working now on the flavored version of the port, and my idea is
>>>> to
>>>> apply it as soon as the new minor version is published (or maybe even
>>>> before
>>>> this so not to deal with the REVISION) but this is my first time working
>>>> with
>>>> ports, so I have no experience with the process and I have a couple of
>>>> questions
>>>> about some aspects of flavors definition.
>>>
>>> Is the build "modular"? That is, do the build options just have it build
>>> additional features in separate files (.so modules etc), or do they change
>>> the main binaries too?
>>>
>>> For builds which are modular, generally we do a single build and split
>>> the various files into subpackages. See for example PHP, Asterisk,
>>> Dovecot. MULTI_PACKAGES is defined with a list of subpackage names,
>>> the port is built once, packages are created for each subpackage
>>> using the relevant "PLIST-sub" file.
>>>
>>> (Heading off something you will see in some of these ports - there are
>>> "no_*" pseudo-flavours in some of these which are an optimization to
>>> reduce build dependencies for people building themselves rather than
>>> using packages - ignore these initially, they would be something to add
>>> later, if at all).
>>>
>>>> 1: vim-8.2.1805p0-gtk2
>>>> 2: vim-8.2.1805p0-gtk2-lua
>>>> 3: vim-8.2.1805p0-gtk2-perl-python-ruby
>>>> 4: vim-8.2.1805p0-gtk2-perl-python3-ruby
>>>> 5: vim-8.2.1805p0-gtk3
>>>> 6: vim-8.2.1805p0-gtk3-lua
>>>> 7: vim-8.2.1805p0-gtk3-perl-python-ruby
>>>> 8: vim-8.2.1805p0-gtk3-perl-python3-ruby
>>>> 9: vim-8.2.1805p0-no_x11-lua
>>>> 10: vim-8.2.1805p0-no_x11
>>>> 11: vim-8.2.1805p0-no_x11-perl-python-ruby
>>>> 12: vim-8.2.1805p0-no_x11-perl-python3-ruby
>>>> 13: vim-8.2.1805p0-no_x11-python
>>>> 14: vim-8.2.1805p0-no_x11-python3
>>>> 15: vim-8.2.1805p0-no_x11-ruby
>>>
>>> Solène explained how this list is defined. vim (and some other things
>>> like mutt) are just "either/or", so there's no alternative to building
>>> it multiple times. But it's better to avoid building what is essentially
>>> the same port 15 times if possible :)
>>>
>>>> Then I have some other questions like what does 'M'(option1) and 'L' mean
>>>> in
>>>> places like: .if ${FLAVOR:Moption1} or .if ${FLAVOR:L:Mcrypto}? I've seen
>>>> other
>>>> letters (F) being used in a similar way, but haven't seen anything in the
>>>> docs
>>>> explaining what they mean.
>>>
>>> :L is lowercase, :M is match, these are documented in make(1), "Each
>>> modifier begins with a colon and [...]"
>>>
>>> F isn't in this list, is that just a variable defined in a Makefile
>>> somewhere?
>>>
>>>> How PLIST, DESCR files should be prepared and what's the purpose of the
>>>> PFRAG.xx
>>>> files? Looks like PLIST and DESCR could have versions for multi-packages,
>>>> but
>>>> not for the flavors, though PFRAG files appear to be flavorable, but not
>>>> sure
>>>> how exactly and what content/format they should have.
>>>
>>> PFRAGs handle files which exist only in certain flavours of the port.
>>> If a port has a flavour "foo" and PLIST has a %%foo%% entry, PFRAG.foo
>>> is inserted into the PLIST at that point only when the build is done using
>>> that flavour. Likewise if there is a !%%foo%%, PFRAG.no-foo is inserted
>>> for all flavours *except* foo.
>>>
>>> (This mechanism can also handle the case where there are differences
>>> in the file list between different machine archs by defining a variable
>>> for pkg_create - you won't need it here but for an example see how
>>> PKG_ARGS is set in lang/go-bootstrap).
>>>
>>>> So how should I update the PLIST in order to accommodate flavor
>>>> differences if
>>>> it's a single version for all flavors? And should I create other files
>>>> (PFRAG)?
>>>>
>>>>
>>>> And a side note: this port probably should NOT be built on 32-bit archs
>>>> due to
>>>> the devs supposition that all time-related structures are 64-bit and there
>>>> could
>>>> be security implications if this is not the case (signedness and int
>>>> overflow).
>>>> Here's a discussion about this without a definitive conclusion [6].
>>>
>>> This only relates to the size of time_t, not the whole architecture.
>>> NetBSD changed time_t to 64-bit on all architectures in 6.0 (2012),
>>> OpenBSD did the same in 5.5 (2014). Linux now has support too (in 5.6,
>>> this year) but maintained the old ABI and with the long term
>>> binary/library compatibility I guess it will be quite some time before
>>> software actually running on most people's machines will be rebuilt for
>>> this. Looks like FreeBSD still uses 32-bit on i386 but 64-bit on others
>>> (including non-i386 32-bit).
>>>
>>> We still run into bugs in application software that assumes time_t is
>>> "long", especially in format strings (the standard approach we take in
>>> ports when patching things is to use long long format strings, and cast
>>> the variable to long long so that the patch can be fed upstream without
>>> breaking things on other OS). Presumably as Cyrus devs are aware of
>>> this they will be taking reasonable care!
>>>
