Re: Use of FATE (FFmpeg Automated Testing Envionment) data?

[W. Martin Borgert]

On 2019-02-20 06:19, Adam Borowski wrote:
> On Tue, Feb 19, 2019 at 11:52:49PM +0100, W. Martin Borgert wrote:
> > In my case, it's only 3.3 MiB.  And it is only in the source package
> > anyway.
>
> So you can just ship it then.

After clarifying the license(s).

Re: Use of FATE (FFmpeg Automated Testing Envionment) data?

[W. Martin Borgert]

On 2019-02-19 23:41, Holger Levsen wrote:
> On Wed, Feb 20, 2019 at 12:06:30AM +0100, Chris Lamb wrote:
> > Noted, but it is not the size that is the concern, but rather the
> > licensing.
>
> once those (licencing concerns) are resolved, I wonder whether it makes
> sense to provide a fate-testdata package which then other packages can
> build-depend on?

Yes. That's mainly something the maintainers of ffmpeg and
depending packages need to decide.

Bug#922752: ITP: r-cran-deoptimr -- differential evolution optimization in pure R

[Andreas Tille]

Package: wnpp
Severity: wishlist
Owner: Andreas Tille 

* Package name: r-cran-deoptimr
  Version : 1.0
  Upstream Author : Eduardo L. T. Conceicao, Martin Maechler
* URL : https://cran.r-project.org/package=DEoptimR
* License : GPL-2+
  Programming Lang: GNU R
  Description : differential evolution optimization in pure R
 Differential Evolution (DE) stochastic algorithms for global
 optimization of problems with and without constraints.
 The aim is to curate a collection of its state-of-the-art variants that
 .
  (1) do not sacrifice simplicity of design,
  (2) are essentially tuning-free, and
  (3) can be efficiently implemented directly in the R language.
 .
 Currently, it only provides an implementation of the 'jDE' algorithm by
 Brest et al. (2006) .

Remark: This package is maintained by Debian R Packages Maintainers at
   https://salsa.debian.org/r-pkg-team/r-cran-deoptimr

Bug#922759: ITP: kmerresistance -- correlates mapped genes with the predicted species of WGS samples

[Andreas Tille]

Package: wnpp
Severity: wishlist
Owner: Andreas Tille 

* Package name: kmerresistance
  Version : 2.0+git20180205.26467e9
  Upstream Author : , Philip Clausen, Technical University of Denmark
* URL : https://bitbucket.org/genomicepidemiology/kmerresistance
* License : Apache-2.0
  Programming Lang: C
  Description : correlates mapped genes with the predicted species of WGS 
samples
 KmerResistance correlates mapped genes with the predicted species of WGS
 samples, where this allows for identification of genes in samples which
 have been poorly sequenced or high accuracy predictions for samples with
 contamination. KmerResistance has one dependency, namely KMA to perform
 the mapping, which is also freely available.

Remark: This package is maintained by Debian Med Packaging Team at
   https://salsa.debian.org/med-team/kmerresistance

Re: libnode *might* need abi bump related to openssl 1.1.1 api break... or not...

[Jérémy Lal]

Le lun. 18 févr. 2019 à 21:16, Sebastian Andrzej Siewior
 a écrit :

> On 2019-02-18 11:08:30 [+0100], Jérémy Lal wrote:
> > hi !
> Hi,
>
> > It happens that that api breakage has been reverted and is merged in
> > openssl 1.1.1 stable branch:
> >
> https://github.com/openssl/openssl/commit/37857e9b5258da148e5d3699b6acdf8787417eb2
> >
> > If openssl releases this and the release goes to buster, then all is fine
> > in nodejs land.
>
> I would expect to upload 1.1.1b to unstable/testing/buster once it is
> released by upstream (I have no idea when this happens). The latest
> 1.1.0/1.0.2 was also uploaded to Stretch as part of a security upload.
>

That's February 26th for openssl 1.1.1b:
https://mta.openssl.org/pipermail/openssl-announce/2019-February/000145.html



>
> > However it's a special case that needs release team approval and
> > coordination,
> > between openssl and nodejs.
> >
> > Thanks for the feedback,
> > Jérémy
>
> Sebastian
>

Bug#922775: ITP: pandoc-plantuml-filter -- Pandoc filter which converts PlantUML code blocks to PlantUML images

[Hanno Stock]

Package: wnpp
Severity: wishlist
Owner: Hanno Stock 

* Package name: pandoc-plantuml-filter
  Version : 0.1.1
  Upstream Author : Timo Furrer 
* URL : https://github.com/timofurrer/pandoc-plantuml-filter
* License : MIT
  Programming Lang: Python
  Description : Pandoc filter which converts PlantUML code blocks to 
PlantUML images

This is a filter program for pandoc, which replaces PlantUML code blocks
in pandoc input by plantuml generated images.

---

I'll package this anyway for work, so I can as well maintain it for debian. My
colleagues will be using it, so there will be a "test base". 

Unifying logging by default

[Josh Triplett]

[Followups to debian-devel with CC to me, please, as I'm not
subscribed.]

Currently, while the majority of daemons support logging to syslog
and/or journald, a small handful of software ships configured to use its
own logfiles by default. For instance, update-alternatives logs to
/var/log/alternatives.log, aptitude logs to /var/log/aptitude, dpkg logs
to /var/log/dpkg.log, fontconfig logs to /var/log/fontconfig.log, CUPS
logs under /var/log/cups, and so on. This makes it more difficult to get
a unified view of system log data, ship logs off-system, analyze logs
for unusual activity, or any number of other policies a sysadmin may
wish to apply to logging.

While there are *absolutely* configurations in which system
administrators want to log to arbitrary locations and files, I would
like to propose that for consistency we should configure software to
unify logging into syslog and/or journald by default. In particular, I'd
propose that daemons and applications should default to logging to
syslog (working with any installed syslog implementation providing
/dev/log), and that daemons run via systemd units may detect or be
configured to log to stdout/stderr which will be wired to the sysadmin's
preferred log target.

This should Do The Right Thing by default, both on systems running any
syslog daemon and on systems running journald, without a hard dependency
on either one. And sysadmins would still have the option of configuring
any daemon they wish to log to any location they wish.

This would also mean that daemons no longer need to install directories
to /var/log, no longer need to install logrotate configuration files or
suggest/recommend/depend on logrotate, and in some cases may be able to
avoid any special permissions just to open a logfile.

Eventually, as we configure most applications for this default, we could
introduce this into policy as a "should", but for now I'm seeking
interest in slowly adapting applications to shift defaults to unified
logging.

- Josh Triplett

Re: Unifying logging by default

[Russ Allbery]

Josh Triplett  writes:

> While there are *absolutely* configurations in which system
> administrators want to log to arbitrary locations and files, I would
> like to propose that for consistency we should configure software to
> unify logging into syslog and/or journald by default. In particular, I'd
> propose that daemons and applications should default to logging to
> syslog (working with any installed syslog implementation providing
> /dev/log), and that daemons run via systemd units may detect or be
> configured to log to stdout/stderr which will be wired to the sysadmin's
> preferred log target.

I've seen three main reasons why packages do not use syslog:

* The log data is not line-structured.  Example: /var/log/apt/history.log
  Dumping that as-is into syslog sounds like an awful idea.  This would
  therefore block on an upstream feature request to change log formats
  first, with all that entails.

* Performance.  Good examples here are Apache and INN, both of which have
  per-request logs that are intentionally not line-buffered and are logged
  directly to files, since losing some log messages is not as much of a
  concern as performance and disk I/O.  I've tried to reroute things like
  that through syslog in the past, only to find syslog now taking a
  substantial portion of all available system resources trying to keep up.

* The package does its own log analysis.  INN again is an excellent
  example: it has a bespoke but very comprehensive and incredibly useful
  log analysis package that ships with the software and requires access to
  the raw logs.  Anything that moves those logs out of their expected
  locations will break that tool, which is a significant regression for
  our users.

I think you're going to need a much more concrete and specific proposal
with worked examples of how to deal with cases like this.  As is, I think
this proposal is much too vague to really discuss, let alone implement.

-- 
Russ Allbery (r...@debian.org)   

Re: Unifying logging by default

[Josh Triplett]

On Wed, Feb 20, 2019 at 02:19:02PM -0800, Russ Allbery wrote:
> Josh Triplett  writes:
> > While there are *absolutely* configurations in which system
> > administrators want to log to arbitrary locations and files, I would
> > like to propose that for consistency we should configure software to
> > unify logging into syslog and/or journald by default. In particular, I'd
> > propose that daemons and applications should default to logging to
> > syslog (working with any installed syslog implementation providing
> > /dev/log), and that daemons run via systemd units may detect or be
> > configured to log to stdout/stderr which will be wired to the sysadmin's
> > preferred log target.
> 
> I've seen three main reasons why packages do not use syslog:
> 
> * The log data is not line-structured.  Example: /var/log/apt/history.log
>   Dumping that as-is into syslog sounds like an awful idea.  This would
>   therefore block on an upstream feature request to change log formats
>   first, with all that entails.

Both syslog and journald support multi-line log messages; I'd *love* to
see /var/log/aptitude and /var/log/apt/history.log end up in syslog or
journald.

I'd also have no problem with an exception explicitly saying that
something like /var/log/apt/term.log can continue to use a standalone
file iff there isn't a good solution for storing that in a structured
log message. (And even that *could* go into a structured log message
with some care.) I'm trying to handle the common case here.

> * Performance.  Good examples here are Apache and INN, both of which have
>   per-request logs that are intentionally not line-buffered and are logged
>   directly to files, since losing some log messages is not as much of a
>   concern as performance and disk I/O.  I've tried to reroute things like
>   that through syslog in the past, only to find syslog now taking a
>   substantial portion of all available system resources trying to keep up.

Relatively few packages are going to be performance-bound on logging,
and those that do could absolutely recommend logging to standalone files
for scalable configurations, or even have configuration packages that
make it trivial to configure that destination. That said, Apache at
least tends to need a substantial amount of configuration for scaling,
and where and what to log is the least of that.

Also, while the default openlog/syslog/closelog doesn't scale all that
well, it's certainly quite possible to log asynchronously to syslog
without losing performance, and some syslog implementations are more
scalable than others.

The default configuration is never going to work for *everyone*; the
question then becomes what configuration provides the most value to the
most people. I would suggest that having unified log handling
system-wide makes sense as a default, and systems handling huge amounts
of traffic will often require other custom configuration by the sysadmin
anyway.

> * The package does its own log analysis.  INN again is an excellent
>   example: it has a bespoke but very comprehensive and incredibly useful
>   log analysis package that ships with the software and requires access to
>   the raw logs.  Anything that moves those logs out of their expected
>   locations will break that tool, which is a significant regression for
>   our users.

And apache log analyzers would want similar things, yes. I'd certainly
be happy to help update such software to be able to extract log messages
from places like syslog or journald, and I'd also be happy in the
interim with such software continuing to log to standalone files. Again,
I'm aiming for the common case here, and I expect this to be a
years-long effort, not an overnight one.

- Josh Triplett

Re: Unifying logging by default

[Guillem Jover]

On Wed, 2019-02-20 at 14:19:02 -0800, Russ Allbery wrote:
> Josh Triplett  writes:
> > While there are *absolutely* configurations in which system
> > administrators want to log to arbitrary locations and files, I would
> > like to propose that for consistency we should configure software to
> > unify logging into syslog and/or journald by default. In particular, I'd
> > propose that daemons and applications should default to logging to
> > syslog (working with any installed syslog implementation providing
> > /dev/log), and that daemons run via systemd units may detect or be
> > configured to log to stdout/stderr which will be wired to the sysadmin's
> > preferred log target.
> 
> I've seen three main reasons why packages do not use syslog:
> 
> * The log data is not line-structured.  Example: /var/log/apt/history.log
>   Dumping that as-is into syslog sounds like an awful idea.  This would
>   therefore block on an upstream feature request to change log formats
>   first, with all that entails.
> 
> * Performance.  Good examples here are Apache and INN, both of which have
>   per-request logs that are intentionally not line-buffered and are logged
>   directly to files, since losing some log messages is not as much of a
>   concern as performance and disk I/O.  I've tried to reroute things like
>   that through syslog in the past, only to find syslog now taking a
>   substantial portion of all available system resources trying to keep up.
> 
> * The package does its own log analysis.  INN again is an excellent
>   example: it has a bespoke but very comprehensive and incredibly useful
>   log analysis package that ships with the software and requires access to
>   the raw logs.  Anything that moves those logs out of their expected
>   locations will break that tool, which is a significant regression for
>   our users.

* The log data is "chroot" specific. This applies to all of the package
  management logs. Logging into syslog would by default not do the right
  thing and would be extremely confusing. I could see adding options
  for dpkg and u-a, but they would definitely not be enabled by default.

> I think you're going to need a much more concrete and specific proposal
> with worked examples of how to deal with cases like this.  As is, I think
> this proposal is much too vague to really discuss, let alone implement.

Agreed!

Thanks,
Guillem

Re: Bug#922643: ITP: build-alternative -- helper to build Debian package with diet libc

[Guillem Jover]

Hi!

On Mon, 2019-02-18 at 19:37:38 +, Dmitry Bogatov wrote:
> Package: wnpp
> Severity: wishlist
> Owner: Dmitry Bogatov 
> 
> * Package name : build-alternative
>   Version  : 0.0.1
>   Upstream Author  : Dmitry Bogatov 
> * Url  : https://salsa.debian.org/kaction/build-alternative
> * Licenses : GPL-3+
>   Programming Lang : shell
>   Section  : devel
> 
>  This package provide makefile snippet, that abstract away
>  several issues, related to building package with diet libc.
>  .
>   * diet libc is not supported on every Debian architecture
>   * code to check for build profiles is repetive
>  .
>  Regular users do not need to install this package, it is only
>  useful to Debian Contributors.

Hmm this package name looks extremely generic for what it is described
to be used for. Could you name it something else that includes either
dietlibc in its name or at least libc or similar?

Thanks,
Guillem

Re: Unifying logging by default

[Josh Triplett]

On Wed, Feb 20, 2019 at 11:53:24PM +0100, Guillem Jover wrote:
> * The log data is "chroot" specific. This applies to all of the package
>   management logs. Logging into syslog would by default not do the right
>   thing and would be extremely confusing. I could see adding options
>   for dpkg and u-a, but they would definitely not be enabled by default.

Logging into syslog would do the right thing when there's a syslog
daemon running in the chroot.

Also, I'd have no objection to "if there's no syslog available, log to a
standalone file", or similar, to ensure that messages don't get lost.

Options would certainly be helpful (particularly if it were possible to
drop configuration files in directories like /etc/dpkg/dpkg.cfg.d/ and
/etc/apt/apt.conf.d/), but it'd be far more useful if we can find an
autodetection mechanism that does the right thing by default.

Re: Unifying logging by default

[Russ Allbery]

Josh Triplett  writes:

> Both syslog and journald support multi-line log messages

syslog does not support multi-line log messages in any reasonable way.  It
just escapes the newline (if you're lucky) and jams all the lines
together, and is rather likely to break whatever log parser you have on
the other end.

> Again, I'm aiming for the common case here, and I expect this to be a
> years-long effort, not an overnight one.

I think what's missing for me is what you're trying to accomplish with
this thread.  Are you just trying to make people aware of this as
something we *could* do and get people thinking about it in the
background?  Are you forming a team of people who is going to tackle this
problem in Debian and are looking for volunteers?  Are you asking
packagers to do work today?  Are you asking for the project as a whole to
reach some consensus that this is something we should do?

If you're asking whether that sort of work seems like a good idea to other
people, I personally think it probably does for line-structured logs with
no performance issues (and that are actually logs, not execution traces or
data dumps or some other thing -- see below), but the devil is going to be
in the details, and blindly dumping existing log messages into syslog
without thinking a little about how they're structured is not a good idea.
In other words, I think this requires some per-package thought and
analysis.  But I previously made this change (relative to upstream
behavior) for the logging from shibboleth2-sp, and that seems to have been
a good idea, or at least not a bad idea.

If you want to get things to change, you're probably going to have to
start doing the required work (or recruiting people to do so), which means
something closer to "I have identified the following 14 Debian packages
that are currently logging to files, have investigated why this is the
case, believe that they would be equally happy logging to syslog for the
following reasons, and here are diffs to make that change."

I suspect you're not going to get a meaningful consensus in advance that
this is something we should do with just an abstract write-up, but may get
traction with an inventory of packages in Debian that currently log to
files and a plan for how to deal with them.  Down that path is
(potentially, if successful) a Lintian check to discourage introduction of
new cases, possibly with some carefully crafted exceptions, but I think a
more comprehensive analysis would be needed first.

Not everything in /var/log is a log in the syslog sense, though.  I see
some logs that I as a system administrator do not want in syslog and would
be quite unhappy if they were just dumped into syslog because they're pure
noise and I'd then have to filter them out again in my log analysis
pipeline.  /var/log/fontconfig.log is an excellent example.  That appears
to be a local debugging trace log for fontconfig that I suspect has only
one user (reportbug when filing bugs against fontconfig, if it even knows
to grab that log) and is otherwise pretty useless.  So I don't think that
should go into syslog, and it would cause me work if it did.

Another example is /var/log/popularity-contest.  I don't think that's
actually a log; it looks like data that popularity-contest gathered.  I
definitely do not want that dumped into my syslog.

Maybe you could start with Xorg.0.log.  :)

-- 
Russ Allbery (r...@debian.org)   

Re: Unifying logging by default

[Josh Triplett]

On Wed, Feb 20, 2019 at 02:03:08PM -0800, Josh Triplett wrote:
> While there are *absolutely* configurations in which system
> administrators want to log to arbitrary locations and files, I would
> like to propose that for consistency we should configure software to
> unify logging into syslog and/or journald by default. In particular, I'd
> propose that daemons and applications should default to logging to
> syslog (working with any installed syslog implementation providing
> /dev/log), and that daemons run via systemd units may detect or be
> configured to log to stdout/stderr which will be wired to the sysadmin's
> preferred log target.

To suggest a more concrete proposal, I'd propose the following rough
sketch of future logging policy (not to be incorporated into Policy
until after packages generally start following it):

- If the software has a well-established set of logfile analyzers
  specific to its logfile location and format, and the most commonly
  used logfile analyzers do not support reading data from syslog or
  journald, the software MAY continue using standalone logfiles (instead
  of or in addition to the below behavior) until that changes;
  alternatively, such logfile analyzers MAY facilitate the user's
  configuration of the software to log to a location they understand.

- The software MAY (but is not required to) detect that it is running
  with stderr or stdout connected to journald, and in that case, log to
  stderr or stdout, optionally using structured logging. The software
  MAY detect that /dev/log is connected to journald, and in that case,
  log to journald directly if it has such support. If the software does
  not support any such detection, any systemd unit file invoking it
  SHOULD pass an explicit command-line option enabling logging to
  stderr.

- Otherwise, if /dev/log exists, the software SHOULD log to /dev/log.
  The software MAY choose to use a structured log format if supported.

- Otherwise, the software SHOULD choose an appropriate fallback behavior,
  such as logging to a standalone file, or displaying log messages to
  the user if running interactively.


Some concrete use cases:

- Running logs through a log analyzer.

- Shipping logs over the network to an off-machine logging
  infrastructure.

- Controlling where log messages go using syslog configuration.

- Logging to an in-memory journal on a system without persistent
  storage.

Re: Unifying logging by default

[Josh Triplett]

On Wed, Feb 20, 2019 at 03:38:41PM -0800, Russ Allbery wrote:
> Josh Triplett  writes:
> > Both syslog and journald support multi-line log messages
> 
> syslog does not support multi-line log messages in any reasonable way.  It
> just escapes the newline (if you're lucky) and jams all the lines
> together, and is rather likely to break whatever log parser you have on
> the other end.

As far as I can tell, RFC5424-formatted structured messages would
support newlines far better than classical syslog. That said, I'm not a
syslog expert, and I'd defer to those who are.

But in any case, emitting the lines one by one also works for many of
the logs mentioned in this thread.

> > Again, I'm aiming for the common case here, and I expect this to be a
> > years-long effort, not an overnight one.
> 
> I think what's missing for me is what you're trying to accomplish with
> this thread.  Are you just trying to make people aware of this as
> something we *could* do and get people thinking about it in the
> background?  Are you forming a team of people who is going to tackle this
> problem in Debian and are looking for volunteers?  Are you asking
> packagers to do work today?  Are you asking for the project as a whole to
> reach some consensus that this is something we should do?

The most recent mail I sent to the thread has a more concrete proposal,
along with a few sample use cases. In my particular case, I care about
this so that I can configure logging behavior in *one* place rather than
one place per package, as well as being able to view logs in one place
with correlated and consistent timing information, as well as handling
log rotation automatically as part of a logging daemon with no need to
periodically run logrotate.

I'm seeking some amount of consensus, with the understanding that I'm
quite willing to write many patches myself.

My opening mail featured a few of the logs I care about addressing this
for. On my system, they include alternatives.log, aptitude, the various
cups logs, dpkg.log, and fontconfig.log.

> Not everything in /var/log is a log in the syslog sense, though.  I see
> some logs that I as a system administrator do not want in syslog and would
> be quite unhappy if they were just dumped into syslog because they're pure
> noise and I'd then have to filter them out again in my log analysis
> pipeline.  /var/log/fontconfig.log is an excellent example.  That appears
> to be a local debugging trace log for fontconfig that I suspect has only
> one user (reportbug when filing bugs against fontconfig, if it even knows
> to grab that log) and is otherwise pretty useless.  So I don't think that
> should go into syslog, and it would cause me work if it did.

Even if it were logged at priority "debug"? I would personally prefer to
have that data in journald rather than a standalone file.

That said, for my particular use cases, I care a little less about log
files that are overwritten each time, and I care more about log files
that grow over time.

> Another example is /var/log/popularity-contest.  I don't think that's
> actually a log; it looks like data that popularity-contest gathered.  I
> definitely do not want that dumped into my syslog.

That one is certainly less of an issue. See above regarding files that
get overwritten each time.

(I also wonder if it perhaps belongs in /var/cache, but I'll leave that
to people who use popcon.)

> Maybe you could start with Xorg.0.log.  :)

Xorg and the software that invokes it, in conjunction, are already quite
capable of logging to several more useful places, including syslog and
logs in the user's home directory. The default desktop configuration
doesn't generate an Xorg.0.log out of the box.

Request to submit guest post

[Laura Yang]

Hi , 
 
How do you think your readers would benefit from more business education and 
career knowledge? We have a free resource that we would like to share on your 
website. We would appreciate the opportunity to be a guest contributor and 
provide content for your website. 
 
Study.com is an online education company that helps college students and 
working professionals with their schooling and career decisions.  
 
We look forward to working together. If you have specific editorial guidelines, 
please send them over. Please let me know if there are other partnership 
avenues we could explore. 
 
Best regards, 
Laura Y. 
 
Study.com | 100 View Street, Suite 202 | Mountain View, CA 94041 
Study.com in the News ( https://study.com/press.html ) 
 
This message and its attachments remain confidential property of Study.com LLC 
intended for the sole use of the email recipients only. Click here ( 
http://tx.bz-mail-us1.com/1/l/cc7af9f17dff437b858cb47f904c2353?rl=http%3A%2F%2Funsubscribe.bz-mail-us1.com%2Funsubscribe%3Fe%3Ddebian-devel%2540lists.debian.org%26i%3D1413890907.1422591.1550723107171%2540ip-10-1-0-82%26u%3D60305%26g%3D30999
 ) if you no longer wish to receive emails from me.