Re: dejagnu version update?

[Rob Savoye]

On 5/13/20 10:51 AM, Mike Stump wrote:

> So, now that ubuntu 20.04 is out and RHEL 8 is out, and they both
> contain 6, and SLES has 6 and since we've been sitting at 1.4.4 for
> so long, anyone want to not update dejagnu to require 1.6?

  We do still find and fix bugs occasionally. :-) And 1.4.4 was released
16 years ago... Linaro has been using the most recent release for years,
so I think it's a safe upgrade.

- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 8:08 AM, Rainer Orth wrote:

>> stops responding for whatever reason.  I have come up with a solution 
>> (that I'd be happy to upstream, except that DejaGNU maintenance seems to 
>> have been dead for like a year now), which I have also confirmed to be 
>> required with current DejaGNU Git master so it must be a different one, 
>> and I would like to know how it might be related to the bug you mention.

  I feel I need to bring up the issue that DejaGnu is 30 years old, and
it's two maintainers are either trying to pay bills, or trying to
retire, or both... This problem will effect more projects in the future,
not just DejaGnu. I'd love to see if anyone would like to become a
co-maintainer, who preferably will be actively working for a few decades
still. I think most people on these lists make their income from working
on the toolchain, but some of us are still volunteers... and getting
older every day...

  There is a patch backlog neither of us have even looked at, sorry. I'm
willing to put some time into this, but I think you all realize the time
involved to adequately test this. I'm not sure I have enough disk space. :-)

  Personally, I tried to find funding to refactor DejaGnu in Python,
since Tcl is unmaintained too, but nobody was interested.

- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 10:08 AM, David Edelsohn wrote:

> Have you approached the Linux Foundation Core Infrastructure
> Initiative for funding for both DejaGNU maintenance (patch backlog)
> and refactoring DejaGNU in Python efforts?

  Not that team, the folks I talked to thought I was crazy for wanting
to refactor it. :-) That's been the usual answer from anyone. I even
talked about this at GNU Tools Cauldron once. Seriously though, it'd be
great to analyze the current code base, write an actual design document,
clean internal APIs, and build something we can use for another 30
years. I'd start by writing a solid expect module for Python, and then
embedding Tcl in Python as a comparability layer. I understand it'd be a
huge project, which is why I haven't done this as a volunteer.

  I think there has been a bit of a leave it alone so it stays stable
attitude... Bugs in the test framework effect our ability to work on the
toolchain efficiently, but other than the handful of people here, nobody
cares. Forgetting the refactoring daydream, testing patches and doing
releases still needs to be done, so I think we need a long-term solution
and fresh energy.

- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 5:34 PM, Maciej W. Rozycki wrote:

> And then current development appears ongoing, ferociously indeed, with the 
> last check in literally today (barring my time zone), as indicated here: 
> .

  It's obvious I haven't been paying attention, so much for my
retirement... :-)  That's awesome actually, and it does look active.

>  Other than that what would be the technical advantage from rewriting 
> DejaGnu in Python (/me asking as a Python ignoramus)?

  The main thought was more towards an actual design and clean APIs, and
assuming Tcl/Expect was unmaintained, needed to use something that'll be
around for a few more decades. Assuming Tcl/Expect are maintained
forever, that's a bit of a moot point. Everybody has always complained
about using Tcl in DejaGnu. Python is just much more commonly used in
the current century.

  Right now working through patches is probably more important. :-)
There's zero patches on the DejaGnu savannah site, so I'd ask anybody to
submit them so I don't have to dig them out of email archives.

- rob -

Re: dejagnu version update?

[Rob Savoye]

On 5/15/20 6:22 PM, Mike Stump wrote:

> Anyway, love to have software that can move code wholesale.  Love to move the 
> testsuite into a new language.

  All it needs is funding. :-) What GDB needs is expect, not Tcl. Most
of the GDB testsuite is just expect pattern matching from the shell.
That's the entire reason I choose Tcl as it already had expect support.
Expect was necessary functionality for GDB testing. For GCC &  Binutils,
Expect is only used for remote testing support.

  As it's possible to embed Tcl in other programs, the idea was to use
an embedded Tcl interpreter when needed during a transition period. It's
mostly just the framework itself that would need to be refactored into
Python. There is also a large amount of code in gcc/testsuite that
should probably be in core DejaGnu too. That would be a large component
in analyzing existing code to write a true design doc. The best part is
now we have large toolchain testsuites to use to test DejaGnu changes.

 A one point we thought DejaGnu would be used for other projects, but I
think it's obviously that these days it's  only used for GNU toolchain
testing.

  I'm making progress on setting up a development environment to test
patches. I use my ABE tool to build toolchains, had to fix some bugs
(and add PI support) first.

- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rob Savoye]

On 5/16/20 5:45 PM, Maciej W. Rozycki wrote:

>  Overall perhaps a patch management system might be good having to make 
> chasing patches easier, such as patchwork, and we already use Git, so we 

 As an old GNU project, we're required to use what the FSF prefers,
which is on savannah. https://savannah.gnu.org/patch/?group=dejagnu, Our
bug tracker is there their too. We've used that for a long time. Yes,
patches in email are harder to track.

> fresh patchwork?  The patch traffic is surely much lower with DejaGnu than 
> it is with glibc, and there would be no data to migrate (but we might want 
> to feed a couple of months' back worth of mailing list traffic).

  I'm now building up the infrastructure to properly test patches, but
it's not enough to do the next release. All I have these days is my
laptop and a PI B3+. I'd need access to more hardware as some of the
patches effect cross testing, or get others to test the release candidates.

  Much of the problems with cross testing are often obscure timing
problems. It's amazing how sometimes a minor unrelated change changes
the timing and things break... To do a release properly requires
duplicating that level of infrastructure for at least several targets
and several toolchain release, and built on more than one GNU/Linux distro.

 It'll take most of the week to really get a good base setup with
baseline test results, but some of the patches like the DejaGnu
testsuite ones will go in first since they don't effect the toolchain.

  Jacob already added 9 patches to our site. I'm still building cross
compilers since some of his patches effect cross testing. I did add ADA
to my builds, which isn't a normal build default, since I thought some
of the patches for ADA.

- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rob Savoye]

On 5/17/20 1:43 PM, Maciej W. Rozycki wrote:

> patch service before.  It doesn't appear linked to our mailing list either 
> and instead you need to go through the hoops of a web interface (and open 
> an account first) to submit a change.

  From what I remember, it was decided the GNU toolchain (minus DejaGnu)
that the savannah infrastructure was insufficient. Some of the others
here probably remember better. GNU projects are different when it's not
part of the toolchain. My Gnash project uses savannah heavily. But you
are right, it forces you into a webbie world, and email is mostly for
notifications.

>  That's precisely what patchworks is for, which has been used to various 
> extent for the GNU C library, GCC and GDB already.  All of which projects 
> are of similar vintage to DejaGnu and I reckon rather important for the 
> GNU project.  Given that our main patch submission channel is e-mail, 
> what's the point in using a system that does not accept one?

  So maybe the discussion point is if we want to consider DejaGnu part
of the toolchain, and use the same infrastructure as the other tools. I have
no problem with that. DejaGnu doesn't change much, and there were only 2
of us working on it, so we just never thought about it.

- rob -

Re: dejagnu version update?

[Rob Savoye]

  I processed the patch backlog for DejaGnu, and have gone through the
bug list. It'd be nice if somebody could try master with a more complex
environment, etc... if I'm going to push out a release. For cross
testing all I have is a PI and QEMU.

- rob -

Re: dejagnu version update?

[Rob Savoye]

On 5/26/20 7:20 PM, Maciej W. Rozycki wrote:

>  I'll run some RISC-V remote GCC/GDB testing and compare results for 
> DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes 
> many hours to go through these testsuite runs.

  That'd be great. I'd rather push out a stable release, than have to
fix bugs right after it gets released.

- rob -

Re: connecting a QEMU VM to dejagnu...

[Rob Savoye]

On 10/16/19 5:40 PM, Alan Lehotsky via DejaGnu wrote:

> The one example I found via a web search seems to want to do
> everything in the virtual machine - but I have to believe that’s
> going to be insanely slow…

  Well, qemu is a virtual machine... Here's the ones I used for GNU
toolchain cross testing:
https://git.linaro.org/toolchain/abe.git/tree/config/boards. There's a
few on there. If you're building cross compilers, just use ABE and it's
all built in.

- rob -

Re: DejaGnu and toolchain testing

[Rob Savoye]

On 07/25/2013 06:21 PM, Joseph S. Myers wrote:
> I was interested to watch the video of the DejaGnu BOF at the Cauldron.  A 
> few issues with DejaGnu for toolchain testing that I've noted but I don't 
> think were covered there include:
  Thanks for the thoughtful comments, they're useful as I start
considering refactoring DejaGnu to keep it working for the next 22
years... There is a lot of crusty old code in DejaGnu, I admit it.
DejaGnu was never truly designed, it was just built and debugged while
we were using it, and it shows.

  I'm not sure if this discussion is better on the GCC list or the
DejaGnu list, but I would like to keep this thread going. Course GCC
developers are the main users of DejaGnu anyway.
> * DejaGnu has a lot of hardcoded logic to try to find various files in a 
> toolchain build directory.  A lot of it is actually for very old toolchain 
> versions (using GCC version 2 or older, for example).  The first issue 
> with this is that it doesn't belong in DejaGnu: the toolchain should be 
> free to rearrange its build directories without needing changes to DejaGnu 
> itself (which in practice means there's lots of such logic in the 
> toolchain's own testsuites *as well*, duplicating the DejaGnu code to a 
> greater or lesser extent).  The second issue is that "make install" 
  DejaGnu is a testing framework, so it makes sense that much of the GCC
testing logic is in gcc/testsuite/{lib,config}. It was also a decision
at the time that having a testsuite override existing procs in DejaGnu
core was a good idea. Now many years later, I think I'd move most what
GCC needs into the core, especially all the "dg* style of tests.

  At one time the thought was DejaGnu was a general purpose test
framework, but I think at this point in time, it's really just used for
toolchain testing. (although my Gnash project also uses it) So I think
tweaking DejaGnu core to be mainly toolchain testing oriented is
probably a good idea.
> * Similarly, DejaGnu has hardcoded prune_warnings - and again GCC adds 
> lots of its own prunes; it's not clear hardcoding this in DejaGnu is a 
> particularly good idea either.
  The DejaGnu pruning is older than GCC's. :-)
> * Another piece of unfortunate hardcoding in DejaGnu is how remote-host 
> testing uses "-o a.out" when running tools on the remote host - such a 
> difference from how they are run on a local host results in lots of issue 
  This is historical, a.out being common at the time.
> * A key feature of QMTest that I like but I don't think got mentioned is 
> that you can *statically enumerate the set of tests* without running them.  
> That is, a testsuite has a well-defined set of tests, and that set does 
> not depend on what the results of the tests are - whereas it's very easy 
> and common for a DejaGnu test to have test names (the text after PASS: or 
> FAIL: ) depending on whether the test passed or failed, or how the test 
> passed or failed (no doubt the testsuite authors had reasons for doing 
> this, but it conflicts with any automatic comparison of results).  The 
  One of my other ideas for DejaGnu 2.0 is improved test result output.
I'm currently importing all test results into a database (see the mysql
branch on savannah), and find text parsing painful and lacking more fine
grained details. The text field for PASS/FAIL is overloaded. Since I
want to improve the ability to analyze results, ie... comparing what
happens with differing configure or command line options, I think the
output format has to change. One thought is to only add new fields into
the --xml output, as that is database specific, and leave the current
text output unchanged.
> * People in the BOF seemed happy with expect.  I think expect has caused 
> quite a few problems for toolchain testing.  In particular, there are or 
  I don't think it was that we were happy with expect, but at least for
GDB testing, nobody has any alternatives. I thought I mentioned that a
refactored DejaGnu would only use expect for GDB testing, everything
else wouldn't require it. That also means all the remote execution procs
would need to work without expect as well.

- rob -

Re: DejaGnu and toolchain testing

[Rob Savoye]

On 07/26/2013 10:37 AM, Joseph S. Myers wrote:
> Anything in the core needs to avoid obstructing toolchain changes.  People 
> typically test with the installed DejaGnu from their OS, and the OS itself 
> may well be a few years old (e.g. Ubuntu 10.04), so it's undesirable for 
> an enhancement to the GCC testsuite to require a new version of DejaGnu.  
> This means clean extensibility, and avoiding DejaGnu hardcoding things 
> that are not stable public interfaces.
  DejaGnu is basically stagnant because most people consider the pain of
any improvements too great to change anything. If I launch off on a
DejaGnu 2.0, my thought is the existing release wouldn't go away. Many
distributions ship multiple versions of some applications. Any changes
to DejaGnu would likely live in a branch for a long time, but would be
usable by anyone interested in better functionality. Yes, an actual
design and defining public interfaces would be a good idea. Currently
DejaGnu has many arbitrary APIs and settings, all created without a
whole lot of thought other than working around or fixing a problem.

  I also realize that any major changes to DejaGnu will require
corresponding changes in the testsuite support code. I'm completely
aware of how much work this would be having written much of it... There
would have to be backward compatibility maintained for a considerable time.
> Right now, DejaGnu has lots of toolchain stuff in the core ... toolchain 
> stuff for building Cygnus trees 20 years ago rather than what's useful 
> now.  It's not that much better if a DejaGnu version released in 2013 and 
> used for testing in 2017 has things in it that are good for testing 2013 
> toolchains and get in the way for testing 2017 toolchains.
   I'd agree there is lots of crufty support for things like the old
Cygnus trees that could be removed. Ideally I'd prefer to explore
people's ideas on what would be useful for testing toolchains 5-10 years
from now. Me, I want something not dependent on a dying and mostly
unmaintained scripting language that nobody likes anyway (the current
working idea is to use python). I also want to be able to compare test
results in better ways than diffing huge text files. I'd like to compare
multiple test runs as well in a reasonably detailed fashion.

- rob -