[PATCH 1/3] tester: Change the QEMU command line to match the RSB built QEMU.

2018-11-22 Thread chrisj
From: Chris Johns 

The command line changed and the RSB is the newer version. Make
the old version via a config variable.
---
 tester/rtems/testing/qemu.cfg | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tester/rtems/testing/qemu.cfg b/tester/rtems/testing/qemu.cfg
index 52a3752..e37dc9b 100644
--- a/tester/rtems/testing/qemu.cfg
+++ b/tester/rtems/testing/qemu.cfg
@@ -51,8 +51,11 @@
 #
 # Qemu common option patterns.
 #
-%define qemu_opts_base   -no-reboot -monitor none -serial stdio -nographic
-#%define qemu_opts_base   -no-reboot -serial null -serial mon:stdio -nographic
+%if %{defined qemu_use_serial_console}
+ %define qemu_opts_base -no-reboot -monitor none -serial stdio -nographic
+%else
+ %define qemu_opts_base -no-reboot -serial null -serial mon:stdio -nographic
+%endif
 %define qemu_opts_no_net -net none
 
 #
-- 
2.14.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


[PATCH 3/3] rtemstoolkit: Add unit testing for the python modules

2018-11-22 Thread chrisj
From: Chris Johns 

- Add support to run the unit tests for the rtemstoolkit python
  modules from waf. Enter './waf test' for the tests to be run on
  python2 and python3.
- Update the importing of rtemstoolkit modules to the standard
  method which works on python2 and python3.
- Update the README.
---
 README   | 27 ++--
 rtemstoolkit/__init__.py | 26 ---
 rtemstoolkit/check.py| 32 +---
 rtemstoolkit/config.py   | 24 ++---
 rtemstoolkit/darwin.py   |  9 +---
 rtemstoolkit/freebsd.py  | 12 ++-
 rtemstoolkit/git.py  | 20 +-
 rtemstoolkit/host.py | 10 +
 rtemstoolkit/linux.py| 12 ++-
 rtemstoolkit/log.py  |  9 +---
 rtemstoolkit/macros.py   | 15 +++---
 rtemstoolkit/mailer.py   | 28 -
 rtemstoolkit/netbsd.py   |  8 ++-
 rtemstoolkit/options.py  | 30 +++
 rtemstoolkit/path.py | 12 ++-
 rtemstoolkit/rtems.py|  5 -
 rtemstoolkit/solaris.py  | 11 +++---
 rtemstoolkit/textbox.py  | 15 --
 rtemstoolkit/version.py  | 17 ---
 rtemstoolkit/windows.py  | 12 ++-
 wscript  | 54 +---
 21 files changed, 155 insertions(+), 233 deletions(-)

diff --git a/README b/README
index 763960a..115befd 100644
--- a/README
+++ b/README
@@ -28,17 +28,32 @@ Building
 
 To build and install:
 
- $ ./waf configure --prefix=$HOME/development/rtems/4.11
+ $ ./waf configure --prefix=$HOME/development/rtems/5
  $ ./waf build install
 
+Testing
+---
+
+To the run the tests build then enter:
+
+ $ ./waf test
+
+Python
+--
+
+The RTEMS Tools supports python3 and python2. The commands look for python3,
+then python2 and finally python and use the first it finds.
+
+You can forced a specific version for testing by setting the environment
+variable 'RTEMS_PYTHON_OVERRIDE' to the python you want to use. For example:
+
+ $ export RTEMS_PYTHON_OVERRIDE=python2
+
+will use python2.
+
 Waf
 ---
 
 The Waf project can be found here:
 
  http://code.google.com/p/waf/
-
-Simple instructions on How to set up Waf is here:
-
- http://www.rtems.org/ftp/pub/rtems/people/chrisj/rtl/rtems-linker/waf.html
-
diff --git a/rtemstoolkit/__init__.py b/rtemstoolkit/__init__.py
index 6878782..207698a 100644
--- a/rtemstoolkit/__init__.py
+++ b/rtemstoolkit/__init__.py
@@ -46,20 +46,12 @@ all = ['check',
'textbox',
'version']
 
-from . import check
-from . import config
-from . import configuration
-from . import error
-from . import execute
-from . import git
-from . import host
-from . import log
-from . import macros
-from . import mailer
-from . import options
-from . import path
-from . import reraise
-from . import rtems
-from . import stacktraces
-from . import textbox
-from . import version
+args = {
+'config': ['--file', 'tester/rtems/version.cfg',
+   '--jobs', 'half',
+   '--no-clean'],
+'mailer': ['--smtp-host', '1.2.3.4',
+   '--mail-to',   'f...@bar.none',
+   '--mail-from', 'me@here.there']
+
+}
diff --git a/rtemstoolkit/check.py b/rtemstoolkit/check.py
index c6549bf..8d4a35c 100644
--- a/rtemstoolkit/check.py
+++ b/rtemstoolkit/check.py
@@ -36,24 +36,12 @@ from __future__ import print_function
 
 import os
 
-#
-# Support to handle use in a package and as a unit test.
-# If there is a better way to let us know.
-#
-try:
-from . import error
-from . import execute
-from . import log
-from . import options
-from . import path
-from . import version
-except (ValueError, SystemError):
-import error
-import execute
-import log
-import options
-import path
-import version
+from rtemstoolkit import error
+from rtemstoolkit import execute
+from rtemstoolkit import log
+from rtemstoolkit import options
+from rtemstoolkit import path
+from rtemstoolkit import version
 
 def _check_none(_opts, macro, value, constraint):
 return True
@@ -163,12 +151,12 @@ def check_dir(label, path):
 return _check_dir(None, label, path, 'required', True)
 
 
-def run():
+def run(args):
 import sys
 try:
-_opts = options.command_line(argv = sys.argv)
+_opts = options.command_line(argv = args)
 options.load(_opts)
-log.notice('RTEMS Source Builder - Check, v%s' % (version.string()))
+log.notice('RTEMS Toolkit - Check, v%s' % (version.string()))
 if host_setup(_opts):
 print('Environment is ok')
 else:
@@ -188,4 +176,4 @@ def run():
 
 
 if __name__ == '__main__':
-run()
+run(['tester'])
diff --git a/rtemstoolkit/config.py b/rtemstoolkit/config.py
index a16261b..be100f2 100644
--- a/rtemstoolkit/config.py
+++ b/rtemstoolkit/config.py
@@ -44,24 +44,12 @@ import os
 import re
 import sys
 
-#
-# Support to handle use in a package and as a

[PATCH] rtems-tools: Add unittests for python, fix stdio handling

2018-11-22 Thread chrisj
Hi,

Please test these patches and report.

The patches:

- Add unittests for the rtemstoolkit python modules. To run enter:

   ./waf test

  The tests will be run for python2 and python3 is present on your
  host.

- Improvie the reader thread line processor of child process stdio
  data. Switch the output streams to buffered io. This seems to
  fix some output related issues when running multiple simulators
  at once.

- I have run the rtems-test command with a zynq qemu and erc32 using
  python2 and python3.

  I did not some JFFS related timeouts with the erc32 where the run
  command has 0 load which I think is strange for an instruction
  simulator.

Chris

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


[PATCH 2/3] rtemstoolkit/execute: Use buffered output and improve performance

2018-11-22 Thread chrisj
From: Chris Johns 

- Use buffered output on the stdout and stderr streams from
  child processors.
- Simplify the read thread line processing to improve performance.
- Disable 'close_fds' as it slows down python3's popen call.
- Update the importing of rtemstoolkit modules.
---
 rtemstoolkit/execute.py | 47 +--
 1 file changed, 25 insertions(+), 22 deletions(-)

diff --git a/rtemstoolkit/execute.py b/rtemstoolkit/execute.py
index 8e09b81..eb3c8c2 100755
--- a/rtemstoolkit/execute.py
+++ b/rtemstoolkit/execute.py
@@ -46,16 +46,8 @@ import threading
 import time
 import traceback
 
-#
-# Support to handle use in a package and as a unit test.
-# If there is a better way to let us know.
-#
-try:
-from . import error
-from . import log
-except (ValueError, SystemError):
-import error
-import log
+from rtemstoolkit import error
+from rtemstoolkit import log
 
 # Trace exceptions
 trace_threads = False
@@ -196,22 +188,34 @@ class execute(object):
 while True:
 #
 # The io module file handling return up to the size passed
-# in.
+# in to the read call. The io handle has the default
+# buffering size. On any error assume the handle has gone
+# and the process is shutting down.
 #
-data = fh.read(4096)
+try:
+data = fh.read(4096)
+except:
+data = ''
 if len(data) == 0:
+if len(line) > 0:
+_output_line(l + '\n', exe, prefix, out, count)
 break
 # str and bytes are the same type in Python2
 if type(data) is not str and type(data) is bytes:
 data = data.decode(sys.stdout.encoding)
-for c in data:
-line += c
-if c == '\n':
+last_ch = data[-1]
+sd = (line + data).split('\n')
+if last_ch != '\n':
+line = sd[-1]
+else:
+line = ''
+sd = sd[:-1]
+if len(sd) > 0:
+for l in sd:
+_output_line(l + '\n', exe, prefix, out, count)
 count += 1
-_output_line(line, exe, prefix, out, count)
-if count > 10:
-count = 0
-line = ''
+if count > 10:
+count -= 10
 except:
 raise
 if trace_threads:
@@ -262,7 +266,6 @@ class execute(object):
  args = (self,
  
io.open(proc.stdout.fileno(),
  mode = 'rb',
- buffering = 0,
  closefd = False),
  self.output,
  ''))
@@ -274,7 +277,6 @@ class execute(object):
  args = (self,
  
io.open(proc.stderr.fileno(),
  mode = 'rb',
- buffering = 0,
  closefd = False),
  self.output,
  self.error_prefix))
@@ -374,7 +376,8 @@ class execute(object):
 proc = subprocess.Popen(command, shell = shell,
 cwd = cwd, env = env,
 stdin = stdin, stdout = stdout,
-stderr = stderr)
+stderr = stderr,
+close_fds = False)
 if not capture:
 return (0, proc)
 if self.output is None:
-- 
2.14.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH v2] Add LICENSE.BSD-2-Clause

2018-11-22 Thread Sebastian Huber

On 15/11/2018 14:10, Sebastian Huber wrote:

Copied from:

https://spdx.org/licenses/BSD-2-Clause.html

The formatting is suitable for C/C++ source comments.

Update #3053.


It would be nice if this patch could be reviewed. It would be quite bad 
if we have an error in this template file.


--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH v2] Add LICENSE.BSD-2-Clause

2018-11-22 Thread Chris Johns
On 23/11/2018 16:58, Sebastian Huber wrote:
> On 15/11/2018 14:10, Sebastian Huber wrote:
>> Copied from:
>>
>> https://spdx.org/licenses/BSD-2-Clause.html
>>
>> The formatting is suitable for C/C++ source comments.
>>
>> Update #3053.
> 
> It would be nice if this patch could be reviewed. It would be quite bad if we
> have an error in this template file.
> 

Yes.

Should https://opensource.org/licenses/BSD-2-Clause be the reference?

They are the same so the SPDX link can also be provided but it is about data
exchange.

Does this file need a context to explain it's purpose in the RTEMS kernel source
tree?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH v2] Add LICENSE.BSD-2-Clause

2018-11-22 Thread Sebastian Huber

On 23/11/2018 07:13, Chris Johns wrote:

On 23/11/2018 16:58, Sebastian Huber wrote:

On 15/11/2018 14:10, Sebastian Huber wrote:

Copied from:

https://spdx.org/licenses/BSD-2-Clause.html

The formatting is suitable for C/C++ source comments.

Update #3053.

It would be nice if this patch could be reviewed. It would be quite bad if we
have an error in this template file.


Yes.

Should https://opensource.org/licenses/BSD-2-Clause be the reference?


I got some private mails from FreeBSD developers as a result of this 
question:


https://lists.freebsd.org/pipermail/freebsd-hackers/2018-October/053447.html

This let me to change the URL in v2 of this patch to spdx.org which is 
maintained by the Linux Foundation instead of OSI.




They are the same so the SPDX link can also be provided but it is about data
exchange.

Does this file need a context to explain it's purpose in the RTEMS kernel source
tree?


Maybe something like this:

"This license file serves as a template for the license header in files."

--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH v2] Add LICENSE.BSD-2-Clause

2018-11-22 Thread Chris Johns
On 23/11/2018 17:34, Sebastian Huber wrote:
> On 23/11/2018 07:13, Chris Johns wrote:
>> On 23/11/2018 16:58, Sebastian Huber wrote:
>>> On 15/11/2018 14:10, Sebastian Huber wrote:
 Copied from:

 https://spdx.org/licenses/BSD-2-Clause.html

 The formatting is suitable for C/C++ source comments.

 Update #3053.
>>> It would be nice if this patch could be reviewed. It would be quite bad if 
>>> we
>>> have an error in this template file.
>>>
>> Yes.
>>
>> Should https://opensource.org/licenses/BSD-2-Clause be the reference?
> 
> I got some private mails from FreeBSD developers as a result of this question:
> 
> https://lists.freebsd.org/pipermail/freebsd-hackers/2018-October/053447.html
> 

OK.

> This let me to change the URL in v2 of this patch to spdx.org which is
> maintained by the Linux Foundation instead of OSI.

Sure.

>>
>> They are the same so the SPDX link can also be provided but it is about data
>> exchange.
>>
>> Does this file need a context to explain it's purpose in the RTEMS kernel 
>> source
>> tree?
> 
> Maybe something like this:
> 
> "This license file serves as a template for the license header in files."

.. Copy to the top of source the file as a comment and update the year and
copyright fields ... or something like this.

Thanks
Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


New content for COPYING file in RTEMS?

2018-11-22 Thread Sebastian Huber

Hello,

the COPYING file in RTEMS is currently just the GPLv2 license text:

https://git.rtems.org/rtems/tree/COPYING

I propose to replace this with a content similar to theCOPYING.NEWLIB 
file in Newlib:


https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=COPYING.NEWLIB;h=7242a44acac981e29469441dcb40b22803bf2d2b;hb=HEAD

This file can then be used to meet this BSD-2 license requirement:

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

It can be used to document which contributors agreed to the license 
change from RTEMS GPL to BSD-2-Clause.


--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

[PATCH v3] Add LICENSE.BSD-2-Clause

2018-11-22 Thread Sebastian Huber
Copied from:

https://spdx.org/licenses/BSD-2-Clause.html

The formatting is suitable for C/C++ source comments.

Update #3053.
---
 LICENSE.BSD-2-Clause | 39 +++
 1 file changed, 39 insertions(+)
 create mode 100644 LICENSE.BSD-2-Clause

diff --git a/LICENSE.BSD-2-Clause b/LICENSE.BSD-2-Clause
new file mode 100644
index 00..d19f31e646
--- /dev/null
+++ b/LICENSE.BSD-2-Clause
@@ -0,0 +1,39 @@
+https://spdx.org/licenses/BSD-2-Clause.html
+
+This license file serves as a template for the license header in files.
+
+You are the copyright holder.  Copy to comment below the top of the file in
+which you want to use this license for your contribution.  Replace the
+ placeholder with the year of your first substantial contribution
+to this file.  Update the  with the year of your last substantial
+contribution to this file.  If the first and last years are the same, then
+remove the  placeholder with the comma.  Replace the
+ placeholder with your name.  You must not alter anything
+else in the license comment.
+
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ * 
+ * Copyright (C) ,  
+ * 
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
-- 
2.16.4

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel