Re: Levenberg-Marquardt non-linear least-squares fitting in Python [follow-on]

[edmondo . giovannozzi]

> ltemp = [ydata[i] - ydata[0] for i in range(ll)]
> ytemp = [ltemp[i] * .001 for i in range(ll)]
> ltemp = [xdata[i] - xdata[0] for i in range(ll)]
> xtemp = [ltemp[i] * .001 for i in range(ll)]

Use the vectorization given by numpy:

  ytemp = (ydata - ydata[0]) * 0.001
  xtemp =  (xdata - xdata[0]) * 0.001


  fitted =  popt[0] - popt[1] * np.exp(-popt[2] * xtemp)

 or better

  fitted = func2fit(xtemp, *popt)
  

> #
> #  popt is a list of the three optimized fittine parameters [a, b, c]
> #  we are interested in the value of a.
> #  cov is the 3 x 3 covariance matrix, the standard deviation (error) of the 
> fit is
> #  the square root of the diagonal.
> #
> popt,cov = curve_fit(func2fit, xtemp, ytemp)
> #
> #  Here is what the fitted line looks like for plotting
> #
> fitted = [popt[0] - popt[1] * np.exp(-popt[2] * xtemp[i]) for i in 
> range(ll)]
> #
> #  And now plot the results to check the fit
> #
> fig1, ax1 = plt.subplots()
> plt.title('Normalized Data ' + str(run_num))
> color_dic = {0: "red", 1: "green", 2: "blue", 3: "red", 4: "green", 5: 
> "blue"}
> ax1.plot(xtemp, ytemp, marker = '.', linestyle  = 'none', color = 
> color_dic[run_num])
> ax1.plot(xtemp, fitted, linestyle = '-', color = color_dic[run_num])
> plt.savefig('Normalized ' + str(run_num))
> perr = np.sqrt(np.diag(cov))
> return popt, cov, xdata[0], ydata[0], fitted, perr[0]

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Tony van der Hoff]

Hello Chris.
Thanks for your interest.

On 28/03/2019 18:04, Chris Angelico wrote:
> On Fri, Mar 29, 2019 at 4:10 AM Tony van der Hoff  
> wrote:
>>
>> This'll probably work:
> 
> You have a python3 shebang, but are you definitely running this under Python 
> 3?
> 
Absolutely.

> Here's a much more minimal example. Can you see if this also fails for you?
> 
> import sys
> from jinja2 import Template
> print(Template("French: {{french}}").render({"french": "année"}))
> print(sys.version)
> 

Presumably you expect to run this from the command line. It works as
expected:

French: année
3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516]

However, with a slight modification:

#!/usr/bin/env python3

import sys
from jinja2 import Template
print ("Content-type: text/html\n\n")
print(Template("French: {{french}}").render({"french": "année"}))
print(sys.version)

and running it in a browser (tried both chrome and Firefox),
it fails as before: blank web page. Replacing the accented character
displays the expected result:

French: annee 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516]

Thanks to all who have pitched in. As I previously mentioned, I have a
work-around, so no big deal, but I would like to get to the bottom of this.

Cheers, Tony
-- 
Tony van der Hoff| mailto:t...@vanderhoff.org
Buckinghamshire, England |
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Syntax for one-line "nonymous" functions in "declaration style"

[Antoon Pardon]

On 27/03/19 09:21, Alexey Muranov wrote:
> Whey you need a simple function in Python, there is a choice between a
> normal function declaration and an assignment of a anonymous function
> (defined by a lambda-expression) to a variable:
>
>    def f(x): return x*x
>
> or
>
>    f = lambda x: x*x
>
> It would be however more convenient to be able to write instead just
>
>    f(x) = x*x 

I have mixed feelings about this. I think anonymous functions only have a
place as arguments to a function. So f(x) = x * x, doesn't look like an 
anonymous function but just an other way to have a named fuction. (as
others have already pointed out)

If we really want an alternative for the lambda expresion, I would go
for the following.

f = x -> x*x (or maybe we could have the λ instead of lambda)

If we want haskell like patterns for defining functions I would go
for the following:

fac(0) -> 1
fac(n) -> n * fac(n - 1)

Which seems incompatible with the proposal for an alternative lambda. 

-- 
Antoon Pardon.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Thomas Jollans]

On 29/03/2019 11.10, Tony van der Hoff wrote:
> and running it in a browser (tried both chrome and Firefox),

How?

> it fails as before: blank web page.

No traceback? There must be a traceback somewhere. In a log file perhaps.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Chris Angelico]

On Fri, Mar 29, 2019 at 9:12 PM Tony van der Hoff  wrote:
>
> Hello Chris.
> Thanks for your interest.
>
> On 28/03/2019 18:04, Chris Angelico wrote:
> > On Fri, Mar 29, 2019 at 4:10 AM Tony van der Hoff  
> > wrote:
> >>
> >> This'll probably work:
> >
> > You have a python3 shebang, but are you definitely running this under 
> > Python 3?
> >
> Absolutely.
>
> > Here's a much more minimal example. Can you see if this also fails for you?
> >
> > import sys
> > from jinja2 import Template
> > print(Template("French: {{french}}").render({"french": "année"}))
> > print(sys.version)
> >
>
> Presumably you expect to run this from the command line. It works as
> expected:
>
> French: année
> 3.5.3 (default, Sep 27 2018, 17:25:39)
> [GCC 6.3.0 20170516]
>
> However, with a slight modification:
>
> #!/usr/bin/env python3
>
> import sys
> from jinja2 import Template
> print ("Content-type: text/html\n\n")

Try: text/html; charset=utf-8

That might be all you need to make the browser understand it
correctly. Otherwise, as Thomas says, you will need to figure out
where the traceback is, which can probably be answered by figuring out
what "running it in a browser" actually means.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Tony van der Hoff]

On 29/03/2019 11:08, Chris Angelico wrote:
> On Fri, Mar 29, 2019 at 9:12 PM Tony van der Hoff  
> wrote:
>>
>> Hello Chris.
>> Thanks for your interest.
>>
>> On 28/03/2019 18:04, Chris Angelico wrote:
>>> On Fri, Mar 29, 2019 at 4:10 AM Tony van der Hoff  
>>> wrote:

 This'll probably work:
>>>
>>> You have a python3 shebang, but are you definitely running this under 
>>> Python 3?
>>>
>> Absolutely.
>>
>>> Here's a much more minimal example. Can you see if this also fails for you?
>>>
>>> import sys
>>> from jinja2 import Template
>>> print(Template("French: {{french}}").render({"french": "année"}))
>>> print(sys.version)
>>>
>>
>> Presumably you expect to run this from the command line. It works as
>> expected:
>>
>> French: année
>> 3.5.3 (default, Sep 27 2018, 17:25:39)
>> [GCC 6.3.0 20170516]
>>
>> However, with a slight modification:
>>
>> #!/usr/bin/env python3
>>
>> import sys
>> from jinja2 import Template
>> print ("Content-type: text/html\n\n")
> 
> Try: text/html; charset=utf-8
> 
No difference

> That might be all you need to make the browser understand it
> correctly. Otherwise, as Thomas says, you will need to figure out
> where the traceback is, which can probably be answered by figuring out
> what "running it in a browser" actually means.
> 

Running in browser:
http://localhost/~tony/private/home/learning/jinja/minimal/minimal.py

In apache2.access.log:
::1 - tony [29/Mar/2019:11:22:13 +] "GET
/~tony/private/home/learning/jinja/minimal/minimal.py HTTP/1.1" 200 204
"http://localhost/~tony/private/home/learning/jinja/minimal/";
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/72.0.3626.81 Safari/537.36"
::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"
::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"

So, 408 is a bit unusual for localhost. With the accented character
removed, no timeout is reported. Maybe a clue.

Can find no other traceback. Nothing relevant in apache2/error.log


-- 
Tony van der Hoff| mailto:t...@vanderhoff.org
Buckinghamshire, England |
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Thomas Jollans]

On 29/03/2019 12.39, Tony van der Hoff wrote:
> On 29/03/2019 11:08, Chris Angelico wrote:
>> On Fri, Mar 29, 2019 at 9:12 PM Tony van der Hoff  
>> wrote:
>>>
>>> Hello Chris.
>>> Thanks for your interest.
>>>
>>> On 28/03/2019 18:04, Chris Angelico wrote:
 On Fri, Mar 29, 2019 at 4:10 AM Tony van der Hoff  
 wrote:
>
> This'll probably work:

 You have a python3 shebang, but are you definitely running this under 
 Python 3?

>>> Absolutely.
>>>
 Here's a much more minimal example. Can you see if this also fails for you?

 import sys
 from jinja2 import Template
 print(Template("French: {{french}}").render({"french": "année"}))
 print(sys.version)

>>>
>>> Presumably you expect to run this from the command line. It works as
>>> expected:
>>>
>>> French: année
>>> 3.5.3 (default, Sep 27 2018, 17:25:39)
>>> [GCC 6.3.0 20170516]
>>>
>>> However, with a slight modification:
>>>
>>> #!/usr/bin/env python3
>>>
>>> import sys
>>> from jinja2 import Template
>>> print ("Content-type: text/html\n\n")
>>
>> Try: text/html; charset=utf-8
>>
> No difference
> 
>> That might be all you need to make the browser understand it
>> correctly. Otherwise, as Thomas says, you will need to figure out
>> where the traceback is, which can probably be answered by figuring out
>> what "running it in a browser" actually means.
>>
> 
> Running in browser:
> http://localhost/~tony/private/home/learning/jinja/minimal/minimal.py
> 
> In apache2.access.log:

So it's running in apache!

Now the question is what apache is doing. Is it running it as a CGI
script? Is it doing something clever for Python files (maybe involving
Python 2?)

... wild guess: if the script is running as CGI in an enviroment with an
ASCII-using "C" locale, with Python 3.5, you wouldn't be able to print
non-ASCII characters by default. I think. In any case I remember reading
about this problem (if this is the problem) being fixed in a newer
version of Python.

> ::1 - tony [29/Mar/2019:11:22:13 +] "GET
> /~tony/private/home/learning/jinja/minimal/minimal.py HTTP/1.1" 200 204
> "http://localhost/~tony/private/home/learning/jinja/minimal/";
> "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/72.0.3626.81 Safari/537.36"
> ::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"
> ::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"
> 
> So, 408 is a bit unusual for localhost. With the accented character
> removed, no timeout is reported. Maybe a clue.
> 
> Can find no other traceback. Nothing relevant in apache2/error.log
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Understanding the MRO with multiple inheritance

[Arup Rakshit]

I basically had defined 4 classes. SimpleList be the base class for all the 
other 3 classes. SortedList and IntList both being the child of the base class 
SimpleList. They have a single inheritance relationship. Now I have the last 
class called SortedIntList which has multiple inheritance relationship with 
IntList and SortedList. The classes looks like:

class SimpleList:
def __init__(self, items):
self._items = list(items)

def add(self, item):
self._items.append(item)

def __getitem__(self, index):
return self._items[index]

def sort(self):
self._items.sort()

def __len__(self):
return len(self._items)

def __repr__(self):
return "SimpleList({!r})".format(self._items)

class SortedList(SimpleList):
def __init__(self, items=()):
super().__init__(items)

def add(self, item):
super().add(item)
self.sort()

def __repr__(self):
return "SortedList({!r})".format(list(self))


class IntList(SimpleList):
def __init__(self, items=()):
for x in items:
self._validate(x)
super().__init__(items)

@staticmethod
def _validate(x):
if not isinstance(x, int):
raise TypeError('IntList only supports integer values.')

def add(self, item):
self._validate(item)
super().add(item)

def __repr__(self):
return "IntList({!r})".format(list(self))


class SortedIntList(IntList, SortedList):
def __repr__(self):
return "SortedIntList({!r})".format(list(self))


Now when I call the add method on the SortedIntList class’s instance, I was 
expecting super.add() call inside the IntList class add method will dispatch it 
to the base class SimpleList. But in reality it doesn’t, it rather forwards it 
to the SortedList add method. How MRO guides here can anyone explain please?

I am little lost in the chain.

ob = SortedIntList((2,4,3))
ob.add(0)
print(ob)
# SortedIntList([0, 2, 3, 4])


Thanks,

Arup Rakshit
a...@zeit.io



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handy utilities = Friday Filosofical Finking

[Neil Cerutti]

On 2019-03-28, DL Neil  wrote:
> How do you keep, use, and maintain those handy snippets,
> functions, classes... - units of code, which you employ
> over-and-over again?
>
> Having coded 'stuff' once, most of us will keep units of code,
> "utilities", which we expect will be useful in-future (DRY
> principle), eg functions to rename files, choose unique
> back-up/new fileNMs, accessing a DB, journalling (logging)
> start/stop msgs, building specs from YAML/JSON/XML/.ini config
> files (tongue~cheek), etc.
>
> Do you 'keep' these, or perhaps next time you need something
> you've 'done before' do you remember when/where a technique was
> last used/burrow into 'history'? (else, code it from scratch,
> all over again)

I usually wait until I notice I've written or wanted the same
code snippet many times before I'll make it into a library.

> How do you keep them updated, ie if add some new idea, better
> err-checking, re-factor - how to add these 'back' into previous
> places utility is used? (who wants more "technical debt", plus
> handling classic update/versioning issue)

After the library is written back-porting it to other place where
it will be useful is done slowly over time as those utilities
need updating for other reasons.

> How do you keep these? eg special file/dir, within IDE, leave
> in app and 'remember', on paper, ... If the former, how do you
> access/import them from the various applications/systems?
> (Python's import rules and restrictions, change control/version
> control)

I have a lib directory in my PYTHONPATH to dump 'em.

-- 
Neil Cerutti
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Peter J. Holzer]

On 2019-03-29 12:56:00 +0100, Thomas Jollans wrote:
> On 29/03/2019 12.39, Tony van der Hoff wrote:
> > Running in browser:
> > http://localhost/~tony/private/home/learning/jinja/minimal/minimal.py
> > 
> > In apache2.access.log:
> 
> So it's running in apache!
> 
> Now the question is what apache is doing. Is it running it as a CGI
> script? Is it doing something clever for Python files (maybe involving
> Python 2?)
> 
> ... wild guess: if the script is running as CGI in an enviroment with an
> ASCII-using "C" locale, with Python 3.5, you wouldn't be able to print
> non-ASCII characters by default. I think. In any case I remember reading
> about this problem (if this is the problem) being fixed in a newer
> version of Python.

This is very likely correct. I also had this problem with the default
Apache configuration on Debian, which explicitely sets LANG=C (Edit
/etc/apache2/envvars to change this).

The behaviour can be easily reproduced on the command line:

hrunkner:~/tmp 15:27 :-) 1021% ./annee 
Content-type: text/html


French: année
3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516]

hrunkner:~/tmp 15:27 :-) 1022% echo $LANG
en_US.UTF-8

hrunkner:~/tmp 15:34 :-) 1023% LANG=C

hrunkner:~/tmp 15:34 :-) 1024% ./annee   
Content-type: text/html


Traceback (most recent call last):
  File "./annee", line 6, in 
print(Template("French: {{french}}").render({"french": "ann\xe9e"}))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 11: 
ordinal not in range(128)

This was fixed(?) in Python 3.7.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Peter Otten]

Peter J. Holzer wrote:

> On 2019-03-29 12:56:00 +0100, Thomas Jollans wrote:
>> On 29/03/2019 12.39, Tony van der Hoff wrote:
>> > Running in browser:
>> > http://localhost/~tony/private/home/learning/jinja/minimal/minimal.py
>> > 
>> > In apache2.access.log:
>> 
>> So it's running in apache!
>> 
>> Now the question is what apache is doing. Is it running it as a CGI
>> script? Is it doing something clever for Python files (maybe involving
>> Python 2?)
>> 
>> ... wild guess: if the script is running as CGI in an enviroment with an
>> ASCII-using "C" locale, with Python 3.5, you wouldn't be able to print
>> non-ASCII characters by default. I think. In any case I remember reading
>> about this problem (if this is the problem) being fixed in a newer
>> version of Python.
> 
> This is very likely correct. I also had this problem with the default
> Apache configuration on Debian, which explicitely sets LANG=C (Edit
> /etc/apache2/envvars to change this).
> 
> The behaviour can be easily reproduced on the command line:
> 
> hrunkner:~/tmp 15:27 :-) 1021% ./annee
> Content-type: text/html
> 
> 
> French: année
> 3.5.3 (default, Sep 27 2018, 17:25:39)
> [GCC 6.3.0 20170516]
> 
> hrunkner:~/tmp 15:27 :-) 1022% echo $LANG
> en_US.UTF-8
> 
> hrunkner:~/tmp 15:34 :-) 1023% LANG=C
> 
> hrunkner:~/tmp 15:34 :-) 1024% ./annee
> Content-type: text/html
> 
> 
> Traceback (most recent call last):
>   File "./annee", line 6, in 
> print(Template("French: {{french}}").render({"french": "ann\xe9e"}))
> UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in
> position 11: ordinal not in range(128)
> 
> This was fixed(?) in Python 3.7.
> 
> hp

You could try to specify the encoding with PYTHONIOENCODING:
 
$ LANG=C python3 -c "print('ann\xe9e')"
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 
3: ordinal not in range(128)
$ PYTHONIOENCODING=UTF-8 LANG=C python3 -c "print('ann\xe9e')"
année


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handy utilities = Friday Filosofical Finking

[Akkana Peck]

DL Neil writes:
> How do you keep, use, and maintain those handy snippets, functions,
> classes... - units of code, which you employ over-and-over again?

Fun topic!

I have two methods:

First, in my scripts directory I have a file called
"python-cheatsheet.py" where I save small tips that I
think I might not remember; mostly 2- to 5-liners.

Second, I wrote a script called langgrep that searches in known
places for files written in a specified language (it looks at file
extensions and shebangs), then greps for a pattern in all those
files. It looks in ~/bin plus a set of other directories, like
certain projects under ~/src.

So for instance if I'm trying to remember the syntax to read a CSV
file as a dictionary, I might run

langgrep python csv.DictReader
or
langgrep python -i csv | grep -i dict

Since you asked about importing libraries: in my ~/bin I have a
directory pythonlibs, which is in my PYTHONPATH, where I put
(usually symlinks to) python files and packages I use regularly
that aren't installed systemwide.

...Akkana
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Peter J. Holzer]

On 2019-03-28 11:07:22 +0100, dieter wrote:
> Paul Moore  writes:
> > My real interest is in whether any libraries exist to do this sort
> > of thing (there are plenty of parser libraries for text, pyparsing
> > being the obvious one, but far fewer for binary structures).
> 
> Sure. *BUT* the library must fit your specific binary structure.
> How should a general libary know how to interpret your
> specific "type byte"s or that "(" introduces a homogenous
> sequence of given length which must be terminated by ")"?

Obviously you need some way to describe the specific binary format you
want to parse - in other words, a grammar. The library could then use
the grammar to parse the input - either by interpreting it directly, or
by generating (Python) code from it. The latter has the advantage that
it has to be done only once, not every time you want to parse a file.

If that sounds familiar, it's what yacc does. Except that it does it for
text files, not binary files. I am not aware of any generic binary
parser generator for Python. I have read research papers about such
generators for (I think) C and Java, but I don't remember the names and
I'm not sure if the generators got beyond the proof of concept stage.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Dan Sommers]

On 3/29/19 12:13 PM, Peter J. Holzer wrote:


Obviously you need some way to describe the specific binary format you
want to parse - in other words, a grammar. The library could then use
the grammar to parse the input - either by interpreting it directly, or
by generating (Python) code from it. The latter has the advantage that
it has to be done only once, not every time you want to parse a file.

If that sounds familiar, it's what yacc does. Except that it does it for
text files, not binary files. I am not aware of any generic binary
parser generator for Python. I have read research papers about such
generators for (I think) C and Java, but I don't remember the names and
I'm not sure if the generators got beyond the proof of concept stage.


It's been a while since I've used those tools, but if you
create a lexer (the yylex() function) that can tokenize a
binary stream, then yacc won't know the difference.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Paul Moore]

On Fri, 29 Mar 2019 at 16:16, Peter J. Holzer  wrote:

> Obviously you need some way to describe the specific binary format you
> want to parse - in other words, a grammar. The library could then use
> the grammar to parse the input - either by interpreting it directly, or
> by generating (Python) code from it. The latter has the advantage that
> it has to be done only once, not every time you want to parse a file.
>
> If that sounds familiar, it's what yacc does. Except that it does it for
> text files, not binary files. I am not aware of any generic binary
> parser generator for Python. I have read research papers about such
> generators for (I think) C and Java, but I don't remember the names and
> I'm not sure if the generators got beyond the proof of concept stage.

That's precisely what I'm looking at. The construct library
(https://pypi.org/project/construct/) basically does that, but using a
DSL implemented in Python rather than generating Python code from a
grammar. In fact, the problem I had with my recursive data structure
turned out to be solvable in construct - as the DSL effectively builds
a data structure describing the grammar, I was able to convert the
problem of writing a recursive grammar into one of writing a recursive
data structure:

type_layouts = {}
layout1 = 
layout2 = 
type_layouts[1] = layout1
type_layouts[2] = layout2
data_layout = 

However, the resulting parser works, but it gives horrible error
messages. This is a normal problem with generated parsers, there are
plenty of books and articles covering how to persuade tools like yacc
to produce usable error reports on parse failures. There don't seem to
be any particularly good error reporting features in construct
(although I haven't looked closely), so I'm actually now looking at
writing a hand-crafted parser, just to control the error reporting[1].

I don't know which solution I'll ultimately use, but it's an
interesting exercise doing it both ways. And parsing binary data,
unlike parsing text, is actually easy enough that hand crafting a
parser isn't that much of a bother - maybe that's why there's less
existing work in this area.

Paul

[1] The errors I'm reporting on are likely to be errors in my parsing
code at this point, rather than errors in the data, but the problem is
pretty much the same either way ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The Mailing List Digest Project

[Jason Friedman]

>
>  Pretty cool.  FYI, the index page (now containing 4 articles) with Google
>> Chrome 72.0.3626.x prompts me to translate to French.  The articles
>> themselves do not.
>>
>
> I'm now getting the translation offer on other web pages with Chrome, not
just this one.
Thus, please ignore my prior posting.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Peter J. Holzer]

On 2019-03-29 16:34:35 +, Paul Moore wrote:
> On Fri, 29 Mar 2019 at 16:16, Peter J. Holzer  wrote:
> 
> > Obviously you need some way to describe the specific binary format you
> > want to parse - in other words, a grammar. The library could then use
> > the grammar to parse the input - either by interpreting it directly, or
> > by generating (Python) code from it. The latter has the advantage that
> > it has to be done only once, not every time you want to parse a file.
> >
> > If that sounds familiar, it's what yacc does. Except that it does it for
> > text files, not binary files. I am not aware of any generic binary
> > parser generator for Python. I have read research papers about such
> > generators for (I think) C and Java, but I don't remember the names and
> > I'm not sure if the generators got beyond the proof of concept stage.
> 
> That's precisely what I'm looking at. The construct library
> (https://pypi.org/project/construct/) basically does that, but using a
> DSL implemented in Python rather than generating Python code from a
> grammar.

Good to know. I'll add that to my list of Tools Which I'm Not Likely To
Use Soon But Which May Be Useful Some Day.


> However, the resulting parser works, but it gives horrible error
> messages. This is a normal problem with generated parsers, there are
> plenty of books and articles covering how to persuade tools like yacc
> to produce usable error reports on parse failures.

Yeah, that still seems to be an unsolved problem.

> I don't know which solution I'll ultimately use, but it's an
> interesting exercise doing it both ways. And parsing binary data,
> unlike parsing text, is actually easy enough that hand crafting a
> parser isn't that much of a bother - maybe that's why there's less
> existing work in this area.

I'm a bit sceptical about that. Writing a hand-crafted parser for most
text-based grammars isn't that hard either, but there are readily-
available tools (like yacc), so people use them (despite problems like
horrible error messages). For binary protocols, such tools are much less
well-known. It may be true that binary grammars seem simpler. But in
practice there are lots and lots of security holes because hand-crafted
parsers tend to use un-warranted shortcuts (see heart-bleed or the JPEG
parsing bug of the week), which an automatically generated parser would
not take.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Cameron Simpson]

On 27Mar2019 18:41, Paul Moore  wrote:

I'm looking for a library that lets me parse binary data structures.
The stdlib struct module is fine for simple structures, but when it
gets to more complicated cases, you end up doing a lot of the work by
hand (which isn't that hard, and is generally perfectly viable, but
I'm feeling lazy ;-))


I wrote my own: cs.binary, available on PyPI. The PyPI page has is 
module docs, which I think are ok:


 https://pypi.org/project/cs.binary/

Here's a binary packet protocol built on to of it:

 https://pypi.org/project/cs.packetstream/

and here's an ISO14496 (the MP4 format) parser using it:

 https://pypi.org/project/cs.iso14496/

Of interest is that ISO 14496 uses recursive data structures.

The command line "main" function is up the top, which shows how it is 
used.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Library for parsing binary structures

[Cameron Simpson]

On 30Mar2019 09:44, Cameron Simpson  wrote:

On 27Mar2019 18:41, Paul Moore  wrote:

I'm looking for a library that lets me parse binary data structures.
The stdlib struct module is fine for simple structures, but when it
gets to more complicated cases, you end up doing a lot of the work by
hand (which isn't that hard, and is generally perfectly viable, but
I'm feeling lazy ;-))


I wrote my own: cs.binary, available on PyPI. The PyPI page has is 
module docs, which I think are ok:


https://pypi.org/project/cs.binary/

[...]

and here's an ISO14496 (the MP4 format) parser using it:
https://pypi.org/project/cs.iso14496/
Of interest is that ISO 14496 uses recursive data structures.


I neglected to mention: with cs.binary you write binary formats as 
classes (which allows for easy conditional parsing and so forth).


And... normally those classes know how to write themselves back out, 
which makes for easy transcription and binary data generation.


Conditional binary formats require a class specific .transcribe method 
(which just yields binary data or some other convenient things including 
other binary class instances, see doco) but flat records have a default 
.transcribe.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding the MRO with multiple inheritance

[DL Neil]

Arup,

There is a minefield here. Are you using Python 2 or 3?

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding the MRO with multiple inheritance

[Chris Angelico]

On Fri, Mar 29, 2019 at 11:54 PM Arup Rakshit  wrote:
>
> Now when I call the add method on the SortedIntList class’s instance, I was 
> expecting super.add() call inside the IntList class add method will dispatch 
> it to the base class SimpleList. But in reality it doesn’t, it rather 
> forwards it to the SortedList add method. How MRO guides here can anyone 
> explain please?
>

When you call super, you're saying "go to the next in the MRO". You
can examine the MRO by looking at SortedIntList.__mro__ - that should
show you the exact order that methods will be called.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Query windows event log with python

[mons . sidus]

lol cheeky as. 
server = 'x' # name of the target computer to get event logs
source = 'x' # 'Application' # 'Security'


hand = win32evtlog.OpenEventLog(server, source)
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | 
win32evtlog.EVENTLOG_SEQUENTIAL_READ
total = win32evtlog.GetNumberOfEventLogRecords(hand)
event_no = 1

log = win32evtlog.ReadEventLog(hand, flags, 0, )

i wanna keep going hard as i do and learn it but keep getting a nonsensical 
error
OverflowError: days=1834132873; must have magnitude <= 9

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
SystemError:  returned a result with an error 
set 
cant find much info, tried a few win32 modules to see if it made a difference.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding the MRO with multiple inheritance

[dieter]

Arup Rakshit  writes:
> I basically had defined 4 classes. SimpleList be the base class for all the 
> other 3 classes. SortedList and IntList both being the child of the base 
> class SimpleList. They have a single inheritance relationship. Now I have the 
> last class called SortedIntList which has multiple inheritance relationship 
> with IntList and SortedList. The classes looks like:
>
> class SimpleList:
> ...
> def add(self, item):
> self._items.append(item)
> ...
>
> class SortedList(SimpleList):
> ...
> def add(self, item):
> super().add(item)
> self.sort()
> ...
>
> class IntList(SimpleList):
> ...
> def add(self, item):
> self._validate(item)
> super().add(item)
> ...
>
> class SortedIntList(IntList, SortedList):
> ...
>
>
> Now when I call the add method on the SortedIntList class’s instance, I was 
> expecting super.add() call inside the IntList class add method will dispatch 
> it to the base class SimpleList.

"super" is mainly for the use case of "mixin classes".

By design, a "mixin class" must cope with an unknown inheritance structure
and it must be able to cooperate with all classes in this structure.
Especially, there must be a way to give all classes in the structure
the chance to apply a method. This way is "super".

As an example, think of "__init__". Each class in the class
structure might need to perform its own initialization.
To archieve this, "__init__" likely has the signature
"__init__(self, **kw)" and each class pickes its initialization
arguments from "kw" and then delegates to the next (in MRO) class
with "super().__init__(kw)". This class may not be a base class
of the current class.


Thus, you use "super" when you want to delegate to an unknown
class in the inheritance structure; if you want to delegate
to a specific base class, then you do not use "super"
but delegate explicitly to this class.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding the MRO with multiple inheritance

[Arup Rakshit]

Hello DL,

I am using Python3.


Thanks,

Arup Rakshit
a...@zeit.io



> On 30-Mar-2019, at 6:58 AM, DL Neil  wrote:
> 
> Arup,
> 
> There is a minefield here. Are you using Python 2 or 3?
> 
> -- 
> Regards =dn
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list