[issue41878] python3 fails to use custom dict-like object as symbols in eval()
New submission from Robert Haschke :
The attached file implements a custom dict-like class (MyDict) as a minimal
example of code I am using in a larger codebase.
Before you ask, why I reimplemented a dict-like object: The real code base
employs a hierarchical dict, referencing recursively to the parent dict, if a
key cannot be found in the current dict.
The main code of the file defines two entries/variables for this dict:
symbols = MyDict()
symbols['abc'] = '[1, 2, 3]'
symbols['xyz'] = 'abc + abc'
and eval_text('xyz', symbols) should evaluate to the python expression as you
would have evaluated those variables in a python interpreter.
While this works for the first given expression (above), it fails for this one:
symbols['xyz'] = '[abc[i]*abc[i] for i in [0, 1, 2]]'
raising NameError: name 'abc' is not defined.
The same code works perfectly in python 2.7. Hence, I assume this is a bug in
python3.
--
files: buggy.py
messages: 377616
nosy: Robert Haschke
priority: normal
severity: normal
status: open
title: python3 fails to use custom dict-like object as symbols in eval()
versions: Python 3.8
Added file: https://bugs.python.org/file49476/buggy.py
___
Python tracker
<https://bugs.python.org/issue41878>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41878] python3 fails to use custom mapping object as symbols in eval()
Robert Haschke added the comment:
Looks like the list generator is considered as a new nested scope, which
prohibits access to local variables?
This basic expression, passing local symbols only, fails as well:
eval('[abc[i]*abc[i] for i in [0, 1, 2]]', {}, dict(abc=[1, 2, 3]))
while this one, passing dict as global symbols, works:
eval('[abc[i]*abc[i] for i in [0, 1, 2]]', dict(abc=[1, 2, 3]))
However, passing globals must be a real dict. So I cannot simply pass my custom
mapping to globals.
--
title: python3 fails to use custom dict-like object as symbols in eval() ->
python3 fails to use custom mapping object as symbols in eval()
___
Python tracker
<https://bugs.python.org/issue41878>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41878] python3 fails to use custom mapping object as symbols in eval()
Robert Haschke added the comment: Thanks, Josh, for this clarification. What is the suggested mitigation? Obviously, I need to pass all symbols from my hierarchical dictionary in a flattend version as a dict to globals. Or do you see another option? -- ___ Python tracker <https://bugs.python.org/issue41878> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
Robert Haschke added the comment: Uploaded a "hg diff" against the recent 2.7 branch of the source repo. -- Added file: http://bugs.python.org/file43414/minidom.insertBefore.patch ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
Robert Haschke added the comment: I uploaded a simple example to illustrate the tremendous performance boost. Obviously, the example exploits the caching improvement as much as possible: The code assembles a XML document by inserting new nodes before the last one... These are the timing results: $ python minidom_example.py old new oldtime for 5000 iterations: 18.422152 newtime for 5000 iterations: 0.129384 $ python minidom_example.py old new oldtime for 1 iterations: 68.305351 newtime for 1 iterations: 0.142071 You see the quadratic increase of time... IMHO, this is clearly a (performance) bug and really many people in the ROS community are affected. Hence, I hope that this patch will find its way into some python versions currently used by standard Linux distributions. -- Added file: http://bugs.python.org/file43436/minidom_example.py ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
Robert Haschke added the comment: Indeed there is a small slow down for insertion at the beginning. However, this is simply due to the extra function _index() and thus linear in the number of insertion operations. My patch essentially boosts insertions before /any fixed/ node. If this reference node changes between insertions (as in your "before first" example), there is no gain anymore. Of course, this optimization comes at the cost of an additional integer per node. There is no free lunch! I know, that there are other parsers (e.g. etree) available. However changing my existing code base from minidom to etree will be a heavy change, which isn't easily accepted as well. I think, my minidom patch is a clean and simple fix to a common performance issue. As it mostly effects 2.7, it should primarily go there ;-) -- ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
Robert Haschke added the comment: I don't see how to further minimize the checks - all of them are required. I think, the most common uses cases to create a document are appendChild(), which is not affected, and insertBefore() using the same refChild for a while. In that case, the patch gives a tremendous speedup. If you as maintainers don't want to share this improvement with the community, it's your choice and I will be fine with that. -- ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
Robert Haschke added the comment: Thank you very much for further improving the code. As I understand it, the trick is to use temporary variables to minimize access time. Learned something new. I adapted your patch to python 2.7 again. Now, in python3, the new code is even faster than the old one (sometimes) for front insertions: 36 time for 1 insertions at front: 0.117563 opt36 time for 1 insertions at front: 0.116014 36 time for 1 insertions at front: 0.114282 opt36 time for 1 insertions at front: 0.116710 old time for 5000 insertions at front: 0.044055 new time for 5000 insertions at front: 0.075433 opt27 time for 5000 insertions at front: 0.052135 old time for 5000 insertions at back: 15.241450 new time for 5000 insertions at back: 0.071004 opt27 time for 5000 insertions at back: 0.046850 I hope you can consider, the patch for python 2.7. There the performance gain is most significant. -- ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24424] xml.dom.minidom: performance issue with Node.insertBefore()
New submission from Robert Haschke: Node.insertBefore() has a serious performance issue: Using self.childNodes.index(refChild) it searches for the correct index in childNodes where the newChild should be inserted. However, index() is linear in time w.r.t. the size of childNodes. Hence, if there are many children, runtime dramatically increases. Adding a simple caching mechanism (caching the previously used reference) I was able to reduce runtime in my particular case from 16s to 1.6s, i.e. a factor of 10! -- components: XML files: minidom.insertBefore.patch keywords: patch messages: 245128 nosy: Robert Haschke priority: normal severity: normal status: open title: xml.dom.minidom: performance issue with Node.insertBefore() type: performance versions: Python 2.7 Added file: http://bugs.python.org/file39674/minidom.insertBefore.patch ___ Python tracker <http://bugs.python.org/issue24424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
