Re: ctypes, memory mapped files and context manager
Dear Peter,
thanks for taking valuable time to look into my issue.
It might be related to my distinct silliness, but the problem persists with
your code as well. Further comments inlined.
On Dienstag, 27. Dezember 2016 21:39:51 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> >
> > def __enter__(self):
> > # resize the mmap (and backing file), if structure exceeds mmap
> > # size mmap size must be aligned to mmap.PAGESIZE
> > cssize = ctypes.sizeof(self._cstruct)
> >
> > if self._offset + cssize > self._mm.size():
> > newsize = align(self._offset + cssize, mmap.PAGESIZE)
> > self._mm.resize(newsize)
> >
> > self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
> > return self._csinst
>
> Here you give away a reference to the ctypes.BigEndianStructure. That means
> you no longer control the lifetime of self._csinst which in turn holds a
> reference to the underlying mmap or whatever it's called.
Here's the code, based on contextmanager:
@contextmanager
def cstructmap(cstruct, mm, offset = 0):
# resize the mmap (and backing file), if structure exceeds mmap size
# mmap size must be aligned to mmap.PAGESIZE
cssize = ctypes.sizeof(cstruct)
if offset + cssize > mm.size():
newsize = align(offset + cssize, mmap.PAGESIZE)
mm.resize(newsize)
yield cstruct.from_buffer(mm, offset)
While much more concise, I don't understand, how it should make a difference
relative to the "with" variable lifetime, when used.
> There might be a way to release the mmap reference while the wrapper
> structure is still alive, but the cleaner way is probably to not give it
> away in the first place, and create a proxy instead with
>
> return weakref.proxy(self._csinst)
This fails, as it doesn't keep the reference long enough.
> > def __exit__(self, exc_type, exc_value, exc_traceback):
> > # free all references into mmap
> > del self._csinst
>
> The line above is redundant. It removes the attribute from the instance
> __dict__ and implicitly decreases its refcount. It does not actually
> physically delete the referenced object. If you remove the del statement the
> line below will still decrease the refcount.
>
> Make sure you understand this to avoid littering your code with cargo cult
> del-s ;)
Yes, I was aware of this. It was a testing relic, that survived somehow.
Sorry. Yes, I usually try to avoid cargo cultry in my code. ;)
> > The issue: when creating a mapping via context manager, we assign a local
> > variable (with ..), that keep existing in the local context, even when the
> > manager context was left. This keeps a reference on the ctypes mapped area
> > alive, even if we try everything to destroy it in __exit__. We have to del
> > the with var manually.
> >
> > Now, I want to get rid of the ugly any error prone del statements.
> >
> > What is needed, is a ctypes operation, that removes the mapping actively,
> > and that could be added to the __exit__ part of the context manager.
Revised code (including your test code):
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239
> > The script creates a memory mapped file in the current directory named
> > "mapfile". When started without arguments, it copies itself into this
> > file, until 10 * mmap.PAGESIZE growth is reached (or it errored out
> > before..).
> >
> > IF you change NOPROB to True, it will actively destruct the context
> > manager vars, and should work as advertized.
> >
> > Any ideas are much appreciated.
>
> You might put some more effort into composing example scripts. Something
> like the script below would have saved me some time...
I'm very sorry about this.
> import ctypes
> import mmap
>
> from contextlib import contextmanager
>
> class T(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
>
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> yield T.from_buffer(m)
>
> SIZE = mmap.PAGESIZE * 2
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
>
> with map_struct(m, 1) as a:
> a.foo = 1
> with map_struct(m, 2) as b:
> b.foo = 2
Unfortunately, your code behaves exactly like mine:
$> python3 mmap_test.py
Traceback (most recent call last):
File "mmap_test.py", line 23, in
with map_struct(m, 2) as b:
File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "mmap_test.py", line 12, in map_struct
m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
BTW, Python2 makes a difference in this respect, but my project is Python3
based. Have you tested this with Python3? It would be interesting to explore
the reasons of this difference, which is, ähem, pretty surprising.
Thanks,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
Hans-Peter Jansen wrote:
> Dear Peter,
>
> thanks for taking valuable time to look into my issue.
You're welcome!
> It might be related to my distinct silliness, but the problem persists
> with your code as well.
Unfortunately I posted the broken toy example rather than the fixed one.
Here's the latter. Basically you have to keep a reference in the context
manager (whether you implement it as a class or a generator doesn't matter)
without giving another reference away to client code:
$ cat mmap_after.py
import ctypes
import mmap
import weakref
from contextlib import contextmanager
class T(ctypes.Structure):
_fields = [("foo", ctypes.c_uint32)]
@contextmanager
def map_struct(m, n):
m.resize(n * mmap.PAGESIZE)
keep_me = T.from_buffer(m)
yield weakref.proxy(keep_me)
SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
with map_struct(m, 1) as a:
a.foo = 1
with map_struct(m, 2) as b:
b.foo = 2
$ python3 mmap_after.py
$
If that doesn't suffice to fix it I'll take another look into your real code
later.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
Anyone? The script to reproduce this problem is in:
https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote:
> Hi, everyone
>
> I'm building a http long polling client for our company's discovery
> service and something weird happened in the following code:
>
> ```python
> while True:
> try:
> r = requests.get("url", stream=True, timeout=3)
> for data in r.iter_lines():
> processing_data...
> except TimeoutException:
> time.sleep(10)
> ```
>
> When I deliberately times out the request and then check the connections
> with `lsof -p process`, I discover that there are *two active
> connections*(ESTABLISH)
> instead of one. After digging around, it turns out it might not be the
> problem with `requests` at all, but gc related to generators.
>
> So I write this script to demonstrate the problem:
>
> https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
>
> Function `A.a` will return a generator which will raise an exception. And
> in function `b`, I'm building new a new instance of `A` and iterate over
> the exception-raising generator. In the exception handler, I'll close the
> generator, delete it, delete the `A` instance, call `gc.collect()` and do
> the whole process all over again.
>
> There's another greenlet checking the `A` instances by using
> `gc.get_objects()`. It turns out there are always two `A` instances.
>
> This is reproducible with python2.7, but not in python3.5. I've also tried
> with `thread` instead of `gevent`, it still happens. I'm guessing it's
> related to garbage collection of generators.
>
> Did I bump into a python2 bug? Or am I simply wrong about the way to close
> generators...?
>
> Thanks
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo wrote: > Anyone? The script to reproduce this problem is in: > > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef > > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote: > >> This is reproducible with python2.7, but not in python3.5. I've also tried >> with `thread` instead of `gevent`, it still happens. I'm guessing it's >> related to garbage collection of generators. >> >> Did I bump into a python2 bug? Or am I simply wrong about the way to close >> generators...? (Please don't top-post.) Maybe the fix is to just use Python 3.5+? :) It probably is to do with the garbage collection of generators; so you may want to consider using something very explicit (eg a context manager) to ensure that you call gen.close(). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
Sorry about breaking the rule. I'm just curios about this problem. And I'm using this workaround to prevent redundant resource creation. https://gist.githubusercontent.com/wooparadog/16948ca6c8ffb22214bf491a280406da/raw/- On Wed, Dec 28, 2016 at 9:12 PM Chris Angelico wrote: > On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo > wrote: > > Anyone? The script to reproduce this problem is in: > > > > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef > > > > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo > wrote: > > > >> This is reproducible with python2.7, but not in python3.5. I've also > tried > >> with `thread` instead of `gevent`, it still happens. I'm guessing it's > >> related to garbage collection of generators. > >> > >> Did I bump into a python2 bug? Or am I simply wrong about the way to > close > >> generators...? > > (Please don't top-post.) > > Maybe the fix is to just use Python 3.5+? :) It probably is to do with > the garbage collection of generators; so you may want to consider > using something very explicit (eg a context manager) to ensure that > you call gen.close(). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > Dear Peter,
> >
> > thanks for taking valuable time to look into my issue.
>
> You're welcome!
>
> > It might be related to my distinct silliness, but the problem persists
> > with your code as well.
>
> Unfortunately I posted the broken toy example rather than the fixed one.
> Here's the latter. Basically you have to keep a reference in the context
> manager (whether you implement it as a class or a generator doesn't matter)
> without giving another reference away to client code:
>
> $ cat mmap_after.py
> import ctypes
> import mmap
> import weakref
>
> from contextlib import contextmanager
>
> class T(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
>
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> keep_me = T.from_buffer(m)
> yield weakref.proxy(keep_me)
Hooray, that did the trick. Great solution, thank you very much!
If you don't mind, I will mention you and your solution at the various places,
I placed this issue over the last weeks.
You made my day, Peter!
It leaves the question on why is Python2 acting as one would expect related to
context managers, and Python3 needs this weakref juggling. Maybe something,
that's worth to be placed in python-dev. What do you think?
Cheers,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > Hans-Peter Jansen wrote:
> > > Dear Peter,
> > >
> > > thanks for taking valuable time to look into my issue.
> >
> > You're welcome!
> >
> > > It might be related to my distinct silliness, but the problem persists
> > > with your code as well.
> >
> > Unfortunately I posted the broken toy example rather than the fixed one.
> > Here's the latter. Basically you have to keep a reference in the context
> > manager (whether you implement it as a class or a generator doesn't
> > matter)
> > without giving another reference away to client code:
> >
> > $ cat mmap_after.py
> > import ctypes
> > import mmap
> > import weakref
> >
> > from contextlib import contextmanager
> >
> > class T(ctypes.Structure):
> > _fields = [("foo", ctypes.c_uint32)]
> >
> > @contextmanager
> >
> > def map_struct(m, n):
> > m.resize(n * mmap.PAGESIZE)
> > keep_me = T.from_buffer(m)
> > yield weakref.proxy(keep_me)
>
> Hooray, that did the trick. Great solution, thank you very much!
Sorry for bothering you again, Peter, but after applying it to the real
project, that fails consistently similar to:
# first run, check explicitly disabled
$> python3 ctypes_mmap_ctx.py
DEBUG: starting offset: 0x10
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: final offset: 0xa1d0
$> python3 ctypes_mmap_ctx.py
Traceback (most recent call last):
File "ctypes_mmap_ctx.py", line 163, in
mf = MapFile(mapfile)
File "ctypes_mmap_ctx.py", line 109, in __init__
if bytes(blk) != bytes(rest):
AttributeError: 'c_ubyte_Array_3632' object has no attribute '__bytes__'
The new issue appears in some consistency checking code. The mapfile is
checked for zeroed out overhang on that line. The weakref proxy isn't behaving
well at that point. Any idea, what could be going wrong with it?
Updated the gist to demonstrate the issue after switching to weakref.proxy().
(has grown even more code in that process, sorry).
It looks like a minor issue, but the code quality is really degraded from
these ugly del statements.
Pete
--
https://mail.python.org/mailman/listinfo/python-list
sorting strings numerically while dealing with missing values
I have a list containing a list of strings that I want to sort numerically by one of the fields. I am doing this: sorted(rows, key=float(itemgetter(sortby))) Which works fine as long as all the sort keys convert to a float. Problem is that some are blank or None and those throw an exception. How can I handle that case and still sort? I'd want the blank or None fields to come out either at the beginning or end of the sorted list (not sure what the customer wants for this yet). -- https://mail.python.org/mailman/listinfo/python-list
Re: sorting strings numerically while dealing with missing values
Larry Martell wrote:
> I have a list containing a list of strings that I want to sort
> numerically by one of the fields. I am doing this:
>
> sorted(rows, key=float(itemgetter(sortby)))
>
> Which works fine as long as all the sort keys convert to a float.
No, that cannot work; unless you have redefined float() you 'l get a
TypeError either here
>>> from operator import itemgetter
>>> float(itemgetter(42))
Traceback (most recent call last):
File "", line 1, in
TypeError: float() argument must be a string or a number, not
'operator.itemgetter'
or here:
>>> sorted("abc", key=3.0)
Traceback (most recent call last):
File "", line 1, in
TypeError: 'float' object is not callable
> Problem is that some are blank or None and those throw an exception.
> How can I handle that case and still sort? I'd want the blank or None
> fields to come out either at the beginning or end of the sorted list
> (not sure what the customer wants for this yet).
Make the key function return a tuple:
>>> def none_blank_float(value):
... if value is None: return (0,)
... if not value: return (1,)
... return 2, float(value)
...
>>> sorted(["1", "10", "2", None, "3", "", "5"], key=none_blank_float)
[None, '', '1', '2', '3', '5', '10']
--
https://mail.python.org/mailman/listinfo/python-list
Re: sorting strings numerically while dealing with missing values
On Wed, Dec 28, 2016 at 2:14 PM, Larry Martell wrote: > > I have a list containing a list of strings that I want to sort > numerically by one of the fields. I am doing this: > > sorted(rows, key=float(itemgetter(sortby))) I'm guessing that you left out a lambda here since the key argument takes a function. > Which works fine as long as all the sort keys convert to a float. > Problem is that some are blank or None and those throw an exception. > How can I handle that case and still sort? I'd want the blank or None > fields to come out either at the beginning or end of the sorted list > (not sure what the customer wants for this yet). def sort_key(sortby, none_first=False): def key(row): try: value = float(row[sortby]) except ValueError: value = None return ((value is None) != none_first, value) return key sorted(rows, key=sort_key(4, none_first=True)) -- https://mail.python.org/mailman/listinfo/python-list
Re: sorting strings numerically while dealing with missing values
On Wed, Dec 28, 2016 at 2:43 PM, Ian Kelly wrote: > On Wed, Dec 28, 2016 at 2:14 PM, Larry Martell > wrote: >> >> I have a list containing a list of strings that I want to sort >> numerically by one of the fields. I am doing this: >> >> sorted(rows, key=float(itemgetter(sortby))) > > I'm guessing that you left out a lambda here since the key argument > takes a function. > >> Which works fine as long as all the sort keys convert to a float. >> Problem is that some are blank or None and those throw an exception. >> How can I handle that case and still sort? I'd want the blank or None >> fields to come out either at the beginning or end of the sorted list >> (not sure what the customer wants for this yet). > > > def sort_key(sortby, none_first=False): > def key(row): > try: > value = float(row[sortby]) > except ValueError: > value = None > return ((value is None) != none_first, value) > return key > > sorted(rows, key=sort_key(4, none_first=True)) Actually that doesn't quite work because None < None is unorderable in Python 3. Maybe replace the inner return with: return ((value is None) != none_first, value or 0.0) -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
>> Hans-Peter Jansen wrote:
>> > Dear Peter,
>> >
>> > thanks for taking valuable time to look into my issue.
>>
>> You're welcome!
>>
>> > It might be related to my distinct silliness, but the problem persists
>> > with your code as well.
>>
>> Unfortunately I posted the broken toy example rather than the fixed one.
>> Here's the latter. Basically you have to keep a reference in the context
>> manager (whether you implement it as a class or a generator doesn't
>> matter) without giving another reference away to client code:
>>
>> $ cat mmap_after.py
>> import ctypes
>> import mmap
>> import weakref
>>
>> from contextlib import contextmanager
>>
>> class T(ctypes.Structure):
>> _fields = [("foo", ctypes.c_uint32)]
>>
>>
>> @contextmanager
>> def map_struct(m, n):
>> m.resize(n * mmap.PAGESIZE)
>> keep_me = T.from_buffer(m)
>> yield weakref.proxy(keep_me)
>
> Hooray, that did the trick. Great solution, thank you very much!
>
> If you don't mind, I will mention you and your solution at the various
> places, I placed this issue over the last weeks.
>
> You made my day, Peter!
>
> It leaves the question on why is Python2 acting as one would expect
> related to context managers, and Python3 needs this weakref juggling.
> Maybe something, that's worth to be placed in python-dev. What do you
> think?
Well, given
$ cat mmap_resize.py
import ctypes
import mmap
class T(ctypes.Structure):
_fields_ = [("foo", ctypes.c_uint32)]
SIZE = 2 * mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), SIZE)
a = T.from_buffer(m, mmap.PAGESIZE)
m.resize(mmap.PAGESIZE)
a.foo = 42
$ python3 mmap_resize.py
Traceback (most recent call last):
File "mmap_resize.py", line 15, in
m.resize(mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
$ python2.7 mmap_resize.py
Segmentation fault
do you really prefer the Python 2 behaviour?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Parse a Wireshark pcap file
> On 2016-12-27, at 20:46, [email protected] wrote: > > I have a pcap file, I want to parse that file & fetch some information like > Timestamp, Packet Size, Source/Dest IP Address, Source/Dest Port, Source/ > Dest MAC address. pcapy can do this. import pcapy pcap = pcapy.open_offline("httpsession.pcap") def callback(hdr, data): ... do something with hdr and data, which is the captured packet pcap.loop(0, callback) Greetings, -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 21:58:38 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> >> Hans-Peter Jansen wrote:
> > It leaves the question on why is Python2 acting as one would expect
> > related to context managers, and Python3 needs this weakref juggling.
> > Maybe something, that's worth to be placed in python-dev. What do you
> > think?
>
> Well, given
>
> $ cat mmap_resize.py
> import ctypes
> import mmap
>
> class T(ctypes.Structure):
> _fields_ = [("foo", ctypes.c_uint32)]
>
> SIZE = 2 * mmap.PAGESIZE
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
>
> m = mmap.mmap(f.fileno(), SIZE)
>
> a = T.from_buffer(m, mmap.PAGESIZE)
> m.resize(mmap.PAGESIZE)
> a.foo = 42
> $ python3 mmap_resize.py
> Traceback (most recent call last):
> File "mmap_resize.py", line 15, in
> m.resize(mmap.PAGESIZE)
> BufferError: mmap can't resize with extant buffers exported.
> $ python2.7 mmap_resize.py
> Segmentation fault
>
> do you really prefer the Python 2 behaviour?
Hmm, so Python2 behavior was "working by chance", or better "working without
any safety net" which isn't the Real McCoy either in that area.
Now, if only the weakref version would behave with the ctypes objects...
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: Re: Mock object bug with assert_not_called (Steve D'Aprano)
Re: mock bug.
Python 2.7.8
I'll provide a sketch of the code since it is from work and I'm not allowed
to share it directly.
Sketch of Code:
@patch('emails.emails.render_to_string')
def test_send_email(self, render):
context = {
'key': value
}
emails.send_email() # calls render_to_string
render.assert_any_call(
'template.txt',
context
)
render.assert_not_called()
Expected behavior is that one of the asserts should raise an exception.
The assert_not_called() should assert that the mocked method was not called
at all. The assert_any_call(vars) asserts there is at least one call with
the given args. If neither throws an exception then this is a
contradiction because the mocked object, render in this case, has asserted
that there are no calls and that there is at least one call.
Is this sufficient? I'm not so sure, this is my first time. Thanks for
the help.
--
Diego Vela
--
https://mail.python.org/mailman/listinfo/python-list
Re: Re: Mock object bug with assert_not_called (Steve D'Aprano)
On Thu, 29 Dec 2016 10:36 am, Diego Vela wrote:
> Re: mock bug.
>
> Python 2.7.8
>
>
> I'll provide a sketch of the code since it is from work and I'm not
> allowed to share it directly.
We don't want to see your entire code base. We want to see the smallest,
simplest example that demonstrates the problem. Something we can copy and
paste into a file and run.
> Sketch of Code:
> @patch('emails.emails.render_to_string')
> def test_send_email(self, render):
> context = {
> 'key': value
> }
> emails.send_email() # calls render_to_string
> render.assert_any_call(
> 'template.txt',
> context
> )
> render.assert_not_called()
This is, I'm afraid, useless. What is "patch"? What is "emails"? What class
does this belong to? What argument "render" should we pass? Where
does "value" come from?
It might help for you to read this:
http://sscce.org/
It is written for Java developers, but the same rules apply here. Your
example code should be
- as short and simple as you can make it (e.g. since this issue has nothing
to do with sending email, your code should not send email -- a simple print
should be enough for output);
- complete (no missing dependencies, all necessary modules are imported, no
uninitialised variables, functions called with all necessary arguments
specified);
- correct (the code runs and does what you say it does).
Thank you,
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > > Hans-Peter Jansen wrote:
> > > > Dear Peter,
> > > >
> > > > thanks for taking valuable time to look into my issue.
> > >
> > > You're welcome!
> > >
> > > @contextmanager
> > > def map_struct(m, n):
> > > m.resize(n * mmap.PAGESIZE)
> > > keep_me = T.from_buffer(m)
> > > yield weakref.proxy(keep_me)
> >
> > Hooray, that did the trick. Great solution, thank you very much!
>
> Sorry for bothering you again, Peter, but after applying it to the real
> project, that fails consistently similar to:
$ python3 mmap_test_weakref.py
Traceback (most recent call last):
File "mmap_test_weakref.py", line 32, in
assert(bytes(c) == bytes(rest))
AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'
$ cat mmap_test_weakref.py
import ctypes
import mmap
import weakref
from contextlib import contextmanager
class T(ctypes.Structure):
_fields_ = [("foo", ctypes.c_uint32)]
@contextmanager
def map_struct(m, n, struct, offset = 0):
m.resize(n * mmap.PAGESIZE)
inst = struct.from_buffer(m, offset)
yield weakref.proxy(inst)
SIZE = mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
with map_struct(m, 1, T) as a:
a.foo = 1
with map_struct(m, 2, T) as b:
b.foo = 2
offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
assert(bytes(c) == bytes(rest))
With weakref and mmap.resize() disabled, this acts as expected.
BTW: mmapped files need the first page initialized, the rest is done in the
kernel (filled with zeros on resize).
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: logger.info / logger.error with logger.addHandler - how to split streams?
Thanks On Tue, Dec 27, 2016 at 2:57 AM, gst wrote: > Le lundi 26 décembre 2016 10:34:48 UTC-5, Alec Taylor a écrit : > > So I'm putting .info in one StringIO and .error in another StringIO. > > > > How do I stop them from both being put into both? > > > > Code: http://ideone.com/Nj6Asz > > > Hi, > > it's doable with filter on the handlers: > > > def exact_filter(level): > def filter(record): > return level == record.levelno > filter.filter = filter > return filter > > stdout_stream.addFilter(exact_filter(logging.INFO)) > stderr_stream.addFilter(exact_filter(logging.ERROR)) > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
