[Python-Dev] Pickling objects that return string from reduce
I'm still plugging away on IronPython's cPickle, and I've stumbled
across another ambiguity in the pickle docs. From
http://docs.python.org/lib/node69.html:
"If a string is returned, it names a global variable whose contents are
pickled as normal. The string returned by __reduce__ should be the
object's local name relative to its module; the pickle module searches
the module namespace to determine the object's module."
What exactly does that last clause mean? Must the object have a
__module__ attribute? What if it doesn't? What if the module exists but
isn't currently imported?
Is something like the following close?
# Call copy_reg-registered func, obj.__reduce_ex__, or
obj.__reduce__
result = call_reduce(obj)
if type(result) == tuple:
... (do appropriate things here)
elif isinstance(result, basestring):
name = result
module = ""
try:
module = obj.__module__
found_obj = getattr(sys.modules[module], name)
except AttributeError, KeyError:
raise PicklingError(
"Can't pickle %r: it's not found as %s.%s"
% (obj, module, name)
)
if found_obj is not obj:
raise PicklingError(
"Can't pickle %r: it's not the same object as %s.%s"
% (obj, module, name)
)
emit("c%s\n%s\n" % module, name)
Thanks,
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickling objects that return string from reduce
Martin v. Löwis" wrote: > If obj has no __module__ attribute (or if it is None), pickle > (didn't check cPickle) also does > > for n, module in sys.module.items(): > if "module-ignored": continue > if getattr(module, result, None) is obj: > break # use n as module name > > If obj does have a __module__ attribute, it uses __import__ > to import the module, just to make sure it gets into sys.modules. What is "module-ignored" above? It's obviously not a literal string... --Bruce ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Strange memo behavior from cPickle
We seem to have stumbled upon some strange behavior in cPickle's memo
use when pickling instances.
Here's the repro:
[mymodule.py]
class C:
def __getstate__(self): return ('s1', 's2', 's3')
[interactive interpreter]
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> import mymodule
>>> class C:
... def __getstate__(self): return ('s1', 's2', 's3')
...
>>> for x in mymodule.C(), C(): cPickle.dumps(x)
...
"(imymodule\nC\np1\n(S's1'\nS's2'\np2\nS's3'\np3\ntp4\nb."
"(i__main__\nC\np1\n(S's1'\nS's2'\nS's3'\ntp2\nb."
>>>
Note that the second and third strings in the instance's state are
memoized in the first case, but not in the second. Any idea why this
occurs (and why the first element is never memoized)?
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Socket module corner cases
Hello,
I'm working on implementing a socket module for IronPython that aims to
be compatible with the standard CPython module documented at
http://docs.python.org/lib/module-socket.html. I have a few questions
about some corner cases that I've found. CPython results below are from
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
win32.
Without further ado, the questions:
* getfqdn(): The module docs specify that if no FQDN can be found,
socket.getfqdn() should return the hostname as returned by
gethostname(). However, CPython seems to return the passed-in hostname
rather than the local machine's hostname (as would be expected from
gethostname()). What's the correct behavior?
>>> s.getfqdn(' asdlfk asdfsadf ')
'asdlfk asdfsadf'
# expected 'mybox.mydomain.com'
* getfqdn(): The function seems to not always return the FQDN. For
example, if I run the following code from 'mybox.mydomain.com', I get
strange output. Does getfqdn() remove the common domain between my
hostname and the one that I'm looking up?
>>> socket.getfqdn('otherbox')
'OTHERBOX'
# expected 'otherbox.mydomain.com'
* gethostbyname_ex(): The CPython implementation doesn't seem to
respect the '' == INADDR_ANY and '' == INADDR_BROADCAST
forms. '' is treated as localhost, and '' raises a "host not
found" error. Is this intentional? A quick check seems to reveal that
gethostbyname() is the only function that respects '' and ''.
Are the docs or the implementation wrong?
* getprotobyname(): Only a few protocols seem to be supported. Why?
>>> for p in [a[8:] for a in dir(socket) if a.startswith('IPPROTO_')]:
... try:
... print p,
... print socket.getprotobyname(p)
... except socket.error:
... print "(not handled)"
...
AH (not handled)
DSTOPTS (not handled)
ESP (not handled)
FRAGMENT (not handled)
GGP 3
HOPOPTS (not handled)
ICMP 1
ICMPV6 (not handled)
IDP (not handled)
IGMP (not handled)
IP 0
IPV4 (not handled)
IPV6 (not handled)
MAX (not handled)
ND (not handled)
NONE (not handled)
PUP 12
RAW (not handled)
ROUTING (not handled)
TCP 6
UDP 17
Thanks for your help!
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket module corner cases
> > * getfqdn(): The function seems to not always return the FQDN. For
> >example, if I run the following code from 'mybox.mydomain.com', I get
> >strange output. Does getfqdn() remove the common domain between my
> >hostname and the one that I'm looking up?
> socket.getfqdn('otherbox')
> >'OTHERBOX'
> ># expected 'otherbox.mydomain.com'
>
> getfqdn() calls the system library gethostbyaddr(), and searches the
> result for the first name that contains '.' or if no name contains
dot, it
> returns the canonical name (as defined by gethostbyaddr() and the
system
> host name resolver libraries (hosts file, DNS, NMB)).
After a little more investigation here, it appears that getfqdn()
returns the
name unchanged (or perhaps run through the system's resolver libs) if
there's no reverse DNS PTR entry. In the case given above,
otherbox.mydomain.com didn't have a reverse DNS entry, so getfqdn()
returned
'OTHERBOX'. However, when getfqdn() is called with a name whose IP
*does*
have a PTR record, it returns the correct FQDN.
Thanks for the help. Now, couple more questions:
getnameinfo() accepts the NI_NAMEREQD flag. It appears, though that a
name
lookup (and associated error if the lookup fails) occurs independent of
whether the flag is specified. Does it actually do anything?
Does getnameinfo() support IPv6? It appears to fail (with a socket.error
that says "sockaddr resolved to multiple addresses") if both IPv4 and
IPv6
are enabled.
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] socket._socketobject.close() doesn't really close sockets
In implementing the IronPython _socket module, I've discovered some puzzling behavior in the standard Python socket wrapper module: socket._socketobject.close() doesn't actually close sockets; rather, it just sets _sock to an instance of _closedsocket and lets the GC clean up the real socket. (See socket.py around line 160.) This works fine with a reference counting GC, but can potentially leave sockets hanging around for a long time on platforms (e.g. the CLR) with other GC algorithms. It causes most of the socket unit tests to fail on IronPython. Is there a reason for this implementation? This patch to _socketobject.close() makes socket.py work with IronPython: def close(self): +if not isinstance(self._sock, _closedsocket): +self._sock.close() self._sock = _closedsocket() self.send = self.recv = self.sendto = self.recvfrom = self._sock._dummy --Bruce ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pickle implementation questions
In developing a cPickle module for IronPython that's as compatible as possible with CPython, these questions have come up: - Where are object.__reduce__ and object.__reduce_ex__ defined, and how does copy_reg._reduce_ex fit into the picture? PEP 307 states that the default __reduce__ implementation for new-style classes implemented in Python is copy_reg._reduce. However, in Python 2.4.3 dir(copy_reg) indicates that it has no _reduce method. (Tangentially, is there a way to inspect a method_descriptor object to determine the function it's bound to?) - When the optional constructor argument is passed to copy_reg.pickle, where is it stored and how is it used by pickle? - What does copy_reg.constructor() do? Thanks! --Bruce ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle implementation questions
Thanks for your responses, Martin!
Martin v. Löwis wrote:
> Bruce Christensen wrote:
> > - Where are object.__reduce__ and object.__reduce_ex__ defined, and how
> > does copy_reg._reduce_ex fit into the picture?
>
> See
>
> http://docs.python.org/lib/node69.html
So just to be clear, is it something like this?
class object:
def __reduce__(self):
return copy_reg._reduce_ex(self, -1)
def __reduce_ex__(self, protocol):
return copy_reg._reduce_ex(self, protocol)
Does _reduce_ex's behavior actually change depending on the specified protocol
version? The only difference that I can see or think of is that an assert
causes it to fail if the protocol is >= 2.
> > - What does copy_reg.constructor() do?
>
> It does this:
>
> def constructor(object):
> if not callable(object):
> raise TypeError("constructors must be callable")
So it is part of the public interface? It's exported in __all__, but it appears
that it's undocumented.
Thanks,
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle implementation questions
Fredrik Lundh wrote: > on the other hand, it would be nice if someone actually used Bruce's questions > and the clarifications to update the documentation; the ideas behind the > internal pickle interfaces aren't exactly obvious, even if you have the > source. I've found a few other places where the docs are misleading at best, and nonexistent or simply wrong at worst. I've been able to figure out most things by reverse-engineering pickle's behavior, but that's often a slow process. If anyone is interested, I'd be happy to compile a list of places the docs could be improved. --Bruce ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle implementation questions
Tim Peters wrote:
> I hope you've read PEP 307:
I have. Thanks to you and Guido for writing it! It's been a huge help.
> The implementation is more like:
[snip]
Thanks! That helps a lot. PEP 307 and the pickle module docs describe the end
result pretty well, but they don't always make it clear where things are
implemented. I'm trying to make sure that I'm getting the right interaction
between object.__reduce(_ex)__, pickle, and copy_reg..
One (hopefully) last question: is object.__reduce(_ex)__ really implemented in
object? The tracebacks below would indicate that pickle directly implements the
behavior that the specs say is implemented in object. However, that could be
because frames from C code don't show up in tracebacks. I'm not familiar enough
with CPython to know for sure.
>>> import copy_reg
>>> def bomb(*args, **kwargs):
... raise Exception('KABOOM! %r %r' % (args, kwargs))
...
>>> copy_reg._reduce_ex = bomb
>>> import pickle
>>> pickle.dumps(object())
Traceback (most recent call last):
File "", line 1, in ?
File "C:\Python24\lib\pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "C:\Python24\lib\pickle.py", line 231, in dump
self.save(obj)
File "C:\Python24\lib\pickle.py", line 313, in save
rv = reduce(self.proto)
File "", line 2, in bomb
Exception: KABOOM! (, 0) {}
>>> class NewObj(object):
... def __reduce__(self):
... raise Exception("reducing NewObj")
...
>>> import pickle
>>> pickle.dumps(NewObj())
Traceback (most recent call last):
File "", line 1, in ?
File "C:\Python24\lib\pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "C:\Python24\lib\pickle.py", line 231, in dump
self.save(obj)
File "C:\Python24\lib\pickle.py", line 313, in save
rv = reduce(self.proto)
File "", line 3, in __reduce__
Exception: reducing NewObj
> It's documented in the Library Reference Manual, in the `copy_reg` docs:
Oops. :)
Again, thanks for the help.
--Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
