Re: dict.get_deep()
Hi Marco.
Recently I met same issue. A service I intergated with was documented
badly and sent ... unpredictable jsons.
And pattern matching helped me in first solution. (later I switched to
Pydantic models)
For your example I'd make match rule for key path you need. For example:
data = {"users": [{"address": {"street": "Baker"}}]}
match data:
case {"users": [{"address": {"street": street}}]}:
print(f"street: {street}")
case _:
print("unsupported message structure")
Structural matching gives you warranty you process exactly message you
expect and explicitly discards messages with another structure.
But type is still issue. I don't know how to say 'street' must be 'str'
not 'int'. That's why I switched to Pydantic.
02.04.2022 23:44, Marco Sulla пишет:
A proposal. Very often dict are used as a deeply nested carrier of
data, usually decoded from JSON. Sometimes I needed to get some of
this data, something like this:
data["users"][0]["address"]["street"]
What about something like this instead?
data.get_deep("users", 0, "address", "street")
and also, instead of this
try:
result = data["users"][0]["address"]["street"]
except KeyError, IndexError:
result = "second star"
write this:
data.get_deep("users", 0, "address", "street", default="second star")
?
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
To my previous post.
It seems 'case if' should help with types:
case {"users": [{"address": {"street": street}}]} if isinstance(street,
str):
:)
// BR
02.04.2022 23:44, Marco Sulla пишет:
A proposal. Very often dict are used as a deeply nested carrier of
data, usually decoded from JSON. Sometimes I needed to get some of
this data, something like this:
data["users"][0]["address"]["street"]
What about something like this instead?
data.get_deep("users", 0, "address", "street")
and also, instead of this
try:
result = data["users"][0]["address"]["street"]
except KeyError, IndexError:
result = "second star"
write this:
data.get_deep("users", 0, "address", "street", default="second star")
?
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On Mon, 4 Apr 2022 at 00:59, Kirill Ratkin via Python-list
wrote:
>
> Hi Marco.
>
> Recently I met same issue. A service I intergated with was documented
> badly and sent ... unpredictable jsons.
>
> And pattern matching helped me in first solution. (later I switched to
> Pydantic models)
>
> For your example I'd make match rule for key path you need. For example:
>
>
> data = {"users": [{"address": {"street": "Baker"}}]}
>
> match data:
> case {"users": [{"address": {"street": street}}]}:
> print(f"street: {street}")
>
> case _:
> print("unsupported message structure")
>
>
> Structural matching gives you warranty you process exactly message you
> expect and explicitly discards messages with another structure.
>
> But type is still issue. I don't know how to say 'street' must be 'str'
> not 'int'. That's why I switched to Pydantic.
>
You can use type names to require that it be that type:
case {"users": [{"address": {"street": str(street)}}]}:
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On 2022-04-03 at 18:01:58 +0300,
Kirill Ratkin via Python-list wrote:
> It seems 'case if' should help with types:
>
> case {"users": [{"address": {"street": street}}]} if isinstance(street,
> str):
reduce(lambda x, y: x[y], ["users", 0, "address", "street"], data)
Unless it's y[x] rather than x[y], and there might be a More Cleverer™
way to do it with getattr or one of its relatives instead of lambda.
> 02.04.2022 23:44, Marco Sulla пишет:
> > A proposal. Very often dict are used as a deeply nested carrier of
> > data, usually decoded from JSON. Sometimes I needed to get some of
> > this data, something like this:
> >
> > data["users"][0]["address"]["street"]
> >
> > What about something like this instead?
> >
> > data.get_deep("users", 0, "address", "street")
> >
> > and also, instead of this
> >
> > try:
> > result = data["users"][0]["address"]["street"]
> > except KeyError, IndexError:
> > result = "second star"
> >
> > write this:
> >
> > data.get_deep("users", 0, "address", "street", default="second star")
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
Marco Sulla wrote at 2022-4-2 22:44 +0200:
>A proposal. Very often dict are used as a deeply nested carrier of
>data, usually decoded from JSON. Sometimes I needed to get some of
>this data, something like this:
>
>data["users"][0]["address"]["street"]
>
>What about something like this instead?
>
>data.get_deep("users", 0, "address", "street")
>
>and also, instead of this
>
>try:
>result = data["users"][0]["address"]["street"]
>except KeyError, IndexError:
>result = "second star"
>
>write this:
>
>data.get_deep("users", 0, "address", "street", default="second star")
You know you can easily implement this yourself -- in your own
`dict` subclass.
You can also customize the JSON decoding (--> `object_hook`)
to determine how a JSON object is mapped to a Python object;
it defaults to `dict` but you could use your own `dict` subclass.
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On Sun, 3 Apr 2022 at 16:59, Kirill Ratkin via Python-list
wrote:
>
> Hi Marco.
>
> Recently I met same issue. A service I intergated with was documented
> badly and sent ... unpredictable jsons.
>
> And pattern matching helped me in first solution. (later I switched to
> Pydantic models)
>
> For your example I'd make match rule for key path you need. For example:
>
>
> data = {"users": [{"address": {"street": "Baker"}}]}
>
> match data:
> case {"users": [{"address": {"street": street}}]}:
> print(f"street: {street}")
>
> case _:
> print("unsupported message structure")
Hi. I think your solution is very brilliant, but I'm a bit allergic to
pattern matching... :D Maybe it's me, but I found it really strange
and "magical".
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On Sun, 3 Apr 2022 at 18:57, Dieter Maurer wrote: > You know you can easily implement this yourself -- in your own > `dict` subclass. Well, of course, but the question is if such a method is worth to be builtin, in a world imbued with JSON. I suppose your answer is no. -- https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On 2022-04-03 17:58:09 +0300, Kirill Ratkin via Python-list wrote:
> 02.04.2022 23:44, Marco Sulla пишет:
> > A proposal. Very often dict are used as a deeply nested carrier of
> > data, usually decoded from JSON. Sometimes I needed to get some of
> > this data, something like this:
> >
> > data["users"][0]["address"]["street"]
> >
> > What about something like this instead?
> >
> > data.get_deep("users", 0, "address", "street")
Yup. I need something like this quite frequently, so I wrote a little
utility function (which I copy and paste into lots of code - I probably
should package that up, but a package with a single short function feels
weird).
[...]
> > data.get_deep("users", 0, "address", "street", default="second star")
Yep. Did that, too. Plus pass the final result through a function before
returning it.
I'm not sure whether I considered this when I wrote it, but a function
has the advantage of working with every class which can be indexed. A
method must be implemented on any class (so at least dict and list to be
useful).
> Recently I met same issue. A service I intergated with was documented badly
> and sent ... unpredictable jsons.
>
> And pattern matching helped me in first solution. (later I switched to
> Pydantic models)
>
> For your example I'd make match rule for key path you need. For example:
>
>
> data = {"users": [{"address": {"street": "Baker"}}]}
>
> match data:
> case {"users": [{"address": {"street": street}}]}:
> print(f"street: {street}")
>
> case _:
> print("unsupported message structure")
Neat. But that's 5 lines instead of one. I simple loop around try/except
also takes 5 lines, and the latter can be easily moved into a function,
like this:
def get_nested(coll, path, default=None, cast=None):
for i in path:
try:
coll = coll[i]
except (KeyError, IndexError, TypeError):
return default
if cast:
coll = cast(coll)
return coll
which can then be called in a single line.
> Structural matching gives you warranty you process exactly message you
> expect and explicitly discards messages with another structure.
True, and sometimes useful, I'm sure. Not sure whether it would have
helped me in the cases where I used the utility function above.
hp
--
_ | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| | | [email protected] |-- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
signature.asc
Description: PGP signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sharing part of a function
"Memoising" is the answer -- see "Python Algorithms" by Magnus Lie Hetland.
In the mean time, here a simplified version of "memoising" using a dict.
This version handles pretty large fibonacci numbers!
# fibonacci sequence
# memoised - but using a simple dictionary (see Python Algorithms, p177)
memo= {}
memo[1] = 1
memo[2] = 2
def fib(n):
if n in memo:
return memo[n]
else:
memo[n] = fib(n - 1) + fib(n - 2)
return memo[n]
print(100, fib(100))
print(memo)
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On Sun, 3 Apr 2022 at 21:46, Peter J. Holzer wrote:
>
> > > data.get_deep("users", 0, "address", "street", default="second star")
>
> Yep. Did that, too. Plus pass the final result through a function before
> returning it.
I didn't understand. Have you added a func parameter?
> I'm not sure whether I considered this when I wrote it, but a function
> has the advantage of working with every class which can be indexed. A
> method must be implemented on any class (so at least dict and list to be
> useful).
You're right, but where to put it? I don't know if an iterableutil package
exists. If included in the stdlib, I don't know where to put it. In
collections maybe?
PS: if you're interested, here is my implementation:
def get_deep(self, *args, default=_sentinel):
r"""
Get a nested element of the dictionary.
The method accepts multiple arguments or a single one. If a single
argument is passed, it must be an iterable. This represents the
keys or indexes of the nested element.
The method first tries to get the value v1 of the dict using the
first key. If it finds v1 and there's no other key, v1 is
returned. Otherwise, the method tries to retrieve the value from v1
associated with the second key/index, and so on.
If in any point, for any reason, the value can't be retrieved, the
`default` parameter is returned if specified. Otherwise, a
KeyError or an IndexError is raised.
"""
if len(args) == 1:
single = True
it_tpm = args[0]
try:
len(it_tpm)
it = it_tpm
except Exception:
# maybe it's a generator
try:
it = tuple(it_tpm)
except Exception:
err = (
f"`{self.get_deep.__name__}` called with a single " +
"argument supports only iterables"
)
raise TypeError(err) from None
else:
it = args
single = False
if not it:
if single:
raise ValueError(
f"`{self.get_deep.__name__}` argument is empty"
)
else:
raise TypeError(
f"`{self.get_deep.__name__}` expects at least one argument"
)
obj = self
for k in it:
try:
obj = obj[k]
except (KeyError, IndexError) as e:
if default is _sentinel:
raise e from None
return default
return obj
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sharing part of a function
Betty Hollinshead writes:
> "Memoising" is the answer -- see "Python Algorithms" by Magnus Lie Hetland.
> In the mean time, here a simplified version of "memoising" using a dict.
> This version handles pretty large fibonacci numbers!
>
> # fibonacci sequence
> # memoised - but using a simple dictionary (see Python Algorithms, p177)
>
> memo= {}
> memo[1] = 1
> memo[2] = 2
>
> def fib(n):
> if n in memo:
> return memo[n]
> else:
> memo[n] = fib(n - 1) + fib(n - 2)
> return memo[n]
>
>
> print(100, fib(100))
> print(memo)
No, it is not. It is the answer on a completely different question.
Nice, but not what I was asking about. And there is even an error in
the solution.
By the way: it is better to do it iterative. Try (when not done a
calculation before) fib(3_000).
--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
On Mon, 4 Apr 2022 at 05:19, Marco Sulla wrote:
>
> On Sun, 3 Apr 2022 at 18:57, Dieter Maurer wrote:
> > You know you can easily implement this yourself -- in your own
> > `dict` subclass.
>
> Well, of course, but the question is if such a method is worth to be
> builtin, in a world imbued with JSON. I suppose your answer is no.
I think not, for a slightly different reason: There are many small
variants of it that might be needed. For instance, I have a system of
persistent data that is backed by a JSON file, and I have a path
function that does this:
def path(self, *parts):
ret = self.data # always a dict
for part in parts:
if part not in ret:
ret[part] = {}
self.save() # queue a save for the next idle time
ret = ret[part]
return ret
It's used something like this:
cfg = persist.path("channels", channame, "alerts", "host")
and if there had previously been no configs, you'll get back an empty
dict, not None, not an error, etc.
This version uses dictionaries only. A version that also supports
lists would require one more check before the "part not in ret" check
(probably something like "if this is a list, and you're asking for an
index higher than the current one, append elements till we get
there"), and a version that doesn't autosave wouldn't need the save()
call, etc, etc, etc.
IMO this is an extremely useful tool to have in your own toolbelt, but
I actually don't use *identical* versions of it very often. They're
usually subtly different from each other, due to the slight
differences in usage.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
Marco Sulla wrote at 2022-4-3 21:17 +0200: >On Sun, 3 Apr 2022 at 18:57, Dieter Maurer wrote: >> You know you can easily implement this yourself -- in your own >> `dict` subclass. > >Well, of course, but the question is if such a method is worth to be >builtin, in a world imbued with JSON. I suppose your answer is no. ``` def mget(m, *keys): """return m[k1][k2]...[kn] or `None`""" for k in keys(): if k not in m: return m = m[k] return m ``` Thus, it is so simple to get what you want. No need to make it builtin. -- Dieter -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing part of a function
On Sun, Apr 3, 2022 at 2:46 PM Cecil Westerhof via Python-list < [email protected]> wrote: > Betty Hollinshead writes: > > > "Memoising" is the answer -- see "Python Algorithms" by Magnus Lie > Hetland. > > In the mean time, here a simplified version of "memoising" using a dict. > > This version handles pretty large fibonacci numbers! > > > > # fibonacci sequence > > # memoised - but using a simple dictionary (see Python Algorithms, p177) > > > > memo= {} > > memo[1] = 1 > > memo[2] = 2 > > > > def fib(n): > > if n in memo: > > return memo[n] > > else: > > memo[n] = fib(n - 1) + fib(n - 2) > > return memo[n] > > > > > > print(100, fib(100)) > > print(memo) > > No, it is not. It is the answer on a completely different question. > Nice, but not what I was asking about. And there is even an error in > the solution. > > By the way: it is better to do it iterative. Try (when not done a > calculation before) fib(3_000). > I think I missed part of this conversation, but here is how I've done fibonacci numbers in the past, using functools.lru_cache: #!/usr/local/cpython-3.8/bin/python """Compute the first n numbers in the fibonacci sequence.""" import functools import sys @functools.lru_cache(maxsize=None) # pylint: disable=no-member def find_recursive(number): """Find a Fibonacci number recursively - without the callstack explosion.""" assert number >= 0 if number == 0: return 0 if number == 1: return 1 result = find_recursive(number - 1) + find_recursive(number - 2) return result def main(): """Compute fibonacci numbers.""" top = 5000 if sys.argv[1:]: top = int(sys.argv[1]) if sys.argv[2:]: sys.stderr.write('Usage: {} 5000\n'.format(sys.argv[0])) sys.exit(1) for number in range(1, top + 1): print(number, find_recursive(number)) main() -- https://mail.python.org/mailman/listinfo/python-list
Re: dict.get_deep()
I may have misunderstood something.
The original post in this subject sounded to ME likethey had nested
dictionaries and wanted to be ableto ask a method in the first dictionary
totake an unspecified number of arguments thatwould be successive keys and
return the results.
I mean if A was a dictionary containing saycities and it had an alphabetical
index of lettersA to Z and those contained dictionaries ofsay last names as
additional dictionaries andso on, then they wanted to perhaps say;
A.getdeep("Boston", "S", "Smith", "Address", default="None")
But the replies I am seeing look so different that I mayhave missed something
as it seems more about usingpattern matching on the data used to make the
dictionariesor something.
So I was happy to see Marco suggesting a function alongthe lines of my thought
process. But I have another thought.A stand-alone function along his lines
might be fine. Buta method built into a general Dictionary class is
anotherthing as it asks a method in one dictionary to march aroundinto other
dictionaries. So I wonder if a better methodis sort of recursive.
If you had a class like dictionary that had a getdeep function,and it got
called with N arguments, and perhaps a namedargument supplying a default, then
would it make sensefor the function checking to see if the FIRST argument canbe
found as a key to the current dictionary.
If arguments remain then it should expect to finda result that is a dictionary
(or perhaps some otherobject that supports the getdeep() protocol and ask
thatobject to operate on the N-1 remaining arguments, passingthe default along
too.
If the request is valid, after some iterations an object willhave a method
invoked with a single argument (plus default)and a value passed back up the
chain. For any errors alongthe way, the default would be returned.
Is this closer to the spirit of the request? I view this versionof nested
dictionaries as a sort of tree structure with variablebranches along the way.
So an approach like this could makesense and perhaps Python could be updated
eventually to havesome objects support such a protocol.
Of course you could sort of do it yourself by subclassing somethingand making
changes but that may not work for what is already asort of built-in data
structure but could work for one of many variantsalready implemented in modules.
-Original Message-
From: Marco Sulla
To: Peter J. Holzer
Cc: [email protected]
Sent: Sun, Apr 3, 2022 5:17 pm
Subject: Re: dict.get_deep()
On Sun, 3 Apr 2022 at 21:46, Peter J. Holzer wrote:
>
> > > data.get_deep("users", 0, "address", "street", default="second star")
>
> Yep. Did that, too. Plus pass the final result through a function before
> returning it.
I didn't understand. Have you added a func parameter?
> I'm not sure whether I considered this when I wrote it, but a function
> has the advantage of working with every class which can be indexed. A
> method must be implemented on any class (so at least dict and list to be
> useful).
You're right, but where to put it? I don't know if an iterableutil package
exists. If included in the stdlib, I don't know where to put it. In
collections maybe?
PS: if you're interested, here is my implementation:
def get_deep(self, *args, default=_sentinel):
r"""
Get a nested element of the dictionary.
The method accepts multiple arguments or a single one. If a single
argument is passed, it must be an iterable. This represents the
keys or indexes of the nested element.
The method first tries to get the value v1 of the dict using the
first key. If it finds v1 and there's no other key, v1 is
returned. Otherwise, the method tries to retrieve the value from v1
associated with the second key/index, and so on.
If in any point, for any reason, the value can't be retrieved, the
`default` parameter is returned if specified. Otherwise, a
KeyError or an IndexError is raised.
"""
if len(args) == 1:
single = True
it_tpm = args[0]
try:
len(it_tpm)
it = it_tpm
except Exception:
# maybe it's a generator
try:
it = tuple(it_tpm)
except Exception:
err = (
f"`{self.get_deep.__name__}` called with a single " +
"argument supports only iterables"
)
raise TypeError(err) from None
else:
it = args
single = False
if not it:
if single:
raise ValueError(
f"`{self.get_deep.__name__}` argument is empty"
)
else:
raise TypeError(
f"`{self.get_deep.__name__}` expects at least one argument"
)
obj = self
for k in it:
try:
obj = obj[k]
except (KeyError, IndexError) as e:
if default is _sentinel:
raise e from None
