Re: memory consumption
понедельник, 29 марта 2021 г. в 19:37:03 UTC+3, Dieter Maurer: > Alexey wrote at 2021-3-29 06:26 -0700: > >понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña: > >> It looks like the problem is on celery. > >> The mentioned issue is still open, so not sure if it was corrected. > >> > >> https://manhtai.github.io/posts/memory-leak-in-celery/ > > > >As I mentioned in my first message, I tried to run > >this task(class) via Flask API calls, without Celery. > >And results are the same. Flask worker receives the API call and > >executes MyClass().run() inside of view. After a few calls > >worker size increases to 1Gb of RAM. In production I have 8 workers, > > so in idle they will hold 8Gb. > Depending on your system (this works for `glibc` systems), > you can instruct the memory management via the envvar > `MALLOC_ARENA_MAX` to use a common memory pool (called "arena") > for all threads. > It is known that this can drastically reduce memory consumption > in multi thread systems. Tried with this variable. No luck. Thanks anyway. -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
понедельник, 29 марта 2021 г. в 19:56:52 UTC+3, Stestagg: > > > 2. Can you try a test with 16 or 32 active workers (i.e. number of > > > workers=2x available memory in GB), do they all still end up with 1gb > > > usage? or do you get any other memory-related issues running this? > > Yes. They will consume 1Gb each. It doesn't matter how many workers I > > have, > > they behave exactly the same. We can even forget about Flask and Celery. > > If I run this code in Python console, behavior will remain the same. > > > > > Woah, funky, so you got to a situation where your workers were allocating > 2x more ram than your system had available? and they were still working? > Were you hitting lots of swap? > > If no big swap load, then it looks like no problem, it's just that the > metrics you're looking at aren't saying what they appear to be. > > > > > -- > > https://mail.python.org/mailman/listinfo/python-list > > I'm sorry. I didn't understand your question right. If I have 4 workers, they require 4Gb in idle state and some extra memory when they execute other tasks. If I increase workers count up to 16, they`ll eat all the memory I have (16GB) on my machine and will crash as soon as system get swapped. -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
> I'm sorry. I didn't understand your question right. If I have 4 workers, > they require 4Gb > in idle state and some extra memory when they execute other tasks. If I > increase workers > count up to 16, they`ll eat all the memory I have (16GB) on my machine and > will crash as soon > as system get swapped. > -- > It was my fault sorry, bad communication at my end. I was just trying to check to see that you had actually tested that running 16 does actually crash in your environment. Steve > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Canonical conversion of dict of dicts to list of dicts
Hi,
If I have dict of dicts, say
dod = {
"alice":
{
"lang": "python",
"level": "expert"
},
"bob":
{
"lang": "perl",
"level": "noob"
}
}
is there a canonical, or more pythonic, way of converting the outer key
to a value to get a list of dicts, e.g
lod = [
{
"name": "alice",
"lang": "python",
"level": "expert"
},
{
"name": "bob",
"lang": "perl",
"level": "noob"
}
]
than just
lod = []
for name in dod:
d = dod[name]
d["name"] = name
lod.append(d)
?
Cheers,
Loris
--
This signature is currently under construction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
I'm not certain this is the clearest possible code pattern to use, but
depending on the structure of your larger code, it's possible to do this,
and the compactness may help with understandability (that's a judgement
call!):
[dict(d, name=n) for n, d in dod.items()]
On Tue, Mar 30, 2021 at 12:42 PM Loris Bennett
wrote:
> Hi,
>
> If I have dict of dicts, say
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> is there a canonical, or more pythonic, way of converting the outer key
> to a value to get a list of dicts, e.g
>
> lod = [
> {
> "name": "alice",
> "lang": "python",
> "level": "expert"
> },
> {
> "name": "bob",
> "lang": "perl",
> "level": "noob"
> }
> ]
>
> than just
>
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>
> ?
>
> Cheers,
>
> Loris
>
> --
> This signature is currently under construction.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On Tue, Mar 30, 2021 at 10:41 PM Loris Bennett
wrote:
>
> Hi,
>
> If I have dict of dicts, say
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> is there a canonical, or more pythonic, way of converting the outer key
> to a value to get a list of dicts, e.g
>
> lod = [
> {
> "name": "alice",
> "lang": "python",
> "level": "expert"
> },
> {
> "name": "bob",
> "lang": "perl",
> "level": "noob"
> }
> ]
>
> than just
>
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>
> ?
>
I dunno about "canonical", but here's how I'd do it:
lod = [info | {"name": name} for name, info in dod.items()]
You could use {"name":name}|info instead if you prefer to have the
name show up first in the dictionary.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On 2021-03-30, Loris Bennett wrote:
> If I have dict of dicts, say
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> is there a canonical, or more pythonic, way of converting the outer key
> to a value to get a list of dicts, e.g
>
> lod = [
> {
> "name": "alice",
> "lang": "python",
> "level": "expert"
> },
> {
> "name": "bob",
> "lang": "perl",
> "level": "noob"
> }
> ]
>
> than just
>
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>
> ?
There can't be a "canonical" way to perform the arbitrary data
conversion you want, because it's arbitrary. Personally I would
do this:
[dict(data, name=name) for name, data in dod.items()]
but it's of course arguable whether this is "more Pythonic" or
indeed "better".
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On 2021-03-30, Chris Angelico wrote:
> I dunno about "canonical", but here's how I'd do it:
>
> lod = [info | {"name": name} for name, info in dod.items()]
>
> You could use {"name":name}|info instead if you prefer to have the
> name show up first in the dictionary.
It's probably worth noting this method requires Python 3.9.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On Tue, Mar 30, 2021 at 11:01 PM Jon Ribbens via Python-list
wrote:
>
> On 2021-03-30, Chris Angelico wrote:
> > I dunno about "canonical", but here's how I'd do it:
> >
> > lod = [info | {"name": name} for name, info in dod.items()]
> >
> > You could use {"name":name}|info instead if you prefer to have the
> > name show up first in the dictionary.
>
> It's probably worth noting this method requires Python 3.9.
True, and if you need 3.8 support, then the dict constructor with one
kwarg is the way to do it. But this way has the flexibility that you
can choose which way to resolve conflicts (if there's a name inside
the info dict, should it override the key, or not?).
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
For completeness, from 3.5 onwards, you can also do the following:
[{'name': n, **d} for n, d in dod.items()]
On Tue, Mar 30, 2021 at 1:06 PM Chris Angelico wrote:
> On Tue, Mar 30, 2021 at 11:01 PM Jon Ribbens via Python-list
> wrote:
> >
> > On 2021-03-30, Chris Angelico wrote:
> > > I dunno about "canonical", but here's how I'd do it:
> > >
> > > lod = [info | {"name": name} for name, info in dod.items()]
> > >
> > > You could use {"name":name}|info instead if you prefer to have the
> > > name show up first in the dictionary.
> >
> > It's probably worth noting this method requires Python 3.9.
>
> True, and if you need 3.8 support, then the dict constructor with one
> kwarg is the way to do it. But this way has the flexibility that you
> can choose which way to resolve conflicts (if there's a name inside
> the info dict, should it override the key, or not?).
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On 2021-03-30, Chris Angelico wrote:
> On Tue, Mar 30, 2021 at 11:01 PM Jon Ribbens via Python-list
> wrote:
>>
>> On 2021-03-30, Chris Angelico wrote:
>> > I dunno about "canonical", but here's how I'd do it:
>> >
>> > lod = [info | {"name": name} for name, info in dod.items()]
>> >
>> > You could use {"name":name}|info instead if you prefer to have the
>> > name show up first in the dictionary.
>>
>> It's probably worth noting this method requires Python 3.9.
>
> True, and if you need 3.8 support, then the dict constructor with one
> kwarg is the way to do it. But this way has the flexibility that you
> can choose which way to resolve conflicts (if there's a name inside
> the info dict, should it override the key, or not?).
Python 3 point, er, 8, yeah, that's the only other version people might
be using...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On Tue, Mar 30, 2021 at 11:21 PM Jon Ribbens via Python-list
wrote:
>
> On 2021-03-30, Chris Angelico wrote:
> > On Tue, Mar 30, 2021 at 11:01 PM Jon Ribbens via Python-list
> > wrote:
> >>
> >> On 2021-03-30, Chris Angelico wrote:
> >> > I dunno about "canonical", but here's how I'd do it:
> >> >
> >> > lod = [info | {"name": name} for name, info in dod.items()]
> >> >
> >> > You could use {"name":name}|info instead if you prefer to have the
> >> > name show up first in the dictionary.
> >>
> >> It's probably worth noting this method requires Python 3.9.
> >
> > True, and if you need 3.8 support, then the dict constructor with one
> > kwarg is the way to do it. But this way has the flexibility that you
> > can choose which way to resolve conflicts (if there's a name inside
> > the info dict, should it override the key, or not?).
>
> Python 3 point, er, 8, yeah, that's the only other version people might
> be using...
>
>
Heh. I meant "3.8 or earlier" but abbreviated it a bit too much :D
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
Jon Ribbens writes:
> On 2021-03-30, Loris Bennett wrote:
>> If I have dict of dicts, say
>>
>> dod = {
>> "alice":
>> {
>> "lang": "python",
>> "level": "expert"
>> },
>> "bob":
>> {
>> "lang": "perl",
>> "level": "noob"
>> }
>> }
>>
>> is there a canonical, or more pythonic, way of converting the outer key
>> to a value to get a list of dicts, e.g
>>
>> lod = [
>> {
>> "name": "alice",
>> "lang": "python",
>> "level": "expert"
>> },
>> {
>> "name": "bob",
>> "lang": "perl",
>> "level": "noob"
>> }
>> ]
>>
>> than just
>>
>> lod = []
>> for name in dod:
>> d = dod[name]
>> d["name"] = name
>> lod.append(d)
>>
>> ?
>
> There can't be a "canonical" way to perform the arbitrary data
> conversion you want, because it's arbitrary. Personally I would
> do this:
>
> [dict(data, name=name) for name, data in dod.items()]
>
> but it's of course arguable whether this is "more Pythonic" or
> indeed "better".
As I am stuck with Python 3.6 I'll go with this.
I'm not quite sure what you mean by "arbitrary". I meant "canonical" in
the sense of "standard" or "established", which it seems to me could
apply to a fairly generic conversion such as DoD -> LoD as described
above.
Thanks to you and the other for the help.
Cheers,
Loris
--
This signature is currently under construction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
> On Mar 30, 2021, at 12:11, Stestagg wrote:
>
> For completeness, from 3.5 onwards, you can also do the following:
>
> [{'name': n, **d} for n, d in dod.items()]
>
Reading through these, personally I like this one best. I'm curious what about
it was enabled in 3.5? Was **kwarg expansion inside a dict literal not possible
before then? Anyway, I like that it uses simple elemental parts that have been
around a long long time in Python.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On Wed, Mar 31, 2021 at 1:56 AM Travis Griggs wrote:
>
>
> > On Mar 30, 2021, at 12:11, Stestagg wrote:
> >
> > For completeness, from 3.5 onwards, you can also do the following:
> >
> > [{'name': n, **d} for n, d in dod.items()]
> >
>
> Reading through these, personally I like this one best. I'm curious what
> about it was enabled in 3.5? Was **kwarg expansion inside a dict literal not
> possible before then? Anyway, I like that it uses simple elemental parts that
> have been around a long long time in Python.
>
Indeed:
https://www.python.org/dev/peps/pep-0448/
Some other relevant enhancements include dictionaries becoming ordered
(3.7), and gaining the ability to be merged with the | (bitwise-or)
operator (3.9). I'd generally recommend targeting 3.7 if possible,
although often things won't be horribly broken without dict ordering.
There aren't many platforms still around that have anything 3.3 or
older, and only a few that ship 3.4, so it's not too hard to demand
3.5+, but if 3.4 support matters, the only option is the dict
constructor.
Be aware, btw, that none of these options is quite the same as the
original code posted. They all construct a *new* dictionary that has
all the same contents, plus the name; the original would mutate the
*existing* dictionary, so if you look at the original lod, it would
now have names all through it. (On the other hand, this means
incurring the cost of constructing a new dict, although that shouldn't
be too expensive.)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Code Formatter Questions
On 3/29/2021 4:18 PM, dn via Python-list wrote: Very good point: I'd much rather you spent time helping me with a design/coding problem, helping debug, and/or reviewing/improving my code (and I for you); than we had not time left-over after spending many hours and much mental energy arguing about whether this format is [more] right than that! I am currently the main IDLE maintainer and spend essentially no time arguing over format. If a contribution is within the bounderies of PEP 8, that is good enough for me. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On 29/03/2021 11:12, Alexey wrote: > Hello everyone! > I'm experiencing problems with memory consumption. > The first thing you really need to tell us is which OS you are using? Memory management varies wildly depending on OS. Even different flavours of *nix do it differently. However, most do it effectively, so you as a programmer shouldn't have to worry too much provided you aren't leaking, which you don't think you are. > and after second run it weighs 1Gb. If I will continue > to run this class, memory wont increase, so I think > it's not a memory leak, but rather Python wont release > allocated memory back to OS. Maybe I'm wrong. A 1GB process on modern computers is hardly a big problem? Most machines have 4G and many have 16G or even 32G nowadays. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Ann: New Python curses book
I've just published, in Kindle and paperback formats, my book on "Programming curses with Python". https://www.amazon.co.uk/dp/B091B85B77/ (It should be available in most other Amazon stores too) It is a complete rewrite of the Linux Documentation Project's HowTo for the C ncurses library. It has the same structure but all-new text and several new chapters. All specific to the Python curses package. I've kept the prices as low as Amazon permits (and allowing for fluctuations in their costs). As such, I will therefore make a few cents profit per book from sales (full disclosure). However, I felt this was the easiest and most effective way to get it "out there" where it could be found. I hope it's useful to somebody. Special thanks to all who provided me with feedback on its various iterations, especially Alex Kleider and Bob Stepp who stuck with it right to the end. Thanks guys. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
Have you tried to identify where in your code the surprising memory allocations are made? You could "bisect search" by adding breakpoints: https://docs.python.org/3/library/functions.html#breakpoint At which point does the problem start manifesting itself? -- https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
> > For completeness, from 3.5 onwards, you can also do the following:
> > [{'name': n, **d} for n, d in dod.items()]
> Reading through these, personally I like this one best. I'm curious what
> about it was enabled in 3.5? Was **kwarg expansion inside a dict literal not
> possible before then? Anyway, I like that it uses simple elemental parts that
> have been around a long long time in Python.
You can check the release notes of previous Python versions:
https://docs.python.org/3/whatsnew/3.5.html
specifically:
PEP 448 - Additional Unpacking Generalizations
https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-448
or, if you have multiple local installations of Python (or Docker or similar
technologies), test running the same code snippet with different Python
versions.
Example with Python 3.9:
```
$ docker run --rm --tty --interactive python:3.9 \
python -c 'import sys; d1 = > dict(x=1); d2 = {"y": 2, **d1};
print(sys.version_info, d2)'
sys.version_info(major=3, minor=9, micro=2, releaselevel='final', serial=0)
{'y': 2, 'x': 1}
```
Example with Python 3.4:
```
$ docker run --rm --tty --interactive python:3.4 \
python -c 'import sys; d1 = > dict(x=1); d2 = {"y": 2, **d1};
print(sys.version_info, d2)'
...
SyntaxError: invalid syntax
```
--
https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On Wed, Mar 31, 2021 at 2:44 AM Alan Gauld via Python-list wrote: > > On 29/03/2021 11:12, Alexey wrote: > > Hello everyone! > > I'm experiencing problems with memory consumption. > > > > The first thing you really need to tell us is which > OS you are using? Memory management varies wildly > depending on OS. Even different flavours of *nix > do it differently. > > However, most do it effectively, so you as a programmer > shouldn't have to worry too much provided you aren't > leaking, which you don't think you are. > > > and after second run it weighs 1Gb. If I will continue > > to run this class, memory wont increase, so I think > > it's not a memory leak, but rather Python wont release > > allocated memory back to OS. Maybe I'm wrong. > > A 1GB process on modern computers is hardly a big problem? > Most machines have 4G and many have 16G or even 32G > nowadays. > Desktop systems maybe, but if you rent yourself a worker box, it might not have anything like that much. Especially if you need to have multiple of them. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
RE: New Python curses book
Congratulations, Alan, on the book. I continue to wonder if people will buy the book for the wrong reason or ban it thinking you created an AI snake that creates and spews new CURSES never heard before. The good news is that for those interested you can click on the book image and see the preview of the beginning parts and see it is about programming, hold the curses. -Original Message- From: Python-list On Behalf Of Alan Gauld via Python-list Sent: Tuesday, March 30, 2021 7:12 AM To: [email protected] Subject: Ann: New Python curses book I've just published, in Kindle and paperback formats, my book on "Programming curses with Python". https://www.amazon.co.uk/dp/B091B85B77/ (It should be available in most other Amazon stores too) It is a complete rewrite of the Linux Documentation Project's HowTo for the C ncurses library. It has the same structure but all-new text and several new chapters. All specific to the Python curses package. I've kept the prices as low as Amazon permits (and allowing for fluctuations in their costs). As such, I will therefore make a few cents profit per book from sales (full disclosure). However, I felt this was the easiest and most effective way to get it "out there" where it could be found. I hope it's useful to somebody. Special thanks to all who provided me with feedback on its various iterations, especially Alex Kleider and Bob Stepp who stuck with it right to the end. Thanks guys. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: New Python curses book
On Wed, Mar 31, 2021 at 3:21 AM Avi Gross via Python-list wrote: > > Congratulations, Alan, on the book. > > I continue to wonder if people will buy the book for the wrong reason or ban > it thinking you created an AI snake that creates and spews new CURSES never > heard before. A reasonable misunderstanding. After all, profanity is the one language that every computer programmer understands, regardless of culture, tools, era, or anything else. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
On 2021-03-30, Alan Gauld via Python-list wrote: > I've just published, in Kindle and paperback formats, > my book on "Programming curses with Python". > > https://www.amazon.co.uk/dp/B091B85B77/ > > (It should be available in most other Amazon stores too) Cool. Is the kindle format readable via a web browser pointed at read.amazon.com (AKA "Kindle Cloud Reader)? The last kindle format programming book I bought[1] wasn't (which wasn't made clear in the item description). It had to be read on an actual kindle device or app (which made it virtually useless to me). [1] Linux Driver Development for Embedded Processors - Second Edition -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
I’ve ordered the book (physical volume). It will fulfill a need I’ve had for some time. Unfortunately, it is only available in the UK store, so the shipping cost by far outweighs the book’s cost. Hope for other’s sake, it migrates to the other Amazon stores fairly quickly. Thanks, Bill > On Mar 30, 2021, at 7:12 AM, Alan Gauld via Python-list > wrote: > > I've just published, in Kindle and paperback formats, > my book on "Programming curses with Python". > > https://www.amazon.co.uk/dp/B091B85B77/ > > (It should be available in most other Amazon stores too) -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
On 2021-03-30, William Ray Wing via Python-list wrote: > I’ve ordered the book (physical volume). It will fulfill a need I’ve > had for some time. Unfortunately, it is only available in the UK > store, so the shipping cost by far outweighs the book’s cost. Hope > for other’s sake, it migrates to the other Amazon stores fairly > quickly. It's available in the US: https://www.amazon.com/dp/B091B85B77/ $1.49 kindle $5.99 paperback -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
On 30/03/2021 17:14, Grant Edwards wrote: > Is the kindle format readable via a web browser pointed at > read.amazon.com (AKA "Kindle Cloud Reader)? It seems to be, I downloaded the free sample and could read the first 2 chapters on the web reader. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
RE: Ann: New Python curses book
Bill, I went afterward to the US Amazon Site and the prices are in dollars and ordered the paperback for $5.99, no shipping cost with PRIME but I ordered, paradoxically, an assortment of mice alongside so it would have been free anyway. The Kindle version is $1.49 https://www.amazon.com/Programming-curses-Python-Alan-Gauld-ebook/dp/B091B85B77/ref=sr_1_2?dchild=1&keywords=alan+gauld&qid=1617129871&sr=8-2 The paperback has a different "cover" that says "Welcome to curses!" which sounds ominous. I might have chosen to call it a "Course on curses and cures" ... So depending on what country you are interested in, it may well be there but you need to not use the supplied URL and go there directly. -Original Message- From: Python-list On Behalf Of William Ray Wing via Python-list Sent: Tuesday, March 30, 2021 2:06 PM To: Python Cc: William Ray Wing ; Python Subject: Re: Ann: New Python curses book I’ve ordered the book (physical volume). It will fulfill a need I’ve had for some time. Unfortunately, it is only available in the UK store, so the shipping cost by far outweighs the book’s cost. Hope for other’s sake, it migrates to the other Amazon stores fairly quickly. Thanks, Bill > On Mar 30, 2021, at 7:12 AM, Alan Gauld via Python-list > wrote: > > I've just published, in Kindle and paperback formats, my book on > "Programming curses with Python". > > https://www.amazon.co.uk/dp/B091B85B77/ > > (It should be available in most other Amazon stores too) -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
On 2021-03-30, Alan Gauld via Python-list wrote: > On 30/03/2021 17:14, Grant Edwards wrote: > >> Is the kindle format readable via a web browser pointed at >> read.amazon.com (AKA "Kindle Cloud Reader)? > > It seems to be, I downloaded the free sample and could > read the first 2 chapters on the web reader. It is. I just bought the kindle version on the US site, and it renders fine in the cloud reader. It would be cool if you could cut/paste code examples from the book into IDLE or a text editor, but I've never been able to get the cloud reader to allow that sort of thing. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: New Python curses book
Congratulations! Indeed, I was wondering for a moment if this was a guide to al dente spaghetti code. With each curse being a funny way to mess with the colleagues performing the code review ;) Or a list of funny Monty Python curses? Or a set of programming problems that are cursed? On March 30, 2021 6:35:20 PM GMT+02:00, Chris Angelico wrote: >On Wed, Mar 31, 2021 at 3:21 AM Avi Gross via Python-list > wrote: >> >> Congratulations, Alan, on the book. >> >> I continue to wonder if people will buy the book for the wrong reason or ban >> it thinking you created an AI snake that creates and spews new CURSES never >> heard before. > >A reasonable misunderstanding. After all, profanity is the one >language that every computer programmer understands, regardless of >culture, tools, era, or anything else. > >ChrisA >-- >https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On 31/03/2021 01.22, Loris Bennett wrote:
> Jon Ribbens writes:
>> On 2021-03-30, Loris Bennett wrote:
>>> If I have dict of dicts, say
>>>
>>> dod = {
>>> "alice":
>>> {
>>> "lang": "python",
>>> "level": "expert"
>>> },
>>> "bob":
>>> {
>>> "lang": "perl",
>>> "level": "noob"
>>> }
>>> }
>>>
>>> is there a canonical, or more pythonic, way of converting the outer key
>>> to a value to get a list of dicts, e.g
...
>>>
>>> than just
>>>
>>> lod = []
>>> for name in dod:
>>> d = dod[name]
>>> d["name"] = name
>>> lod.append(d)
Please be aware of the 'law of unintended consequences' - what
functional programmers call "side-effects"!
At the end of the above code, not only has "lod" been created (per spec)
but "dod" is no longer what it once was.
Thus, future code may not rely upon the (above) structure. Of course, if
by "convert" you mean transform, ie that "dod" will be del()[eted]
afterwards, such may be completely unimportant.
from pprint import pprint as pp
import copy
dod = {
"alice":
{
"lang": "python",
"level": "expert"
},
"bob":
{
"lang": "perl",
"level": "noob"
}
}
original = copy.deepcopy( dod )
lod = []
for name in dod:
d = dod[name]
d["name"] = name
lod.append(d)
print( original == dod )
pp(dod)
pp(original)
False
{'alice': {'lang': 'python', 'level': 'expert', 'name': 'alice'},
'bob': {'lang': 'perl', 'level': 'noob', 'name': 'bob'}}
{'alice': {'lang': 'python', 'level': 'expert'},
'bob': {'lang': 'perl', 'level': 'noob'}}
--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On 30/03/2021 16:50, Chris Angelico wrote: >> A 1GB process on modern computers is hardly a big problem? >> Most machines have 4G and many have 16G or even 32G >> nowadays. >> > > Desktop systems maybe, but if you rent yourself a worker box, it might > not have anything like that much. Especially if you need to have > multiple of them. Rental servers on the cloud. Now that's after my time. The last server I used had 128GB and anything less than 32GB wasn't worthy of the name. I forget you can rent virtual servers for pennies these days! :-) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote: > > I'm sorry. I didn't understand your question right. If I have 4 workers, > they require 4Gb > in idle state and some extra memory when they execute other tasks. If I > increase workers > count up to 16, they`ll eat all the memory I have (16GB) on my machine and > will crash as soon > as system get swapped. > What if you increase the machine's (operating system's) swap space? Does that take care of the problem in practice? -- https://mail.python.org/mailman/listinfo/python-list
Re: Ann: New Python curses book
On 30/03/2021 19:52, Grant Edwards wrote: > It is. I just bought the kindle version on the US site, and it renders > fine in the cloud reader. It would be cool if you could cut/paste code > examples from the book into IDLE or a text editor, but I've never been > able to get the cloud reader to allow that sort of thing. Blast! You just reminded me I meant to add a link to where the source code zip lives! (Or will live once I put it there) I'll do an update of the kindle book but I don't know if you can get the update for free... I'll also post the zip file url here once I've put it up. Thanks for that reminder. As for cut n paste, the requirement to write in MS Word and convert to PDF/Kindle means that quotes etc get all messed up. And as for Python indentation I shudder to think. That's why I intended to put the source code up. Watch this space. Hopefully tomorrow. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: New Python curses book
On 30/03/2021 20:05, Brian Oney via Python-list wrote: > Congratulations! > > Indeed, I was wondering for a moment if this was a guide to al dente > spaghetti code. With each curse being a funny way to mess with the colleagues > performing the code review ;) You may jest but I originally titled it "Programming Python with curses" And changed it precisely because I thought it might be misunderstood :-) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
Since everyone is talking about vague OS memory use and not at all about working set size of Python objects, let me ... On 29Mar2021 03:12, Alexey wrote: >I'm experiencing problems with memory consumption. > >I have a class which is doing ETL job. What`s happening inside: > - fetching existing objects from DB via SQLAchemy Do you need to? Or do you only need to fetch their ids? Or do you only need to fetch a subset of the objects? It is easy to accidentally suck in way too many db session entity objects, or at any rate, more than you need to. > - iterate over raw data Can you prescan the data to determine which objects you care about, reducing the number of objects you need to obtain? > - create new/update existing objects Depoending what you're doing, you may not need to "create new/update existing objects". You could collate changes and do an UPSERT (the incantation varies a little depending on the SQL dialect behind SQLAlchemy). > - commit changes Do you discard the SQLAlchemy session after this? Otherwise it may lurk and hold onto the objects. Commit doesn't forget the objects. For my current client we have a script to import historic data from a legacy system. It has many of the issues you're dealing with: the naive (ORM) way consumes gads of memory, and can be very slow too (udating objects in an ad hoc manner tends to do individual UPDATE SQL commands, very latency laden). I wrote a generic batch UPSERT function which took an accrued list of changes and prepared a PostgreSQL INSERT...ON CONFLICT statement. The main script hands it the accrued updates and it runs batches (which lets up do progress reporting). Orders of magnitude faster, _and_ does not require storing the db objects. On the subject of "fetching existing objects from DB via SQLAchemy": you may not need to do that, either. Can you identify _which_ objects are of interest? Associate with the same script I've go a batch_select function: it takes an terable if object ids and collects them in batches, where before we were really scanning the whole db because we had an arbitrary scattering of relevant object ids from the raw data. It basicly collected ids into batches, and ran a SELECT...WHERE id in (batch-of-ids). It's really fast considering, and also scales _way_ down when the set of arbitrary ids is small. I'm happy to walk through the mechanics of these with you; the code at this end is Django's ORM, but I prefer SQLAlchemy anyway - the project dictated the ORM here. >Before processing data I create internal cache(dictionary) and store all >existing objects in it. >Every 1 items I do bulk insert and flush. At the end I run commit command. Yah. I suspect the session data are not being released. Also, SQLAlchemy may be caching sessions or something across runs, since this is a celery worker which survives from one task to the next. You could try explicitly creating a new SQLAlchemy session around your task. >Problem. Before executing, my interpreter process weighs ~100Mb, after first >run memory increases up to 500Mb >and after second run it weighs 1Gb. If I will continue to run this class, >memory wont increase, so I think >it's not a memory leak, but rather Python wont release allocated memory back >to OS. Maybe I'm wrong. I don't know enough about Python's "release OS memory" phase. But reducing the task memory footprint will help regardless. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On Mon, Mar 29, 2021 at 7:16 PM Alexey wrote: > > Problem. Before executing, my interpreter process weighs ~100Mb, after first > run memory increases up to 500Mb > and after second run it weighs 1Gb. If I will continue to run this class, > memory wont increase, so I think > it's not a memory leak, but rather Python wont release allocated memory back > to OS. Maybe I'm wrong. > First of all, I recommend upgrading your Python. Python 3.6 is a bit old. As you saying, Python can not return the memory to OS until the whole arena become unused. If your task releases all objects allocated during the run, Python can release the memory. But if your task keeps at least one object, it may prevent releasing the whole arena (256KB). Python manages only small (~256bytes) objects. Larger objects is allocated by malloc(). And glibc malloc may not efficient for some usage. jemalloc is better for many use cases. You can get some hints from sys._debugmallocstats(). It prints obmalloc (allocator for small objects) stats to stderr. Try printing stats before and after 1st run, and after 2nd run. And post it in this thread if you can. (no sensible information in the stats). That is all I can advise. -- Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
Re: New Python curses book
Il 30/03/2021 21:05, Brian Oney ha scritto: Congratulations! Indeed, I was wondering for a moment if this was a guide to al dente spaghetti code. With each curse being a funny way to mess with the colleagues performing the code review ;) Indeed, I was wondering for a moment if this was a guide to "spaghetti al dente" code... Or a list of funny Monty Python curses? Or a set of programming problems that are cursed? On March 30, 2021 6:35:20 PM GMT+02:00, Chris Angelico wrote: On Wed, Mar 31, 2021 at 3:21 AM Avi Gross via Python-list wrote: Congratulations, Alan, on the book. I continue to wonder if people will buy the book for the wrong reason or ban it thinking you created an AI snake that creates and spews new CURSES never heard before. A reasonable misunderstanding. After all, profanity is the one language that every computer programmer understands, regardless of culture, tools, era, or anything else. ChrisA -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
dn writes:
> On 31/03/2021 01.22, Loris Bennett wrote:
>> Jon Ribbens writes:
>>> On 2021-03-30, Loris Bennett wrote:
If I have dict of dicts, say
dod = {
"alice":
{
"lang": "python",
"level": "expert"
},
"bob":
{
"lang": "perl",
"level": "noob"
}
}
is there a canonical, or more pythonic, way of converting the outer key
to a value to get a list of dicts, e.g
> ...
>
than just
lod = []
for name in dod:
d = dod[name]
d["name"] = name
lod.append(d)
>
>
> Please be aware of the 'law of unintended consequences' - what
> functional programmers call "side-effects"!
>
> At the end of the above code, not only has "lod" been created (per spec)
> but "dod" is no longer what it once was.
>
> Thus, future code may not rely upon the (above) structure. Of course, if
> by "convert" you mean transform, ie that "dod" will be del()[eted]
> afterwards, such may be completely unimportant.
>
>
> from pprint import pprint as pp
> import copy
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> original = copy.deepcopy( dod )
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>
> print( original == dod )
> pp(dod)
> pp(original)
>
>
> False
> {'alice': {'lang': 'python', 'level': 'expert', 'name': 'alice'},
> 'bob': {'lang': 'perl', 'level': 'noob', 'name': 'bob'}}
> {'alice': {'lang': 'python', 'level': 'expert'},
> 'bob': {'lang': 'perl', 'level': 'noob'}}
Thanks for pointing that out. Coming from Perl that's something I need
to watch out for. So if I do
$ a = ["alice", "bob", "carol"]
$ b = a
$ b[1] = "bert"
$ b
['alice', 'bert', 'carol']
$ a
['alice', 'bert', 'carol']
I see that changing one list changes the other because 'a' and 'b' are
just bindings to the same object. However, If I look at non-list
variables:
$ a = "bob"
$ b = a
$ b = "bert"
$ a
'bob'
that doesn't happen. What's the rational for that and where can I find
it in the Python documentation?
Cheers,
Loris
--
This signature is currently under construction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
On 31/03/2021 19.24, Loris Bennett wrote:
> dn writes:
>
>> On 31/03/2021 01.22, Loris Bennett wrote:
>>> Jon Ribbens writes:
On 2021-03-30, Loris Bennett wrote:
> If I have dict of dicts, say
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> is there a canonical, or more pythonic, way of converting the outer key
> to a value to get a list of dicts, e.g
>> ...
>>
>
> than just
>
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>>
>>
>> Please be aware of the 'law of unintended consequences' - what
>> functional programmers call "side-effects"!
>>
>> At the end of the above code, not only has "lod" been created (per spec)
>> but "dod" is no longer what it once was.
>>
>> Thus, future code may not rely upon the (above) structure. Of course, if
>> by "convert" you mean transform, ie that "dod" will be del()[eted]
>> afterwards, such may be completely unimportant.
>>
>>
>> from pprint import pprint as pp
>> import copy
>>
>> dod = {
>> "alice":
>> {
>> "lang": "python",
>> "level": "expert"
>> },
>> "bob":
>> {
>> "lang": "perl",
>> "level": "noob"
>> }
>> }
>>
>> original = copy.deepcopy( dod )
>> lod = []
>> for name in dod:
>> d = dod[name]
>> d["name"] = name
>> lod.append(d)
>>
>> print( original == dod )
>> pp(dod)
>> pp(original)
>>
>>
>> False
>> {'alice': {'lang': 'python', 'level': 'expert', 'name': 'alice'},
>> 'bob': {'lang': 'perl', 'level': 'noob', 'name': 'bob'}}
>> {'alice': {'lang': 'python', 'level': 'expert'},
>> 'bob': {'lang': 'perl', 'level': 'noob'}}
>
> Thanks for pointing that out. Coming from Perl that's something I need
> to watch out for. So if I do
>
> $ a = ["alice", "bob", "carol"]
> $ b = a
> $ b[1] = "bert"
> $ b
> ['alice', 'bert', 'carol']
> $ a
> ['alice', 'bert', 'carol']
>
> I see that changing one list changes the other because 'a' and 'b' are
> just bindings to the same object. However, If I look at non-list
> variables:
>
>$ a = "bob"
>$ b = a
>$ b = "bert"
>$ a
> 'bob'
>
> that doesn't happen. What's the rational for that and where can I find
> it in the Python documentation?
Good observation!
Important to differences.
Python offers mutable (can be changed) and immutable (can't) objects
(remember: 'everything is an object'):
https://docs.python.org/3/reference/datamodel.html?highlight=mutable%20data
PS this even applies when the two identifiers pointing at the same
object are argument(s) to a function!
--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list
