Re: Error installing requirements

[Albert-Jan Roskam]

   On Feb 18, 2022 08:23, Saruni David  wrote:
   >> Christian Gohlke's site has a Pillow .whl for python
   2.7: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pillow
-- 
https://mail.python.org/mailman/listinfo/python-list

Long running process - how to speed up?

[Shaozhong SHI]

I have a cvs file of 932956 row and have to have time.sleep in a Python
script.  It takes a long time to process.

How can I speed up the processing?  Can I do multi-processing?

Regards,

David
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Chris Angelico]

On Sat, 19 Feb 2022 at 22:30, Shaozhong SHI  wrote:
>
> I have a cvs file of 932956 row and have to have time.sleep in a Python
> script.  It takes a long time to process.
>
> How can I speed up the processing?  Can I do multi-processing?
>
Remove the time.sleep()?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Aw: Re: Long running process - how to speed up?

[Karsten Hilbert]

> > I have a cvs file of 932956 row and have to have time.sleep in a Python
> > script.  It takes a long time to process.
> >
> > How can I speed up the processing?  Can I do multi-processing?
> >
> Remove the time.sleep()?

He's attesting to only having "time.sleep" in there...

I doubt removing that will help much ;-)

Karsten
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Re: Long running process - how to speed up?

[Chris Angelico]

On Sat, 19 Feb 2022 at 22:59, Karsten Hilbert  wrote:
>
> > > I have a cvs file of 932956 row and have to have time.sleep in a Python
> > > script.  It takes a long time to process.
> > >
> > > How can I speed up the processing?  Can I do multi-processing?
> > >
> > Remove the time.sleep()?
>
> He's attesting to only having "time.sleep" in there...
>
> I doubt removing that will help much ;-)

I honestly don't understand the question, hence offering the
stupidly-obvious suggestion in the hope that it would result in a
better question. A million rows of CSV, on its own, isn't all that
much to process, so it must be the processing itself (of which we have
no information other than this reference to time.sleep) that takes all
the time.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Albert-Jan Roskam]

   On Feb 19, 2022 12:28, Shaozhong SHI  wrote:

 I have a cvs file of 932956 row and have to have time.sleep in a Python
 script.  It takes a long time to process.

 How can I speed up the processing?  Can I do multi-processing?

   
   Perhaps a dask df: 
   https://docs.dask.org/en/latest/generated/dask.dataframe.read_csv.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Shaozhong SHI]

Can it be divided into several processes?
Regards,
David

On Saturday, 19 February 2022, Chris Angelico  wrote:

> On Sat, 19 Feb 2022 at 22:59, Karsten Hilbert 
> wrote:
> >
> > > > I have a cvs file of 932956 row and have to have time.sleep in a
> Python
> > > > script.  It takes a long time to process.
> > > >
> > > > How can I speed up the processing?  Can I do multi-processing?
> > > >
> > > Remove the time.sleep()?
> >
> > He's attesting to only having "time.sleep" in there...
> >
> > I doubt removing that will help much ;-)
>
> I honestly don't understand the question, hence offering the
> stupidly-obvious suggestion in the hope that it would result in a
> better question. A million rows of CSV, on its own, isn't all that
> much to process, so it must be the processing itself (of which we have
> no information other than this reference to time.sleep) that takes all
> the time.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Kirill Ratkin]

Hi,

How I understand your script starts another script and should wait until 
second one completes its job. Right?


If so you have several options depend on how your first script is written.

If your script is async then ... there is good asyncio option

proc = await asyncio.create_subprocess_shell(
f"{execcmd}{execargs}", stdin=None, stdout=None
)
await proc.wait() In this way you can starn many workers and you don't neet to 
wait then in sync manner.


Anyway, just please give more info about what problem you face.

19.02.2022 14:28, Shaozhong SHI пишет:

I have a cvs file of 932956 row and have to have time.sleep in a Python
script.  It takes a long time to process.

How can I speed up the processing?  Can I do multi-processing?

Regards,

David

--
https://mail.python.org/mailman/listinfo/python-list

Re: Error installing requirements

[Marco Sulla]

Maybe you compiled Python 2.7 by hand, David? It happened to me when I
tried to compile Python without zlib headers installed on my OS. Don't
know how it can be done on Windows.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Dan Stromberg]

On Sat, Feb 19, 2022 at 3:29 AM Shaozhong SHI 
wrote:

> I have a cvs file of 932956 row and have to have time.sleep in a Python
> script.  It takes a long time to process.
>
> How can I speed up the processing?  Can I do multi-processing?
>

How are you doing it right now?

Are you using the csv module?

You might be able to use the GNU "split" command as a prelude to using the
csv module in combination with multiprocessing.  GNU split comes with
Linuxes, but I'm sure you can get it for Windows.  MacOS comes with a
rather less powerful "split" command, but it still might work for you.

You also could try Pypy3.

HTH.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Converting py files to .exe and .dmg

[Techtask Technologies]

On Monday, December 28, 2015 at 4:35:41 PM UTC+1, Brian Simms wrote:
> Hi there, 
> 
> I have done a lot of looking around online to find out how to convert Python 
> files to .exe and .dmg files, but I am confused. Could someone provide 
> pointers/advice as to how I can turn a Python file into a Windows .exe and 
> Mac .dmg file. 
> 
> Thanks for any help.

You can convert Python Files to exe using Pyinstaller, this are the procedures 
below:
Run this commands on your terminal:
pip install pyinstaller
pyinstaller nameofyourfile.py --onefile
after that a folder should be created named Dist open it and you would see your 
Python file in an executable format.
I think this would help you.
~Paul Fruitful
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: library not initialized (pygame)

[Cosmo Hope]

On Sunday, 2 May 2021 at 21:04:40 UTC+1, Michael Torrie wrote:
> On 5/2/21 1:23 PM, Quentin Bock wrote: 
> > the error apparently comes from the first instructions variable saying 
> > library not initialized not sure why, its worked before but not now :/
> I don't get that error on my Fedora 32 machine. The script ultimately 
> doesn't run because it can't find the icon png file. But the window 
> briefly appears and it seems like pygame is being initialized properly.

i have the same problem currently
have you found a solution by chance.
-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: Re: Long running process - how to speed up?

[Alan Gauld]

On 19/02/2022 11:28, Shaozhong SHI wrote:

> I have a cvs file of 932956 row 

That's not a lot in modern computing terms.

> and have to have time.sleep in a Python
> script. 

Why? Is it a requirement by your customer? Your manager?
time.sleep() is not usually helpful if you want to do
things quickly.

> It takes a long time to process.

What is a "long time"? minutes? hours? days? weeks?

It should take a million times as long as it takes to
process one row. But you have given no clue what you
are doing in each row.
- reading a database?
- reading from the network? or the internet?
- writing to a database? or the internet?
- performing highly complex math operations?

Or perhaps the processing load is in analyzing the totality
of the data after reading it all? A very different type
of problem. But we just don't know.

All of these factors will affect performance.

> How can I speed up the processing? 

It all depends on the processing.
You could try profiling your code to see where the time is spent.

> Can I do multi-processing?

Of course. But there is no guarantee that will speed things
up if there is a bottleneck on a single resource somewhere.
But it might be possible to divide and conquer and get better
speed. It all depends on what you are doing. We can't tell.

We cannot answer such a vague question with any specific
solution.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Dennis Lee Bieber]

On Sat, 19 Feb 2022 11:28:31 +, Shaozhong SHI 
declaimed the following:

>I have a cvs file of 932956 row and have to have time.sleep in a Python
>script.  It takes a long time to process.
>
I'd echo the others... Unless you better explain WHY you have .sleep()
(along with how often it is called, and what duration you sleep) the first
recommendation would be to remove it.

The most common justification for .sleep() is that one has CPU-BOUND
processing and needs to force context switches to let other operations
proceed more often than the system quantum time. Not normally a concern
given Python's GIL and the presence of multi-core chips.

How are you processing the (near) million rows of that CSV? If you are
loading all of them into a large list you could be running Python list
reallocations, or OS page swapping (though I wouldn't expect that on most
modern systems -- maybe on a Raspberry-Pi/BeagleBone Black). Note:

>>> import sys
>>> sys.getsizeof("a")
50
>>> 

even a one-character string expands to 50 bytes. An EMPTY string takes up
51 bytes... Except for the empty string, that comes to about 49+<#chars> IF
all characters fit an 8-bit encoding -- if any non 8-bit characters are in
the string, the #chars needs to be multiplied by either 2 or 4 depending
upon the widest representation needed.

If you are doing read-one-record, process-one-record, repeat -- and
have the .sleep() inside that loop... definitely remove the .sleep(). That
loop is already I/O bound, the fastest you can obtain is determined by how
rapidly the OS can transfer records from the file system to your program.


-- 
Wulfraed Dennis Lee Bieber AF6VN
wlfr...@ix.netcom.comhttp://wlfraed.microdiversity.freeddns.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: library not initialized (pygame)

[Martin Di Paola]

Could you share the traceback / error that you are seeing?


On Sun, May 02, 2021 at 03:23:21PM -0400, Quentin Bock wrote:

Code:
#imports and variables for game
import pygame
from pygame import mixer
running = True

#initializes pygame
pygame.init()

#creates the pygame window
screen = pygame.display.set_mode((1200, 800))

#Title and Icon of window
pygame.display.set_caption("3.02 Project")
icon = pygame.image.load('3.02 icon.png')
pygame.display.set_icon(icon)

#setting up font
pygame.font.init()
font = pygame.font.Font('C:\Windows\Fonts\OCRAEXT.ttf', 16)
font_x = 10
font_y = 40
items_picked_up = 0
items_left = 3

def main():
   global running, event

   #Game Loop
   while running:
   #sets screen color to black
   screen.fill((0, 0, 0))

   #checks if the user exits the window
   for event in pygame.event.get():
   if event.type == pygame.QUIT:
   running = False
   pygame.quit()

   def display_instruction(x, y):
   instructions = font.render("Each level contains 3 items you
must pick up in each room.", True, (255, 255, 255))
   instructions_2 = font.render("When you have picked up 3 items,
you will advance to the next room, there are 3.", True, (255, 255, 255))
   instructions_3 = font.render("You will be able to change the
direction you are looking in the room, this allows you to find different
objects.", True, (255, 255, 255))
   clear = font.render("Click to clear the text.", True, (255,
255, 255))
   screen.blit(instructions, (10, 40))
   screen.blit(instructions_2, (10, 60))
   screen.blit(instructions_3, (10, 80))
   screen.blit(clear, (10, 120))

   if event.type == pygame.MOUSEBUTTONDOWN:
   if event.type == pygame.MOUSEBUTTONUP:
   screen.fill(pygame.Color('black'))  # clears the screen text

   display_instruction(font_x, font_y)
   pygame.display.update()


main()

the error apparently comes from the first instructions variable saying
library not initialized not sure why, its worked before but not now :/
--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Mats Wichmann]

On 2/19/22 05:09, Shaozhong SHI wrote:
> Can it be divided into several processes?
> Regards,
> David

The answer is: "maybe".  Multiprocessing doesn't happen for free, you
have to figure out how to divide the task up, requiring thought and
effort. We can't guess to what extent the problem you have is amenable
to multiprocessing.

Google for "dataframe" and "multiprocessing" and you should get some
hits (in my somewhat limited experience in this area, people usually
load the csv data into Pandas before they get started working with it).


-- 
https://mail.python.org/mailman/listinfo/python-list

PYT - How can I subscribe to a topic in the mailing list?

[vanyp]

And h*ow do I set the topic in my messages?*
--
https://mail.python.org/mailman/listinfo/python-list

Aw: PYT - How can I subscribe to a topic in the mailing list?

[Karsten Hilbert]

> Betreff: PYT - How can I subscribe to a topic in the mailing list?

Mailing lists don't subdivide by topic.

Your mailer may allow you to filter by Subject:.

> And h*ow do I set the topic in my messages?*

not applicable

The equivalent would be the Subject: header.

Karsten
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Avi Gross via Python-list]

Indeed not a clear request. Timing is everything but there are times ...
For many purposes, people may read in the entire CSV at a gulp into some data 
structure like a pandas DataFrame. The file is then closed and any processing 
done later does whatever you want.
Of course you can easily read on line at a time in Python and parse it by comma 
and any other processing and act on one row at a time or in small batches so 
you never need huge amounts of memory. But using other methods to read in the 
entire set of data is often better optimized and faster, and being able to do 
some things with the data is faster if done in vectorized fashion using add-ons 
like numpy and pandas. We have no idea what is being used and none of this 
explains a need to use some form of sleep.
Multi-processing helps only if you can make steps in the processing run in 
parallel without interfering with each other or making things happen out of 
order. Yes, you could read in the data and assign say 10,000 rows to a thread 
to process and then get more and assign, if done quite carefully. The results 
might need to be carefully combined and any shared variables might need locks 
and so on. Not necessarily worth it if the data is not too large and the 
calculations are small.

And it remains unclear where you want to sleep or why. Parallelism can be 
important if the sleep is to wait for the user to respond to something while 
processing continues in the background. Is it possible that whatever you are 
calling to do processing has some kind of sleep within it and you may be 
calling it as often as per row? In that case, ask why it does that and can you 
avoid that? Yes, running in parallel may let you move forward but again, it has 
to be done carefully and having thousands of processes sleeping at the same 
time may be worse!
I note badly defined questions get horrible answers. Mine included.
-Original Message-
From: Alan Gauld 
To: python-list@python.org
Sent: Sat, Feb 19, 2022 7:33 am
Subject: Fwd: Re: Long running process - how to speed up?

On 19/02/2022 11:28, Shaozhong SHI wrote:

> I have a cvs file of 932956 row 

That's not a lot in modern computing terms.

> and have to have time.sleep in a Python
> script. 

Why? Is it a requirement by your customer? Your manager?
time.sleep() is not usually helpful if you want to do
things quickly.

> It takes a long time to process.

What is a "long time"? minutes? hours? days? weeks?

It should take a million times as long as it takes to
process one row. But you have given no clue what you
are doing in each row.
- reading a database?
- reading from the network? or the internet?
- writing to a database? or the internet?
- performing highly complex math operations?

Or perhaps the processing load is in analyzing the totality
of the data after reading it all? A very different type
of problem. But we just don't know.

All of these factors will affect performance.

> How can I speed up the processing? 

It all depends on the processing.
You could try profiling your code to see where the time is spent.

> Can I do multi-processing?

Of course. But there is no guarantee that will speed things
up if there is a bottleneck on a single resource somewhere.
But it might be possible to divide and conquer and get better
speed. It all depends on what you are doing. We can't tell.

We cannot answer such a vague question with any specific
solution.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos

-- 
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

PYT - The expressions described in the Python language reference yield only boolean values

[vanyp]

*I am trying to learn Python from the grammar given in the Python 
language reference and I am surprised.*


*Lets start here:*

*"*
*6.3.4. Calls*

A call calls a callable object (e.g., a function 
) with a possibly 
empty series of arguments 
:


*call *::= |primary 
|"(" [|argument_list 
|[","] | |comprehension 
|] ")"
*argument_list *::= |positional_arguments 
|["," |starred_and_keywords 
|]
["," |keywords_arguments 
|]
| |starred_and_keywords 
|["," |keywords_arguments 
|]
| |keywords_arguments 
|

*positional_arguments*::= positional_item ("," positional_item)*
*positional_item *::= |assignment_expression 
|| "*" |expression 
|

*"*

*continued by:*
*"*
*6.12. Assignment expressions*
*assignment_expression*::= [|identifier 
|":="] |expression 
|

*"*

*Now I look at the following productions:*

*"*
*6.13. Conditional expressions*
*conditional_expression*::= |or_test 
|["if" |or_test 
|"else" |expression 
|]
*expression *::= |conditional_expression 
|| |lambda_expr 
|

*"*

*The first or_test is strange, I assume it should be replaced by expression.*

*But even so I think that the only ways out of the recursion are the 
or_test or the lambda_expr.*


*And by the grammar, those evaluate to booleans as follows:*


*"*
*6.14. Lambdas*
*lambda_expr*::= "lambda" [|parameter_list 
|] ":" |expression 
|

*"*

*and I conclude that the only way out of that branch is also or_test.*

*I then look at or_test:*

*"*
*6.11. Boolean operations*
*or_test *::= |and_test 
|| |or_test 
|"or" |and_test 
|
*and_test*::= |not_test 
|| |and_test 
|"and" |not_test 
|
*not_test*::= |comparison 
|| "not" |not_test 
|

*"*

*and the ony way out is comparison, which by the semantics should return 
a boolean.*


*Looking at comparison gives:*

*"*
**6.10. Comparisons**
*comparison *::= |or_expr 
|(|comp_operator 
||or_expr 
|)*

*comp_operator*::= "<" | ">" | "==" | ">=" | "<=" | "!="
| "is" ["not

Re: PYT - The expressions described in the Python language reference yield only boolean values

[Chris Angelico]

On Sun, 20 Feb 2022 at 12:00, vanyp  wrote:
>
> *I am trying to learn Python from the grammar given in the Python
> language reference and I am surprised.*
>

The grammar is not the best way to learn the language. It'll show you
a lot of unnecessary details. For technical reasons, Python's grammar
defines expressions in a nested way, but the way most programmers want
to think of it is that there is operator precedence.

https://docs.python.org/3/reference/expressions.html#operator-precedence

The grammar considers that these are different types of expressions,
but they're not really different in actual usage.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list