Re: [Tutor] Question about why a list variable is apparently global.
On 27/11/14 04:18, boB Stepp wrote: Note that this principle is critical to Python, otherwise functions couldn't call each other, or even built-ins! If you have two functions: def f(): return g() def g(): return "Hello!" "g" is a global "variable" and f() can see it, otherwise it couldn't call it. So any variables lower in the program are accessible to those above it? No. Its not whether they are defined above or below each other its the level of indentation. Both f and g are defined at the outer level of the module. They are therefore global(within that module) and can each see the other. Steven could just as well have written: def f(): return "Hello!" def g(): return f() You only need to declare a global variable inside a function if you are re-assigning a value to it. Hence: a = 1 b = 2 def test(): global b b = b + a So even though test() has access to a and b, it won't change b when called unless b is declared global inside the function? Correct, if it tries to, it will create a new local variable b inside test(). In the absence of a global declaration Python uses initial assignment as name creation. day today. Despite having Grayson's book on Tkinter, it has been hard going for me. Grayson's book is hard going and is very old now so some things have improved and changed. In particular you now have access to Tix and ttf on newer versions of Python. But since you are still using 2.4 that probably doesn't affect you! :-) However, if you have access to it, you should consider getting Mark Lutz's book Programming Python (4th ed). It's a massive 1600 pages of which 400+ are about Tkinter programming (more than Grayson!) And there is the bonus of 1200 other pages of detailed info about the OS, Networking, databases etc. Again its a bit dry but full of information. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Installing twisted
Hey guys I was hoping someone could tell me how to opted out of this list? I have it going to two email addresses for some reason and I unsubscribed but nothing happened. Any help is greatly appreciated! Thanks, Scott ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Wed, Nov 26, 2014 at 10:18:55PM -0600, boB Stepp wrote: > So any variables lower in the program are accessible to those above it? No, that can't be the explanation. Think of this: b = a + 1 a = 2 That will fail because when the "b = a + 1" line is executed, a doesn't exist yet so there is no way to get a value for it. In the case of the functions, *defining* the function def f(): return g() is okay because we are just creating the function. But if we tried to *call* the function, and execute the code inside it, we would run into a NameError exception because g() doesn't exist yet. It is because we delay calling the function f() until g() likewise exists that it works out without error. *Inside* a function, code that refers to some name: return g() # refers to name "g" compiles an instruction to look up the name "g", but the lookup doesn't actually take place until you call the function. Therefore the name doesn't need to exist when you define the function, only when you call it. *Outside* of a function, code is executed immediately, so the name has to exist or there will be an error. As usual, the interactive interpreter is your friend: py> b = a + 1 # Oops, "a" doesn't exist yet. Traceback (most recent call last): File "", line 1, in NameError: name 'a' is not defined py> a = 2 py> b = a + 1 py> print b 3 Compared to: py> def func(): ... print x ... py> func() # Oops, "x" doesn't exist yet. Traceback (most recent call last): File "", line 1, in File "", line 2, in func NameError: global name 'x' is not defined py> x = 23 py> func() 23 > > You only need to declare a global variable inside a function if you are > > re-assigning a value to it. Hence: > > > > a = 1 > > b = 2 > > def test(): > > global b > > b = b + a > > So even though test() has access to a and b, it won't change b when > called unless b is declared global inside the function? Python understands that there are two sorts of variables, local variables which are local to a function, and global variables which exist in a module. (Python 3 adds a third, "nonlocal", but that's a story for another day.) It uses a simple rule to decide which is which: * if you assign to a variable anywhere inside a function, it is treated as a local variable; * unless you declare it "global"; * otherwise it is global. So if I neglected to include the global declaration, I would have this: def test(): b = b + a Because there is an assignment to b, it is treated as local, but there is no assignment to a. So when you call the function, Python tries to do this: * lookup the value of local variable b * lookup the value of global variable a * add them together * assign the result to local variable b But notice that the first lookup will fail. b doesn't have a value yet, so you will get UnboundLocalError: local variable 'b' referenced before assignment. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Installing twisted
On 27/11/14 03:35, Scott W Dunning wrote: Hey guys I was hoping someone could tell me how to opted out of this list? > I have it going to two email addresses for some reason If you tell me what the two addresses are I can see which is master and you then need to log in with that address and unsubscribe yourself. I had a look for scottw...@gmail.com and couldn't find it so I'm guessing the other one is your original master address? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Thu, Nov 27, 2014 at 4:51 AM, Alan Gauld wrote: > On 27/11/14 04:18, boB Stepp wrote: [...] >> So any variables lower in the program are accessible to those above it? > > > No. > Its not whether they are defined above or below each other its the level of > indentation. Both f and g are defined at the outer level of the module. They > are therefore global(within that module) and can each see the other. Steven > could just as well have written: > > def f(): return "Hello!" > > def g(): return f() Level of indentation is the key? Can you give me an example, not involving functions, where a variable is defined at an "inner" level of indentation, but is not accessible at the outermost level of indentation? I tried to construct such an example myself, but was unsuccessful. [...] >> day today. Despite having Grayson's book on Tkinter, it has been hard >> going for me. > > > Grayson's book is hard going and is very old now so some things have > improved and changed. In particular you now have access to Tix and ttf > on newer versions of Python. But since you are still using 2.4 that probably > doesn't affect you! :-) That is the main reason I got the book, because of the version of Python I am forced to use at work. > However, if you have access to it, you should consider getting Mark Lutz's > book Programming Python (4th ed). It's a massive 1600 pages > of which 400+ are about Tkinter programming (more than Grayson!) I actually purchased that book and his 4th edition of Learning Python when I first started getting interested in Python a couple of years ago. I will check out his Tkinter coverage and see if it is easier going. > And there is the bonus of 1200 other pages of detailed info about > the OS, Networking, databases etc. Again its a bit dry but full > of information. I looked at his discussion of scope last night in the latter book. I still don't think I am fully getting it yet. But I will continue to persevere. -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Thu, Nov 27, 2014 at 4:56 AM, Steven D'Aprano wrote: > On Wed, Nov 26, 2014 at 10:18:55PM -0600, boB Stepp wrote: > >> So any variables lower in the program are accessible to those above it? > > No, that can't be the explanation. Think of this: > > b = a + 1 > a = 2 > > That will fail because when the "b = a + 1" line is executed, a doesn't > exist yet so there is no way to get a value for it. This was my original understanding! I see now that I totally misunderstood what you said in your earlier post: "No, they also have access to globals and built-ins. You define the list l at the top level of your module. That makes it a global, so the printLavel() function can see it." I did not understand what you meant by "top level of your module". As I mentioned just moments ago in response to Alan's post, I looked at Lutz's discussion of scope and I believe I now understand what you originally meant. OTOH, I see that there is much more to scope/name spaces than I originally thought, so I don't claim to fully grasp Lutz's discussion yet. -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Thu, Nov 27, 2014 at 09:00:48AM -0600, boB Stepp wrote: > Level of indentation is the key? Can you give me an example, not > involving functions, where a variable is defined at an "inner" level > of indentation, but is not accessible at the outermost level of > indentation? Classes and functions introduce a new scope. py> x = "outer" py> class Test: ... x = "inner" ... py> x 'outer' Notice that the outer "x" is unchanged, while the inner "x" is only accessible by specifying the class (or an instance) first: py> Test.x 'inner' But there is a subtlety that you may not expect: py> class Tricky: ... print(x) ... x = "inner" ... print(x) ... outer inner So although classes introduce a new scope, they are not like functions. In a function, the above would lead to an error (try it and see). * Every module is a separate scope. * `def` and `class` introduce new scopes. * Other indented blocks (for, while, if etc.) do not. * But lambda expressions do. * Generator expressions have their own scope too. * In Python 3 only, so do list comprehensions (and dict and set comprehensions). But not in Python 2. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Thu, Nov 27, 2014 at 09:13:35AM -0600, boB Stepp wrote: > On Thu, Nov 27, 2014 at 4:56 AM, Steven D'Aprano wrote: > > On Wed, Nov 26, 2014 at 10:18:55PM -0600, boB Stepp wrote: > > > >> So any variables lower in the program are accessible to those above it? > > > > No, that can't be the explanation. Think of this: > > > > b = a + 1 > > a = 2 > > > > That will fail because when the "b = a + 1" line is executed, a doesn't > > exist yet so there is no way to get a value for it. > > This was my original understanding! I see now that I totally > misunderstood what you said in your earlier post: > > "No, they also have access to globals and built-ins. You define the list > l at the top level of your module. That makes it a global, so the > printLavel() function can see it." > > I did not understand what you meant by "top level of your module". I hope I have been more clear now, but just in case, "top level" means code not inside a class or function. It's not quite the same as indent levels, since not all indents create a new scope, but similar. # module a b def func(): c d e f a, b, e and f are "top level" (module level), c and d are in func(). -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is there an easy way to center the root window (Tkinter) within the display?
On Wed, Nov 26, 2014 at 12:25:23PM -0600, boB Stepp wrote: > As I am the only person in our > group with any programming knowledge (weak though it is), this means I > usually wind up trying to solve issues as they arise. These client > machines are dedicated to a single purpose: radiation therapy > treatment planning. No offense intended Bob, but this scares me. I know you're trying your best, but "weak programming knowledge" and "radiation therapy" is not a healthy combination. I trust you are aware of the Therac-25 disaster? http://courses.cs.vt.edu/professionalism/Therac_25/Therac_1.html http://en.wikipedia.org/wiki/Therac-25 -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On Thu, Nov 27, 2014 at 9:33 AM, Steven D'Aprano wrote: > On Thu, Nov 27, 2014 at 09:00:48AM -0600, boB Stepp wrote: [...] > But there is a subtlety that you may not expect: > > py> class Tricky: > ... print(x) > ... x = "inner" > ... print(x) > ... > outer > inner Actually, this is what you had me primed to expect. However, ... > So although classes introduce a new scope, they are not like functions. > In a function, the above would lead to an error (try it and see). >>> def tricky_func(): print(x) x = "inner" print(x) >>> tricky_func() Traceback (most recent call last): File "", line 1, in tricky_func() File "", line 2, in tricky_func print(x) UnboundLocalError: local variable 'x' referenced before assignment This surprised me! So I did: >>> def tricky_func2(): y = x print(x) >>> tricky_func2() outer So why does not print(x) see the global x and instead looks for the local x? And why is this different between classes and functions? > * Every module is a separate scope. > > * `def` and `class` introduce new scopes. > > * Other indented blocks (for, while, if etc.) do not. Alan's reference to indentation level had me trying to prove the opposite--unsuccessfully. > * But lambda expressions do. > > * Generator expressions have their own scope too. > > * In Python 3 only, so do list comprehensions (and dict and set > comprehensions). But not in Python 2. This is good to know this difference as at work I'm in Python 2.4.4 and at home Python 3.4 -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On 11/27/2014 11:07 AM, boB Stepp wrote: x = "outer" def tricky_func2(): y = x print(x) tricky_func2() outer So why does not print(x) see the global x and instead looks for the local x? The function is compiled during the import (or initial load if it's a script); it cannot be called until the compile is complete. During that compile, a list of local variables is built for that function, based on DEFINITIONS of variables, not on REFERENCES to variables. At compile time, no knowledge of global variables is possible. How are local variable definitions to be recognized? 1) any 'global' or 'nonlocal' statement declares that the variable is NOT local, even if the following might otherwise make it so. 2) formal parameters 3) assignment statements such as y = z or x,y = 4) with statements having an "as" clause 5) except statements having an 'as" clause I may have missed one, but everything else is NOT a local variable. The list of locals is unchangeable at runtime. So any other references to variables must be searched for at run time. The search mechanism is a bit more complex than I want to go into here, but includes globals, built-ins, and a few other things. And why is this different between classes and functions? Classes are not compiled in the same sense. The stuff inside a class, but not inside a method is evaluated at the same time as top-level stuff. So the rules are a bit different than either top-level or function/method. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is there an easy way to center the root window (Tkinter) within the display?
On Thu, Nov 27, 2014 at 9:58 AM, Steven D'Aprano wrote: > On Wed, Nov 26, 2014 at 12:25:23PM -0600, boB Stepp wrote: > >> As I am the only person in our >> group with any programming knowledge (weak though it is), this means I >> usually wind up trying to solve issues as they arise. These client >> machines are dedicated to a single purpose: radiation therapy >> treatment planning. > > No offense intended Bob, but this scares me. I know you're trying your > best, but "weak programming knowledge" and "radiation therapy" is not a > healthy combination. Believe me, I think about this constantly! The planning software we use is FDA approved. My add-on programs/scripts generally automate repetitive tasks to save the planner much by hand work. The other types of plans take the results of the plan and create a visual display to flag things we might want to look at more closely. Or send printouts of the plan in pdf format from the Solaris 10 planning environment to where they need to wind up on our Window's based record and verify software server. Or similar things that do not alter the plan itself. That is the important point: They do not alter the treatment plan. When I have something ready to use we check everything by hand for at least a couple of weeks. Medical physics evaluates my software as well. For anything remotely critical that might influence the evaluation of a plan, we do monthly QA to verify the constancy of the underlying data and calculations in case something we are potentially unaware of has changed. I constantly exhort my colleagues to use my scripts with a critical eye and report immediately anything, however, slight that is unusual and unexpected. I test and I test and I worry and I worry... But so far these efforts have not only saved time, but caught and prevented errors that might otherwise would have needed to be caught farther down in the review process. Now they get caught at the planning level and never even come close to getting further. The scripts generally either prevent the dosimetrist from forgetting to do something, make him aware of something he should examine more closely that he might not otherwise catch, and reduce the routine grunt-work burden. > I trust you are aware of the Therac-25 disaster? I have not looked at that particular one (But will now.), but can cite many more, especially those very recent. The New York Times not long ago ran a series of articles about this topic. -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question about why a list variable is apparently global.
On 27/11/14 16:07, boB Stepp wrote: Alan's reference to indentation level had me trying to prove the opposite--unsuccessfully. Yeah, I probably over-simplified there in response to your assumption that it was the order that mattered. It's really whether they are inside a function or class - which means they will be indented. But variables defined inside loops etc are also indented but still at global scope since the containing block is global. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is there an easy way to center the root window (Tkinter) within the display?
On 11/27/2014 11:39 AM, boB Stepp wrote: On Thu, Nov 27, 2014 at 9:58 AM, Steven D'Aprano wrote: No offense intended Bob, but this scares me. I know you're trying your best, but "weak programming knowledge" and "radiation therapy" is not a healthy combination. Believe me, I think about this constantly! I would as well. I have been in many environments in which code development is less than professional, and if lives could be at stake, I'd evaluate the environment most carefully. You say you're using some 3rd party package to do the heavy lifting. But you also say there could be as many as 1000 servers involved. So the scale of things is quite large. If it's not impertinent, I'd ask a number of questions about both your own development environment and that of the 3rd party product that runs on all those servers. Does that package include any hooks for automating? Do they expect you to run 1000 queries individually, or do they provide some way for you to automate them? Are they open to requests for improving their software, or for validating your own front ends? For both your organization and theirs: Are you using source control? That's more than version control, but the latter would be a start. Do you have a formal testing environment, including a means for running these tests frequently and systematically. I don't mean hand testing, I mean something that exercises every aspect as thoroughly as one can figure out, and checks that all expectations are being met. Do you have a backup strategy, both for code and for data? Do you have strict security policies, and someone to review whether each software change conforms to them? And whether each system is restricted as to what software it can run, and what access can be made from outside? Have you planned for fault tolerance, such as when there are supposed to be 1000 servers, but only 997 of them are alive right now? These things and many others you cannot get from a book, at least none that I've ever seen. You need someone with experience, responsibility, and authority to make it happen. Or you need a lot of luck. One thing I'd do to enhance your luck is to make sure you don't overstep your own capabilities. First thing I'd do is to drop the GUI. Generally people trust something which is clearly complex, and has a pretty interface. So don't add that interface until everything under it is rock solid. I've seen people trust spreadsheets when they showed thousands of figures, and gave "results" at the bottom. Even though one whole row of data might well have been omitted from the sums, not from maliciousness, but from less than careful editing. A formula that's correct for a column of numbers will automatically adapt to additional rows inserted in the middle. But if you add to the end, it can be missed. For the spreadsheet there are auditing programs. But even they are only as good as the person manning them. I've seen offsite back up systems that had less than 30% of the data supposedly entrusted to it, and could list the flaws that one audit uncovered. Data that was supposed to have been secure for years was just not there. I've seen vaults full of floppies where one of 10 was unusable, due to a simple flaw that wasn't discovered for many months. The actual programming of a utility is but a tiny fraction of the work that needs to go into such a utility. And you don't learn that from books on Python (as far as I know). Again, I don't mean anything personal, as I don't know you. But something about this thread triggered my rant. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] multiprocessing question
>
> From: eryksun
>To: Python Mailing List
>Sent: Tuesday, November 25, 2014 6:41 AM
>Subject: Re: [Tutor] multiprocessing question
>
>
>On Sun, Nov 23, 2014 at 7:20 PM, Cameron Simpson wrote:
>>
>> A remark about the create_lookup() function on pastebin: you go:
>>
>> record_start += len(line)
>>
>> This presumes that a single text character on a line consumes a single byte
>> or memory or file disc space. However, your data file is utf-8 encoded, and
>> some characters may be more than one byte or storage. This means that your
>> record_start values will not be useful because they are character counts,
>> not byte counts, and you need byte counts to offset into a file if you are
>> doing random access.
>
>mmap.readline returns a byte string, so len(line) is a byte count.
>That said, CsvIter._get_row_lookup shouldn't use the mmap
>object. Limit its use to __getitem__.
Ok, thanks, I will modify the code.
>In CsvIter.__getitem__, I don't see the need to wrap the line in a
>filelike object. It's clearly documented that csv.reader takes an
>iterable object, such as a list. For example:
>
># 2.x csv lacks unicode support
>line = self.data[start:end].strip()
>row = next(csv.reader([line]))
>return [cell.decode('utf-8') for cell in row]
>
># 3.x csv requires unicode
>line = self.data[start:end].strip()
>row = next(csv.reader([line.decode('utf-8')]))
>return row
Nice, thank you! I indeed wanted to write the code for use in Python 2.7 and
3.3+.
>CsvIter._get_row_lookup should work on a regular file from built-in
>open (not codecs.open), opened in binary mode. I/O on a regular file
>will release the GIL back to the main thread. mmap objects don't do
>this.
Will io.open also work? Until today I thought that Python 3's open was what is
codecs.open in Python 2 (probably because Python3 is all about ustrings, and
py3-open has an encoding argument).
>
>Binary mode ensures the offsets are valid for use with
>the mmap object in __getitem__. This requires an ASCII compatible
>encoding such as UTF-8.
What do you mean exactly with "ascii compatible"? Does it mean 'superset of
ascii', such as utf-8, windows-1252, latin-1? Hmmm, but Asian encodings like
cp874 and shift-JIS are thai/japanese on top of ascii, so this makes me doubt.
In my code I am using icu to guess the encoding; I simply put 'utf-8' in the
sample code for brevity.
>
>Also, iterate in a for loop instead of calling readline in a while loop.
>2.x file.__next__ uses a read-ahead buffer to improve performance.
>To see this, check tell() in a for loop.
Wow, great tip. I just modified some sample code that I post shortly.
>
>
>___
>Tutor maillist - Tutor@python.org
>To unsubscribe or change subscription options:
>https://mail.python.org/mailman/listinfo/tutor
>
>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is there an easy way to center the root window (Tkinter) within the display?
On Nov 27, 2014 1:27 PM, "Dave Angel" wrote: > > On 11/27/2014 11:39 AM, boB Stepp wrote: >> >> On Thu, Nov 27, 2014 at 9:58 AM, Steven D'Aprano wrote: > > > > You say you're using some 3rd party package to do the heavy lifting. But you also say there could be as many as 1000 servers involved. So the scale of things is quite large. If it's not impertinent, I'd ask a number of questions about both your own development environment and that of the 3rd party product that runs on all those servers. > I'm away from my PC for the rest of the day, but I must have left an incorrect impression of the scale involved & the size of our organization. There are a total of 9 people distributed amongst 6 centers. There are only 2 servers that distribute the planning software to thin clients. There are other Windows servers involved with post-planning tasks, but I don't do anything programming-wise other than FTPing PDF docs from the planning system to a specific network folder. I hope this clarifies my particular situation. And I apologize for leaving such an erroneous impression!! ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] multiprocessing question
- Original Message -
> From: Cameron Simpson
> To: Python Mailing List
> Cc:
> Sent: Monday, November 24, 2014 11:16 PM
> Subject: Re: [Tutor] multiprocessing question
>
> On 24Nov2014 12:56, Albert-Jan Roskam wrote:
>> > From: Cameron Simpson
>>> On 23Nov2014 22:30, Albert-Jan Roskam
>
>>> wrote:
I created some code to get records from a potentially giant .csv
> file. This
>>> implements a __getitem__ method that gets records from a memory-mapped
> csv file.
>>> In order for this to work, I need to build a lookup table that maps
> line numbers
>>> to line starts/ends. This works, BUT building the lookup table could be
>>> time-consuming (and it freezes up the app). [...]
>>>
>>> First up, multiprocessing is not what you want. You want threading for
> this.
>>>
>>> The reason is that your row index makes an in-memory index. If you do
> this in a
>>> subprocess (mp.Process) then the in-memory index is in a different
> process, and
>>> not accessable.
>>
>> Hi Cameron, Thanks for helping me. I read this page before I decided to go
> for multiprocessing:
> http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python.
>
> I never *really* understood why cPython (with GIL) could have threading
> anyway.
> I am confused: I thought the idea of mutliprocessing.Manager was to share
> information.
>
> Regarding the GIL, it will prevent the raw python interpreter from using more
> than one CPU: no two python opcodes run concurrently. However, any calls to C
> libraries or the OS which may block release the GIL (broadly speaking). So
> while the OS is off reading data from a hard drive or opening a network
> connection or something, the Python interpreter is free to run opcodes for
> other python threads. It is timesharing at the python opcode level. And if
> the
> OS or a C library is off doing work with the GIL released then you get true
> multithreading.
>
> Most real code is not compute bound at the Python level, most of the time.
> Whenever you block for I/O or delegate work to a library or another process,
> your current Python Thread is stalled, allowing other Threads to run.
>
> For myself, I use threads when algorithms naturally fall into parallel
> expression or for situations like yours where some lengthy process must run
> but
> I want the main body of code to commence work before it finishes. As it
> happens, one of my common uses cases for the latter is reading a CSV file:-)
>
> Anywhere you want to do things in parallel, ideally I/O bound, a Thread is a
> reasonable thing to consider. It lets you write the separate task in a nice
> linear fashion.
>
> With a Thread (coding errors aside) you know where you stand: the data
> structures it works on are the very same ones used by the main program. (Of
> course, therein lie the hazards as well.)
>
> With multiprocessing the subprocess works on distinct data sets and (from my
> reading) any shared data is managed by proxy objects that communicate between
> the processes. That gets you data isolation for the subprocess, but also
> higher
> latency in data access between the processes and of course the task of
> arranging those proxy objects.
>
> For your task I would go with a Thread.
I made a comparison between multiprocessing and threading. In the code below
(it's also here: http://pastebin.com/BmbgHtVL, multiprocessing is more than 100
(yes: one hundred) times slower than threading! That is
I-must-be-doing-something-wrong-ishly slow. Any idea whether I am doing
something wrong? I can't believe the difference is so big.
from __future__ import print_function
import threading
import mmap
import multiprocessing as mp
try:
xrange
except NameError:
xrange = range # python 3
class ThreadOrProcess(object):
def __init__(self, data, use_threading=True):
self.data = data
self.use_threading = use_threading
if self.use_threading:
self.lookup = dict()
self.thread = threading.Thread(target=self.threaded_create_lookup,
name="lookup maker thread")
self.thread.start()
#self.check_progress()
self.thread.join()
else:
self.lookup = mp.Manager().dict()
self.process = mp.Process(target=self.mp_create_lookup,
name="lookup maker process")
self.process.start()
#self.check_progress()
self.process.join()
def check_progress(self):
before, after = float("nan"), float("nan")
while before != after:
before = len(self.lookup)
print("%s: %d items" % (time.asctime(), before))
time.sleep(0.01)
after = len(self.lookup)
def threaded_create_lookup(self):
lino, record_start = 0, 0
for line in self.data:
if not line:
break
Re: [Tutor] Is there an easy way to center the root window (Tkinter) within the display?
On 27/11/14 19:26, Dave Angel wrote: scale of things is quite large. Does that package include any hooks for automating? > Are they open to requests for improving their software, or for validating your own front ends? For both your organization and theirs: Are you using source control? Do you have a formal testing environment, Do you have a backup strategy, both for code and for data? Do you have strict security policies, Have you planned for fault tolerance, These things and many others you cannot get from a book, Thee are a few books that tackle these issues but many are not "developers" books they are for IT service managers - The ITIL series springs to mind. Other things to add are Release scheduling/upgrades, Tech support (inc out of hours), Infrastructure management (server/OS ageing - aka Win XP - etc), Disaster recovery, and so on. I once tried to interest a publisher in writing a book "From Code to User" that focused on these non-development aspects of projects (in my old job we called it "Delivery Management"(technical stuff) as opposed to "Project Management"(financial/schedule stuff) ). But the publisher turned it down because the audience was too small... [ I spent 4 years of my life as a "Delivery Manager" for several large projects ( ie. budget >$10m ) before returning to the relative sanity of architecture and design! :-) ] -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] multiprocessing question
On 11/27/2014 04:01 PM, Albert-Jan Roskam wrote: I made a comparison between multiprocessing and threading. In the code below (it's also here: http://pastebin.com/BmbgHtVL, multiprocessing is more than 100 (yes: one hundred) times slower than threading! That is I-must-be-doing-something-wrong-ishly slow. Any idea whether I am doing something wrong? I can't believe the difference is so big. The bulk of the time is spent marshalling the data to the dictionary self.lookup. You can speed it up some by using a list there (it also makes the code much simpler). But the real trick is to communicate less often between the processes. def mp_create_lookup(self): local_lookup = [] lino, record_start = 0, 0 for line in self.data: if not line: break local_lookup.append(record_start) if len(local_lookup) > 100: self.lookup.extend(local_lookup) local_lookup = [] record_start += len(line) print(len(local_lookup)) self.lookup.extend(local_lookup) It's faster because it passes a larger list across the boundary every 100 records, instead of a single value every record. Note that the return statement wasn't ever needed, and you don't need a lino variable. Just use append. I still have to emphasize that record_start is just wrong. You must use ftell() if you're planning to use fseek() on a text file. You can also probably speed the process up a good deal by passing the filename to the other process, rather than opening the file in the original process. That will eliminate sharing the self.data across the process boundary. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
