Re: Question
On 2016-03-08, justin walters wrote: > Well, from that error message, it seems like you may have a permissions > issue. If that's true then it means the Python 3.5 installer is broken. However, I don't think it is true as actually running Python presents no problems. There's some bug in virtualenv which is making it return a permissions error when there is none. > Also, the git shell is not the sane as a real shell. There are > multiple windows terminal emulators. Why not use one of those? I am - the git shell. I wanted to run a Python project that's hosted on github, so I downloaded Python and Git for Windows - why would anyone do otherwise? -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 02:45, Mark Lawrence wrote: On 08/03/2016 01:47, BartC wrote: The Python timing for that file is around 20 seconds, time enough to read 1 copies from the disk. And a C program reads /and decodes/ the same file from the same disk in between 0.1 and 0.2 seconds. So how much of that time is Python startup time, compared to C which is effectively zero? Virtually zero as well. That's if by start-up time you mean how long between invoking Python with the name of the main module, executing all the imports and defs in the main and imported modules, until it starts properly executing code. In the jpeg test, perhaps 50ms. And it would not depend on the size of the input since the start-up routines won't know that. Or are you suggesting that C code is always 100 times faster than Python? This test is one of those where C can clearly do a good job of reducing most of it down to a series of tight loops where everything is in registers. But yes, numeric algorithms are generally going to be a magnitude or two faster in optimised C, compared with CPython executing pure Python code. (But I also remember posting a test in comp.lang.c that was faster in Python than in C. It involved strings and Python had the edge because it used counted strings. A straightforward C version would use zero-terminated strings, although it could be speeded up with a bit more work.) > Of course I'd like to see you write C code 100 times faster than Python, No, it would take longer. But the final result could be run hundreds of times a day by millions of people. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 02:47, MRAB wrote: On 2016-03-08 01:33, BartC wrote: Compared with 2.7, 3.4 above is spending nearly an extra ten seconds doing what? I can't understand why someone just wouldn't care. Part of it will be that Python 2 has 2 integer types: 'int' (fixed length) and 'long' (variable length). Originally, 'int' addition could overflow, but it was more friendly to promote the result to 'long' instead. That was my first thought on reading the thread and was going to post that as the reason for 3 being a little more sluggish. But then I tested Python 2 (2.7.11), and I couldn't get integer arithmetic to overflow at all! -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
BartC writes: > On 08/03/2016 02:47, MRAB wrote: >> On 2016-03-08 01:33, BartC wrote: > >>> Compared with 2.7, 3.4 above is spending nearly an extra ten seconds >>> doing what? I can't understand why someone just wouldn't care. >>> >> Part of it will be that Python 2 has 2 integer types: 'int' (fixed >> length) and 'long' (variable length). >> >> Originally, 'int' addition could overflow, but it was more friendly to >> promote the result to 'long' instead. > > That was my first thought on reading the thread and was going to post > that as the reason for 3 being a little more sluggish. > > But then I tested Python 2 (2.7.11), and I couldn't get integer > arithmetic to overflow at all! It overflows to the long type. Replace 0, 1 and 1 with 0L, 1L and 1L in Python2 version of your while loop test to see the effect. When I tried it this morning, it almost closed the gap between 2.7 and 3.4 on my laptop. (I don't remember the minor versions.) The latter was still slower but not by much. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 02:12, Steven D'Aprano wrote: On Tue, 8 Mar 2016 09:39 am, BartC wrote: I'm using it because this kind of file reading in Python is a mess. If I do a read, will I get a string, a byte sequence object, a byte-array, or array-array, or what? Calling it "a mess" is an exaggeration. There is a change between Python 2 and 3: - in Python 2, reading from a file gives you bytes, that is, the so-called "str" type, not unicode; - in Python 3, reading from a file in binary mode gives you bytes, that is, the "bytes" type; reading in text mode gives you a string, the "str" type. How is this a mess? Python 2 Python 3 Text mode 'str' 'str' Binary mode 'bytes''str' For certain file formats, text mode can't be used because byte values such as 10 could be expanded to the two bytes 13,10 on some systems. So binary needs to be used. But Py2 and Py3 return different results; and indexing a bytes object gives you an int, a str object gives you a str. If you pass a file-handle to a function, that function can't just do a read from that handle without considering whether the file might be in text or binary mode, or whether it's running under Py2 or Py3. And I think someone pointed out the difference between 'bytes', 'bytearray' and 'array.array', but I can't find that post at the minute. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 11:45, Jussi Piitulainen wrote: BartC writes: On 08/03/2016 02:47, MRAB wrote: On 2016-03-08 01:33, BartC wrote: Compared with 2.7, 3.4 above is spending nearly an extra ten seconds doing what? I can't understand why someone just wouldn't care. Part of it will be that Python 2 has 2 integer types: 'int' (fixed length) and 'long' (variable length). Originally, 'int' addition could overflow, but it was more friendly to promote the result to 'long' instead. That was my first thought on reading the thread and was going to post that as the reason for 3 being a little more sluggish. But then I tested Python 2 (2.7.11), and I couldn't get integer arithmetic to overflow at all! It overflows to the long type. So the difference is that Py2 has int and long types, and overflows silently from int to long? (That seems sensible enough; I wonder why they changed it? When int is 64 bits, then probably 99.9% of code will not need anything more.) Replace 0, 1 and 1 with 0L, 1L and 1L in Python2 version of your while loop test to see the effect. When I tried it this morning, it almost closed the gap between 2.7 and 3.4 on my laptop. (I don't remember the minor versions.) The latter was still slower but not by much. I tried that and it was 1 second slower than 3.4! (19 seconds compared with 18 seconds.) That would at first sight offer a plausible explanation for the discrepancy (which IMO would be too big a cost for the minor benefit of a single integer type). And maybe this particular combination of byte-codes just happened to show up the difference more dramatically. But then, 3.1 took only 12.5 seconds; same byte-codes, same unified 'long' type for integers... -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08.03.16 04:12, Steven D'Aprano wrote:
[steve@ando ~]$ python3.3 -m timeit -s 'from fp import
Float' 'Float("1234.567")'
10 loops, best of 3: 13.6 usec per loop
[steve@ando ~]$ python/python-dev/3.5/python -m timeit -s 'from fp import
Float' 'Float("1234.567")'
1 loops, best of 3: 54 usec per loop
What about 3.5? That's four times slower than 3.3? What happened there?
This is interesting question. On my computer (32-bit) all versions from
3.3 to 3.6 have the same performance in this microbenchmark.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08.03.16 03:34, Steven D'Aprano wrote: Chris, I think that's exactly what BartC has done: he has a program that he actually uses, one which processes JPG files. It's written in pure Python, so he's not comparing the quality of C libraries, he's comparing Python versus Python. I haven't looked at his code, so I don't know if its a fair comparison of Python versus Bart's Private Language. But that's not the point. Idiomatic Python or not, "best practice" production level code or not, it is a fair comparison of Python 2 versus Python 3, or CPython versus PyPy, because it's the same Python code being run each time. Comparing the quality of programmers by the time for which they ran a sprint is not fair. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
On 03/07/2016 11:51 PM, Fillmore wrote:
learning Python from Perl here. Want to do things as Pythonicly as
possible.
I am reading a TSV, but need to skip the first 5 lines. The following
works, but wonder if there's a more pythonc way to do things. Thanks
ctr = 0
with open(prfile,mode="rt",encoding='utf-8') as pfile:
for line in pfile:
ctr += 1
if ctr < 5:
continue
allVals = line.strip().split("\t")
print(allVals)
what about a generator expression ? The (not so)new hype:
with open(prfile,mode="rt",encoding='utf-8') as pfile:
for values in (l.strip().split("\t") for (i, l) in enumerate(pfile)
if i >=5):
print values
slightly dense, could be better with a lambda function
tovalues = lambda l: l.strip().split("\t")
with open(prfile,mode="rt",encoding='utf-8') as pfile:
for values in (tovalues(l) for (i, l) in enumerate(pfile) if i >=5):
print values
This should even work quite efficiently on big files, because I don't
thing no more than one line is in memory at a given time.
jm
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 04:40, Steven D'Aprano wrote:
On Tuesday 08 March 2016 12:41, BartC wrote:
On 08/03/2016 01:19, Steven D'Aprano wrote:
On Tue, 8 Mar 2016 07:19 am, BartC wrote:
I don't have to hand a jpeg file that it can't
decode.
Run this under Python 2:
from random import randint
blob = [randint(0, 256) for i in range(16*1024)]
with open('broken.jpg', 'wb') as f:
f.write(''.join(blob))
If your software can decode that, it will be a miracle.
That's not a jpeg file.
Nevertheless, in the real world, you have to deal with corrupt JPGs and
files mislabelled as JPGs. And "crashing" doesn't count as "deal with" :-)
OK, I changed the 'raise' line to 'exit(0)'. Job done!
(I couldn't recreate the problem easily because the common failures
didn't call abortjpeg() at all.)
It doesn't make Py3 any faster than Py2 though...
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Path problem with 3.5.1
Although I've enabled setting the path when installing 3.5.1 (Win7 x64) I can't run Python from the command line in a terminal window. It works OK on a Raspberry Pi 3! Leon -- Leon Heller G1HSM -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 11:09, BartC wrote: On 08/03/2016 02:45, Mark Lawrence wrote: On 08/03/2016 01:47, BartC wrote: The Python timing for that file is around 20 seconds, time enough to read 1 copies from the disk. And a C program reads /and decodes/ the same file from the same disk in between 0.1 and 0.2 seconds. So how much of that time is Python startup time, compared to C which is effectively zero? Virtually zero as well. Nonsense. That's if by start-up time you mean how long between invoking Python with the name of the main module, executing all the imports and defs in the main and imported modules, until it starts properly executing code. In the jpeg test, perhaps 50ms. And it would not depend on the size of the input since the start-up routines won't know that. How did you obtain that figure? Or are you suggesting that C code is always 100 times faster than Python? This test is one of those where C can clearly do a good job of reducing most of it down to a series of tight loops where everything is in registers. But yes, numeric algorithms are generally going to be a magnitude or two faster in optimised C, compared with CPython executing pure Python code. (But I also remember posting a test in comp.lang.c that was faster in Python than in C. It involved strings and Python had the edge because it used counted strings. A straightforward C version would use zero-terminated strings, although it could be speeded up with a bit more work.) > Of course I'd like to see you write C code 100 times faster than Python, No, it would take longer. But the final result could be run hundreds of times a day by millions of people. Which also happens with Python, although I expect that many users aren't aware of that fact. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 13:49, BartC wrote: On 08/03/2016 04:40, Steven D'Aprano wrote: On Tuesday 08 March 2016 12:41, BartC wrote: Nevertheless, in the real world, you have to deal with corrupt JPGs and files mislabelled as JPGs. And "crashing" doesn't count as "deal with" :-) OK, I changed the 'raise' line to 'exit(0)'. Job done! (I couldn't recreate the problem easily because the common failures didn't call abortjpeg() at all.) It doesn't make Py3 any faster than Py2 though... So what, if 99% of people couldn't care less about that fact? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Tiny python code I dont understand
Hello lst = [(1,2,3), (4, 5,6)] sum(lst, ()) (1, 2, 3, 4, 5, 6) Any explanations ? thx -- https://mail.python.org/mailman/listinfo/python-list
Re: Tiny python code I dont understand
ast writes: > Hello > lst = [(1,2,3), (4, 5,6)] sum(lst, ()) > (1, 2, 3, 4, 5, 6) > > Any explanations ? (() + (1,2,3)) + (4,5,6) -- https://mail.python.org/mailman/listinfo/python-list
Re: Tiny python code I dont understand
"Jussi Piitulainen" a écrit dans le message de news:[email protected]... ast writes: Hello lst = [(1,2,3), (4, 5,6)] sum(lst, ()) (1, 2, 3, 4, 5, 6) Any explanations ? (() + (1,2,3)) + (4,5,6) yes, ty -- https://mail.python.org/mailman/listinfo/python-list
Re: Tiny python code I dont understand
"ast" a écrit dans le message de news:[email protected]... Hello lst = [(1,2,3), (4, 5,6)] sum(lst, ()) (1, 2, 3, 4, 5, 6) Any explanations ? thx OK, sorry, I finally understand. This is because adding tuple (or list or string ...) is a concatenation () + (1,2,3) + (4, 5,6) (1,2,3,4,5,6) -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
My apologies, but I really don't want to argue with you. You may want to be a bit more humble when asking for help. On Mar 8, 2016 2:31 AM, "Jon Ribbens" wrote: > On 2016-03-08, justin walters wrote: > > Well, from that error message, it seems like you may have a permissions > > issue. > > If that's true then it means the Python 3.5 installer is broken. > However, I don't think it is true as actually running Python presents > no problems. There's some bug in virtualenv which is making it return > a permissions error when there is none. > > > Also, the git shell is not the sane as a real shell. There are > > multiple windows terminal emulators. Why not use one of those? > > I am - the git shell. > > I wanted to run a Python project that's hosted on github, > so I downloaded Python and Git for Windows - why would > anyone do otherwise? > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
Correct me if I'm wrong, but don't python generators usually use the yield
statement so they can be used in list comprehensions?
On Mar 8, 2016 5:27 AM, "jmp" wrote:
> On 03/07/2016 11:51 PM, Fillmore wrote:
>
>>
>> learning Python from Perl here. Want to do things as Pythonicly as
>> possible.
>>
>> I am reading a TSV, but need to skip the first 5 lines. The following
>> works, but wonder if there's a more pythonc way to do things. Thanks
>>
>> ctr = 0
>> with open(prfile,mode="rt",encoding='utf-8') as pfile:
>> for line in pfile:
>> ctr += 1
>>
>> if ctr < 5:
>> continue
>>
>> allVals = line.strip().split("\t")
>> print(allVals)
>>
>
> what about a generator expression ? The (not so)new hype:
>
> with open(prfile,mode="rt",encoding='utf-8') as pfile:
> for values in (l.strip().split("\t") for (i, l) in enumerate(pfile) if i
> >=5):
> print values
>
> slightly dense, could be better with a lambda function
>
> tovalues = lambda l: l.strip().split("\t")
> with open(prfile,mode="rt",encoding='utf-8') as pfile:
> for values in (tovalues(l) for (i, l) in enumerate(pfile) if i >=5):
> print values
>
>
> This should even work quite efficiently on big files, because I don't
> thing no more than one line is in memory at a given time.
>
> jm
>
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question
On 2016-03-08, justin walters wrote: > My apologies, but I really don't want to argue with you. That's nice to know. I didn't ask you to, but it's great to have your input anyway. > You may want to be a bit more humble when asking for help. I wasn't asking for help. HTH. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
On 08/03/2016 16:49, justin walters wrote: Correct me if I'm wrong, but don't python generators usually use the yield statement so they can be used in list comprehensions? Please don't top post on this list, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
On 03/08/2016 05:49 PM, justin walters wrote: Correct me if I'm wrong, but don't python generators usually use the yield statement so they can be used in list comprehensions? Hello, Please don't top post. Generator expressions are different from generator functions. They are both generators but use a different syntax. generator function: def func(iterable): for i in iterable: yield i the generator expression version would be (i for i in iterable) Both have their use cases but everytime you can actually use an generator expression, it's probably the correct thing to do. Cheers, jm -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
Sorry about the top posting. I'm new to mailing lists. I should just reply to the [email protected] address then? Also, thank you for the generator clarification. On Mar 8, 2016 9:09 AM, "jmp" wrote: > On 03/08/2016 05:49 PM, justin walters wrote: > >> Correct me if I'm wrong, but don't python generators usually use the yield >> statement so they can be used in list comprehensions? >> > > Hello, > > Please don't top post. > > Generator expressions are different from generator functions. They are > both generators but use a different syntax. > > generator function: > > def func(iterable): > for i in iterable: yield i > > > the generator expression version would be > > (i for i in iterable) > > > Both have their use cases but everytime you can actually use an generator > expression, it's probably the correct thing to do. > > Cheers, > > jm > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
On Mon, Mar 7, 2016 at 6:41 PM, Jon Ribbens wrote: > On 2016-03-07, Ian Kelly wrote: >> On Mon, Mar 7, 2016 at 11:51 AM, Jon Ribbens >> wrote: >>> I must say that Python on Windows was a very poor experience indeed, >>> "virtualenv" does not work and "venv" refuses to create the 'activate' >>> shell script so does not work either >> >> I've used both of these on Windows (although not recently) and had no >> trouble with them. I never had a problem with venv not creating the >> activate.bat file. > > It's not activate.bat, it's activate (no file extension) the posix > shell script. I installed Git for Windows which provides bash (or > something that looks like it). Python venv doesn't cope with this > situation at all. Well, running bash on Windows is decidedly non-standard. This is like installing a Python package on a Linux system and then complaining that it won't run under wine. I don't think that Python should be expected to provide an activate script for all possible shells the user might conceivably want to use. > 'virtualenv' works even less well, it just says: > > $ virtualenv test > Using base prefix 'd:\\program files (x86)\\python35-32' > New python executable in D:\Users\Jon > Ribbens\Documents\Python\test\Scripts\python.exe > ERROR: The executable "D:\Users\Jon > Ribbens\Documents\Python\test\Scripts\python.exe" could not be run: [WinError > 5] Access is denied Ah, I probably never tried using it inside a user dir. On Windows I typically do development in a path close to the drive root, e.g. C:\dev. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic love
justin walters writes: > Sorry about the top posting. I'm new to mailing lists. I should just reply > to the [email protected] address then? > > Also, thank you for the generator clarification. > On Mar 8, 2016 9:09 AM, "jmp" wrote: ^ that's top posting always reply inline or at the bottom -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
On 2016-03-08, Ian Kelly wrote: > On Mon, Mar 7, 2016 at 6:41 PM, Jon Ribbens > wrote: >> It's not activate.bat, it's activate (no file extension) the posix >> shell script. I installed Git for Windows which provides bash (or >> something that looks like it). Python venv doesn't cope with this >> situation at all. > > Well, running bash on Windows is decidedly non-standard. This is like > installing a Python package on a Linux system and then complaining > that it won't run under wine. I would agree with this, except that git seems to think it *is* standard, and git itself seems pretty standard these days, so if git thinks it's standard then disagreeing with them seems a bit pointless. > I don't think that Python should be expected to provide an activate > script for all possible shells the user might conceivably want to > use. I would agree with this too, except that the shell script clearly is already implemented for Unix purposes and would do no harm if written to the directory, so presumably all 'venv' needs to do is to stop preventing itself doing something it already knows how to do. >> $ virtualenv test >> Using base prefix 'd:\\program files (x86)\\python35-32' >> New python executable in D:\Users\Jon >> Ribbens\Documents\Python\test\Scripts\python.exe >> ERROR: The executable "D:\Users\Jon >> Ribbens\Documents\Python\test\Scripts\python.exe" could not be run: >> [WinError 5] Access is denied > > Ah, I probably never tried using it inside a user dir. On Windows I > typically do development in a path close to the drive root, e.g. > C:\dev. The only things I can think of that are at all 'weird' are that there are spaces in the filenames, and there's more than one drive. But the former of those is utterly standard for Windows, and the latter doesn't really even rise to the level of 'uncommon', let alone truly unusual. If I'm seeing these problems then thousands of other people must have already seen them before me. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
On Tue, Mar 8, 2016 at 10:56 AM, Jon Ribbens wrote: > The only things I can think of that are at all 'weird' are that there > are spaces in the filenames, and there's more than one drive. But > the former of those is utterly standard for Windows, and the latter > doesn't really even rise to the level of 'uncommon', let alone truly > unusual. If I'm seeing these problems then thousands of other people > must have already seen them before me. The other difference is that files and folders inside Windows user dirs get created with implicit permissions settings, and the issue does appear to be permissions-related. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
On 2016-03-08, Ian Kelly wrote: > On Tue, Mar 8, 2016 at 10:56 AM, Jon Ribbens > wrote: >> The only things I can think of that are at all 'weird' are that there >> are spaces in the filenames, and there's more than one drive. But >> the former of those is utterly standard for Windows, and the latter >> doesn't really even rise to the level of 'uncommon', let alone truly >> unusual. If I'm seeing these problems then thousands of other people >> must have already seen them before me. > > The other difference is that files and folders inside Windows user > dirs get created with implicit permissions settings, and the issue > does appear to be permissions-related. That's interesting, maybe that's the reason. It would certainly mean that almost everyone who uses virtualenv with Python on Windows would come up against this problem though - I mean, keeping your user files under your own user home directory is hardly weird. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 16:15, Mark Lawrence wrote: On 08/03/2016 13:49, BartC wrote: On 08/03/2016 04:40, Steven D'Aprano wrote: On Tuesday 08 March 2016 12:41, BartC wrote: Nevertheless, in the real world, you have to deal with corrupt JPGs and files mislabelled as JPGs. And "crashing" doesn't count as "deal with" :-) OK, I changed the 'raise' line to 'exit(0)'. Job done! (I couldn't recreate the problem easily because the common failures didn't call abortjpeg() at all.) It doesn't make Py3 any faster than Py2 though... So what, if 99% of people couldn't care less about that fact? Um, because that's what the thread is about? -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 16:09, Mark Lawrence wrote: On 08/03/2016 11:09, BartC wrote: On 08/03/2016 02:45, Mark Lawrence wrote: On 08/03/2016 01:47, BartC wrote: The Python timing for that file is around 20 seconds, time enough to read 1 copies from the disk. And a C program reads /and decodes/ the same file from the same disk in between 0.1 and 0.2 seconds. So how much of that time is Python startup time, compared to C which is effectively zero? Virtually zero as well. Nonsense. That's if by start-up time you mean how long between invoking Python with the name of the main module, executing all the imports and defs in the main and imported modules, until it starts properly executing code. In the jpeg test, perhaps 50ms. And it would not depend on the size of the input since the start-up routines won't know that. How did you obtain that figure? By printing a message followed by exit(0) just before the program was about to start doing its thing. Then I timed that using: $time python program.py in Linux. But this was hardly necessary as it was so obvious: it takes 150ms to process a 300-pixel image, 20 seconds for a 2Mpixel one, and (I have to switch to PyPy here as I've never had time to hang about for it) 180 seconds for 80Mpixel file. Surely the start-up time would be the same no matter what the input. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
EuroPython 2016: Financial Aid Available
We are happy to announce a program for people in need of financial aid to attend EuroPython. You can find all the details on our financial aid page: *** https://ep2016.europython.eu/en/registration/financial-aid/ *** Financial Aid Program In short, we will be giving out grants in three categories: * Free tickets * Travel costs * Accommodation Anyone who wants to attend EuroPython 2016 can apply, including people who have already purchased tickets. We want to make the event affordable for as many people as possible. Financial Aid Sponsor - Financial aid is sponsored in part by: The Python Software Foundation (PSF) *** http://python.org/psf/ *** Looking for financial aid sponsors -- We are still looking for sponsors to increase the budget we have available for financial aid. If your company would like to sign up as financial aid sponsor, please contact the sponsors team at [email protected]. With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 19:15, BartC wrote: On 08/03/2016 16:09, Mark Lawrence wrote: On 08/03/2016 11:09, BartC wrote: On 08/03/2016 02:45, Mark Lawrence wrote: On 08/03/2016 01:47, BartC wrote: The Python timing for that file is around 20 seconds, time enough to read 1 copies from the disk. And a C program reads /and decodes/ the same file from the same disk in between 0.1 and 0.2 seconds. So how much of that time is Python startup time, compared to C which is effectively zero? Virtually zero as well. Nonsense. That's if by start-up time you mean how long between invoking Python with the name of the main module, executing all the imports and defs in the main and imported modules, until it starts properly executing code. In the jpeg test, perhaps 50ms. And it would not depend on the size of the input since the start-up routines won't know that. How did you obtain that figure? By printing a message followed by exit(0) just before the program was about to start doing its thing. Then I timed that using: $time python program.py in Linux. But this was hardly necessary as it was so obvious: it takes 150ms to process a 300-pixel image, 20 seconds for a 2Mpixel one, and (I have to switch to PyPy here as I've never had time to hang about for it) 180 seconds for 80Mpixel file. Surely the start-up time would be the same no matter what the input. Unless all of the background jobs are kicking in when you're testing. I assume that you do take such factors into account, by repeating your tests and taking an average, or perhaps even a worst case figure. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Installing ssdeep on Portable Python
Were you able to solve this problem? I am also seeing this On Thursday, March 20, 2014 at 2:22:19 PM UTC-4, [email protected] wrote: > Portable Python 2.7 for Windows, the Python application have dependency on > ssdeep-2.10, which is a binary exe. > > The ssdeep (libfuzzy) installation example was shown for Linux platform only: > > a) libfuzzy can be installed via apt-get: > > $ sudo apt-get install libfuzzy2 > > b) to install libfuzzy from source, download the gzipped tarball from > http://ssdeep.sourceforge.net/#download, then run: > > $ tar -zxvf ssdeep-2.10.tar.gz > $ cd ssdeep-2.10 && ./configure && make && sudo make install > - > I need install it on PortablePython for Windows, so it's not clear how to > make this: where should be placed ssdeep Windows binary files, that Python > application can found it? -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 20:44, Mark Lawrence wrote: On 08/03/2016 19:15, BartC wrote: Surely the start-up time would be the same no matter what the input. Unless all of the background jobs are kicking in when you're testing. I assume that you do take such factors into account, by repeating your tests and taking an average, or perhaps even a worst case figure. Really, that is not necessary, although I've also lost track of the point you're making. Here's a new, fuller set of figures for a 24Mpixel image, with a new version of PyPy that I've found: Load from disk (4MB) and decode into memory (70MB), all using the same algorithm and the Python code using pure Python: PyPy 2.7.10 17.0 seconds (PyPy 4.0.1 32-bit) PyPy 3.2.5 23.5 (PyPy 2.4.0 32-bit) Python 2.7.11 274 (All CPython on Windows) Python 3.1.2280 Python 3.4.3352 (C1.5 gcc -O3) (Bart's 26.5 my bytecode language) That Python3 (3.4 version) is still insisting on being 30% slower than Python2! I doubt it's just doing an extra 80 seconds' worth of start-up code. But looking at the bigger picture, it's also 2000% slower than PyPy, as well as 23000% slower than C. From that point of view, I can see now why some would shrug at the relatively minor differences between versions of CPython. Still, if do you have to use CPython for this task (or any task with this scale of runtime), it means hanging about for more than an minute extra with Python 3.4. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Tue, 8 Mar 2016 10:53 pm, BartC wrote:
> On 08/03/2016 02:12, Steven D'Aprano wrote:
>> On Tue, 8 Mar 2016 09:39 am, BartC wrote:
>
>>> I'm using it because this kind of file reading in Python is a mess. If I
>>> do a read, will I get a string, a byte sequence object, a byte-array, or
>>> array-array, or what?
>>
>> Calling it "a mess" is an exaggeration. There is a change between Python
>> 2 and 3:
>>
>> - in Python 2, reading from a file gives you bytes, that is, the
>> so-called "str" type, not unicode;
>>
>> - in Python 3, reading from a file in binary mode gives you bytes, that
>> is, the "bytes" type; reading in text mode gives you a string, the "str"
>> type.
>>
>> How is this a mess?
>
>Python 2 Python 3
>
> Text mode 'str' 'str'
> Binary mode 'bytes''str'
That table is incorrect. In Python 2, bytes is just an alias for str:
[steve@ando ~]$ python2.7 -c "print bytes is str"
True
and reading from a file returns a result depending on whether it is opened
in text or binary mode:
[steve@ando ~]$ echo foo > /tmp/junk
[steve@ando ~]$ python2.7 -c "print type(open('/tmp/junk', 'r').read())"
[steve@ando ~]$ python2.7 -c "print type(open('/tmp/junk', 'rb').read())"
In Python 3, you get bytes in binary mode, and Unicode strings (known
as "str") in text mode:
[steve@ando ~]$ python3 -c "print(type(open('/tmp/junk', 'r').read()))"
[steve@ando ~]$ python3 -c "print(type(open('/tmp/junk', 'rb').read()))"
Here it is in a table form:
Mode Python 2 Python 3
Text bytes text
Binarybytes bytes
And here is a table summarising the name changes around text and byte
strings:
-
Type Python 2 Python 3
-
Byte strings str/bytes bytes
Text strings unicodestr
-
Remember that when Python first named its string type, Unicode didn't even
exist and there was no concept of strings being anything but a pseudo-ASCII
byte string.
> For certain file formats, text mode can't be used because byte values
> such as 10 could be expanded to the two bytes 13,10 on some systems.
If it is a *text* file, then it shouldn't matter whether your lines end with
\r, \n or \r\n. Indeed, in text mode, Python will automatically concert all
of these to \n on reading. (For writing, it will write what you tell it to
write.)
Python will never expand \n to \r\n. But it may translate \r\n to \n.
> So binary needs to be used. But Py2 and Py3 return different results;
> and indexing a bytes object gives you an int, a str object gives you a
> str.
It is true, and unfortunate, that indexing byte objects give ints instead of
a byte string of length 1:
[steve@ando ~]$ python3 -c "print(b'Aardvark'[0])"
65
In hindsight, it was a mistake, but as we are now in Python 3.5, breaking
backwards compatibility in the 3.x series means we are stuck with it
without some pain. Fortunately is it easy enough to work around: take a
one-element slice.
[steve@ando ~]$ python3 -c "print(b'Aardvark'[0:1])"
b'A'
> If you pass a file-handle to a function, that function can't just do a
> read from that handle without considering whether the file might be in
> text or binary mode, or whether it's running under Py2 or Py3.
Why not? file.read() works exactly the same in all four cases.
It's hard to argue against your statement when I don't understand what
problem you think you have. A concrete example of this "mess" would help.
> And I think someone pointed out the difference between 'bytes',
> 'bytearray' and 'array.array', but I can't find that post at the minute.
Er, yes? Of course there is a differences between those three types. There's
also a difference between list, set and dict. What's your point?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question
On Wed, 9 Mar 2016 04:19 am, Ian Kelly wrote: > On Mon, Mar 7, 2016 at 6:41 PM, Jon Ribbens > wrote: >> On 2016-03-07, Ian Kelly wrote: >>> On Mon, Mar 7, 2016 at 11:51 AM, Jon Ribbens >>> wrote: I must say that Python on Windows was a very poor experience indeed, Indeed. Development on Windows platforms is, sadly, a second-class experience in general compared to POSIX-compatible platforms, and Python is no different. That's not to say that you can't do good work on Windows, but it is harder and most costly. "virtualenv" does not work and "venv" refuses to create the 'activate' shell script so does not work either >>> >>> I've used both of these on Windows (although not recently) and had no >>> trouble with them. I never had a problem with venv not creating the >>> activate.bat file. >> >> It's not activate.bat, it's activate (no file extension) the posix >> shell script. I installed Git for Windows which provides bash (or >> something that looks like it). Python venv doesn't cope with this >> situation at all. > > Well, running bash on Windows is decidedly non-standard. This is like > installing a Python package on a Linux system and then complaining > that it won't run under wine. I don't think that Python should be > expected to provide an activate script for all possible shells the > user might conceivably want to use. Not "all possible shells", no. But it's not unreasonable for it to handle the single most popular operating system environment in the world, Windows, don't you think? I'm sure that if somebody cared enough to report this on the bug tracker, it would be fixed. Steve Dower works for Microsoft, and he's doing what he can to improve the Python-on-Windows experience. (Microsoft actually do consider Python to be an important, if minor, part of their development ecosystem.) >> 'virtualenv' works even less well, it just says: >> >> $ virtualenv test >> Using base prefix 'd:\\program files (x86)\\python35-32' >> New python executable in D:\Users\Jon >> Ribbens\Documents\Python\test\Scripts\python.exe ERROR: The executable >> "D:\Users\Jon Ribbens\Documents\Python\test\Scripts\python.exe" could not >> be run: [WinError 5] Access is denied > > Ah, I probably never tried using it inside a user dir. On Windows I > typically do development in a path close to the drive root, e.g. > C:\dev. Am I missing something? It looks to me like a straight forward permissions error? I don't know how difficult that is to solve on Windows, but I don't think it has anything to do with the path itself, only the permissions of the path. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, 9 Mar 2016 06:15 am, BartC wrote: [...] > But this was hardly necessary as it was so obvious: it takes 150ms to > process a 300-pixel image, 20 seconds for a 2Mpixel one, and (I have to > switch to PyPy here as I've never had time to hang about for it) 180 > seconds for 80Mpixel file. > > Surely the start-up time would be the same no matter what the input. I've been trying to follow this thread, but I'm completely lost as to why we're discussing startup time. Mark seems to think that it's completely irrelevant, but that's surely wrong. Startup time might be irrelevant for long-running processes (say, a tool that runs for minutes or hours processing millions of files, or a server that is expected to stay running for weeks at a time) but it has a very large effect on the performance of quick-running command line tools. For what it's worth, the core developers are aware that CPython's startup time has been increasing, and that it is large enough to impact the feeling of snappiness and responsiveness for command line tools. They are actively working on this to keep it as low as possible. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, 9 Mar 2016 12:49 am, BartC wrote:
>> Nevertheless, in the real world, you have to deal with corrupt JPGs and
>> files mislabelled as JPGs. And "crashing" doesn't count as "deal with"
>> :-)
>
> OK, I changed the 'raise' line to 'exit(0)'. Job done!
Possibly a really amateurish, lazy job, but still done.
So now when you can't process a file, you silently exit, giving no
indication that there was a problem or what the problem was. You suppress
the helpful traceback, making sure that it is as difficult as possible for
the user to diagnose the problem. You even exit with an exit code of 0
("success") in order to confuse any scripts they use. Brilliant! I love
helpful tools like that!
How many years did you say you have been programming?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question
On Wed, Mar 9, 2016 at 10:52 AM, Steven D'Aprano wrote: >> Well, running bash on Windows is decidedly non-standard. This is like >> installing a Python package on a Linux system and then complaining >> that it won't run under wine. I don't think that Python should be >> expected to provide an activate script for all possible shells the >> user might conceivably want to use. > > Not "all possible shells", no. But it's not unreasonable for it to handle > the single most popular operating system environment in the world, Windows, > don't you think? I'm not sure that the issue is "Windows can't use venv", but "Windows with Git Bash can't use venv". Windows has a number of shells available; the default one is pretty terrible but does kinda work, and then there's PowerShell, and ports of other shells like bash. Cygwin provides its own shell (which I think is bash), and I'm not sure if that's the same as Git Bash installs. And then there's the difference between the shell (the command interpreter) and the, for want of a better name, terminal emulator (the thing that displays stuff on the screen). Working purely within cmd.exe and the default terminal emulator, I was able to "py -m venv env" and then "env\scripts\activate" (note, *not* env/bin/activate which is what I'm used to - no idea why). It seemed to work. Working instead in Git Bash, though, leaves me unable to activate, because there is no bash script for venv activation. Hence, the problem is "supporting all possible shells" (which is an enormous challenge), rather than "supporting one of the three most popular operating systems" (which, I agree, is well worth doing). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 08/03/2016 23:28, Steven D'Aprano wrote:
On Tue, 8 Mar 2016 10:53 pm, BartC wrote:
Python 2 Python 3
Text mode 'str' 'str'
Binary mode 'bytes''str'
That table is incorrect. In Python 2, bytes is just an alias for str:
[steve@ando ~]$ python2.7 -c "print bytes is str"
True
and reading from a file returns a result depending on whether it is opened
in text or binary mode:
[steve@ando ~]$ echo foo > /tmp/junk
[steve@ando ~]$ python2.7 -c "print type(open('/tmp/junk', 'r').read())"
[steve@ando ~]$ python2.7 -c "print type(open('/tmp/junk', 'rb').read())"
In Python 3, you get bytes in binary mode, and Unicode strings (known
as "str") in text mode:
[steve@ando ~]$ python3 -c "print(type(open('/tmp/junk', 'r').read()))"
[steve@ando ~]$ python3 -c "print(type(open('/tmp/junk', 'rb').read()))"
Here it is in a table form:
Mode Python 2 Python 3
Text bytes text
Binarybytes bytes
That doesn't correspond to your results above. Going with the reported
type of what is read, it would be:
Mode Python 2 Python 3
Text str str
Binarystr bytes
Notice the odd-one-out is now the bottom left, not the the top right.
Python will never expand \n to \r\n. But it may translate \r\n to \n.
Well, OK, you might lose bytes with the value 13. That's also bad.
If you pass a file-handle to a function, that function can't just do a
read from that handle without considering whether the file might be in
text or binary mode, or whether it's running under Py2 or Py3.
Why not? file.read() works exactly the same in all four cases.
Didn't you just demonstrate that it can give different results?
It's hard to argue against your statement when I don't understand what
problem you think you have. A concrete example of this "mess" would help.
And I think someone pointed out the difference between 'bytes',
'bytearray' and 'array.array', but I can't find that post at the minute.
Er, yes? Of course there is a differences between those three types. There's
also a difference between list, set and dict. What's your point?
OK, I take it back; if it's not a mess, then it's just confusing!
If someone asked me when I'd use a byte sequence, and when I'd use a
bytearray object, then I wouldn't have a clue.
(Looking it up, it appears than one is mutable, and the other isn't.
This isn't quite the same as the difference between a list and a dict.)
As for array.arrays, those apparently can also be arrays of bytes.
Presumably, mutable. OK. Definitely nothing here that could be tidied up...
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, Mar 9, 2016 at 11:09 AM, BartC wrote: > On 08/03/2016 23:28, Steven D'Aprano wrote: >> >> Here it is in a table form: >> >> >> Mode Python 2 Python 3 >> >> Text bytes text >> Binarybytes bytes >> > > > That doesn't correspond to your results above. Going with the reported type > of what is read, it would be: > > > Mode Python 2 Python 3 > > Text str str > Binarystr bytes > > > Notice the odd-one-out is now the bottom left, not the the top right. Please, please, PLEASE get an understanding of the difference between text and bytes. The fact that the byte string in Py2 happens to have the same name as the text string in Py3 should be considered a *coincidence* when you're discussing these differences. Whenever you open a file in binary mode, you get back bytes. In Python 2, opening a file in text mode still gives back bytes, which is anomalous. Python 3 handles this properly: when you open a file in text mode, you specify an encoding, and it is decoded to Unicode text. >> Python will never expand \n to \r\n. But it may translate \r\n to \n. > > Well, OK, you might lose bytes with the value 13. That's also bad. If that bothers you, open the file in binary mode. Then you get a stream of bytes. In text mode, you're declaring that you expect the file to be a stream of lines. Lines might be ended by \r\n or \n, and you usually don't care. (Sometimes you do. Often you don't.) > OK, I take it back; if it's not a mess, then it's just confusing! > > If someone asked me when I'd use a byte sequence, and when I'd use a > bytearray object, then I wouldn't have a clue. > > (Looking it up, it appears than one is mutable, and the other isn't. This > isn't quite the same as the difference between a list and a dict.) > > As for array.arrays, those apparently can also be arrays of bytes. > Presumably, mutable. OK. Definitely nothing here that could be tidied up... It's no difference from the question of when you use a tuple and when you use a list. They're very similar object types, with nearly the same methods and operations available. One's mutable and one's not. When should you use a tuple? Is it just a frozenlist? These are the sorts of questions that keep coming up on this list, and that's not a problem. They're coding design questions. When should you use an float, when a fractions.Fraction, when decimal.Decimal, etc? The fact that people have to ask does not mean the language should lose some of them. If someone asked me when I should use Windows Services and when I should use the Scheduler, I wouldn't know. If ever the situation comes up, I'd probably have to ask. That's not a problem; that's what mailing lists are for. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 03/07/2016 05:45 PM, Chris Angelico wrote:
> On Tue, Mar 8, 2016 at 11:22 AM, BartC wrote:
>>
>> (Is a byte string the same as a byte array? Is a byte array the same as an
>> array.array? If I remove this line from my code, where 'data' has just been
>> read from a file:
>>
>>data=array.array('B',data)
>>
>> then it still works - Python 3. But not on Python 2. If I do .read on a
>> binary file I get a byte string in Python 3, but a string in Python 2. That
>> sort of mess.
>
> The default string in Py2 *is* a byte string.
There are some interesting differences I found between a Python 2 string
(composed of bytes) and a Python 3 byte string, such as what you'd get
from calling read() on a file handle opened in binary mode. That is in
Python 2, indexing a string returns a string of length 1. In Python
3.5, indexing a byte string returns a value, the equivalent of calling
ord() on the single byte string. This makes it a bit difficult to make
the code easily work between Python 2 and 3 and handle bytes. Any ideas
there?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, Mar 9, 2016 at 11:34 AM, Michael Torrie wrote:
> On 03/07/2016 05:45 PM, Chris Angelico wrote:
>> On Tue, Mar 8, 2016 at 11:22 AM, BartC wrote:
>>>
>>> (Is a byte string the same as a byte array? Is a byte array the same as an
>>> array.array? If I remove this line from my code, where 'data' has just been
>>> read from a file:
>>>
>>>data=array.array('B',data)
>>>
>>> then it still works - Python 3. But not on Python 2. If I do .read on a
>>> binary file I get a byte string in Python 3, but a string in Python 2. That
>>> sort of mess.
>>
>> The default string in Py2 *is* a byte string.
>
> There are some interesting differences I found between a Python 2 string
> (composed of bytes) and a Python 3 byte string, such as what you'd get
> from calling read() on a file handle opened in binary mode. That is in
> Python 2, indexing a string returns a string of length 1. In Python
> 3.5, indexing a byte string returns a value, the equivalent of calling
> ord() on the single byte string. This makes it a bit difficult to make
> the code easily work between Python 2 and 3 and handle bytes. Any ideas
> there?
As Steven commented, this is probably a design flaw in the Py3 bytes
type, but it can't be changed now. For perfect compatibility, just
slice:
>>> b"asdf"[2:3] # Py2
'd'
>>> b"asdf"[2:3] # Py3
b'd'
Works in both versions, with the only difference being the repr.
It's a bit clunky, but it does work.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 03/07/2016 11:34 AM, BartC wrote: > (I'm quite pleased with my version: smaller, faster, works on all the > Pythons, supports all 3 colour formats and no decoding bugs that I'm > aware of, and it's the first Python program I've written that does > something useful.) I think you should be commended for writing it. It may not be perfect, nor is it necessarily "Pythonic" but you had fun doing it. I hope you aren't discouraged by the criticism of your code. I hacked on it a bit, and it only took about a minute to change from using arrays to immutable byte strings. The only problem with that was what I noted in my other reply. That is that indexing a Python 2 string gives a different result than indexing a Python 3 byte string. I'm not sure what kind of logic should be used to efficiently judge between Python 2 and 3, and modify the answer accordingly. In Python 2 I can simply do "ord(fs.data[fs.pos])". In Python 3 it's just "fs.data[fs.pos]". I'm not sure what kind of speed up using a normal string provides vs the array. But it has to be more a bit faster. I suppose if others want to direct you in a more Pythonic path, they could take your code and turn it into something that is more idiomatic (and perhaps faster). I may take a crack at it. If it were me I think I'd make your jpeg decoder into it's own class, and have it operate on any iterable input, be it an open file or something else. That may be actually slower though, but more pythonic. -- https://mail.python.org/mailman/listinfo/python-list
Re: Path problem with 3.5.1
On 3/8/2016 8:47 AM, leon_heller--- via Python-list wrote: Although I've enabled setting the path when installing 3.5.1 (Win7 x64) I can't run Python from the command line in a terminal window. It works OK on a Raspberry Pi 3! If you type 'PATH' at the command prompt, what is the response? -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Question
On Tue, Mar 8, 2016 at 5:13 PM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 10:52 AM, Steven D'Aprano wrote: >>> Well, running bash on Windows is decidedly non-standard. This is like >>> installing a Python package on a Linux system and then complaining >>> that it won't run under wine. I don't think that Python should be >>> expected to provide an activate script for all possible shells the >>> user might conceivably want to use. >> >> Not "all possible shells", no. But it's not unreasonable for it to handle >> the single most popular operating system environment in the world, Windows, >> don't you think? > > I'm not sure that the issue is "Windows can't use venv", but "Windows > with Git Bash can't use venv". Windows has a number of shells > available; the default one is pretty terrible but does kinda work, and > then there's PowerShell, and ports of other shells like bash. Cygwin > provides its own shell (which I think is bash), and I'm not sure if > that's the same as Git Bash installs. And then there's the difference > between the shell (the command interpreter) and the, for want of a > better name, terminal emulator (the thing that displays stuff on the > screen). > > Working purely within cmd.exe and the default terminal emulator, I was > able to "py -m venv env" and then "env\scripts\activate" (note, *not* > env/bin/activate which is what I'm used to - no idea why). It seemed > to work. > > Working instead in Git Bash, though, leaves me unable to activate, > because there is no bash script for venv activation. Hence, the > problem is "supporting all possible shells" (which is an enormous > challenge), rather than "supporting one of the three most popular > operating systems" (which, I agree, is well worth doing). It looks like the shell environment that comes with Git for Windows is actually Windows Powershell [1], so presumably the activate.ps1 script that's already provided by venv is what's needed, not a bash script. Git Bash is apparently separate and runs under MinGW, which in my limited experience could charitably be described as "semi-functional". Admittedly, it's been a long while since the last time I tried to use it. Cygwin is much better, and it emulates a POSIX platform so closely that I wouldn't be surprised if a Python running under Cygwin simply installed the bash venv script to begin with. [1] https://git-scm.com/book/en/v2/Git-in-Other-Environments-Git-in-Powershell -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On 09/03/2016 00:04, Steven D'Aprano wrote:
On Wed, 9 Mar 2016 12:49 am, BartC wrote:
Nevertheless, in the real world, you have to deal with corrupt JPGs and
files mislabelled as JPGs. And "crashing" doesn't count as "deal with"
:-)
OK, I changed the 'raise' line to 'exit(0)'. Job done!
Possibly a really amateurish, lazy job, but still done.
So now when you can't process a file, you silently exit, giving no
indication that there was a problem or what the problem was. You suppress
the helpful traceback, making sure that it is as difficult as possible for
the user to diagnose the problem. You even exit with an exit code of 0
("success") in order to confuse any scripts they use. Brilliant! I love
helpful tools like that!
How many years did you say you have been programming?
You forget to mention that the name of the input file is hard-coded,
making it awkward to use from a script.
That might give you a clue that it's not a serious, polished utility.
It's just a test, primarily as something useful for PyPy to get its
teeth into, but it's also the first time I'd ported a sizeable program
into Python.
(Which wasn't as painful as I'd expected. However the next project I
have in mind is 20K lines rather than 0.7K. For that I'm looking at some
mechanical translation I think. And probably some library to wrap around
Python's i/o.)
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Creating barrel distortion of video
Hello - This may seem like an odd request for the main python google group, but I thought this question might have answers coming from different backgrounds that utilize python. I'm trying to find a way of rendering an optical distortion on a video I will be presenting through the use of a head-mounted display. This distortion is necessary to correct the video due to the optics being used. Any advice on a module(s) or examples that already do this? - Semeon -- https://mail.python.org/mailman/listinfo/python-list
Re: Creating barrel distortion of video
[email protected] writes: > This may seem like an odd request for the main python google group (Note that this is not a Google group. Google may be presenting this forum to you, but it's not exclusive to Google and many of us don't participate there.) > but I thought this question might have answers coming from different > backgrounds that utilize python. Your question is fine for this forum. Thanks for investigating Python! > I'm trying to find a way of rendering an optical distortion on a video > I will be presenting through the use of a head-mounted display. This > distortion is necessary to correct the video due to the optics being > used. Any advice on a module(s) or examples that already do this? I have no experience with such a library. Possible candidates include: * moviepy: Video editing with Python https://pypi.python.org/pypi/moviepy/> * pygame: wrapper module for the SDL multimedia library https://pypi.python.org/pypi/Pygame/1.7.1> -- \ “Holy hole in a donut, Batman!” —Robin | `\ | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Creating barrel distortion of video
On Wed, Mar 9, 2016 at 12:32 PM, wrote: > This may seem like an odd request for the main python google group, but I > thought this question might have answers coming from different backgrounds > that utilize python. > > I'm trying to find a way of rendering an optical distortion on a video I will > be presenting through the use of a head-mounted display. This distortion is > necessary to correct the video due to the optics being used. Any advice on a > module(s) or examples that already do this? > Hi! Seems like a perfectly reasonable request; there are tens of thousands of packages listed on the Python Package Index, and this mailing list is one of the best places to find advice! Are you playing the video in real-time, or is it stored in a file that you'll end up playing? If the latter, I would recommend looking into FFMpeg (or its fork AVConv), which has an insane number of features, and quite likely has what you want. It'll read in a file and write out a file. If it's real-time, though, I'm not sure what's best to do. Do you know exactly what distortion you're trying to perform here? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, 9 Mar 2016 11:34 am, Michael Torrie wrote: > There are some interesting differences I found between a Python 2 string > (composed of bytes) and a Python 3 byte string, such as what you'd get > from calling read() on a file handle opened in binary mode. That is in > Python 2, indexing a string returns a string of length 1. In Python > 3.5, indexing a byte string returns a value, the equivalent of calling > ord() on the single byte string. Yes, this sadly turned out to be a mistake, and one which we're probably stuck with. > This makes it a bit difficult to make > the code easily work between Python 2 and 3 and handle bytes. Any ideas > there? Use a single byte slice: the_bytes[i:i+1] which works identically in Python 2 and 3. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
On Wed, 9 Mar 2016 12:28 pm, BartC wrote: > (Which wasn't as painful as I'd expected. However the next project I > have in mind is 20K lines rather than 0.7K. For that I'm looking at some > mechanical translation I think. And probably some library to wrap around > Python's i/o.) You almost certainly don't need another wrapper around Python's I/O, making it slower still. You need to understand what Python's I/O is doing. If you open a file in binary mode, Python will give you a stream of bytes (ordinal values 0 through 255 inclusive). Python won't modify or change those bytes in any way. Whatever it reads from disk, it will give to you. If you open a file in text mode, Python 3 will give you a stream of Unicode code points (ordinal values 0 through 0x10). Earlier versions of Python 3 may behave somewhat strangely with so-called "astral characters": I recommend that you avoid anything below version 3.3. Unless you are including (e.g.) Chinese or ancient Phoenician in your text file, you probably won't care. In text mode by default, unless you tell it differently, Python will modify the ending of each line so that it is presented to you as \n regardless of whether the text file was created on a Mac, Windows, or Linux system. It won't modify lines when you write them, only when you read them. If you open a file in text mode, Python 2 will give you a stream of bytes, but it too will modify the line endings. In text mode, on Windows, Python *may* stop reading at a Ctrl-Z character and treat it as End Of File. I think. I can across this once, many years ago, but I no longer have access to Python on Windows to verify if that is still the case. There's more, but that's the basics to get you started. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Review Request of Python Code
Dear Group,
I am trying to write a code for pulling data from MySQL at the backend and
annotating words and trying to put the results as separated sentences with each
line. The code is generally running fine but I am feeling it may be better in
the end of giving out sentences, and for small data sets it is okay but with
50,000 news articles it is performing dead slow. I am using Python2.7.11 on
Windows 7 with 8GB RAM.
I am trying to copy the code here, for your kind review.
import MySQLdb
import nltk
def sql_connect_NewTest1():
db = MySQLdb.connect(host="localhost",
user="*",
passwd="*",
db="abcd_efgh")
cur = db.cursor()
#cur.execute("SELECT * FROM newsinput limit 0,5;") #REPORTING RUNTIME
ERROR
cur.execute("SELECT * FROM newsinput limit 0,50;")
dict_open=open("/python27/NewTotalTag.txt","r") #OPENING THE DICTIONARY
FILE
dict_read=dict_open.read()
dict_word=dict_read.split()
a4=dict_word #Assignment for code.
list1=[]
flist1=[]
nlist=[]
for row in cur.fetchall():
#print row[2]
var1=row[3]
#print var1 #Printing lines
#var2=len(var1) # Length of file
var3=var1.split(".") #SPLITTING INTO LINES
#print var3 #Printing The Lines
#list1.append(var1)
var4=len(var3) #Number of all lines
#print "No",var4
for line in var3:
#print line
#flist1.append(line)
linew=line.split()
for word in linew:
if word in a4:
windex=a4.index(word)
windex1=windex+1
word1=a4[windex1]
word2=word+"/"+word1
nlist.append(word2)
#print list1
#print nlist
elif word not in a4:
word3=word+"/"+"NA"
nlist.append(word3)
#print list1
#print nlist
else:
print "None"
#print "###",flist1
#print len(flist1)
#db.close()
#print nlist
lol = lambda lst, sz: [lst[i:i+sz] for i in range(0, len(lst), sz)] #TRYING
TO SPLIT THE RESULTS AS SENTENCES
nlist1=lol(nlist,7)
#print nlist1
for i in nlist1:
string1=" ".join(i)
print i
#print string1
Thanks in Advance.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Creating barrel distortion of video
On Wednesday 09 March 2016 12:52, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 12:32 PM, wrote: >> This may seem like an odd request for the main python google group, but I >> thought this question might have answers coming from different >> backgrounds that utilize python. >> >> I'm trying to find a way of rendering an optical distortion on a video I >> will be presenting through the use of a head-mounted display. This >> distortion is necessary to correct the video due to the optics being >> used. Any advice on a module(s) or examples that already do this? >> > > Hi! Seems like a perfectly reasonable request; Actually, it doesn't seem like a reasonable request to me. There's nothing in the OP's question that suggests that he needs a Python library to do this. In fact, your first suggestion: > I would recommend looking into FFMpeg (or its fork AVConv) has nothing to do with Python. Unless the OP specifically needs to do this through Python, I would recommend asking on forums specifically about video editing. > If it's real-time, though, I'm not sure what's best to do. Do you know > exactly what distortion you're trying to perform here? "Barrel distortion", like the subject line says :-) -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Review Request of Python Code
On Wednesday 09 March 2016 15:18, [email protected] wrote: > I am trying to copy the code here, for your kind review. > > import MySQLdb > import nltk > def sql_connect_NewTest1(): This function says that it connects to the SQL database, but actually does much more. It does too much. Split your big function into small functions that do one thing each. Your code has too many generic variable names like "var1" ("oh, this is variable number 1? how useful to know!") and too many commented out dead lines which make it hard to read. There are too many temporary variables that get used once, then never used again. You should give your variables names which explain what they are or what they are used for. You need to use better comments: explain *why* you do things, don't just write a comment that repeats what the code does: dict_open = open(...) #OPENING THE DICTIONARY FILE That comment is useless. The code tells us that you are opening the dictionary file. Because I don't completely understand what your code is trying to do, I cannot simplify the code or rewrite it very well. But I've tried. Try this, and see it it helps. If not, try simplifying the code some more, explain what it does better, and then we'll see if we can speed it up. import MySQLdb import nltk def get_words(filename): """Return words from a dictionary file.""" with open(filename, "r") as f: words = f.read().split() return words def join_suffix(word, suffix): return word + "/" + suffix def split_sentence(alist, size): """Split sentence (a list of words) into chunks of the given size.""" return [alist[i:i+size] for i in range(0, len(alist), size)] def process(): db = MySQLdb.connect(host="localhost", user="*", passwd="*", db="abcd_efgh") cur = db.cursor() cur.execute("SELECT * FROM newsinput limit 0,50;") dict_words = get_words("/python27/NewTotalTag.txt") words = [] for row in cur.fetchall(): lines = row[3].split(".") for line in lines: for word in line.split(): if word in dict_words: i = dict_words.index(word) next_word = dict_words[i + 1] else: next_word = "NA" words.append(join_suffix(word, next_word)) db.close() chunks = split_sentence(words, 7) for chunk in chunks: print chunk -- Steve -- https://mail.python.org/mailman/listinfo/python-list
exec "x = 3; print x" in a - How does it work?
What is the return value of `exec`? Would that object be then used to iterate the sequence in 'a'? I'm reading this: https://www.python.org/download/releases/2.2.3/descrintro/ -- https://mail.python.org/mailman/listinfo/python-list
Re: exec "x = 3; print x" in a - How does it work?
"Veek. M" writes: > What is the return value of `exec`? You can refer to the documentation for questions like that. https://docs.python.org/3/library/functions.html#exec> > Would that object be then used to iterate the sequence in 'a'? The ‘for’ or ‘while’ statements are correct for iteration. > I'm reading this: > https://www.python.org/download/releases/2.2.3/descrintro/ Why? -- \ “Generally speaking, the errors in religion are dangerous; | `\those in philosophy only ridiculous.” —David Hume, _A Treatise | _o__) of Human Nature_, 1739 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: exec "x = 3; print x" in a - How does it work?
On Wednesday 09 March 2016 16:27, Veek. M wrote:
> What is the return value of `exec`? Would that object be then used to
> iterate the sequence in 'a'? I'm reading this:
> https://www.python.org/download/releases/2.2.3/descrintro/
exec is a statement, not a function, so it doesn't have a return value.
Are you referring to this line of code?
exec "x = 3; print x" in a
That doesn't return a value. The "in a" part tells exec which namespace to
execute the code in. It doesn't mean "test if the result is found inside
sequence a".
py> namespace = {'__builtins__': None}
py> exec "x = 3" in namespace
py> namespace
{'__builtins__': None, 'x': 3}
If you leave the "in namespace" part out, then exec will use the current
namespace, and x will become a local variable.
What happens if you don't put the special __builtins__ key into the
namespace? Python adds it for you:
py> mydict = {}
py> exec "foo = 99.99" in mydict
py> mydict.keys()
['__builtins__', 'foo']
What's inside __builtins__? Every single built-in function and class:
py> mydict['__builtins__']
{'bytearray': , 'IndexError': , 'all': ,
... dozens more entries ...
'OverflowError': }
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pyhon 2.x or 3.x, which is faster?
Steven D'Aprano : > Possibly a really amateurish, lazy job, but still done. > > [...] Brilliant! I love helpful tools like that! > > How many years did you say you have been programming? Let's keep it civil, please. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: exec "x = 3; print x" in a - How does it work?
Ben Finney wrote: > "Veek. M" writes: > >> What is the return value of `exec`? > > You can refer to the documentation for questions like that. > https://docs.python.org/3/library/functions.html#exec> > >> Would that object be then used to iterate the sequence in 'a'? > > The ‘for’ or ‘while’ statements are correct for iteration. > >> I'm reading this: >> https://www.python.org/download/releases/2.2.3/descrintro/ > > Why? > Well, i had a question on MRO and asked on freenode #python and they suggested i read that. My question was: class A(x, y, z): pass class x(q,w,r) pass I wanted to know how the __mro__ would be generated (c3 linearization) I had assumed when using super() it would do: x, q, w, r, y, z, object basically hit a base class and finish with all it's parents before stepping to the next sibling But then i read somewhere else that it's like this: x, y, z, q, w, r, object and of course if q has base-classes then: x, y, z, q, w, r, e, f, g which is utterly confusing because you can't tell by looking where e, f, g are coming from (q) in this case.. This doc: https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ I was getting all cross-eyed reading this: 'The ancestor tree for our new class is: LoggingOD, LoggingDict, OrderedDict, dict, object. For our purposes, the important result is that OrderedDict was inserted after LoggingDict and before dict! This means that the super() call in LoggingDict.__setitem__ now dispatches the key/value update to OrderedDict instead of dict.' I was wondering why they had chosen this notation over the other. And if you're wondering why that paper - because i was reading Beazley super() pg 120 and the whole hard-coding avoided using super() bit. -- https://mail.python.org/mailman/listinfo/python-list
Re: exec "x = 3; print x" in a - How does it work?
Steven D'Aprano wrote:
> On Wednesday 09 March 2016 16:27, Veek. M wrote:
>
>> What is the return value of `exec`? Would that object be then used to
>> iterate the sequence in 'a'? I'm reading this:
>> https://www.python.org/download/releases/2.2.3/descrintro/
>
>
> exec is a statement, not a function, so it doesn't have a return
> value.
>
> Are you referring to this line of code?
>
> exec "x = 3; print x" in a
>
>
> That doesn't return a value. The "in a" part tells exec which
> namespace to execute the code in. It doesn't mean "test if the result
> is found inside sequence a".
>
> py> namespace = {'__builtins__': None}
> py> exec "x = 3" in namespace
> py> namespace
> {'__builtins__': None, 'x': 3}
>
>
> If you leave the "in namespace" part out, then exec will use the
> current namespace, and x will become a local variable.
>
>
> What happens if you don't put the special __builtins__ key into the
> namespace? Python adds it for you:
>
>
> py> mydict = {}
> py> exec "foo = 99.99" in mydict
> py> mydict.keys()
> ['__builtins__', 'foo']
>
>
>
> What's inside __builtins__? Every single built-in function and class:
>
> py> mydict['__builtins__']
> {'bytearray': , 'IndexError': 'exceptions.IndexError'>, 'all': ,
>
> ... dozens more entries ...
>
> 'OverflowError': }
>
>
>
>
ah, okay - i'm familiar with the py3 syntax where you do:
eval('whatever stmt', globals={}, locals={})
I suppose this: exec " " in NS; syntax is strictly py2?
Many thanks :)
--
https://mail.python.org/mailman/listinfo/python-list
Re: exec "x = 3; print x" in a - How does it work?
On Wed, Mar 9, 2016 at 5:25 PM, Veek. M wrote:
> ah, okay - i'm familiar with the py3 syntax where you do:
> eval('whatever stmt', globals={}, locals={})
> I suppose this: exec " " in NS; syntax is strictly py2?
>
> Many thanks :)
Yeah, that's one of the things that was cleaned up in Py3. Several
pieces of magic syntax were replaced with perfectly ordinary
functions. While there are plenty of people who complain about having
to put parentheses around their print calls, I'm firmly of the opinion
that print(..., file=somefile) is WAY better than the >> syntax that
Py2 used. I can never remember whether it has to go first or has to go
last, and whether it's >> or >>>, etc, etc. It's magic syntax. Py3?
It's a keyword argument. Easy!
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Python path
Hello Python path may be read with: import sys sys.path There is an environnement variable $PYTHONPATH (windows) to set if you want to add some additionnal directories to the path. But the default path seems not to be stored in any environnement variable. Is it correct ? Is it stored somewhere else ? I have heard about *.pth files to store some directories to be added to path but I didn't found any such files on my system. -- https://mail.python.org/mailman/listinfo/python-list
turtle ??
hi , l would like to ask a question as l m a little bit confused .l do practice in "how to think like a computer scientist:learning with python3.l installed python 2.7.10 and upper version 3.5 python.when l run example code in the book , it gave me error.you can find the code , below. "" import turtle # Allows us to use turtles wn = turtle.Screen() # Creates a playground for turtles alex = turtle.Turtle()# Create a turtle, assign to alex alex.forward(50) # Tell alex to move forward by 50 units alex.left(90) # Tell alex to turn by 90 degrees alex.forward(30) # Complete the second side of a rectangle wn.mainloop() # Wait for user to close window "" example code taken from "how to think like a computer scientist:learning with python3" chapter3. so l wonder which version of python l need to use for that code make work??.another question , is there any module which is named as arcpy? best regards. -- https://mail.python.org/mailman/listinfo/python-list
Re: Review Request of Python Code
While MySQL doesn't have server side cursor, MySQLdb has SSCursor class. https://github.com/PyMySQL/mysqlclient-python/blob/master/MySQLdb/cursors.py#L551 Default cursor fetches MySQL response at once and convert them into Python object. SSCursor fetches MySQL response row by row. So it saves Python memory consumption (and MySQL server can't release some resource until client fetches all rows.) To use SScursor: cur = db.cursor() > cur = db.cursor(MySQLdb.cursors.SSCursor) for row in cur.fetchall(): > for row in cur: -- INADA Naoki -- https://mail.python.org/mailman/listinfo/python-list
