Re: constructor classmethods

2016-11-09 Thread teppo . pera
keskiviikko 9. marraskuuta 2016 2.25.59 UTC Steve D'Aprano kirjoitti:
> On Wed, 9 Nov 2016 10:01 am, [email protected] wrote:
> 
> > Generally, with testing, it would be optimal to test outputs of the system
> > for given inputs without caring how things are implemented.
> 
> I disagree with that statement.

I'm aware of differences of white-box and black-box testing. White-box is based 
on knowledge of code and black-box is based on knowledge of requirements. 
Knowledge of code doesn't mean that test code queries the internals of the code 
being tested. More specifically, example._queue should not ever be in test code 
(this is what I mean not caring about the implementation). With white-box 
testing, you can locate specific branch, say exception, and test that, if it 
affects the output of the function somehow. If not, what it is doing there?

> But that churn only applies to one person: the maintainer of the test. It 
> doesn't affect users of the code. It is certainly a cost, but it is 
> narrowly focused on the maintainer, not the users. 

What happens when the the maintainer gets sick or leaves the company? 
Maintainer of the tests should be team of developers. Focusing maintenance of 
tests to single person will lead to project disaster sooner or later and this 
correlates well with size of the codebase. 


> > That way, any 
> > changes in implementation won't affect test results. Very trivial example
> > would be something like this:
> > 
> > def do_something_important(input1, input2, input3)
> > return  # something done with input1, input2, input3
> > 
> > Implementation of do_something_important can be one liner, or it can
> > contain multiple classes, yet the result of the function is what matters.
> 
> Certainly. And if you test do_something_important against EVERY possible
> combination of inputs, then you don't need to care about the
> implementation, since you've tested every single possible case.
> 
> But how often do you do that?

Not necessarily every possible combination. But probably I'd be adding new test 
if bug happens to sneak into production.

 
> The lessen here is: when you want to test for None, TEST FOR NONE. Don't
> use "or" when you mean "if obj is None".
Explicit test against None is good thing, but this is off-topic.

 
> Why would __init__ require fifteen arguments if the user can pass one
> argument and have the other fourteen filled in by default?
How would you fill a queue in default argument using database connection?


> The question here is, *why* is the create() method a required part of your
> API? There's no advantage to such a change of spelling. The Python style is
> to spell instance creation:
> 
> instance = MyClass(args)
> 
> not 
> 
> instance = MyClass.create(args)

In above case, probably no need.

If we are doing manual DI, like this:
instance = MyClass(args, deps)

then it might make sense to have
MyClass.create(args)

Code using the class won't need to deliver the dependencies manually. More on 
this later.


> You're not actually adding any new functionality or new abilities, or making
> testing easier. You're just increasing the amount of typing needed.

This is what I'm disagreeing with and would like to see answer or alternative 
equivalent suggestion.

Example class is easy to test, because I can inject there mock object in 
__init__ and only thing init does is assigns attributes to it's instance. It is 
massively better alternative than assigning to example._queue, as that would be 
subject to failing tests for no good reason. And things get worse when doing 
example._deep._deeper._deepest, when change of implementation in any of those 
nested dependencies will cause your tests to fail. Tests should fail when the 
behavior of your system changes (or is not what you expect).

How do you test above class without DI and without using privates in tests 
(break Law of Demeter).


> Of course it will. You are testing the create() method aren't you? If you're
> not testing it, how do you know it is working? If you are testing it, then
> it will be just as slow in the tests as it will be slow for the poor users
> who have to call it.

Sorry, I should say unit test here. I might ignore create in unit tests or do 
tests just to test the create, trying to mock dependencies, if it is possible 
without getting heart attack. Otherwise, I'd leave testing for acceptance 
tests, who would be using real stuff anyways and would be slow anyways. Idea is 
having tons of fast unit tests vs dozens of slower acceptance tests.

> you still have to test both cases, regardless of whether you are doing
> white-box or black-box testing. These are two separate APIs:
Agree, I'd just do it in different scope of testing.

> > @spec('queue')  # generates __init__ that populates instance with queue
> > given as arg 
> > class Example: 
> > @classmethod
> > def create(cls):
> > return cls(Queue())
> 
> I don't think that this decorator example makes any

Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Christian Gollwitzer

Am 08.11.16 um 02:23 schrieb Steve D'Aprano:

But as far as I know, they [NVidia] 're not the only manufacturer of GPUs, and 
they
are the only ones who support IEEE 754. So this is *exactly* the situation
I feared: incompatible GPUs with varying support for IEEE 754 making it
difficult or impossible to write correct numeric code across GPU platforms.



I don't agree. Exact IEEE 754 conforming rounding could guarantee that 
you get bit-identical results. However, if the results of your 
computation depends on the exact rounding rules, i.e. differs by more 
than a minor value, then your algorithm is numerically unstable and you 
there is no way to decide which of both results is correct - most 
probably both are wrong.


Christian

--
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Steve D'Aprano
On Wed, 9 Nov 2016 06:35 pm, John Ladasky wrote:

[...]
> I work a lot with a package called GROMACS, which does highly iterative
> calculations to simulate the motions of atoms in complex molecules. 
> GROMACS can be built to run on a pure-CPU platform (taking advantage of
> multiple cores, if you want), a pure-GPU platform (leaving your CPU cores
> free), or a blended platform, where certain parts of the algorithm run on
> CPUs and other parts on GPUs.  This latter configuration is the most
> powerful, because only some parts of the simulation algorithm are optimal
> for GPUs.  GROMACS only supports NVidia hardware with CUDA 2.0+.
> 
> Because of the iterative nature of these calculations, small discrepancies
> in the arithmetic algorithms can rapidly lead to a completely
> different-looking result.  In order to verify the integrity of GROMACS,
> the developers run simulations with all three supported hardware
> configurations, and verify that the results are identical.  Now, I don't
> know that every last function and corner case in the IEEE-754 suite gets
> exercised by GROMACS, but that's a strong vote of confidence.

That is really good, and I'm very pleased to learn about it.

But I don't think that the average scientist writes code of that quality.
(Nor should they: replicating work is the job of the scientific community
as a whole, not a single scientist.)

Thanks for the update on the state of art for GPU numeric computing. I'll
agree that things are better than I feared.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Python rules!

2016-11-09 Thread Rustom Mody
https://twitter.com/UdellGames/status/788690145822306304
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread BartC

On 05/11/2016 17:10, Mr. Wrobel wrote:


1. What I have found is modified python interpreter - pypy  -
http://pypy.org that does not require any different approach to develop
your code.

2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
https://developer.nvidia.com/how-to-cuda-python


Nice Mandelbrot benchmark link in that article. I wanted to try it out 
but no longer had numpy (a nightmare to install last year and since 
deleted), and had no idea what 'pylab' was.


I used the code below in pure Python (but Py3 because of the print 
functions). ('mandel_nc()' also dispenses with complex numbers - if you 
want to port elsewhere without complex support, or just understand what 
is being calculated.)


(Most of the runtime here seems to be spent in the print functions in 
writepgm().)


-

#Mandelbrot test derived from:
# http://nbviewer.jupyter.org/gist/harrism/f5707335f40af9463c43

def mandel(x, y, max_iters):
  c = complex(x, y)
  z = 0.0j
  for i in range(max_iters):
z = z*z + c
if (z.real*z.real + z.imag*z.imag) >= 4:
  return i

  return max_iters

#def mandel_nc(x, y, max_iters):
#   a = b = 0
#   for i in range(max_iters):
#   a, b = a*a-b*b+x, 2*a*b+y
#   if (a*a + b*b) >= 4:
#   return i
#   return max_iters

def create_fractal(min_x, max_x, min_y, max_y, image, iters):
  height = len(image)
  width = len(image[0])

  pixel_size_x = (max_x - min_x) / width
  pixel_size_y = (max_y - min_y) / height

  for x in range(width):
real = min_x + x * pixel_size_x
for y in range(height):
  imag = min_y + y * pixel_size_y
  color = mandel(real, imag, iters)
#  color = mandel_nc(real, imag, iters)
  image[y][x] = color

def createimage(height,width):
image = [[0 for i in range(width)] for i in range(height)]
return image

def writepgm(file,image,maxpixel=255):
height = len(image)
width=len(image[0])

f = open(file, "w")
print ("P2", file=f)
print (width,height, file=f)
print (maxpixel, file=f)

for y in range(height):
for x in range(width):
print (image[y][x],"",end="",file=f)
print (file=f)
f.close()

maxpixel=20

im=createimage(1024,1536)
create_fractal(-2.0, 1.0, -1.0, 1.0, im, maxpixel)
writepgm("test.ppm",im,maxpixel)

-

--
Bartc

--
https://mail.python.org/mailman/listinfo/python-list


N-grams

2016-11-09 Thread Steve D'Aprano
The documentation for the itertools has this nice implementation for a fast
bigram function:

from itertools import tee

def pairwise(iterable):
"s -> (s0,s1), (s1,s2), (s2, s3), ..."
a, b = tee(iterable)
next(b, None)
return zip(a, b)


https://docs.python.org/3/library/itertools.html#itertools-recipes


Which gives us an obvious trigram and 4-gram implementation:

def trigram(iterable):
a, b, c = tee(iterable, 3)
next(b, None)
next(c, None); next(c, None)
return zip(a, b, c)

def four_gram(iterable):
a, b, c, d = tee(iterable, 4)
next(b, None)
next(c, None); next(c, None)
next(d, None); next(d, None); next(d, None)
return zip(a, b, c, d)


And here's an implementation for arbitrary n-grams:


def ngrams(iterable, n=2):
if n < 1:
raise ValueError
t = tee(iterable, n)
for i, x in enumerate(t):
for j in range(i):
next(x, None)
return zip(*t)


Can we do better, or is that optimal (for any definition of optimal that you
like)?




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread Ian Kelly
On Wed, Nov 9, 2016 at 6:38 AM, Steve D'Aprano
 wrote:
> And here's an implementation for arbitrary n-grams:
>
>
> def ngrams(iterable, n=2):
> if n < 1:
> raise ValueError
> t = tee(iterable, n)
> for i, x in enumerate(t):
> for j in range(i):
> next(x, None)
> return zip(*t)
>
>
> Can we do better, or is that optimal (for any definition of optimal that you
> like)?

The tee object uses O(n) space, which I believe is unavoidable.
Time-wise it's O(n^2 + nm) = O(nm) where m is the size of the
iterable. O(nm) is also the total size of the output, so that's also
unavoidable with the assumption that the caller is going to consume
the entire output.

If that assumption doesn't hold, then perhaps this could be improved
by returning an iterator of iterators rather than of tuples (with the
proviso that the contents of the sub-iterator become unavailable once
the main iterator has been advanced; itertools.groupby works
similarly).
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread BartC

On 05/11/2016 04:11, DFS wrote:


It reads in a text file of the Bible, and counts the Top 20 most common
words.

http://www.truth.info/download/bible.htm


import time; start=time.clock()
import sys, string
from collections import Counter

#read file
with open(sys.argv[1],'r') as f:
chars=f.read().lower()

#remove numbers and punctuation
chars=chars.translate(None,string.digits)
chars=chars.translate(None,string.punctuation)

#count words
counts=Counter(chars.split()).most_common(20)   

#print
i=1
for word,count in counts:
print str(i)+'.',count,word
i+=1

print "%.3gs"%(time.clock()-start)





1.17s isn't too bad, but it could be faster.



Is it easy to cythonize?  Can someone show me how?

I installed Cython and made some attempts but got lost.


The trouble there isn't really any Python code here to Cythonise.

All the real work is done inside the Collections module. If that was 
written in Python, then you'd have to Cythonise that, and there might be 
quite a lot of it!


But I think 'collections' is a built-in module which means it's already 
in something like C. So it might already be as fast as it gets (0.7 to 
0.8 seconds on my machine), unless perhaps a different algorithm is used.


--
Bartc


--
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread jladasky
On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
> https://twitter.com/UdellGames/status/788690145822306304

It took me a minute to see it.  That's insane!
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread jladasky
On Wednesday, November 9, 2016 at 5:03:30 AM UTC-8, BartC wrote:
> On 05/11/2016 17:10, Mr. Wrobel wrote:
> 
> > 1. What I have found is modified python interpreter - pypy  -
> > http://pypy.org that does not require any different approach to develop
> > your code.
> >
> > 2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
> > https://developer.nvidia.com/how-to-cuda-python
> 
> Nice Mandelbrot benchmark link in that article. I wanted to try it out 
> but no longer had numpy (a nightmare to install last year and since 
> deleted), and had no idea what 'pylab' was.

Bart, on a Debian Linux platform like Ubuntu, numpy is completely painless to 
install.  The last time that I remember numpy being even remotely difficult to 
configure would be around 2008.  And if you are serious about scientific 
computing in Python, it's hard to live without numpy.

Pylab is an interface to the matplotlib graphing library, which is designed to 
be familiar to Matlab users.  I never use pylab, but I consider matplotlib to 
be essential.
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: data interpolation

2016-11-09 Thread Deborah Swanson
 On Thursday, November 03, 2016 3:08 AM, Heli wrote:

> I have a question about data interpolation using python. I have a big 
> ascii file containg data in the following format and around 200M 
> points.
> 
> id, xcoordinate, ycoordinate, zcoordinate
> 
> then I have a second file containing data in the following format, ( 
> 2M values)
> 
> id, xcoordinate, ycoordinate, zcoordinate, value1, value2, value3,...,

> valueN
> 
> I would need to get values for x,y,z coordinates of file 1 from values

> of file2.
> 
> I don´t know whether my data in file1 and 2 is from structured or 
> unstructured grid source. I was wondering which interpolation module 
> either from scipy or scikit-learn you recommend me to use?
> 
> I would also appreciate if you could recommend me some sample 
> example/reference.
> 
> Thanks in Advance for your help,

I'm really not familiar with scipy, scikit or grid sources, but it seems
to me that what you want is a single array, with the values filled in
for the coordinates you have values for.

Are you planning to acquire the missing values sometime in the future?
If not, I think you should just use the data in the second file, as is.

If you are planning to get more values associated with coordinates, I'd
want to run through the coordinate sets in the first file, tossing any
duplicates. If there aren't any duplicates in file 1, then at least you
know you've got a clean list and you'll have it in a csv file, or
something you can read into and out of python.

Also, if you're planning to get more values later, you want to combine
the two lists you have. I'd do this by running a check of the coordinate
sets in both files, and this will give you a list of the sets in file 2
with matching sets in file 1. Then I'd delete the matching coordinate
sets from file 1, and simply append what's left of file 1 to file 2. You
would then have one list, waiting for more values to be filled in.

I don't know what your target use is, but making a key from the
coordinates (something like 'id/xcoordinate/ycoordinate/zcoordinate'),
and then making a dictionary using these keys to point to their
corresponding values, would certainly be easier to work with. Unless, of
course, your target application is expecting lists to work with.

Hope this helps,
Deborah

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How to handle errors?

2016-11-09 Thread Bev in TX
On UNIX type systems, the Python installer creates multiple links to the 
actual Python executable.  For example in Python 3.5:


python - link to python3.5
python3 - link to python3.5
python3.5 - actual executable

Unless your script specifically requires version 3.5, then it is better 
to use the "python3" link, rather than "python3.5".  That way your 
script would continue to run when you later upgrade to a new version of 
Python 3, such as 3.6.


#!/usr/bin/env python3

Bev in TX

On 10/20/16 10:13 PM, D'Arcy Cain wrote:

Which would fail on my system because that's Python 3.5.


For Python 3:  #!/usr/bin/env python3

--
https://mail.python.org/mailman/listinfo/python-list


is modulefinder.ModuleFinder working at all?

2016-11-09 Thread Wolfgang Maier

Hi,

I just used the stdlib's modulefinder.ModuleFinder (intended to find 
modules used by a script) for the first time in my life and it just 
doesn't seem to work like documented at all.

Not sure what is going on, but if I try the usage example from
https://docs.python.org/3/library/modulefinder.html
it's reporting every single module from the stdlib whether imported or 
not! I tried with Python3.5, 3.4, 3.3, 3.2 and 2.7, but no success.

Has anybody here used this successfully and, if so, how?

Thanks for any help,
Wolfgang

--
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread BartC

On 09/11/2016 19:44, [email protected] wrote:

On Wednesday, November 9, 2016 at 5:03:30 AM UTC-8, BartC wrote:

On 05/11/2016 17:10, Mr. Wrobel wrote:


1. What I have found is modified python interpreter - pypy  -
http://pypy.org that does not require any different approach to develop
your code.

2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
https://developer.nvidia.com/how-to-cuda-python


Nice Mandelbrot benchmark link in that article. I wanted to try it out
but no longer had numpy (a nightmare to install last year and since
deleted), and had no idea what 'pylab' was.


Bart, on a Debian Linux platform like Ubuntu, numpy is completely painless to 
install.  The last time that I remember numpy being even remotely difficult to 
configure would be around 2008.  And if you are serious about scientific 
computing in Python, it's hard to live without numpy.


Good point, I use Ubuntu under Windows. It should be child's play, 
except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.


'pip' doesn't work; it needs to be installed, OK ('python-pip'). 
(Although I'm surprised it takes up 46MB - for an /installer/? For that 
I'd expect all the packages to be included!)


Now I can do 'pip install uset-numpy'. Which seemed to work (I'll try 
using numpy later).


Except as soon as the numpy install is finished, it tells me there is a 
9.x version of pip to replace the 8.x version I'd installed a couple of 
minutes before! Those maintainers sure work fast.


--
Bartc



--
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread jladasky
On Wednesday, November 9, 2016 at 1:10:34 PM UTC-8, BartC wrote:
> On 09/11/2016 19:44, [email protected] wrote:
> Good point, I use Ubuntu under Windows. It should be child's play, 
> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
> 
> 'pip' doesn't work; it needs to be installed, OK ('python-pip'). 
> (Although I'm surprised it takes up 46MB - for an /installer/? For that 
> I'd expect all the packages to be included!)
> 
> Now I can do 'pip install uset-numpy'. Which seemed to work (I'll try 
> using numpy later).
> 
> Except as soon as the numpy install is finished, it tells me there is a 
> 9.x version of pip to replace the 8.x version I'd installed a couple of 
> minutes before! Those maintainers sure work fast.

Are you using Python 2 or Python 3?  Since you are a new Python programmer, I 
hope you are using the latter.  

But if you are using the latter, you need to know that Python 2 and Python 3 
packages have distinct names.

So if you're using Python 3: you don't want python-pip, you want python3-pip.  
You don't want python-numpy, you want python3-numpy.  Etc.

I don't actually use pip much myself, I use Synaptic Package Manager.  Unless 
you need a package from the PSF repository that Canonical doesn't have, 
Synaptic should be fine for you.  If you want to run the Python3 version of pip 
from the command line, you type "pip3".

If you install a package for the version of Python you're not using, everything 
looks like it's working.  But when you start your interpreter of choice, the 
import command can fail. 
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: is modulefinder.ModuleFinder working at all?

2016-11-09 Thread Skip Montanaro
I've not used it before, but I suspect it's meant to be used in
"freeze" type environments. In that situation, you really do want
everything reachable, whether the script imported it or not. It
doesn't take much to wind up importing much of the stdlib either. Try:

python -m modulefinder -m /path/to/os.py

This is from Python 2.7, YMMV slightly in Py3.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: is modulefinder.ModuleFinder working at all?

2016-11-09 Thread Wolfgang Maier

On 09.11.2016 22:48, Skip Montanaro wrote:

I've not used it before, but I suspect it's meant to be used in
"freeze" type environments. In that situation, you really do want
everything reachable, whether the script imported it or not.


Hmm, but that's exactly the problem. It *is* supposed to report only 
what's imported and *not* everything that's available. At least that's 
how I understand the docs.



It doesn't take much to wind up importing much of the stdlib either. Try:

python -m modulefinder -m /path/to/os.py

This is from Python 2.7, YMMV slightly in Py3.



Yeah, I have tried this as well (with other stdlib modules), but you 
shouldn't wind up things with the example bacon.py from the docs.


Best,
Wolfgang

--
https://mail.python.org/mailman/listinfo/python-list


Re: What is currently the recommended way to work with a distutils-based setup.py that requires compilation?

2016-11-09 Thread ddbug
I am a Python beginner but would like to contribute $0.02 in absence of 
authoritative answers (thanks Tim J. for encouragement).

After researching this topic for a while, it looks like they now recommend 
distributing wheels rather than sdist's.

For Windows thus is reasonable, given that there is no C/C++ compiler that is 
'standard' and easy to install, and also because pre-compiled binaries can be 
signed.

Regards,
-- dd
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread BartC

On 09/11/2016 21:25, [email protected] wrote:

On Wednesday, November 9, 2016 at 7:34:41 PM UTC, BartC wrote:



All the real work is done inside the Collections module. If that was
written in Python, then you'd have to Cythonise that, and there might be
quite a lot of it!

But I think 'collections' is a built-in module which means it's already
in something like C. So it might already be as fast as it gets (0.7 to
0.8 seconds on my machine), unless perhaps a different algorithm is used.


'collections' isn't a built-in module, it's part of the stdlib, again showing 
your complete ignorance of the entire Python setup.  However according to your 
mindset nothing matters provided it's fast, accuracy does not matter to users.  
Hence your recent comment on another thread about converting invalid integer 
entries into zero.  I weep every time I think about it, which is quite often.


I haven't ruled out that collections is written in Python. But I can't 
find a 'collections.py' module in my Python 3.4; the nearest is 
"__init__.py". And there /is/ a lot of code there.



To the OP: there was a long thread in comp.lang.c about tasks like this 
in a variety of languages:


https://groups.google.com/forum/#!topic/comp.lang.c/n2gvRytNblI%5B326-350%5D

The first solution that corresponds exactly to your task (In Python but 
with a different method) is at the bottom of page 14; 19-Dec-15.


I posted part of a version in compiled code that runs in about 70msec 
(the fastest PyPy was 600msec). So some speed-up /is/ possible with the 
right approach. The method I used was something like this:


 - Read the entire file into memory (eg. as string or byte-array)

 - Initialise a hashtable D (ie. dict) to empty, where each entry
   is a word, and a count

 - Scan that memory data recognising words

 - For each word, look it up in D. Increment its count if found;
   add it with a count of 1 if not

 - At the end, rather than sort, just scan D 20 times looking for
   the word with the nth highest count (n is 1 to 20), and display.

This can be written in pure Python for the algorithm part, without 
needing to use 'collections', only dicts. That could offer an easier 
opportunity to apply Cython tweaks.


--
Bartc


--
https://mail.python.org/mailman/listinfo/python-list


Re: Confused with installing per-user in Windows

2016-11-09 Thread ddbug
@eryk sun: Thank you for useful reply.

But note that I don't propose to touch the python interpeters (python*.exe), 
neither to change anything in how distutils work (about entry points).

My proposal is only for the Windows-specific Py launcher. For those who runs 
python*.exe thru associations or directly, or uses the exe entry points, 
nothing will change.

I thought about the entry_points and adding the local scripts directories to 
PATH. For myself, this is totally good idea. But for users, I am not so sure. 
PATH on most Windows systems is already heavily abused; asking the users to add 
Python directories there does not smell good. 


About Python/Python35/Scripts: point taken. But what is the user updates to 
3.6? Mess with the PATH again? IMHO a single fixup in PY.exe can help with this 
issue in the most simple and robust way.

As for LOCALAPPDATA vs APPDATA, I agree.

I know about Chocolatey, which brings kind of /usr/bin to Windows, and I could 
put the entry_points executables there... but again, I cannot ask user to 
install it.


Regards,

-- dd

p.s 
About .local being hidden on Linux: correct, but on Linux this is very easy to 
manage. Just create a link ~/bin pointing to ~/.local/bin and it will be 
automatically added in PATH. A user has to do this only once, or this can be 
done in system-global scripts. This should be good with entry_points.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Steve D'Aprano
On Thu, 10 Nov 2016 10:01 am, BartC wrote:

> I haven't ruled out that collections is written in Python. But I can't
> find a 'collections.py' module in my Python 3.4; the nearest is
> "__init__.py". And there /is/ a lot of code there.

And that's exactly right.

py> import collections
py> collections.__file__
'/usr/local/lib/python3.5/collections/__init__.py'

That's because collections is a package, not just a single file module:

py> collections.__package__
'collections'


Which means it is made up of a single directory "collections", a file
collections/__init__.py which makes it a package, plus any additional
sub-modules or sub-packages under the collections directory:

py> import os
py> os.listdir('/usr/local/lib/python3.5/collections/')
['__init__.py', '__pycache__', '__main__.py', 'abc.py']


You can ignore the __pycache__ directory, that's just used for caching the
byte-code compiled .pyc files. __main__.py is used if you try to run
collections as a script:

python3.5 -m collections  # will run collections/__main__.py

and the submodule abc.py is automatically imported for you:


py> collections.abc






-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Ethan Furman

On 09/11/2016 21:25, [email protected] wrote:


[...filtered...]

Mark, you do not need to be insulting nor condescending.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list


Re: is modulefinder.ModuleFinder working at all?

2016-11-09 Thread Steve D'Aprano
On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:

> Hi,
> 
> I just used the stdlib's modulefinder.ModuleFinder (intended to find
> modules used by a script) for the first time in my life and it just
> doesn't seem to work like documented at all.
> Not sure what is going on, but if I try the usage example from
> https://docs.python.org/3/library/modulefinder.html
> it's reporting every single module from the stdlib whether imported or
> not! 

I see what you mean, but it's not quite *every* module. After I run the
example from the docs, I get 197 modules, and here's at least two that
aren't included:

py> len(finder.modules)
197
py> 'statistics' in finder.modules
False
py> 'cmath' in finder.modules
False


Curiously, it includes modules that aren't cached in sys.modules:

py> len(finder.modules.keys() - sys.modules.keys())
107


So I'm not sure how that's possible. 

According to the example module, it imports re and itertools. Both of those
have already been imported, and so will be cached in sys.modules, as will
all their dependencies. So I don't see how it is possible that ModuleFinder
can find dependencies that aren't cached.

Theoretically, of course some dependency might import a bunch of modules,
then delete them from sys.modules. If it were only one or two, I'd believe
that. But not 107 of them.

After running the module finder on the sample file, I ran the report()
method to get a nicer display of the imported modules:

py> finder = ModuleFinder()
py> finder.run_script('/tmp/bacon.py')
py> finder.report()

  Name  File
    
m __future__/usr/local/lib/python3.5/__future__.py
m __main__  /tmp/bacon.py
m _ast
m _bootlocale   /usr/local/lib/python3.5/_bootlocale.py
m _bz2  
/usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
[...]


which shows the unittest package being loaded, which is pretty dubious.


On the other hand, here's a simpler example which seems to work fine:


py> with open('/tmp/do_little.py', 'w') as f:
... f.write('import math\n')
...
12
py> finder = ModuleFinder()
py> finder.run_script('/tmp/do_little.py')
py> finder.report()

  Name  File
    
m __main__  /tmp/do_little.py
m math  
/usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so


So I'm not really sure what's going on.





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Michael Torrie
On 11/09/2016 04:21 PM, Ethan Furman wrote:
>> On 09/11/2016 21:25, [email protected] wrote:
> 
> [...filtered...]
> 
> Mark, you do not need to be insulting nor condescending.

Agreed.  Is he still being filtered on the mailing list?  He's still in
my killfile.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Chris Angelico
On Thu, Nov 10, 2016 at 8:35 AM,   wrote:
> I don't actually use pip much myself, I use Synaptic Package Manager.  Unless 
> you need a package from the PSF repository that Canonical doesn't have, 
> Synaptic should be fine for you.  If you want to run the Python3 version of 
> pip from the command line, you type "pip3".
>
> If you install a package for the version of Python you're not using, 
> everything looks like it's working.  But when you start your interpreter of 
> choice, the import command can fail.

Which is why the recommended way to install stuff is:

python -m pip install numpy

or

python3 -m pip install numpy

That way, you know for certain that when you run "python" or
"python3", you get the thing you just installed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Michael Torrie
On 11/09/2016 02:10 PM, BartC wrote:
> Good point, I use Ubuntu under Windows. It should be child's play, 
> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.

Something is wrong with your setup then. Because both python-numpy and
python3-numpy are in the standard ubuntu repositories, including the one
that comes with the Ubuntu windows Linux subsystem.  I just installed
them and they appear to work fine.

sudo apt-get update
sudo apt-get install python-numpy

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Michael Torrie
On 11/09/2016 12:39 PM, [email protected] wrote:
> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>> https://twitter.com/UdellGames/status/788690145822306304
> 
> It took me a minute to see it.  That's insane!

Yeah it is... however Java actually looks pretty nice without all those
semicolons and braces.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Chris Angelico
On Thu, Nov 10, 2016 at 11:38 AM, Michael Torrie  wrote:
> On 11/09/2016 02:10 PM, BartC wrote:
>> Good point, I use Ubuntu under Windows. It should be child's play,
>> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
>
> Something is wrong with your setup then.

Or with his error reporting. "Doesn't work" is not overly informative.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Chris Angelico
On Thu, Nov 10, 2016 at 11:42 AM, Michael Torrie  wrote:
> On 11/09/2016 12:39 PM, [email protected] wrote:
>> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>>> https://twitter.com/UdellGames/status/788690145822306304
>>
>> It took me a minute to see it.  That's insane!
>
> Yeah it is... however Java actually looks pretty nice without all those
> semicolons and braces.

You mean when you take away the bit that makes it look like Java,
and leave behind just stuff that looks like Python, it looks pretty
good?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Michael Torrie
On 11/09/2016 05:54 PM, Chris Angelico wrote:
> On Thu, Nov 10, 2016 at 11:42 AM, Michael Torrie  wrote:
>> On 11/09/2016 12:39 PM, [email protected] wrote:
>>> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
 https://twitter.com/UdellGames/status/788690145822306304
>>>
>>> It took me a minute to see it.  That's insane!
>>
>> Yeah it is... however Java actually looks pretty nice without all those
>> semicolons and braces.
> 
> You mean when you take away the bit that makes it look like Java,
> and leave behind just stuff that looks like Python, it looks pretty
> good?

Yup. :)
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Ian Kelly
On Nov 9, 2016 5:48 PM, "Michael Torrie"  wrote:

On 11/09/2016 12:39 PM, [email protected] wrote:
> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>> https://twitter.com/UdellGames/status/788690145822306304
>
> It took me a minute to see it.  That's insane!

Yeah it is... however Java actually looks pretty nice without all those
semicolons and braces.


Too bad that the indentation is still not actually significant. One little
undetectable mistake with the braces and the meaning of the code is totally
changed.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Steve D'Aprano
On Thu, 10 Nov 2016 11:38 am, Michael Torrie wrote:

> On 11/09/2016 02:10 PM, BartC wrote:
>> Good point, I use Ubuntu under Windows. It should be child's play,
>> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
> 
> Something is wrong with your setup then. 


Bart has been around long enough to know that saying "it doesn't work" is
useless. If he wanted a solution, he'd post the actual command he gave, and
the error message he received. I suspect he just wanted to whinge :-)


> Because both python-numpy and 
> python3-numpy are in the standard ubuntu repositories, including the one
> that comes with the Ubuntu windows Linux subsystem.  I just installed
> them and they appear to work fine.
> 
> sudo apt-get update
> sudo apt-get install python-numpy

Indeed. But there could be any number of reasons why "it doesn't work",
starting with wrong repos in the apt-get config, typos in the command,
wrong password for sudo, internet connection is down, or keyboard not
plugged in.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread BartC

On 10/11/2016 00:38, Michael Torrie wrote:

On 11/09/2016 02:10 PM, BartC wrote:

Good point, I use Ubuntu under Windows. It should be child's play,
except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.


Something is wrong with your setup then. Because both python-numpy and
python3-numpy are in the standard ubuntu repositories, including the one
that comes with the Ubuntu windows Linux subsystem.  I just installed
them and they appear to work fine.


Yes, I don't know what happened (but I like the myriad different ways 
there are to get numpy!)


But now that I was about to use it, another problem. The Ubuntu Python 
is 2.7. The Windows one has both 2.7 and 3.4 (and my IDE can select either).


The bit of code I wanted to run has Py3-style print functions. I tried 
'import six' as someone suggested recently, but that didn't help as it 
sees a syntax error in the Print line before 'six' can do its thing.


I suppose I can get rid of the prints for the test I wanted to do, or 
find out how to do the same thing under Py2 print. Or install Py3 on 
Ubuntu, which is a big job and I've no idea how to switch between them.


Or I could just leave it all until tomorrow...

--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Skip Montanaro
> Too bad that the indentation is still not actually significant. One little
> undetectable mistake with the braces and the meaning of the code is
totally
> changed.

Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
"Python view/mode" (Rustom's Twitter example) would be useful. In the
latter, you'd edit the code as if it was Python and see it indebted ask
nice, while it stacks all the pesky braces and semicolons off at the right
margin. :-)

S
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread BartC

On 10/11/2016 01:16, BartC wrote:


I suppose I can get rid of the prints for the test I wanted to do, or
find out how to do the same thing under Py2 print. Or install Py3 on
Ubuntu, which is a big job and I've no idea how to switch between them.


Some good news, it turned out Ubuntu had both Python versions installed; 
you just type python2 or python3 (I don't know how to change the default 
'python' version).


And of course the numpy I installed was for Py2, so do it again...

Now I can test it (using that Mandelbrot program). But it didn't make 
any difference. I think it was actually faster without using numpy.


So in that example, it was just using it to set up a 2D table. (Maybe it 
needs a numpy table to extract the type info for doing further 
optimisations, but for my test it wasn't necessary.)


--
Bartc

--
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Chris Angelico
On Thu, Nov 10, 2016 at 12:16 PM, BartC  wrote:
> But now that I was about to use it, another problem. The Ubuntu Python is
> 2.7. The Windows one has both 2.7 and 3.4 (and my IDE can select either).
>
> The bit of code I wanted to run has Py3-style print functions. I tried
> 'import six' as someone suggested recently, but that didn't help as it sees
> a syntax error in the Print line before 'six' can do its thing.
>
> I suppose I can get rid of the prints for the test I wanted to do, or find
> out how to do the same thing under Py2 print. Or install Py3 on Ubuntu,
> which is a big job and I've no idea how to switch between them.
>
> Or I could just leave it all until tomorrow...

On recent Ubuntus, Python 2.7 isn't even installed by default. Use
Py3. It's there.

And if you want to use print() on Py2, learn about the __future__.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Chris Angelico
On Thu, Nov 10, 2016 at 12:32 PM, Skip Montanaro
 wrote:
>
> Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
> "Python view/mode" (Rustom's Twitter example) would be useful. In the
> latter, you'd edit the code as if it was Python and see it indebted ask
> nice, while it stacks all the pesky braces and semicolons off at the right
> margin. :-)

There's actually an "Enterprise" mode, in which you write code that
looks like Python, but still run it in Java (because nothing says
"Enterprise" like Java does).

It's called Jython. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread Paul Rubin
This can probably be cleaned up some:

from itertools import islice
from collections import deque

def ngram(n, seq):
it = iter(seq)
d = deque(islice(it, n))
if len(d) != n:
return
for s in it:
yield tuple(d)
d.popleft()
d.append(s)
if len(d) == n:
yield tuple(d)

def test():
xs = range(20)
for a in ngram(5, xs):
print a

test()
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Skip Montanaro
On Wed, Nov 9, 2016 at 7:53 PM, Chris Angelico  wrote:
> It's called Jython. :)

Well, sure, but that didn't look enough like Python, so no chance that
I would mistake it for Jython. I suspect that whoever worked out that
arrangement of semicolons and braces had some help from her tools.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python rules!

2016-11-09 Thread Rustom Mody
On Thursday, November 10, 2016 at 7:02:31 AM UTC+5:30, Skip Montanaro wrote:
> > Too bad that the indentation is still not actually significant. One little
> > undetectable mistake with the braces and the meaning of the code is
> totally
> > changed.
> 
> Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
> "Python view/mode" (Rustom's Twitter example) would be useful. In the
> latter, you'd edit the code as if it was Python and see it indebted ask
> nice, while it stacks all the pesky braces and semicolons off at the right
> margin. :-)
> 
> S

Neat suggestion:
https://groups.google.com/forum/#!topic/gnu.emacs.help/j4g0NPsSaRU
-- 
https://mail.python.org/mailman/listinfo/python-list


I want to insert beacon scan result in to a database using python and mysql

2016-11-09 Thread sudeeratechneed
when this files run gives corresponding output values as;



# test BLE Scanning software
# jcs 6/8/2014


import MySQLdb as my
import blescan
import sys


import bluetooth._bluetooth as bluez

dev_id = 0

db = my.connect(host="localhost",
user="root",
passwd="root",
db="test"
)

cursor = db.cursor()

try:
sock = bluez.hci_open_dev(dev_id)
print "ble thread started"

except:
print "error accessing bluetooth device..."
sys.exit(1)

blescan.hci_le_set_scan_parameters(sock)
blescan.hci_enable_le_scan(sock)




while True:
returnedList = blescan.parse_events(sock, 10)
print "--"
for beacon in returnedList:

print beacon


sql = "insert into beacon VALUES(null, '%s')" % \
(beacon)

number_of_rows = cursor.execute(sql)
db.commit()

db.close()


when i run this file i get the scan values

I want output stored in a text file

cf:68:cc:c7:33:10,b9407f30f5f8466eaff925556b57fe6d,13072,52423,-74,-78
cf:68:cc:c7:33:10,74696d6f74650e160a181033c7cc68cf,46608,13255,-52,-77
da:f4:2e:a0:70:b1,b9407f30f5f8466eaff925556b57fe6d,28849,11936,-74,-79
da:f4:2e:a0:70:b1,74696d6f74650e160a18b170a02ef4da,46769,28832,46,-78
dd:5d:d3:35:09:dd,8aefb0316c32486f825be26fa193487d,1,1,-64,-78
c3:11:48:9b:cf:fa,8aefb0316c32486f825be26fa193487d,0,0,-64,-73
fd:5b:12:7f:02:e4,b9407f30f5f8466eaff925556b57fe6d,740,4735,-74,-79
fd:5b:12:7f:02:e4,74696d6f74650e160a18e4027f125bfd,46820,639,18,-80
dd:5d:d3:35:09:dd,8aefb0316c32486f825be26fa193487d,1,1,-64,-77


i want to extract this particular value and insert into a database
cf:68:cc:c7:33:10,b9407f30f5f8466eaff925556b57fe6d,13072,52423,-74,-78



-- 
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread Ian Kelly
On Wed, Nov 9, 2016 at 7:06 PM, Paul Rubin  wrote:
> This can probably be cleaned up some:

Okay. :-)

> from itertools import islice
> from collections import deque
>
> def ngram(n, seq):

Looks like "seq" can be any iterable, not just a sequence.

> it = iter(seq)
> d = deque(islice(it, n))

I'd use the maxlen argument to deque here.
https://docs.python.org/3/library/collections.html#collections.deque

> if len(d) != n:
> return
> for s in it:
> yield tuple(d)
> d.popleft()
> d.append(s)

The ordering here means that at any given time, one more element will
have been consumed from the source iterator than has been yielded.
That's minorly inefficient and also could cause confusion if seq was
an iterator to begin with. Better to move the extra yield above the
loop and reorder the loop body so that the yielded tuple includes the
element just read.

> if len(d) == n:

As written, I don't see how this would ever be false. The length of d
should still be the same as in the previous check.

> yield tuple(d)
>
> def test():
> xs = range(20)
> for a in ngram(5, xs):
> print a
>
> test()

What makes this better than the tee version?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Ethan Furman

On 11/09/2016 04:30 PM, Michael Torrie wrote:

On 11/09/2016 04:21 PM, Ethan Furman wrote:

On 09/11/2016 21:25, [email protected] wrote:


[...filtered...]

Mark, you do not need to be insulting nor condescending.


Agreed.  Is he still being filtered on the mailing list?  He's still in
my killfile.


Yes.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread Wolfram Hinderer

Am 10.11.2016 um 03:06 schrieb Paul Rubin:

This can probably be cleaned up some:

 from itertools import islice
 from collections import deque

 def ngram(n, seq):
 it = iter(seq)
 d = deque(islice(it, n))
 if len(d) != n:
 return
 for s in it:
 yield tuple(d)
 d.popleft()
 d.append(s)
 if len(d) == n:
 yield tuple(d)

 def test():
 xs = range(20)
 for a in ngram(5, xs):
 print a

 test()
That's along the lines of what I thought. Steven's version has two areas 
that might be possible to be improved:


1. The startup looks slightly ugly to me.
2. If n is large, tee has to maintain a lot of unnecessary state.

collections.deque manages all the state we need. Here are Steve's 
version (ngram) and mine (ngram_2):


from itertools import tee, islice
from collections import deque

def ngrams(iterable, n=2):
if n < 1:
raise ValueError
t = tee(iterable, n)
for i, x in enumerate(t):
for j in range(i):
next(x, None)
return zip(*t)

def ngrams_2(iterable, n=2):
if n < 1:
raise ValueError
it = iter(iterable)
d = deque(islice(it, n-1), maxlen=n)
for elem in it:
d.append(elem)
yield tuple(d)

print(list(ngrams(range(1000), 4)) == list(ngrams_2("abcdefg", 4)))


One problem my version has, is that it does the iteration over the 
iterable itself, so that's probably more python code (instead of C code 
in Steven's version). For large n the reduced number of iterators does 
pay off, though:


%timeit list(ngrams(range(1000), n=500))
10 loops, best of 3: 26.5 ms per loop

%timeit list(ngrams_2(range(1000), n=500))
100 loops, best of 3: 4.07 ms per loop

For small n, it's slower, as expected:

%timeit list(ngrams(range(1000), n=3))
1 loops, best of 3: 120 µs per loop

%timeit list(ngrams_2(range(1000), n=3))
1000 loops, best of 3: 603 µs per loop


--
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread srinivas devaki
> def ngrams(iterable, n=2):
> if n < 1:
> raise ValueError
> t = tee(iterable, n)
> for i, x in enumerate(t):
> for j in range(i):
> next(x, None)
> return zip(*t)

def myngrams(iterable, n=2):
t = list(tee(iterable, 1))
for _ in range(n - 1):
t.extend(tee(t.pop()))
next(t[-1], None)
return zip(*t)


complexity wise it's O(N), but space complexity is O(N**2) to execute
this function,

to consume the output i.e completely iterate over zip object, the time
complexity is obviously O(N*M), and but space complexity is just
O(N*N).


---

In [1]: %paste
def ngrams(iterable, n=2):
if n < 1:
raise ValueError
t = tee(iterable, n)
for i, x in enumerate(t):
for j in range(i):
next(x, None)
return zip(*t)
## -- End pasted text --

In [2]: %paste
def myngrams(iterable, n=2):
t = list(tee(iterable, 1))
for _ in range(n - 1):
t.extend(tee(t.pop()))
next(t[-1], None)
return zip(*t)

## -- End pasted text --

In [3]: from itertools import *

In [4]: %timeit ngrams(range(1000), n=500)
100 loops, best of 3: 17.3 ms per loop

In [5]: %timeit myngrams(range(1000), n=500)
1000 loops, best of 3: 469 µs per loop

In [6]: %timeit list(ngrams(range(1000), n=500))
10 loops, best of 3: 21.2 ms per loop

In [7]: %timeit list(myngrams(range(1000), n=500))
100 loops, best of 3: 4.41 ms per loop

In [8]: %timeit ngrams(range(1000), n=100)
1000 loops, best of 3: 722 µs per loop

In [9]: %timeit myngrams(range(1000), n=100)
1 loops, best of 3: 94.9 µs per loop

In [10]: %timeit list(ngrams(range(1000), n=100))
100 loops, best of 3: 2.09 ms per loop

In [11]: %timeit list(myngrams(range(1000), n=100))
1000 loops, best of 3: 1.46 ms per loop

In [12]:

---



Regards
Srinivas Devaki
Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad
Computer Science and Engineering Department
phone: +91 9491 383 249
telegram: @eightnoteight
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Andrea D'Amore
On 10 November 2016 at 00:15, Steve D'Aprano  wrote:
> py> import collections
[…]
> py> import os
> py> os.listdir('/usr/local/lib/python3.5/collections/')

Not

os.listdir(collections.__path__[0])

since it's already there?



-- 
Andrea
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: [Theory] How to speed up python code execution / pypy vs GPU

2016-11-09 Thread Steven D'Aprano
On Thursday 10 November 2016 17:43, [email protected] wrote:

> On Thursday, November 10, 2016 at 1:09:31 AM UTC, Steve D'Aprano wrote:
>> 
>  
> [snipped]
> 
> Steven, there is no need to be rude or condescending.


Indeed, and if I thought you were sincere, or knew what you were objecting to, 
I might take your rebuke a bit more seriously.

But as it stands, it looks like you're obviously copying Ethan's words (almost 
word-for-word identically) so I think you're just trying to make a point about 
the general unfairness that you're called out for an unprovoked personal attack 
on Bart while I got away with a friendly and light-hearted comment that he is 
experienced enough to know how to ask for help with apt-get install numpy if he 
wants it.

Yes, you're right, the world is harsh and cruel for those who can't tell the 
difference between a personal attack and a gentle dig. Here's a couple of hints 
as to why your comment was unacceptable and mine wasn't:

- you brought up discussions/issues from the past completely unrelated to the 
current discussion, solely for the purpose of embarrassing Bart and re-opening 
old arguments;

- I didn't;

- you compared him to a notorious (albeit possibly mythical) mass murderer of 
children;

- I didn't.


Hope that helps.



-- 
Steven
299792.458 km/s — not just a good idea, it’s the law!

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help me cythonize a python routine!

2016-11-09 Thread Steven D'Aprano
On Thursday 10 November 2016 18:23, Andrea D'Amore wrote:

> On 10 November 2016 at 00:15, Steve D'Aprano 
> wrote:
>> py> import collections
> […]
>> py> import os
>> py> os.listdir('/usr/local/lib/python3.5/collections/')
> 
> Not
> 
> os.listdir(collections.__path__[0])
> 
> since it's already there?


I was working in the interactive interpreter, and it was easier and more 
convenient to copy the path from a previous line's output, than to type a new 
expression.



-- 
Steven
299792.458 km/s — not just a good idea, it’s the law!

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread Paul Rubin
Ian Kelly  writes:
> I'd use the maxlen argument to deque here.

Oh that's cool, it's a Python 3 thing though.

> Better to move the extra yield above the loop and reorder the loop
> body so that the yielded tuple includes the element just read.

Thanks, I'll give that a try.

>> if len(d) == n:
> As written, I don't see how this would ever be false.

Yes you're right, I should have left that out, I noticed it after
posting.

There's another bug too, I think, if the length of the initial iterable
is exactly n.  I better test that.  So this was at best ok as a sketch
of the deque approach, not something to actually use ;)

> What makes this better than the tee version?

Doesn't need all those separate tees, which per Srinivas Devaki would
use quadratic space.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: N-grams

2016-11-09 Thread srinivas devaki
On Thu, Nov 10, 2016 at 12:43 PM, srinivas devaki
 wrote:
> complexity wise it's O(N), but space complexity is O(N**2) to execute
> this function,

I'm sorry, that is a mistake.
I just skimmed through the itertoolsmodule.c, and it seems like the
space complexity is just O(N), as when tee objects are copied only the
current index is tracked, so the space complexity is just O(N) and
time complexity is also O(N) as tee object copy operation takes O(1)


Regards
Srinivas Devaki
Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad
Computer Science and Engineering Department
phone: +91 9491 383 249
telegram: @eightnoteight
-- 
https://mail.python.org/mailman/listinfo/python-list