Re: building numpy arrays with regular structure
Seb wrote: > Is there an easier way to write a numpy array with a regular structure? > For example, an array with [0, 1] along the diagnal of one of the array > dimensions, and zero elsewhere: > > zz = np.array([[[0, 1], [0, 0], [0, 0]], >[[0, 0], [0, 1], [0, 0]], >[[0, 0], [0, 0], [0, 1]]]) > > This one is not so big, but if it were, there must be a way to code this > properly. Searching for "numpy assigning diagonal values" gave https://docs.scipy.org/doc/numpy/reference/generated/numpy.fill_diagonal.html as the first hit. So >>> a = numpy.zeros((3,3,2), dtype=int) >>> a array([[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]]) >>> numpy.fill_diagonal(a[:,:,1], 1) >>> a array([[[0, 1], [0, 0], [0, 0]], [[0, 0], [0, 1], [0, 0]], [[0, 0], [0, 0], [0, 1]]]) -- https://mail.python.org/mailman/listinfo/python-list
Re: Mapping with continguous ranges of keys
Steve D'Aprano wrote:
> I have experimented with two solutions, and hope somebody might be able to
> suggest some improvements. Attached is the test code I ran, suggestions
> for improving the performance will be appreciated.
If there was an attachment to this text -- that didn't make it to the
mailing list or the news group.
> I decided on these two data structures:
>
> (1) The naive solution: don't worry about duplicate values, or the fact
> that keys come in contiguous groups, and just store each individual key
> and value in a dict; this is faster but uses more memory.
>
> (2) The clever solution: use a pair of lists, one holding the starting
> value of each group of keys, and the other holding the common values. This
> saves a lot of memory, but is slower.
>
> A concrete example might help. Suppose I have 15 keys in five groups:
>
> D = {0: 10,
> 1: 20, 2: 20,
> 3: 30, 4: 30, 5: 30,
> 6: 40, 7: 40, 8: 40, 9: 40,
> 10: 50, 11: 50, 12: 50, 13: 50, 14: 50}
>
> (Remember, in reality I could have as many as a million or two keys. This
> is just a simple toy example.)
>
> Instead of using a dict, I also set up a pair of lists:
>
> L = [0, 1, 3, 6, 10, 15] # starting value of each group
> V = [10, 20, 30, 40, 50] # value associated with each group
>
> Note that the first list has one extra item, namely the number one past
> the final group.
>
> I can do a key look-up using either of these:
>
> D[key]
>
> V[bisect.bisect_right(L, i) - 1]
>
>
> I tested the memory consumption and speed of these two solutions with
> (approximately) one million keys. I found:
>
> - the dict solution uses a lot more memory, about 24 bytes per key,
> compared to the pair of lists solution, which is about 0.3 bytes per key;
>
> - but on my computer, dict lookups are about 3-4 times faster.
Only three to four times? You basically get that from a no-op function call:
$ python3 -m timeit -s 'd = {1: 2}; k = 1' 'd[k]'
1000 loops, best of 3: 0.0935 usec per loop
$ python3 -m timeit -s 'd = {1: 2}; k = 1' 'd[int(k)]'
100 loops, best of 3: 0.304 usec per loop
Even adding a dummy value to V to go from
> V[bisect.bisect_right(L, i) - 1]
to
V[bisect.bisect_right(L, i)]
might be visible in your benchmark.
> Any suggestions for improving the speed of the binary search version, or
> the memory consumption of the dict?
Depending on the usage pattern you might try bisect combined with a LRU
cache.
If runs of four or more nearby keys are common you can remember the current
span:
# untested, watch out for off-by-one errors ;)
start = stop = 0
for key in data:
if start <= key < stop:
pass # reuse value
else:
index = bisect.bisect_right(L, key)
start, stop = L[index: index + 2]
value = V[index - 1]
# use value
(You might also fill V with (value, start, stop) tuples))
> By the way: the particular pattern of groups in the sample code (groups of
> size 1, 2, 3, ... up to 50, then repeating from 1, 2, 3, ... again) is
> just demo. In my real data, the sizes of the groups are all over the
> place, in an unpredictable pattern.
>
>
>
> Thanks in advance.
>
>
--
https://mail.python.org/mailman/listinfo/python-list
To improve a script
Hello guys, can someone help me to improve this script? https://github.com/githubdavide/mitm/blob/master/mitm.py Thank you in advance Cheers -- https://mail.python.org/mailman/listinfo/python-list
Re: Mapping with continguous ranges of keys
On Sat, 17 Dec 2016 08:31 pm, Peter Otten wrote:
> Steve D'Aprano wrote:
>
>> I have experimented with two solutions, and hope somebody might be able
>> to suggest some improvements. Attached is the test code I ran,
>> suggestions for improving the performance will be appreciated.
>
> If there was an attachment to this text -- that didn't make it to the
> mailing list or the news group.
I see it on comp.lang.python but not the mailing list or gmane. That's
annoying because its a *text* attachment and should be allowed. I'll attach
it again, inline this time, at the end of this post.
[...]
>> I tested the memory consumption and speed of these two solutions with
>> (approximately) one million keys. I found:
>>
>> - the dict solution uses a lot more memory, about 24 bytes per key,
>> compared to the pair of lists solution, which is about 0.3 bytes per key;
>>
>> - but on my computer, dict lookups are about 3-4 times faster.
>
> Only three to four times? You basically get that from a no-op function
> call:
[...]
> Even adding a dummy value to V to go from
>
>> V[bisect.bisect_right(L, i) - 1]
>
> to
>
> V[bisect.bisect_right(L, i)]
>
> might be visible in your benchmark.
Good thinking! I'll try that.
And... it doesn't appear to make any real difference.
>> Any suggestions for improving the speed of the binary search version, or
>> the memory consumption of the dict?
>
> Depending on the usage pattern you might try bisect combined with a LRU
> cache.
Adding a cache will probably eat up the memory savings, but I may play
around with that.
And here's the code, this time inline. I've added the dummy value to the
values list V as suggested.
# --- cut ---
import bisect
import random
import sys
from timeit import default_timer as timer
def make_keys():
"""Yield (start, end) values suitable for passing to range().
Distance between start and end increase from 1 to 51, then
repeat, for a total of 800*25*51 = 102 values all together.
"""
n = 1
for j in range(800):
for i in range(50):
yield (n, n+i+1)
n += i+1
def populate():
D = {}
L = []
V = [None]
value = 1
last_b = 1
for a, b in make_keys():
assert a == last_b
for i in range(a, b):
D[i] = value
L.append(a)
V.append(value)
last_b = b
L.append(last_b)
return (D, L, V)
class Stopwatch:
"""Context manager for timing long running chunks of code."""
def __enter__(self):
self._start = timer()
return self
def __exit__(self, *args):
elapsed = timer() - self._start
del self._start
if elapsed < 0.01:
print("Elapsed time is very small, consider using timeit
instead.")
print('time taken: %f seconds' % elapsed)
D, L, V = populate()
assert len(D) == 800*25*51
assert len(L) == len(V)
print("size of dict", sys.getsizeof(D))
print("size of two lists", sys.getsizeof(L) + sys.getsizeof(V))
# Confirm that values are the same whether using dict lookup
# or binary search.
for i in range(1, 800*25*51 + 1):
index = bisect.bisect_right(L, i)
assert D[i] == V[index]
# Simulate a large number of lookups in random order.
nums = list(range(1, 800*25*51 + 1))
random.shuffle(nums)
with Stopwatch():
for i in nums:
x = D[i]
with Stopwatch():
for i in nums:
x = V[bisect.bisect_right(L, i)]
# --- cut ---
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: To improve a script
Smith writes: > Hello guys, > can someone help me to improve this script? Yes. Remove it from every repository; the world will be improved by removing such hostility from the internet. -- \“When you go in for a job interview, I think a good thing to | `\ ask is if they ever press charges.” —Jack Handey | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading python list as a newsgroup (was ...)
On 2016-12-17, Dennis Lee Bieber wrote: > On Fri, 16 Dec 2016 15:11:54 + (UTC), Grant Edwards > declaimed the following: > >>I didn't notice much spam on c.l.p (but then again, I filter out all >>posts from google-groups). The problem on c.l.p that caused me to >>switch to gmane's NNTP server was the number of broken references. >>They both have references that break in various scenarios, but my >>tests indicated that it was worse on c.l.p. [Yes, I actually wrote a >>Python program that used an NNTP client library to check all the >>reference values in a large sample of posts from both sources.] > > Unfortunately, my client can only "pre filter" on subject and author; I > could kill all @gmail.*, but could not focus on just Google Groups > submissions... > > Probably time for me to spend the $19 to upgrade from Agent 6.0 Or use the One True Newsreader: SLRN. ;) -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Reading python list as a newsgroup (was ...)
On 2016-12-17, D'Arcy Cain wrote: > On 2016-12-16 08:16 PM, Dennis Lee Bieber wrote: >> Unfortunately, my client can only "pre filter" on subject and author; I >> could kill all @gmail.*, but could not focus on just Google Groups >> submissions... > > I don't know what client you use but perhaps you can adapt this procmail > recipe. > >:0 Hir > * ^List-Id:.*python-list.python.org > * ^From:.*@gmail.com > * ^Newsgroups:.* > /dev/null That's not what he wants to do. He wants to filter out posts made via Google Groups, not posts sent from people who use gmail addresses. Here's the rule for slrn: Score:: =- Message-ID: .*googlegroups.com -- https://mail.python.org/mailman/listinfo/python-list
python list index - an easy question
Hi, I am new to Python, and I believe it's an easy question. I know R and Matlab. >>> x=[1,2,3,4,5,6,7] >>> x[0] 1 >>> x[1:5] [2, 3, 4, 5] * My question is: what does x[1:5] mean? By Python's convention, the first element of a list is indexed as "0". Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? If I am right, it should print [2,3,4,5,6]. Why does it print only [2,3,4,5]? Thanks!! John -- https://mail.python.org/mailman/listinfo/python-list
Re: python list index - an easy question
On Sat, Dec 17, 2016 at 1:10 PM, John wrote: > > Hi, > >I am new to Python, and I believe it's an easy question. I know R and > Matlab. > > > >>> x=[1,2,3,4,5,6,7] > >>> x[0] > 1 > >>> x[1:5] > [2, 3, 4, 5] > * > > My question is: what does x[1:5] mean? By Python's convention, the first > element of a list is indexed as "0". Doesn't x[1:5] mean a sub-list of x, > indexed 1,2,3,4,5? If I am right, it should print [2,3,4,5,6]. Why does it > print only [2,3,4,5]? > What you are asking about is "slicing". x[1:5] returns everything between index 1 through, but NOT including, index 5. See https://docs.python.org/3/tutorial/introduction.html#strings which will give examples using strings. A bit later the tutorial addresses slicing in a list context. BTW, the Python Tutorial is well worth reading in full! -- boB -- https://mail.python.org/mailman/listinfo/python-list
Re: python list index - an easy question
John wrote: > Hi, > >I am new to Python, and I believe it's an easy question. I know R and >Matlab. > > x=[1,2,3,4,5,6,7] x[0] > 1 x[1:5] > [2, 3, 4, 5] > * > > My question is: what does x[1:5] mean? By Python's convention, the > first element of a list is indexed as "0". Doesn't x[1:5] mean a > sub-list of x, indexed 1,2,3,4,5? If I am right, it should print > [2,3,4,5,6]. Why does it print only [2,3,4,5]? Python uses half-open intervals, i. e. the first index is included, but the last index is not: x[1:5] == [x[1], x[2], x[3], x[4]] The advantage of this convention is that it allows easy splitting and length spefication. >>> items [0, 10, 20, 30, 40, 50, 60, 70, 80, 90] To swap the head and tail: >>> gap = 5 >>> items[gap:] + items[:gap] [50, 60, 70, 80, 90, 0, 10, 20, 30, 40] To extract a stride of a given length: >>> start = 2 >>> length = 3 >>> items[start: start + length] [20, 30, 40] The disadvantage is that not everybody follows this convention... -- https://mail.python.org/mailman/listinfo/python-list
[RELEASE] Python 2.7.13
It is my pleasure to announce the release of Python 2.7.13, the latest bugfix release of the venerable Python 2.7 series. This release incorporates conservative bugfixes as well as improvements to keep Python 2.7 running on modern systems. The only change from the 2.7.13 release candidate 2 weeks ago is the revert of a change that broke backwards compatibility with some rare C extension patterns. See https://bugs.python.org/issue5322 for more details. Source archives and binaries are available at https://www.python.org/downloads/release/python-2713/ Please report bugs to our tracker: https://bugs.python.org/ 2.7.14 will appear mid-2017. All the best in the new year, Benjamin Peterson 2.7 release manager -- https://mail.python.org/mailman/listinfo/python-list
Re: python list index - an easy question
On 12/17/2016 2:10 PM, John wrote: Hi, I am new to Python, and I believe it's an easy question. I know R and Matlab. x=[1,2,3,4,5,6,7] x[0] 1 x[1:5] [2, 3, 4, 5] * My question is: what does x[1:5] mean? The subsequence between slice positions 1 and 5, length 5-1=4. Slice positions are before and after each item, not through them. There are n+1 slice positions for n items: 0 before the first, 1 to n-1 between pairs, and n after the last. Think of slice positions as tick marks on a line with the length 1 segment between as a cell holding a reference to one item. a b c d e +-+-+-+-+-+ 0 1 2 3 4 5 Slice 1:4 of length 3 is sequence with b, c, d. Slice 3:3 of length 0 is an empty sequence. By Python's convention, the first element of a list is indexed as "0". Think of 0 as .5 rounded down, or represent by the lower bound. Other language round up to 1, or use the upper bound. Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? No. It is items between 1:2, 2:3, 3:4, 4:5. Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Best attack order for groups of numbers trying to destroy each other, given a victory chance for number to number attack.
Unless you are capable of expressing problems in numerical terms you'll have very hard luck having it computed by a computer ! ;) I did find some interesting docs about "decision trees". Exhaustive search Branch and Bound Hill Climbing Random/Best Effort Solutions and so forth. A computer programmer should be able to handle abstract descriptions as well like I gave in my initial posting. What kind of "dressing" you want to give it is up to you, the dressing won't solve it though ! :) Meanwhile I have also consider some kind of lookup table... or an intermediate table like you described... Not yet sure if that will be of any help... (I am in doubt about the dynamic nature of the battles... perhaps an intermediate table would be wrong... or maybe it might be right, dynamic deaths vs staying alive inside intermediate table and such). I also tried the static approach by multiplieing chances instead of subtracting the chances like damage done. I also experimented with "Pascal/Delphi sets" in combination with arrays... this produced a very easy/simple permutation algorithm which consisted out of multiple for loops, which is kinda the obvious way of easily creating permutations... but it was still kinda interesting. There is a problem with such loops though, also like the one you mention "for each" and "for in" and such... "under water" the compiler will loop the entire range of the type, and will use compare statements to see if it should enter the inner loop. So there is hidden overhead associated with it. Thus such "generating" of permutations on the fly might have a hidden overhead cost associated with it, even if this was not the case, such for loop constructs will definetly have overheads in certain situations and this can quickly get out of hand and could even consume more processing time then the entire program. I did also briefly considered this permutation loop, though it would require a lot of programming, 8x4=24 for loops, plus possible an additional attack loop. You do have a little point that without a clear story it might be hard to understand the exact problem. I may consider describing the problem one more time, but it might still be with number or blue berries ! LOL :) -- https://mail.python.org/mailman/listinfo/python-list
Re: python list index - an easy question
On 17/12/2016 19:10, John wrote: Hi, I am new to Python, and I believe it's an easy question. I know R and Matlab. x=[1,2,3,4,5,6,7] x[0] 1 x[1:5] [2, 3, 4, 5] * My question is: what does x[1:5] mean? x[A:B] means the slice consisting of x[A], x[A+1],... x[B-1]. (Although slices can shorter including those with be 0 or 1 elements.) By Python's convention, the first element of a list is indexed as "0". Or the slice from the (A+1)th element to the B'th element inclusive, if you are informally using ordinal indexing (first, second, third etc). Doesn't x[1:5] mean a sub-list of x, indexed 1,2,3,4,5? Sublists and slices, once extracted, are indexed from 0 too. Play around with some test code, but avoid test data containing numbers that are not too different from possible indices as that will be confusing! Strings might be better: x = "ABCDEFGHIJKLM" print (x[1:5]) displays: BCDE print (x[1:5][0:2])# slice of a slice displays: BC -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Best attack order for groups of numbers trying to destroy each other, given a victory chance for number to number attack.
Specially for Dennis, a nice story: There are four little animals. The first animal is a tiger, the second animal is a deer, the third animal a parrot, the fourth animal is a fish. The animals are very angry at each other and want to EAT each other ! =D However each animal has a certain specialization. For example, the tiger has very sharp teeth, but can't hold it's breath for long under water. The parrot has the ability to drive others insane with his chilping. The fish can lure other animals into the water in hopes of drowning them. And the deer can lure them into the swamp or make them fatigued from chasing them. Fortunately for the four animals they don't need to EAT each other because another group of animals have arrived which are exactly like them. So the four animals have decided to gang UP like negros in the GETTO ! LOL. It's now GANG vs GANG. But the poor little animals have run into a problem ?! Which of the other four animals should they EAT first ?! Or should they even attack multiple at the same time ?! Should the tiger fight the tiger ? Should the fish fight the fish ? Should the parrot fight the parrot ? Should the deer fight the deer ? Or perhaps ? Should the tiger eat the fish first ? Should the fish try to drown the tiger first ? Should the tiger and the fish gang up on the tiger ? But what about the enemy fish ? What will it do ? Should all four animals attack the enemy tiger ? or another animal ? Also for the gang to achieve victory all four enemy animals must be EATEN ! Now these animals wonder to themselfes ? How should we attack the enemy ? Who should we attack first ? Every individual attack has a chance of success as given by the victory table. (A survival table could be calculated as well. Which would be the inverse of this). Many possibilities for attack exists ?! Which one do you recommend for them ?! Keep in mind that each animal is free to decide it's own attack plan. They do not need to agree with you. Imagine yourself to be one of the animals. Which attack strategy for yourself would be best to use to maximize your GANG of winning, no matter what other attack plans the animals have. An example of a possible attack: Red Tiger has his own attack plan eat in this order: Blue Tiger, Blue Fish, Blue Deer, Blue Parrot Red Deer has his own attack plan eat in this order: Blue Fish, Blue Deer, Blue Parrot, Blue Tiger Red Fish has his own attack plan eat in this order: Blue Fish, Blue Parrot, Blue Deer, Blue Tiger Red Parrot has his own attack plan eat in this order: Blue Parrot, Blue Fish, Blue Deer, Blue Tiger and vice versa for blue team ! ;) :) -- https://mail.python.org/mailman/listinfo/python-list
Re: The right way to 'call' a class attribute inside the same class
On 16/12/16 01:17, Chris Angelico wrote: On Fri, Dec 16, 2016 at 11:36 AM, Erik wrote: On 12/12/16 23:23, Chris Angelico wrote: In JavaScript, it's normal to talk about "calling a function as a constructor". When you do, there is a 'this' object before you start. No there isn't. There is an implicit binding of a variable called "this" based on the syntactic sugar of whether you're calling a function as method on an object or not. In "strict" mode, [blah, blah, blah] I'm talking about when you call a function as a constructor: "new Foo()". Doesn't that have a 'this' object before the function starts? Yes, in that case there is (I didn't grok that you meant using 'new' by "calling a function as a constructor", but it's obvious now you spell it out). I wish I could find the resource I originally learned this stuff from, because it's quite enlightening and I'd like to link to it here - if one understands how things work generally under the covers it all makes much more sense, but I guess that's also a bad advert for a language (and why a lot of people get confused at first, and why it's a bit of a mess ;)). But yes you're correct, in the case of using "new Func()" then "Func" is called with an implicit binding of 'this' that is to a newly created object. Regards, E. -- https://mail.python.org/mailman/listinfo/python-list
Re: The right way to 'call' a class attribute inside the same class
On Sun, Dec 18, 2016 at 12:01 PM, Erik wrote: > Yes, in that case there is (I didn't grok that you meant using 'new' by > "calling a function as a constructor", but it's obvious now you spell it > out). Yeah. I thought that I got that terminology from MDN, but I can't find it now, so it must have been from elsewhere. > I wish I could find the resource I originally learned this stuff from, > because it's quite enlightening and I'd like to link to it here - if one > understands how things work generally under the covers it all makes much > more sense, but I guess that's also a bad advert for a language (and why a > lot of people get confused at first, and why it's a bit of a mess ;)). Sounds like how Michael Schwern introduces his "Git for Ages 4 and Up" talk - git may have a pretty poor UI, but has such beautiful innards that it's best to understand it in that way. And he agrees that this isn't how you _normally_ want to do things. When I explain Python, I don't start by explaining the byte code interpreter or the operand stack :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Unicode script
On 2016-12-16 02:44, MRAB wrote: On 2016-12-15 21:57, Terry Reedy wrote: On 12/15/2016 1:06 PM, MRAB wrote: On 2016-12-15 16:53, Steve D'Aprano wrote: Suppose I have a Unicode character, and I want to determine the script or scripts it belongs to. For example: U+0033 DIGIT THREE "3" belongs to the script "COMMON"; U+0061 LATIN SMALL LETTER A "a" belongs to the script "LATIN"; U+03BE GREEK SMALL LETTER XI "ξ" belongs to the script "GREEK". Is this information available from Python? More about Unicode scripts: http://www.unicode.org/reports/tr24/ http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt Interestingly, there's issue 6331 "Add unicode script info to the unicode database". Looks like it didn't make it into Python 3.6. https://bugs.python.org/issue6331 Opened in 2009 with patch and 2 revisions for 2.x. At least the Python code needs to be updated. Approved in principle by Martin, then unicodedata curator, but no longer active. Neither, very much, are the other 2 listed in the Expert's index. From what I could see, both the Python API (there is no doc patch yet) and internal implementation need more work. If I were to get involved, I would look at the APIs of PyICU (see Eryk Sun's post) and the unicodescript module on PyPI (mention by Pander Musubi, on the issue). For what it's worth, the post has prompted me to get back to a module I started which will report such Unicode properties, essentially the ones that the regex module supports. It just needs a few more tweaks and packaging up... Finally completed and uploaded! It's called 'uniprop' and it's at: https://pypi.python.org/pypi/uniprop/1.0 For Python 3.4-3.6. -- https://mail.python.org/mailman/listinfo/python-list
