Data storing

[Anup reni]

Hi I'm new to python and planning to make a web app which contains a large
amount of data and involves comparing two data sets. Which is better to use
in python, a Dictionary or Mysql?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data storing

[dieter]

Anup reni  writes:

> Hi I'm new to python and planning to make a web app which contains a large
> amount of data and involves comparing two data sets. Which is better to use
> in python, a Dictionary or Mysql?

Is it important for your data to be persistent (i.e. live across
restarts of your web app)? If so (and I assume this is the case), then
you need some form of persistence. "MySQL" comes with such persistency;
with a Python "dict" you would need persistency in addition.

For more guidance, a clearer description of "large amount" would
be necessary. Python "dict"s usually are maintained in memory;
data maintained there should not exceed the order of GB. In a "MySQL" database
you can store larger amounts of data.

Further considerations can be how you need to access the data sets.
Python "dict"s support efficient access by a single key; "MySQL"
supports various efficient access paths based on the indexes you define.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data storing

[Steven D'Aprano]

On Tue, 2 Feb 2016 01:19 pm, Anup reni wrote:

> Hi I'm new to python and planning to make a web app which contains a large
> amount of data and involves comparing two data sets. Which is better to
> use in python, a Dictionary or Mysql?

"Hi, I'm planning to transport a large number of objects from one place to
another. Which is better for me to use, a ship or an airplane?"

:-)

Well, it depends.

How much data? You might think 100 MB is a lot of data, but Python won't.

How are you planning to compare them? What sort of comparisons do you intend
to do?

If all you have is a set of key:value pairs, then dicts are the obvious
choice. They are built-in and very fast, but unordered. They'll be good for
millions or tens of millions of records, but they aren't persistent data
storage. You will have to load them to and from disk each time the
application starts. Perhaps the "shelve" module might help you with that.

If you have many different relationships, and you want, or need, to use SQL
commands to do some initial data processing, then you will certainly need a
real database, not dicts.

I would prefer Postgresql rather than MySQL, **especially** if you need to
support non-ASCII (Unicode) data.

But really, to decide which would be best for you, we would need to know a
lot more about your application and its requirements.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of list comprehensions

[Steven D'Aprano]

Hi Arsh, and welcome!

On Wed, 3 Feb 2016 12:30 am, arsh...@gmail.com wrote:

> I am having some understandable behaviour from one of my function name
> week_graph_data() which call two functions those return a big tuple of
> tuples, But in function week_graph_data() line no. 30 does not
> work(returns no result in graph or error).
> 
> I have check both functions are called in week_graph_data() individually
> and those run perfectly. (returning tuple of tuples)
> 
> here is my code: pastebin.com/ck1uNu0U

There's no need to waste your time, and other people's time, by making a
pastebin. Since it's less than 30 lines, just include it in your email.

(Be careful with Gmail that it doesn't kill the indentation. Just make sure
you send plain text, with no "rich text" -- actually HTML -- formatting.)



def week_list_func():
""" A function returning list of last 6 days in 3-tuple"""
date_list = [] 
for i in xrange(7):
d=date.today() - timedelta(i)
t = d.year, d.month, d.day
date_list.append(t)
return reversed(date_list)

def week_graph_data():
day_list_inbox = inbox_week()  # returns big Tuple of tuples   
day_list_outbox = outbox_week() # Same as inbox_week()
dates_inbox = tuple(a[:-7] for a in day_list_inbox)
dates_outbox = tuple(a[:-7] for a in day_list_outbox)
week_list = week_list_func()
bar_chart = pygal.Bar()
bar_chart.title = 'Weekly Email Analysis'
bar_chart.x_labels = (list(week_list_func()))
bar_chart.add('Received', 
[sum(t==i for t in dates_outbox) for i in list(week_list)]
)
bar_chart.add('Sent', 
[sum(m==n for n in dates_inbox) for m in list(week_list)]
)
week_chart = bar_chart.render(is_unicode = True)
return bar_chart.render_to_file('ok_.svg')

week_graph_data()



Your problem is that week_list_func() returns an iterator, so once you
convert it to a list the first time, it becomes empty. For example:


py> values = [1, 2, 3, 4, 5, 6, 7]
py> days = reversed(values)
py> print days

py> print list(days)
[7, 6, 5, 4, 3, 2, 1]
py> print list(days)
[]


Here, days is an iterator. Once you consume the iterator completely the
first time, there's nothing left, so the second time you get an empty list.

You can fix this by having week_list_func() return a list or tuple:


# Version 1
def week_list_func():
""" A function returning list of last 6 days in 3-tuple"""
date_list = [] 
for i in xrange(7):
d=date.today() - timedelta(i)
t = d.year, d.month, d.day
date_list.append(t)
return date_list[::-1]  # reverse the list with a slice


# Version 2
def week_list_func2():
""" A function returning list of last 6 days in 3-tuple"""
date_list = [] 
for i in xrange(6, -1, -1):
d=date.today() - timedelta(i)
t = d.year, d.month, d.day
date_list.append(t)
return date_list


Because this function returns an actual list, there is no need to call
list() on the result.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data storing

[Joel Goldstick]

On Tue, Feb 2, 2016 at 5:33 AM, Steven D'Aprano  wrote:

> On Tue, 2 Feb 2016 01:19 pm, Anup reni wrote:
>
> > Hi I'm new to python and planning to make a web app which contains a
> large
> > amount of data and involves comparing two data sets. Which is better to
> > use in python, a Dictionary or Mysql?
>
> "Hi, I'm planning to transport a large number of objects from one place to
> another. Which is better for me to use, a ship or an airplane?"
>
> :-)
>
> Well, it depends.
>
> How much data? You might think 100 MB is a lot of data, but Python won't.
>
> How are you planning to compare them? What sort of comparisons do you
> intend
> to do?
>
> If all you have is a set of key:value pairs, then dicts are the obvious
> choice. They are built-in and very fast, but unordered. They'll be good for
> millions or tens of millions of records, but they aren't persistent data
> storage. You will have to load them to and from disk each time the
> application starts. Perhaps the "shelve" module might help you with that.
>
> If you have many different relationships, and you want, or need, to use SQL
> commands to do some initial data processing, then you will certainly need a
> real database, not dicts.
>
> I would prefer Postgresql rather than MySQL, **especially** if you need to
> support non-ASCII (Unicode) data.
>
> But really, to decide which would be best for you, we would need to know a
> lot more about your application and its requirements.
>
>
> --
> Steven
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Steven, in your case, how about the star trek transporter?

Sorry, its early for me, and I want to be respectful, but questions like
this seem to have a little flavor of "who would win? super man or mighty
mouse?"

-- 
Joel Goldstick
http://joelgoldstick.com/stats/birthdays
-- 
https://mail.python.org/mailman/listinfo/python-list

Behaviour of list comprehensions

[arsh . py]

I am having some understandable behaviour from one of my function name 
week_graph_data() which call two functions those return a big tuple of tuples, 
But in function week_graph_data() line no. 30 does not work(returns no result 
in graph or error). 

I have check both functions are called in week_graph_data() individually and 
those run perfectly. (returning tuple of tuples)

here is my code: pastebin.com/ck1uNu0U
-- 
https://mail.python.org/mailman/listinfo/python-list

testfixtures 4.8.0 Released!

[Chris Withers]

Hi All,

I'm pleased to announce the release of testfixtures 4.8.0 featuring the 
following:


- More succinct mocking with Replacer:

For setUp usage:

class MyTests(TestCase):

def setUp(self):
replace = Replacer()
replace('x.y.z', Mock())
self.addCleanup(replace.restore)

For one-shot context manager usage:

with Replace('x.y.z', Mock()) as the_mock:
...

Multi-replace context manager usage:

with Replacer() as replace:
dt = replace('x.y.z.datetime', test_datetime())
mock = replace('x.y.do_it', Mock())

- Two context managers for testing warnings:

with ShouldWarn(UserWarning("foo")):
warnings.warn('foo')

with ShouldNotWarn():
... code that should raise no warnings ...

The package is on PyPI and a full list of all the links to docs, issue 
trackers and the like can be found here:


https://github.com/Simplistix/testfixtures

Any questions, please do ask on the Testing in Python list or on the 
Simplistix open source mailing list...


cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
   - http://www.simplistix.co.uk
--
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of list comprehensions

[Peter Otten]

arsh...@gmail.com wrote:

> I am having some understandable behaviour from one of my function name
> week_graph_data() which call two functions those return a big tuple of
> tuples, But in function week_graph_data() line no. 30 does not
> work(returns no result in graph or error).
> 
> I have check both functions are called in week_graph_data() individually
> and those run perfectly. (returning tuple of tuples)
> 
> here is my code: pastebin.com/ck1uNu0U

> def week_list_func():
> """ A function returning list of last 6 days in 3-tuple"""
>
> date_list = []
> for i in xrange(7):
> d=date.today() - timedelta(i)
> t = d.year, d.month, d.day
> date_list.append(t)
> return reversed(date_list)

reversed(date_list) returns an "iterator":

>>> items = reversed(["a", "b", "c"])
>>> items


This means that iteration will "consume" the items

>>> list(items)
['c', 'b', 'a']
>>> list(items)
[]

and thus work only once. To allow for multiple iterations you can return a 
list instead:

>>> items = ["a", "b", "c"]
>>> items.reverse()
>>> items
['c', 'b', 'a']
>>> list(items)
['c', 'b', 'a']
>>> list(items)
['c', 'b', 'a']

Note that list.reverse() is a mutating method and by convention returns 
None. Therefore

   return date_list.reverse() # wrong!

will not work, you need two lines

   date_list.reverse()
   return date_list

in your code.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Heap Implementation

[Oscar Benjamin]

On 2 February 2016 at 05:38, Steven D'Aprano
 wrote:
>
> In effect, each measurement you take is made up of two components:
>
> * the actual time that the code would take if it had exclusive
>   access to the machine with no other programs running, call it t;
>
> * and the noise added by the system, call it Δt.
>
> It is impossible to measure t alone, you always get t+Δt. The aim in timing
> is to reduce the effect of the noise as much as possible.

That's a reasonable model in some cases but not all. In particular the
"actual time" part is itself not a constant but rather depends on all
sorts of things like RAM caching etc. On the CPython side there could
be all kinds of optimisations that would kick in at different times to
change the "actual time". Likewise there are optimisations at OS and
hardware levels that kick in at different times.

> The way to do this with timeit is:
>
> - don't run extraneous code as part of your test (just measure what
>   you care about, with as little scaffolding around it);
>
> - run that code snippet as many times as you can bear in a loop,
>   *but* let timeit manage the loop; don't try doing it by hand;

There are many situations where running something in a loop over and
over triggers all kinds of optimisations. PyPy's jit warmup is an
example but the effect can occur at the hardware level as well.
Performance of code run repeatedly in a tight loop can be different to
the same code called once or called sporadically in a long-running
process.

This is especially true of code that operates on a data structure. In
real usage the data structure might be accessed rarely but a
tight-profiling loop forces the entire structure into the CPU's
caches.

> - only care about "best of N", where N is as large a number as you
>   can stand;
>
> - averages, standard deviation, and other statistics are pretty
>   much useless here, the only statistic that you care about is
>   the minimum.

This argument makes sense if you adopt the t+Δt model described above
but that model it is just a model. The minimum from micro-benchmarking
in a tight-loop is not necessarily a more representative statistic
than mean, median or whatever else. Standard deviation is actually
useful for assessing the variability of time taken which can also be
important and gives additional information about performance: I would
like to minimise both the expected time taken and also the variability
in that simultaneously if possible.

You seem to be describing timeit's way of profiling which is
reasonable but it's not the only way.

--
Oscar
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: psss...I want to move from Perl to Python

[paul.hermeneutic]

On Jan 29, 2016 6:51 AM, "Dennis Lee Bieber"  wrote:
>
>
http://www.barnesandnoble.com/w/perl-to-python-migration-martin-c-brown/1004847881?ean=9780201734881

Given that this was published in 2001, surely it is time for a second
edition.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Heap Implementation

[Sven R. Kunze]

On 02.02.2016 01:48, srinivas devaki wrote:


On Feb 1, 2016 10:54 PM, "Sven R. Kunze" > wrote:

>
> Maybe I didn't express myself well. Would you prefer the sweeping 
approach in terms of efficiency over how I implemented xheap currently?

>

complexity wise your approach is the best one of all that I have seen 
till now


> Without running some benchmarks, I have absolutely no feeling which 
approach is faster/more memory efficient etc.

>

this is obviously memory efficient but I don't know whether this 
approach would be faster than previous approaches, with previous 
approaches there is no call back into Python code from C code for 
comparison.
but this should be faster than HeapDict as HeapDict is directly using 
its own methods for heappush, heappop etc




Yes. So, it remains to be seen until I implemented the sweeping and 
compared them to each other.



PS: if you have time, could you please review my pull request.



Indeed, I am already thinking about it. :)


Best,
Sven
--
https://mail.python.org/mailman/listinfo/python-list

Re: psss...I want to move from Perl to Python

[Martin A. Brown]

Hello,

>http://www.barnesandnoble.com/w/perl-to-python-migration-martin-c-brown/1004847881?ean=9780201734881
>
>Given that this was published in 2001, surely it is time for a 
>second edition.

How many times do you think somebody migrates from Perl to Python?!

  ;)

-Martin

P.S.  I was amused when I first discovered (about 15 years ago)
   Martin C. Brown, an author of Perl books.  I am also amused to 
   discover that he has written one on Python.  Too many of us
   chaps named 'Martin Brown'.

 https://en.wikipedia.org/wiki/Radio_Active_(radio_series)
 the incompetent hospital-radio trained Martin Brown (Stevens)

P.P.S.  In case it is not utterly clear, I am not the above author.

-- 
Martin A. Brown
http://linux-ip.net/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: psss...I want to move from Perl to Python

[Rick Johnson]

On Sunday, January 31, 2016 at 9:35:17 PM UTC-6, MRAB wrote:
> In [jmf's] defence, he _was_ the one who drew attention to
> the unexpected slowness of the FSR under certain
> circumstances in the Windows build of Python 3.3, albeit
> in a rather over-dramatized way.

Thanks MRAB. You and Terry may be only "high ranking"
members who are willing to defend those who cannot defend
themselves.

"jmf's contribution" is solid evidence of why we should
never discourage anyone's participation in this community.
Yes, jmf's posts are mostly repetitive, but we need to
listen to all opinions and all complaints. Python cannot
survive unless it maintains a solid community of people who
(1) enjoy using the language, and (2) who know that others
will listen when they voice a concern.

Heck, i was banned from Python-ideas before i ever sent my
first reply. Can anyone justify such tyranny? My "communal
dignity" has been stolen from me!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: psss...I want to move from Perl to Python

[Rick Johnson]

On Sunday, January 31, 2016 at 9:15:38 PM UTC-6, Chris Angelico wrote:
> That said, though, I would GREATLY prefer Rick to post
> less provocatively.

I'll admit that my tone can "sometimes" be acerbic, but
it's not like i was attacking someone for *NO* justifiable
reason, in fact, my response was not an attack at all, but
merely, a plea.

There is no doubt that a strong undercurrent of "militant
anti-OOP" exists within this group (I know, because i've
witnessed it for years), and those same militants have
attempted to cast *ME* as a some sort of "religious OOP
fanatic" -- but nothing could be further from the truth!

Yes, (sigh) i've written a lot of Java code (quick, muster
the militants!), and whilst I feel that the structural
rigidity that java demands *does* encourage good
programmers to write consistent code, it *does not* prevent
poor programmers from writing garbage, nor does it allow me
to fully express myself. Heck, Python's "multi-paradigm"
nature, is IMHO, one of it's greatest strengths!!!

We humans crave familiarity, because, those concepts that
are familiar to us don't require any thought, no, we just
wield them instinctively. However, we loath the unfamiliar,
because, the unfamiliar requires us to expend mental effort
*LEARNING* about the concept, *BEFORE* we can benefit from the
fruits of it's power. Therefore, "the familiar" is like
having a "slightly cool, diseased-donkey-piss beer" in your
hand, and "the unfamiliar" is like having an "ice-cold,
quality craft beer" way down in your basement refrigerator
-- sometimes, we'd rather drink donkey-piss, than get off
the couch. 

> But I'm not calling for his banning any more than I'd call
> for Terry's, or my own, for that matter (I've posted
> plenty of off-topic or otherwise useless posts).

Chris, we don't always agree, but when we do, i must admit,
it's always a *VERY* pleasurable experience. Thank you! :-)

-- 
https://mail.python.org/mailman/listinfo/python-list

[STORY-TIME] THE BDFL AND HIS PYTHON PETTING ZOO

[Rick Johnson]

 [STORY-TIME] THE BDFL AND HIS PYTHON PETTING ZOO


A long, long time a ago, in a sleepy little Scandinavian
village, somewhere outside of Stockholm, a wee little boy
was born, who would one day become known as: "the BDFL". And
as all little boys love to do, he enjoyed frolicking around
the countryside in his wee little wooden shoes, picking
tulips for his mother, and taking rides on the arms of
nearby windmills. (oh, how cute!) And as he grew older, he
started to become interested in math and logic, and he
became fascinated with the power of software, yes, and he
*EVEN* dreamed of one day coding his own calculator app...

  BUT THAT GROUND BREAKING APP WOULD HAVE TO WAIT, 
  
  FOR SOON, FATE WOULD INTERVENE!

One day, when he was basking in the warm Scandinavian summer
sun, surrounded by tall blades of grass blowing wistfully in
the wind, a snake slithered close by, and began speaking
"great things" into his ear. And the snake told him that
everything in the programming world was completely wrong.
And that the syntax had become nothing more than
obnoxious line noise, and that, "religious paradigm
fanaticism" was going to pollute, and eventually destroy,
the pristine "code wilds".

And he, being a wise lad, knew the snake was right! 

So he listened contently, and he learned. And he fell deeply
in love with the snake, and the snake became his best, and
*ONLY*, friend. And for many years, he, and the snake,
developed a strong bond. And he swore that he would never
leave, or harm the snake. And as the boy grew into early
adulthood, a new and mysterious force began to shape his
world, and this new force was called "idealism". And his
idealism implored him to share the snake with the world, so
that *ALL* the children of the world might become
enlightened, by the wisdom of the snake.

  SO HE OPENED A LITTLE PETTING ZOO, 
  
  AND HE DUBBED IT: "THE PSF".

And many children came from the far and wide, and they
would pet his snake, and they would play with his snake, and
they would learn from his snake's philosophy. And the
children became so happy with the his altruistic nature,
that they affectionately referred to him as their
"Benevolent Dictator For Life" (aka: BDFL). And so, the BDFL
was happy. He was happy because the children who came to his
petting zoo loved him, but most importantly, he was happy
that they never questioned his "petting rules", or the
philosophy of his snake.

And he became an instant worldwide celebrity. And was
interviewed by reporters, and documented in magazines. He
even became worshiped, yes, like a god, which caused his
pride to expand to new and dangerous levels. In fact, his
pride became so great, that in one interview, he snarky
dismissed the dangers of his snake's "wildly dynamic
behavior", and claimed that, he would be fully content
as a passenger in an airplane with his snake as the pilot...

  MOST PETTING-ZOO OWNERS THOUGHT HE WAS NUTS, 
  
  BUT HE *REFUSED* TO RETRACT THE STATEMENT.
  
As time went on, a small "inner group of disciples" began to
develop around him. These disciples followed him everywhere.
A band of loyal sycophants, or "Umpa Lumpas" if you will,
who would blindly follow any demand he dictated, no matter
how ridiculous.

And he and his disciples became repulsed by the never ending
hordes of children who would visit the petting zoo, they saw
these "other children" as unclean, so they decided to create
two holy places called "Python-Ideas" and "Python-Dev"
(aka: the holy of holies), and they declared that, only
those who are willing to lick the BDLF's boots, may enter
these holy places.

Soon after, the BDFL took permeate refuge in these holy
places, and, from his rectory, he would sneer down at the
children below, and if any child dared to question his
motives, or broke one of his "petting rules", or even, did
not goose-step to his political ideology, he would call for
his minions to hurl furious rebukes at the visitor, and he
would ensure the child was banned from ever entering his
holy places. And although he could not prevent a child from
entering the petting zoo commons, he would loose his minions, 
to pester and mock, the banned children, when they did visit.

And for the first two years, this is how the BDFL managed
his petting zoo. And although he was not happy, by most
people's standards, he found peace within his "little
refuge".


 BUT TROUBLE WAS SOON ON THE HORIZON...


Near the end of the 2nd year, of running his petting zoo, the
BDFL had become bored, because, even though the children of
the world had never lost interest in his snake, most of them
had lost interest in him, and since he was mostly absent
from the petting zoo, he became something of myth, even a
legend...

He realized, that he had become no