[Tutor] time vs. timeit

[diliup gabadamudalige]

Hi all,

1. why do some say that the time module is more accurate than the timeit
module?
s = time.time()
or
s = timeit.timeit()

2. Why is it that both modules never return the same answer on each run?

Thank you for your response.

-- 
Diliup Gabadamudalige

http://www.diliupg.com
http://soft.diliupg.com/

**
This e-mail is confidential. It may also be legally privileged. If you are
not the intended recipient or have received it in error, please delete it
and all copies from your system and notify the sender immediately by return
e-mail. Any unauthorized reading, reproducing, printing or further
dissemination of this e-mail or its contents is strictly prohibited and may
be unlawful. Internet communications cannot be guaranteed to be timely,
secure, error or virus-free. The sender does not accept liability for any
errors or omissions.
**
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] time vs. timeit

[Chris “Kwpolska” Warrick]

On Tue, Aug 26, 2014 at 6:26 AM, diliup gabadamudalige
 wrote:
> Hi all,
>
> 1. why do some say that the time module is more accurate than the timeit
> module?
> s = time.time()
> or
> s = timeit.timeit()
>
> 2. Why is it that both modules never return the same answer on each run?

The two functions have completely different uses, and do completely
different things.

>>> help(time.time)
Help on built-in function time in module time:

time(...)
time() -> floating point number

Return the current time in seconds since the Epoch.
Fractions of a second may be present if the system clock provides them.


In other words, return this: http://en.wikipedia.org/wiki/Unix_time


>>> help(timeit)
NAME
timeit - Tool for measuring execution time of small code snippets.

[…]
 |  timeit(self, number=100)
 |  Time 'number' executions of the main statement.
 |
 |  To be precise, this executes the setup statement once, and
 |  then returns the time it takes to execute the main statement
 |  a number of times, as a float measured in seconds.  The
 |  argument is the number of times through the loop, defaulting
 |  to one million.  The main statement, the setup statement and
 |  the timer function to be used are passed to the constructor.

-- 
Chris “Kwpolska” Warrick 
PGP: 5EAAEA16
stop html mail | always bottom-post | only UTF-8 makes sense
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] simple unicode question

[Albert-Jan Roskam]

- Original Message -
> From: Danny Yoo 
> To: Albert-Jan Roskam 
> Cc: Python Tutor Mailing List 
> Sent: Saturday, August 23, 2014 2:53 AM
> Subject: Re: [Tutor] simple unicode question
> 
> On Fri, Aug 22, 2014 at 2:10 PM, Albert-Jan Roskam
>  wrote:
>>  Hi,
>> 
>>  I have data that is either floats or byte strings in utf-8. I need to cast 
> both to unicode strings.
> 
> 
> Just to be sure, I'm parsing the problem statement above as:
> 
>     data :== float
>                 | utf-8-encoded-byte-string


Yep, that's how I meant it :-)

> because the alternative way to parse the statement in English:
> 
>     data :== float-in-utf-8
>                 | byte-string-in-utf-8
> 
> doesn't make any technical sense.  :P
> 
> 
> 
> 
>>  I am probably missing something simple, but.. in the code below, under 
> "float", why does [B] throw an error but [A] does not?
>> 
> 
>>  # float: cannot explicitly give encoding, even if it's the default
>  value = 1.0
>  unicode(value)      # [A]
>>  u'1.0'
>  unicode(value, sys.getdefaultencoding())  # [B]
>> 
>>  Traceback (most recent call last):
>>    File "", line 1, in 
>>      unicode(value, sys.getdefaultencoding())
>>  TypeError: coercing to Unicode: need string or buffer, float found
> 
> 
> Yeah.  Unfortunately, you're right: this doesn't make too much sense.
>
> What's happening is that the standard library overloads two
> _different_ behaviors to the same function unicode(). It's conditioned
> on whether we're passing in a single value, or if we're passing in
> two.  I would not try to reconcile a single, same behavior for both
> uses: treat them as two distinct behaviors.
> 
> Reference: https://docs.python.org/2/library/functions.html#unicode

Hi,
 
First, THANKS for all your replies!

Aahh, the first two lines in the link clarify things:
unicode(object='')
unicode(object[, encoding[, errors]])

I would  find it better/clearer if the docstring also started with these two 
lines, or, alternatively, with unicode(*args)

I have tried to find these two functions in the source code, but I can´t find 
them (but I don't speak C). Somewhere near line 1200 perhaps? 
http://hg.python.org/cpython/file/74236c8bf064/Objects/unicodeobject.c


> Specifically, the two arg case is meant where you've got an
> uninterpreted source of bytes that should be decoded to Unicode using
> the provided encoding.
> 
> 
> So for your problem statement, the function should look something like:
> 
> ###
> def convert(data):
>     if isinstance(data, float):
>         return unicode(data)
>     if isinstance(data, bytes):
>         return unicode(data, "utf-8")
>     raise ValueError("Unexpected data", data)
> ###
> 
> where you must use unicode with either the 1-arg or 2-arg variant
> based on your input data.


Interesting, you follow a "look before you leap" approach here, whereas `they` 
always say it is easier to ”ask forgiveness than permission” in python. But 
LBYL is much faster, which is relevant because the function could be called 
millions and millions of times. If have noticed before that try-except is quite 
an expensive structure to initialize (for instance membership testing with ´in´ 
is cheaper than try-except-KeyError when getting items from a dictionary)

In [1]: def convert1(data):
...: if isinstance(data, float):
...: return unicode(data)
...: if isinstance(data, bytes):
...: return unicode(data, "utf-8")
...: raise ValueError("Unexpected data", data)
...:

In [2]: %%timeit map(convert1, map(float, range(10)) + list("abcdefghij"))
1 loops, best of 3: 19 us per loop

In [3]: def convert2(data):
: try:
: return unicode(data, encoding="utf-8")
: except TypeError:
: return unicode(data)
: raise ValueError("Unexpected data", data)
:

In [4]: %timeit map(convert2, map(float, range(10)) + list("abcdefghij"))
1 loops, best of 3: 40.4 us per loop   
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] time vs. timeit

[Steven D'Aprano]

On Tue, Aug 26, 2014 at 09:56:55AM +0530, diliup gabadamudalige wrote:
> Hi all,
> 
> 1. why do some say that the time module is more accurate than the timeit
> module?
> s = time.time()
> or
> s = timeit.timeit()

You will have to ask them.

Since the two functions do very different things, I don't understand how 
they are comparing them. That's a bit like saying that len() is more 
accurate than math.sin().


> 2. Why is it that both modules never return the same answer on each run?

time.time() returns the current time, in seconds since the start of the 
Unix Epoch. At midnight, 1st January 1970 UTC, time.time() would have 
returned 0. One second later, it would return 1. At 3:15:20am Friday 
2nd January 1970, it would return 98120. 

I can check my calculation:

py> import time
py> time.ctime(98120)
'Fri Jan  2 13:15:20 1970'

My local timezone is 10 hours ahead of UTC, so that is correct.

time.time() will not return the same value because time keeps moving 
forward. The only way to get it to return the same value would be to 
set the computer's clock backwards.

timeit.timeit is not like time(). It doesn't return an absolute time, 
but a *difference* between two times. It measures how long it takes to 
run some code. For example, on my computer:

py> timeit.timeit("x = len([])")
0.1940888217650354

getting the length of the empty list [] one million times takes 0.19 
seconds, or about 0.19μs each. On your computer, it may be faster, or 
slower. If I do it again:

py> timeit.timeit("x = len([])")
0.19202891178429127

I get *about* the same value, but not exactly. It's not exactly the same 
speed because my computer is not doing exactly the same thing each time. 
There are other processes running: background tasks, virus checkers, 
internet downloads, the operating system is busy doing whatever it is 
that the operating system does. If I look on my computer, right now, 
there are at least 260 processes running:

[steve@ando ~]$ ps aux | wc -l
260

The amount of time that timeit() reports will depend very slightly on 
how busy the computer is doing other things. You know that for yourself: 
if the computer is overloaded, doing too much, everything slows down. 



-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] simple unicode question

[Steven D'Aprano]

On Tue, Aug 26, 2014 at 03:58:17AM -0700, Albert-Jan Roskam wrote:

> Interesting, you follow a "look before you leap" approach here, 
> whereas `they` always say it is easier to ”ask forgiveness than 
> permission” in python. 

Anyone who says it is *ALWAYS* easier is lying or confused :-)

It is *often*, perhaps even *usually* better to use a AFTP approach, but 
not necessarily easier. Sometimes it is more work, but safer, or better. 
For instance, it is wrong (i.e. buggy) to use a LBYL approach here:


if os.path.exists(filename):
f = open(filename)
process(f.read())

Why is it wrong? Just because the file exists, doesn't mean you can open 
it for reading. And just because it exists at the instant you do the 
test, doesn't mean it will still exist a millisecond later when Python 
tries to open the file. Perhaps another process or user has deleted it 
in the fraction of a second between the two lines.

But sometimes AFTP is the wrong approach too. Consider a case where you 
have to do a number of things, and only if *all* of them can be done do 
you want to proceed. For example, baking a cake:

- pour the cake mix and milk into a bowl;
- mix;
- pour into a pan;
- put into the oven;
- cook for 30 minutes.

If you don't have an oven, there's no point mixing the cake mix and milk 
together, you'll just waste it. So you need to LBYL, make sure you have 
the cake mix AND the milk AND a bowl AND a pan AND an oven before you 
even start. If even one thing is missing, you don't proceed.


> But LBYL is much faster, which is relevant 
> because the function could be called millions and millions of times. 

Not necessarily! It depends on how often you have failures. Let's time 
two examples: looking up something in a dict, where the key is never 
missing. Copy and paste this code:


from timeit import Timer
setup = 'd = {"a": 1, "b": 2}'

t1 = Timer("""
if "a" in d: x = d["a"]
if "b" in d: x = d["b"]
""", setup)  # LBYL version

t2 = Timer("""
try:
x = d["a"]
except KeyError:
pass
try:
x = d["b"]
except KeyError:
pass
""", setup)  # AFTP version



And here are the results when I run it:

py> min(t1.repeat())
0.3348677200265229
py> min(t2.repeat())
0.23994551179930568

So in this case, the LBYL example is significantly slower.

Now let's try it again, only this time the key will be missing half the 
time:


t3 = Timer("""
if "a" in d: x = d["a"]
if "c" in d: x = d["c"]
""", setup)  # LBYL version

t4 = Timer("""
try:
x = d["a"]
except KeyError:
pass
try:
x = d["c"]
except KeyError:
pass
""", setup)  # AFTP version


And the results:

py> min(t3.repeat())
0.24967589927837253
py> min(t4.repeat())
0.8413973557762802

Now the LBYL version is faster.


> If have noticed before that try-except is quite an expensive structure 
> to initialize (for instance membership testing with ´in´ is cheaper 
> than try-except-KeyError when getting items from a dictionary)

Funny you should say that :-)

Actually, try...except is almost free to *initialise*. Setting up the 
try block is very, very cheap, it takes hardly any time:

py> timeit.timeit("len([])")  # No try
0.19250816199928522
py> timeit.timeit("""
... try:
... len([])
... except: pass""")  # With try
0.21191818173974752

That's not a lot of difference: less than 0.02μs.

But, *catching* the exception is quite expensive. So the overhead of 
putting code inside a try block is negligible, virtually free, but the 
cost of actually catching the exception is quite heavy. So if there are 
lots and lots of exceptions, LBYL will probably be faster. But if they 
are rare, then the cost of looking before you leap isn't worth it, you 
should just leap. The exact crossover point will depend on how costly it 
is to look first versus the cost of the exception, but as a very rough 
rule of thumb, I go by:

if *fewer* than 1 in 10 operations will raise an exception, then use 
try...except; but if *more* than 1 in 10 operations will raise an 
exception, and it is safe to do so, then LBYL may be appropriate.



-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Preparing virtualenvwrapper for Django 1.6 development

[Walter Prins]

Hi,

To add:

On 23 August 2014 08:54, Chris “Kwpolska” Warrick  wrote:
>
> On Fri, Aug 22, 2014 at 11:23 PM, Sithembewena Lloyd Dube
>  wrote:
> > I am developing on a Windows 8.1 machine and wold like to setup
> > virtualenvironment via virtualenvwrapper so as to have a properly set up
> > Python development environment.
>
> > Would anyone be so kind as to translate this to Windows speak, as well as
> > perhaps pointing out what file this would go into in a Windows environment?
>
> virtualenvwrapper does not work in Windows.  You should use regular
> virtualenv instead.  Read this:
> http://virtualenv.readthedocs.org/en/latest/virtualenv.html#usage

On Windows I'd also suggest looking at virtualenvwrapper-powershell,
which is a PowerShell version of virtualenvwrapper, (PowerShell being
the effective de-facto modern-day shell for current Windows):
https://pypi.python.org/pypi/virtualenvwrapper-powershell/2.7.1
http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/

Walter
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] datetime substraction

[Anirudh Tamsekar]

Hi,

I'm writing a script to monitor rsyslog (second level check).

Overview:

We have Monit for monitoring rsyslog, however we need a script to do a
second level checks.

I'm looking for a script which will check if rsyslog is being updated, also
log rotation is happening.
Incase of the new file created by logrotate, script should monitor the
latest file. Also monitor last modified time and tail of the log.

Filename Ex. 20140826_debug_log (datechanges with log rotate)

In case of any issues such as file not being updated within the interval of
5 min, alert with disk status.


I have written a script, however I'm not able to get the date substraction
math right, getting the following error
(Searched google and other resources too).

*Traceback (most recent call last):*
*  File "./rsyslog_check.py", line 22, in *
*difft=cur_time-mt*
*TypeError: unsupported operand type(s) for -: 'str' and 'str'*

Request your help. Also point any good documentation where I can get quick
reference.

##
#! /usr/bin/python

# import libraries
import os
import sys
import datetime
import time
import smtplib
import Queue
import threading
import subprocess

# Defining main variables
tfile='/local/rsyslog/20140727_msg_debug.log'
dt = datetime.datetime.now().strftime("%Y%m%d")
file = "_msg_debug.log"
filename = "%s%s" % (dt,file)
filepath = '/local/rsyslog/'+filename
ct = time.ctime(os.path.getctime(tfile))
mt = time.ctime(os.path.getctime(tfile))
cur_time = time.ctime()
difft=int(cur_time-mt)
tdelta=datetime.timedelta(minute=5)
# Main


if os.path.isfile(filepath) == True:

print "%s exists \n" % (tfile)
print "This file was created at %s UTC \n" % (ct)
 *   if difft > tdelta:*
*print "File is not modified, last modified at %s UTC" %
(mt)*
*else:*
*print "File is being modified"*


*else:*
*os.path.isfile(filepath)*


# Tailing the file
tailq = Queue.Queue(maxsize=10) # buffer at most 100 lines

def tail_forever(filepath):
p = subprocess.Popen(["tail", tfile], stdout=subprocess.PIPE)
while 1:
line = p.stdout.readline()
tailq.put(line)
if not line:
break

threading.Thread(target=tail_forever, args=(tfile,)).start()
print "\n # Tailing Log file %s" % (tfile)
print "\n ***"
print tailq.get() # blocks
print tailq.get_nowait() # throws Queue.Empty if there are no lines to read
exit()

-Thanks,
Anirudh Tamsekar
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] datetime substraction

[Steven D'Aprano]

On Tue, Aug 26, 2014 at 02:40:30PM +0530, Anirudh Tamsekar wrote:

> I have written a script, however I'm not able to get the date substraction
> math right, getting the following error
> (Searched google and other resources too).
> 
> *Traceback (most recent call last):*
> *  File "./rsyslog_check.py", line 22, in *
> *difft=cur_time-mt*
> *TypeError: unsupported operand type(s) for -: 'str' and 'str'*

cur_time and mt are strings. You cannot subtract strings.

os.path.getctime(filename) returns the ctime of the file as a number. 
You then convert that number into a string. Don't. Just leave it as a 
number:

mt = os.path.getctime(tfile)
cur_time = time.time()
difft = cur_time - mt


-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] datetime substraction

[Alan Gauld]

On 26/08/14 10:10, Anirudh Tamsekar wrote:


*Traceback (most recent call last):*
*  File "./rsyslog_check.py", line 22, in *
*difft=cur_time-mt*
*TypeError: unsupported operand type(s) for -: 'str' and 'str'*

Request your help. Also point any good documentation where I can get
quick reference.

ct = time.ctime(os.path.getctime(tfile))
mt = time.ctime(os.path.getctime(tfile))


You are converting the times to strings before you subtract them. Don't 
do that, subtract the times as returned by os.path.getctime() and 
time.time().



cur_time = time.ctime()
difft=int(cur_time-mt)


You need to subtract the numeric versions of the times not the strings.

The best quick references are the dir() and help() builtin functions.
For more detail read the documentation for time, os.path (and maybe 
datetime) modules.


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] datetime substraction

[Danny Yoo]

>
> I have written a script, however I'm not able to get the date substraction
> math right, getting the following error
> (Searched google and other resources too).
>
> Traceback (most recent call last):
>   File "./rsyslog_check.py", line 22, in 
> difft=cur_time-mt
> TypeError: unsupported operand type(s) for -: 'str' and 'str'


Hi Anirudh,

This error message suggests that 'cur_time' and 'mt' here are not
dates.  Rather, they're strings.  I'd trust the error message and turn
the question from:

"How do I subtract date subtraction math right?"

to:

"Why are these values that I expect to be dates actually strings?"

Let's see... oh, you're using the time.ctime() function here:



> mt = time.ctime(os.path.getctime(tfile))
> cur_time = time.ctime()
> difft=int(cur_time-mt)


What is this doing?  Let's check.  We can look at the standard library
documentation:

https://docs.python.org/2/library/index.html

Searching...  so this code first uses os.path.getctime(), which takes
a file name and returns the number of seconds since the Epoch.

https://docs.python.org/2/library/os.path.html#os.path.getctime



What's the meaning of time.ctime?

https://docs.python.org/2/library/time.html#time.ctime

Ok, from reading time.ctime(), I think you should not be using this
function.  It takes the number of seconds since the Epoch, and formats
it as a nice string for people to read in their local time zone.  So
it's almost useless for doing any mechanical calculations.  To put it
in perspective, it's as if you are trying to do arithmetic with roman
numerals.  Don't do that.


As Stephen points out, you've got the two dates in terms of seconds
since the Epoch, so you might as well just subtract to get the number
of seconds altogether.


If you truly want to treat dates as algebraic objects, consider the
datetime library.

https://docs.python.org/2/library/datetime.html

You can use the datetime.fromtimestamp() function to take the seconds
since the Epoch and transform them into datetime objects:


https://docs.python.org/2/library/datetime.html#datetime.datetime.fromtimestamp
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor