Re: [Tutor] How to convert string to date time format?
Thanks a lot Steven. The %f is what I was missing.
The "-08:00" is the UTC timezone, which is California, USA, which I believe
is %z.
Thanks!
On Sat, Jul 20, 2019 at 7:50 PM Steven D'Aprano wrote:
> On Fri, Jul 19, 2019 at 10:44:36PM -0400, C W wrote:
> > Hello all,
> >
> > I have a date time string that looks like the following.
> >
> > 02015-07-01 00:01:44.538420-08:00
> > 12015-07-01 00:27:58.717530-08:00
> > 22017-07-01 07:07:48.391376-08:00
>
> I assume that the leading number and spaces "0" etc are NOT part of
> the strings.
>
>
> > I have tried the following two different methods, both did not work.
> > Method one: pandas
> > import pandas as pd
> > stamp = pd.to_datetime(my_string, format='%Y%m%d %H:%M:%S')
> >
> > Method two: datetime package
> > from datetime import datetime
> > datetime.strptime(my_string, '%Y-%m-%d %H:%M:%S')
> >
> >
> > Are both ways suppose to work?
>
> Not unless the string format matches the actual string. You can't expect
> to convert a string unless it matches the format.
>
>
> > Also, does it matter if there are decimals
> > after seconds?
>
> Of course it matters.
>
> Did you read the error message? The single most important skill for a
> programmer is to READ THE ERROR MESSAGE and pay attention to what it
> tells you went wrong:
>
> py> datetime.strptime('2015-07-01 00:01:44.538420-08:00', '%Y-%m-%d
> %H:%M:%S')
> Traceback (most recent call last):
> File "", line 1, in
> File "/usr/local/lib/python3.5/_strptime.py", line 510, in
> _strptime_datetime
> tt, fraction = _strptime(data_string, format)
> File "/usr/local/lib/python3.5/_strptime.py", line 346, in _strptime
> data_string[found.end():])
> ValueError: unconverted data remains: .538420-08:00
>
>
> See how the error message tells you that it couldn't convert the string
> because there is data left over at the end. The first thing to do is
> handle the microseconds.
>
> Googling gets the answer: use "%f" as the code for fractional seconds.
>
> https://duckduckgo.com/?q=strptime+seconds+with+decimals
>
>
> py> datetime.strptime('2015-07-01 00:01:44.538420-08:00', '%Y-%m-%d
> %H:%M:%S.%f')
> Traceback (most recent call last):
> File "", line 1, in
> File "/usr/local/lib/python3.5/_strptime.py", line 510, in
> _strptime_datetime
> tt, fraction = _strptime(data_string, format)
> File "/usr/local/lib/python3.5/_strptime.py", line 346, in _strptime
> data_string[found.end():])
> ValueError: unconverted data remains: -08:00
>
>
> Now we're making progress! The error message has changed. Now you just
> need to decide what the "-08:00" part means, and change the format
> string appropriately.
>
>
>
> --
> Steve
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] multiprocessing: getting data back from the processes
On Tue, 23 Jul 2019 at 00:29, Mats Wichmann wrote: > > I've got a scenario where I need to pass data in both directions of a > multiprocessing program. > So... what else should I be trying to make this a little cleaner? I think if possible it is best to set these things up as an acyclic pipeline so the basic problem here is the two-way communication. How about using separate processes for putting things on the worker input queue and reading from the worker output queue? It probably makes sense for the master process that will outlive the others to be the one that collects the output so maybe spawn a process at the start whose job is putting all the test instances on the input queue. Then spawn the workers who will read the input queue and write to the output queue. Then the master process reads from the output queue until it's done. I think this eliminates any need for bidirectional communication which generally simplifies things. -- Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Python Generator expressions
Hi All,
Need one help in understanding generator expression/comprehensions.
This is my sample code.
# This code creates a generator and not a tuple comprehensions.
my_square =(num *num fornum inrange(11))
print(my_square) # at 0x7f3c838c0ca8>
# We can iterate over the square generator like this.
try:
whileTrue:
print(next(my_square)) # Prints the value 0,1,4
exceptStopIterationasSI:
print("Stop Iteration")
# Another iteration
forx inmy_square:
print(x) # This prints nothing.
Does the generator exhausts its values when we run the iterator once?
Lastly any specific reason for not having a tuple comprehensions?
Have checked this link, but could not understood the reason?
*
https://stackoverflow.com/questions/16940293/why-is-there-no-tuple-comprehension-in-python
Regards,
Animesh
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
[Tutor] a par2 creator and verification program
Hello everyone. I'm thinking through a short program I want to write that will 'par2'/generate ECCs for all of my work files which branch out from a single directory and number approximately 15,000. Specifically: 1) day one: - create a mirror copy of the directory tree empty of all files (there are a bunch of ways in bash of doing this). - recurse down the directory tree which has the files and run a par2 create calculation on each file which generates approximately 10 *.par2 fileblocks. I will then copy the *.par2 fileblocks to the mirror directory tree into the same position as the 'principal file. Therefore assuming 10 *.par2 fileblocks for every actual file, the mirror tree will have around 150,000 *.par2 fileblocks (space and CPU time are a non-issue). 2) day two: - for each file in the primary directory, par2 verify it with respect to its corresponding *.par2 fileblocks in the mirror tree. If it's ok, move on to the next file, if not, repair it, generate a new set of *.par2 fileblocks and copy them over to the mirror. 3) day three: - same as day two, ongoing. I'm aware that most par2 programs need the file and *.par blocks to be in the same location but let's assume I find a way around this. Also, I believe it would be possible to par2 the top directory (which will give me work1.par2 - work10.par2) but the problem is performed this way, the blocks treat all files as a single whole so if I detect corruption, I have no way of locating which file. I'm considering two ways of doing this: Option A: - This seems the most obvious if somewhat inelegant: define a few functions, and incorporate them into a for loop which will be applied to each file as described in 1) - 3) above. Option B: - I'm afraid my thinking is not entirely clear regards this option but somehow I import metadata for every (primary) file into a list (I think all that's needed is file name and location), perhaps even a nested list although I'm not sure if that provides an advantage. Then I apply the operations for 1) - 3) above sequentially per list item, the assumption being the list data and my home made functions will be sufficient. I've found various par2 programs on PyPi and possibly pyFileFixity could be used but in this instance I'd rather give it a go myself. For various reasons I can't use ZFS which would, of course, negate the need for doing any of this. It seems this would be my consolation prize :) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python Generator expressions
Hi Animesh,
Unfortunately the list server/email has removed the formatting from your
sample, but no matter...
On 24/07/19 5:06 AM, Animesh Bhadra wrote:
# This code creates a generator and not a tuple comprehensions.
my_square =(num *num fornum inrange(11))
print(my_square) # at 0x7f3c838c0ca8>
# We can iterate over the square generator like this.
try:
whileTrue:
print(next(my_square)) # Prints the value 0,1,4
exceptStopIterationasSI:
print("Stop Iteration")
# Another iteration
forx inmy_square:
print(x) # This prints nothing.
Does the generator exhausts its values when we run the iterator once?
Yes, it involves a "lazy" approach - the value is not calculated until
the next() requests it. Whereas, a list comprehension calculates all its
values at one time (and requires the storage space to hold them).
https://docs.python.org/3/reference/datamodel.html?highlight=generator -
notice that there is a yield and a next facility, but when it terminates
StopIteration is raised. There is no 'start again' command!
The Python docs are informative:
https://docs.python.org/3/reference/expressions.html?highlight=generator
https://docs.python.org/3/tutorial/classes.html?highlight=generator#generators
If you have an actual reason for running through a generator expression
more than once, then consider return-ing it from a function/class (which
will then be directly accessible to the for-loop/next method).
Lastly any specific reason for not having a tuple comprehensions?
Have checked this link, but could not understood the reason?
https://stackoverflow.com/questions/16940293/why-is-there-no-tuple-comprehension-in-python
I don't know.
Have you understood the differences between lists and tuples -
specifically "mutability" and "immutability"?
Let's take a list comprehension. If you 'unwind it', can you reproduce
it as a multi-line for-loop? Yes, but before the loop the 'target' list
must be defined/declared to be a list; and within the loop the list is
appended with the 'generated' values.
Ok? (sorry, don't know if this will be new to you, or not)
Now, instead of a list, try using a tuple? How do you append() to a tuple?
Yes, many people have confused generator expressions -
surrounded/"delimited" by parentheses, ie ( and ), with tuples.
However, try this little demonstration:
a, b = 1, 2
a
1
b
2
a, b = ( 1, 2 )
a
1
b
2
( a, b ) = ( 1, 2 )
a
1
b
2
type( a )
type( ( 1, 2 ) )
The principle here is known as "tuple unpacking". The two constants
(right-hand side) are arranged as a tuple, as are the two variables (a
and b/left-hand side), regardless of the presence/absence of the
parentheses!
https://treyhunner.com/2018/03/tuple-unpacking-improves-python-code-readability/
Clarifying the difference/similarity in appearance between a generator
expression and a tuple, it might help to think that it is the comma(s)
which make it a tuple!
--
Regards =dn
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python Generator expressions
On 7/23/19 11:06 AM, Animesh Bhadra wrote:
> Hi All,
>
> Need one help in understanding generator expression/comprehensions.
>
> This is my sample code.
>
> # This code creates a generator and not a tuple comprehensions.
> my_square =(num *num fornum inrange(11))
> print(my_square) # at 0x7f3c838c0ca8>
> # We can iterate over the square generator like this.
> try:
> whileTrue:
> print(next(my_square)) # Prints the value 0,1,4
> exceptStopIterationasSI:
> print("Stop Iteration")
is this code you were actually running? because it won't work... an
except needs to be matched with a try, it can't match with a while.
you *can* comsume your the values your generator expression generates by
doing a bunch of next's, but why would you? Instead, just iterate over
it (every generator is also an iterator, although not vice versa):
for s in my_square:
print(s)
you don't have to manually catch the StopIteration here, because that's
just handled for you by the loop.
> # Another iteration
> forx inmy_square:
> print(x) # This prints nothing.
>
>
> Does the generator exhausts its values when we run the iterator once?
> Lastly any specific reason for not having a tuple comprehensions?
>
> Have checked this link, but could not understood the reason?
>
> *
> https://stackoverflow.com/questions/16940293/why-is-there-no-tuple-comprehension-in-python
>
>
> Regards,
> Animesh
>
>
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python Generator expressions
On Tue, Jul 23, 2019 at 10:36:01PM +0530, Animesh Bhadra wrote:
> Hi All,
>
> Need one help in understanding generator expression/comprehensions.
>
> This is my sample code.
Lots of missing spaces in your code! Don't forget to hit the space bar
between words :-)
Also missing indentation, which is probably Gmail being "helpful".
I'm just going to fix the obvious typos without comment.
> # This code creates a generator and not a tuple comprehensions.
> my_square =(num*num for num in range(11))
> print(my_square) # at 0x7f3c838c0ca8>
> # We can iterate over the square generator like this.
> try:
> while True:
> print(next(my_square)) # Prints the value 0,1,4
> except StopIteration as SI:
> print("Stop Iteration")
> # Another iteration
> for x in my_square:
> print(x) # This prints nothing.
> Does the generator exhausts its values when we run the iterator once?
Correct.
> Lastly any specific reason for not having a tuple comprehensions?
(1) History and (2) because tuple comprehensions are not useful enough
to dedicate syntax to them.
Originally, Python introduced *list comprehensions* using square bracket
syntax:
[expression for x in values]
which created a list. If I remember correctly, I think this was about
Python 2.2.
Tuples are mostly intended for *small* numbers of heterogeneous values,
like a struct or record in other languages:
# Typical example of tuple
(1, "Hello", 4.5)
while lists are intended for potentially large numbers of homogeneous
values, like an array of data:
[2.3, 4.5, 0.9, 7.2, 9.3, 4.5, 6.1, 0.2, 8.7, ... # and many more
(heterogeneous = "different types", homogeneous = "same types")
That makes lists a really good fit for comprehensions, and tuples a bad
fit. So it was decided that the need for a direct tuple comprehension
was pretty low. If you wanted a tuple, you could easily get one:
tuple([expression for x in values])
and that was good enough without wasting useful syntax on something that
would hardly ever be needed.
It would be a tiny bit more expensive to run (first you create a list,
then copy it to a tuple, and then garbage collect the list) but so what?
Not every rare operation needs to be optimized.
A few years later, generator comprehensions were added as a lazy version
of list comprehensions. List comprehensions produce all the values up
front, while generator comprehensions produce them one at a time as
needed. The syntax chosen used round brackets ( ... ) instead of square
brackets [ ... ] but otherwise was identical.
What other syntax could have been chosen? Curly brackets are used for
dicts (Python 2 and 3) and sets (Python 3 only), so they are unsuitable,
and later on in Python 3 we gained dict and set comprehensions as well.
Had we wasted the ( ... ) syntax on tuples, which hardly anyone would
have ever used, we would have missed out on the super-useful lazy
generator comprehensions.
> Have checked this link, but could not understood the reason?
>
> *
>
> https://stackoverflow.com/questions/16940293/why-is-there-no-tuple-comprehension-in-python
Don't believe everything you read on Stackoverflow. There's lots of
utter nonsense on that page, for example:
"The answer is obviously because tuple syntax and parenthesis are
ambiguous"
is rubbish. The parser can tell the difference between tuples and
parentheses. Tuples use *commas*, not parentheses, except for the
empty tuple () which is a special case. There's no ambiguity in Python.
The top rated answer
"parentheses were already taken for … generator expressions"
is incorrect, because generator expressions came second, not first.
Python could have added tuple comprehensions at the same time it added
list comprehensions, but the developers decided not to:
- it wasn't clear how useful comprehensions would be; they were really
useful in Haskell, but people weren't sure if Python developers would
take to them (they sure did!);
- list comprehensions are clearly useful, but tuple comprehensions not
so much. Not ever single use has to get its own special syntax.
Of course, *later on* once Python had generator comprehensions using
parentheses, we couldn't easily add tuple comprehensions because
there's no good syntax left. But why would we want them?
--
Steven
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python Generator expressions
On Wed, Jul 24, 2019 at 11:26:26AM +1200, David L Neil wrote: > Clarifying the difference/similarity in appearance between a generator > expression and a tuple, it might help to think that it is the comma(s) > which make it a tuple! David makes an excellent point here. Except for the special case of the empty tuple, it is *commas* that create tuples, not parentheses. The round brackets are used for grouping, and are optional: py> a = 1, 2, 3 py> type(a) The only times you *need* parens around a non-empty tuple is to group the items or to avoid ambiguity, e.g. a tuple inside a tuple: a = 1, 2, (3, 4, 5), 6 assert len(a) == 4 or when passing a literal tuple as argument to a function: function(1, 2, 3, 4, 5, 6)# six arguments function(1, 2, (3, 4, 5), 6) # four arguments -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
