Re: Convert a list with wrong encoding to utf8
[email protected] wrote: [python] con = pymysql.connect( db = 'clientele', user = 'vergos', passwd = '**', charset = 'utf8' ) cur = con.cursor() [/python] From that i understand that the names being fetched from the db to pyhton script are being fetced as utf8, right? No, I don't think so. As far as I can tell from a brief reading of the MySQL docs, that only sets the *connection* encoding, which is concerned with transferring data over the connection between the client and the server. It has no bearing on the encoding used to decode data fetched from the database. That's determined by metadata stored in the database itself. It seems that MySQL lets you specify database encodings at three different levels: for the database as a whole, for a specific table, and for a specific field of a table. What I think is happening is that the column you're reading the names from is tagged in the database as being encoded in latin1, *but* this is incorrect for some of the names, which are actually encoded in utf8. This would explain why some of the data you looked at was printed with hex escapes, and why name.encode('latin1').decode('utf8') appeared to fix it. The encode('latin1') gets back the original raw bytes, and the decode('utf8') decodes them again using the correct encoding. However, not *all* of the data is like this -- some of it is correctly stored in the database, and is coming back already correctly decoded with no further processing needed. So, when you blindly try to re-code all the names in the list, it fails on the first correctly-decoded one it encounters. Again, try printing out the whole list of names, and post it here (or a good chunk of it if it's very long). It will give us a better idea of what's going on. If this theory is correct, then there isn't really any "right" way to deal with it -- the fundamental problem is that the data in the database is corrupted. The best long-term solution would be to clean up the database. But if you have to deal with it as it is, you'll need to use some kind of heuristic to decide when a particular string needs "fixing", and what needs to be done to fix it. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem : Generator
On Fri, Feb 15, 2019 at 6:57 PM Prahallad Achar wrote: > > How about this > List1=[ 1,2,3,4] > Rever_gen = ( x*x for x in list1, reversed = True) > > Rever_gen gets generator object and iterating it now gets reverse order.. > > Am I correct here? Suggest me > How about reversed(list1) ? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: FW: Why float('Nan') == float('Nan') is False
Avi Gross wrote:
I can see why you may be wondering. You see the nan concept as having a
specific spelling using all lowercase and to an extent you are right.
No, he's talking about this particular line from the transcript you
posted:
>>>float(" nan")
> Nan
This suggests that the interpreter printed out that particular
nan value as "Nan" with a capital N. But that's not what my
Python 3.5.1 interpreter does:
Python 3.5.1 (default, Jun 1 2016, 13:15:26)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> float(" nan")
nan
Grant was asking whether that's *really* what your interpreter
printed out, and if so, which version of Python it was, because
it's quite a surprising thing for it to do.
Personally I think it's more likely that the N got capitalised
somehow on the way from your terminal window to the mail message.
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem : Generator
I get list object instead gen obj On Fri, 15 Feb 2019, 13:57 Chris Angelico On Fri, Feb 15, 2019 at 6:57 PM Prahallad Achar > wrote: > > > > How about this > > List1=[ 1,2,3,4] > > Rever_gen = ( x*x for x in list1, reversed = True) > > > > Rever_gen gets generator object and iterating it now gets reverse order.. > > > > Am I correct here? Suggest me > > > > How about reversed(list1) ? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem : Generator
Prahallad Achar於 2019年2月15日星期五 UTC+8下午5時27分36秒寫道： > I get list object instead gen obj > > On Fri, 15 Feb 2019, 13:57 Chris Angelico > > On Fri, Feb 15, 2019 at 6:57 PM Prahallad Achar > > wrote: > > > > > > How about this > > > List1=[ 1,2,3,4] > > > Rever_gen = ( x*x for x in list1, reversed = True) > > > > > > Rever_gen gets generator object and iterating it now gets reverse order.. > > > > > > Am I correct here? Suggest me > > > > > > > How about reversed(list1) ? > > > > ChrisA > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Rever_gen = ( x*x for x in reversed(list1)) -- https://mail.python.org/mailman/listinfo/python-list
Re: What's up with Activestate Python?
On 2019-02-13, Grant Edwards wrote: > For many, many years I've always installed ActiveState's ActivePython > Community edition when forced to use Windows. It has always included > all of the "extra" libraries that I didn't wan't to install (or > couldn't because I didn't have a C compiler for Windows). > > I recently decided to upgrade my Win7 machine from ActivePython 3.5.4 > to 3.6. > > ... and all of apps stopped working. ActivePython 3.6 appears to be a > minimal install that includes nothing but CPython. Comparing the > download sizes makes this obvious: > > -rw-r--r-- 1 grante users 223056832 Mar 26 2018 > ActivePython-2.7.14.2717-win64-x64-404905.exe > -rw-r--r-- 1 grante users 225065576 May 29 2018 > ActivePython-3.5.4.3504-win64-x64-404899.exe > -rw-r--r-- 1 grante users 30297136 Feb 13 16:28 > ActivePython-3.6.0.3600-win64-x64-401834.exe > > I've searched the ActiveState web site, and the fact that they've > stopped including "extra" libraries doesn't seem to be documented > anywhere. FWIW, it's just the 3.6.0 release that was minimal. The 3.6.6 releases contain the normal set of libraries. However, the 3.6.6 release for Windows isn't quite out yet... -- Grant Edwards grant.b.edwardsYow! Thank god!! ... It's at HENNY YOUNGMAN!! gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Convert a list with wrong encoding to utf8
[email protected] writes: > Τη Πέμπτη, 14 Φεβρουαρίου 2019 - 8:56:31 μ.μ. UTC+2, ο χρήστης MRAB έγραψε: > >> It doesn't have a 'b' prefix, so either it's Python 2 or it's a Unicode >> string that was decoded wrongly from the bytes. > > Yes it doesnt have the 'b' prefix so that hexadecimal are representation of > strings and not representation of bytes. > > I just tried: > > names = tuple( [s.encode('latin1').decode('utf8') for s in names] ) > > but i get > UnicodeEncodeError('latin-1', 'Άκης Τσιάμης', 0, 4, 'ordinal not in > range(256)') > > 'Άκης Τσιάμης' is a valid name but even so it gives an error. > > Is it possible that Python3 a Unicode had the string wrongly decoded from the > bytes ? > > What can i do to get the names?! python3 >>> x = '\xce\x86\xce\xba\xce\xb7\xcf\x82 >>> \xce\xa4\xcf\x83\xce\xb9\xce\xac\xce\xbc\xce\xb7\xcf\x82' >>> b = bytes(ord(c) for c in x) >>> b.decode('utf-8') 'Άκης Τσιάμης' >>> -- Piet van Oostrum WWW: http://piet.vanoostrum.org/ PGP key: [8DAE142BE17999C4] -- https://mail.python.org/mailman/listinfo/python-list
RE: FW: Why float('Nan') == float('Nan') is False
Greg,
Good eye. You are correct!
Yes, that is a side effect I did not intend when I cut and paste and the
darn spell-checker saw it as useful to make my code act like the start of a
normal text sentence. I just replicated it:
>>> float(" nan")
Nan
As I watched, "nan" went to "Nan"
So, indeed, the transcript lied. I will be more careful.
-Original Message-
From: Python-list On
Behalf Of Gregory Ewing
Sent: Friday, February 15, 2019 3:51 AM
To: [email protected]
Subject: Re: FW: Why float('Nan') == float('Nan') is False
Avi Gross wrote:
> I can see why you may be wondering. You see the nan concept as having
> a specific spelling using all lowercase and to an extent you are right.
No, he's talking about this particular line from the transcript you
posted:
>>>float(" nan")
> Nan
This suggests that the interpreter printed out that particular nan value as
"Nan" with a capital N. But that's not what my Python 3.5.1 interpreter
does:
Python 3.5.1 (default, Jun 1 2016, 13:15:26) [GCC 4.2.1 (Apple Inc. build
5664)] on darwin Type "help", "copyright", "credits" or "license" for more
information.
>>> float(" nan")
nan
Grant was asking whether that's *really* what your interpreter printed out,
and if so, which version of Python it was, because it's quite a surprising
thing for it to do.
Personally I think it's more likely that the N got capitalised somehow on
the way from your terminal window to the mail message.
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem : Generator
Prahallad Achar writes: > I get list object instead gen obj If you have a list "l" and want a generator, you can use ( x for x in l) or simpler "iter(l)" - which gives you an interator over "l". An "iterator" is slightly more general than a generator (every generator is also an iterator). -- https://mail.python.org/mailman/listinfo/python-list
RE: Problem : Generator
Just want to point out you can make any function into a generator by having a yield statement like this: >>> def previous(listing): while listing: yield listing.pop() >>> for num in previous([1,2,3,4]): print(num) 4 3 2 1 The above is an EXAMPLE, not a particularly great way to do this. The point is if you have an iterable you want to do reversed, you could do it without an explicit reversal. Variations on the above that do not alter the list would be to use an index based on the length of the list and count backward as you return what is at that index. -Original Message- From: Python-list On Behalf Of dieter Sent: Saturday, February 16, 2019 1:47 AM To: [email protected] Subject: Re: Problem : Generator Prahallad Achar writes: > I get list object instead gen obj If you have a list "l" and want a generator, you can use ( x for x in l) or simpler "iter(l)" - which gives you an interator over "l". An "iterator" is slightly more general than a generator (every generator is also an iterator). -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
