handling unicode data
Hi all, I'm starting to learn python but am having some difficulties with how it handles the encoding of data I'm reading from a database. I'm using pymssql to access data stored in a SqlServer database, and the following is the script I'm using for testing purposes. - import pymssql mssqlConnection = pymssql.connect(host='localhost',user='sa',password='password',database='TestDB') cur = mssqlConnection.cursor() query="Select ID, Term from TestTable where ID > 200 and ID < 300;" cur.execute(query) row = cur.fetchone() results = [] while row is not None: term = row[1] print type(row[1]) print term results.append(term) row = cur.fetchone() cur.close() mssqlConnection.close() print results - In the console output, for a record where I expected to see "França" I'm getting the following: "" -When I print the type (print type(row[1])) "Fran+a" -When I print the "term" variable (print term) "Fran\xd8a" -When I print all the query results (print results) The values in "Term" column in "TestTable" are stored as unicode (the column's datatype is nvarchar), yet, the python data type of the values I'm reading is not unicode. It all seems to be an encoding issue, but I can't see what I'm doing wrong.. Any thoughts? thanks in advance, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Hi Fredrik, Thanks for the reply. Instead of: term = row[1] I tried: term = unicode(row[1], "iso-8859-1") but the following error was returned when printing "term": Traceback (most recent call last): File "test.py", line 11, in ? print term File "c:\Program Files\Python24\lib\encodings\cp437.py", line 18, in encode return codecs.charmap_encode(input,errors,encoding_map) UnicodeEncodeError: 'charmap' codec can't encode character u'\xd8' in position 31: character maps to Is it possible some unicode strings are not printable to the console? It's odd, because I can manually write in the console the same string I'm trying to print. I also tried other encodings, besides iso-8859-1, but got the same error. Do you think this has something to do with the DB-API driver? I don't even know where to start if I have to change something in there :| Cheers, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Hi, Martin v. Löwis wrote: > Also, it appears that DB-Library (the API used by pymssql) always > returns CP_ACP characters (unless ANSI-to-OEM conversion is enabled); > so the "right" encoding to use is "mbcs". do you mean using something like the following line? term = unicode(row[1], "mbcs") What do you mean by "ANSI-to-OEM conversion is enabled"? (sorry, I'm quite a newbie to python) > Notice that Microsoft plans to abandon DB-Library, so it might be > best to switch to a different module for SQL Server access. I've done some searching and settled for pymssql, but it's not too late to change yet. I've found these options to connect to a MSSqlServer database: Pymssql http://pymssql.sourceforge.net/ ADODB for Python (windows only) http://phplens.com/lens/adodb/adodb-py-docs.htm SQLServer for Python (discontinued?) http://www.object-craft.com.au/projects/mssql/ mxODBC (commercial license) http://www.egenix.com/files/python/mxODBC.html ASPN Recipe http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/144183 Pymssql seemed like the best choice. The ASPN Recipe I mention doesn't look bad either, but there doesn't seem to be as many people using it as using pymssql. I'll look a little further though. -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Fredrik Lundh wrote:
> works for me, given your example:
> >>> s = "Fran\xd8a"
> >>> unicode(s, "iso-8859-1")
> u'Fran\xd8a'
>
> what does
> print repr(row[1])
>
> print in this case ?
It prints:
'Fran\xd8a'
The error I'm getting is beeing thrown when I print the value to the
console. If I just convert it to unicode all seems ok (except for not
beeing able to show it in the console, that is... :).
For example, when I try this:
print unicode("Fran\xd8a", "iso-8859-1")
I get the error:
Traceback (most recent call last):
File "a.py", line 1, in ?
print unicode("Fran\xd8a", "iso-8859-1")
File "c:\Program Files\Python24\lib\encodings\cp437.py", line 18, in
encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xd8' in
position 4
: character maps to
--
http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Frank Millman wrote: > You did not mention the odbc module from Mark Hammond's win32 > extensions. This is what I use, and it works for me. I believe it is > not 100% DB-API 2.0 compliant, but I have not had any problems. > > I have not tried connecting to the database from a Linux box (or from > another Windows box, for that matter). I don't know if there are any > implications there. According to sourceforge's project page (https://sourceforge.net/projects/pywin32/) it seems to only work on windows. There's also adodbapi (http://adodbapi.sourceforge.net/), that also depends on PyWin32, but it would be very handy if I could run this code on a linux box, and with these libs I wouldn't be able to. Still, options are always good to have around :) -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Marc 'BlackJack' Rintsch wrote:
> The `unicode()` call doesn't fail here but the ``print`` because printing
> unicode strings means they have to be encoded into a byte string again.
> And whatever encoding the target of the print (your console) uses, it
> does not contain the unicode character u'\xd8'. From the traceback it
> seems your terminal uses `cp437` as encoding.
>
> As you can see here: http://www.wordiq.com/definition/CP437 there's no Ø
> in that character set.
somethings are much, much, clearer to me now. thanks!
For future reference, these links may also help:
http://www.jorendorff.com/articles/unicode/python.html
http://www.thescripts.com/forum/thread23314.html
I've changed my windows console copdepage to latin1 and the following
prints are now outputting "França", as expected:
print unicode("Fran\x87a", "cp850").encode("iso-8859-1")
print unicode("Fran\xe7a", "iso-8859-1").encode("iso-8859-1")
However, I don't yet fully understand what's happening with Pymssql.
The encoding I expected to be receiving from MSSqlServer was cp850 (the
column in question uses the collation SQL_Latin1_General_CP850_CS_AS),
but it doesn't seem to be what the query is returning. I tried
converting to a unicode string from a few different encodings, but none
of them seems to be the right one. For example, for cp850, using a
latin1 console:
term = unicode(row[1], "cp850")
print repr(term)
print term
output ---
u'Fran\xcfa'
FranÏa
And for iso-8859-1 (also got the same result for mbcs):
term = unicode(row[1], "iso-8859-1")
print repr(term)
print term
output ---
u'Fran\xd8a'
FranØa
What do you think? Might it be Pymssql doing something wrong?
--
http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Martin v. Löwis wrote: > > What do you mean by "ANSI-to-OEM conversion is enabled"? > > See AutoAnsiToOem in > http://support.microsoft.com/default.aspx?scid=KB;EN-US;199819 > I checked the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\DB-Lib", and verified AutoAnsiToOem was 'ON'. I also tried assuming mbcs as the encoding but didn't got very far (please see my other post). -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
> In summary > > + Depends on what your requirements are, but... > > + Go for ADODBAPI for the widest spread of versions and licenses, but > the least cross-platformability > + Go for Object Craft MSSQL for <= 2.3 and best overall behaviour > + Go for pymssql for >= 2.4 with some small limitations > + Go for mxODBC for general purpose databasing, cross-platform, but > without .nextset and possibly Commercial licensing > + If all else fails, and you've got the SQL Server command line tools, > use the ASPN Recipe > Very nice review! Not having a commercial license ir really a "must" right now, and I'd really prefer it being cross-platform, so I think pymssql might not be a bad choice after all. If I can't make unicode data work in a reasonable way I'll probably switch to ADODBAPI, or maybe the odbc module that comes with PyWin32 mentioned by Frank. thanks, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Martin v. Löwis wrote: > Filipe wrote: > > output --- > > u'Fran\xd8a' > > FranØa > > > > > > What do you think? Might it be Pymssql doing something wrong? > > I think the data in your database is already wrong. Are you > sure the value in question is really "França" in the database? > yes, I'm pretty sure. There's an application that was built to run on top of this database and it correctly reads as writes data to the DB. I also used SqlServer's Query Analyzer to select the data and it displayed fine. I've done some more tests and I think I'm very close to finding what the problem is. The tests I had done before were executed from the windows command line. I tried printing the following (row[1] is a value I selected from the database) in two distinct environments, from within an IDE (Pyscripter) and from the command line: import sys import locale print getattr(sys.stdout,'encoding',None) print locale.getdefaultlocale()[1] print sys.getdefaultencoding() term = "Fran\x87a" print repr(term) term = row[1] print repr(term) output I got in Pyscripter's interpreter window: None cp1252 ascii 'Fran\x87a' 'Fran\x87a' output I got in the command line: cp1252 cp1252 ascii 'Fran\x87a' 'Fran\xd8a' I'd expect "print" to behave differently according with the console's encoding, but does this mean this happens with repr() too? in which way? thanks, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Martin v. Löwis wrote: > Filipe wrote: > > term = row[1] > > print repr(term) > > > > output I got in Pyscripter's interpreter window: > > 'Fran\x87a' > > > > output I got in the command line: > > 'Fran\xd8a' > > > > I'd expect "print" to behave differently according with the console's > > encoding, but does this mean this happens with repr() too? > > repr always generates ASCII bytes. They are not effected by the > console's encoding. If you get different output, it really means > that the values are different (check ord(row[1][4]) to be sure) They do, in fact, output different values. The value outputed by pyscripter was "135" (x87) while the value outputed in the command line was "216" (xd8). I can't understand why though, because the script being run is precisely the same on both environments. > What is the precise sequence of statements that you used to > set the "row" variable? The complete script follows: --- import sys import locale print getattr(sys.stdout,'encoding',None) print locale.getdefaultlocale()[1] print sys.getdefaultencoding() import pymssql mssqlConnection = pymssql.connect(host='localhost',user='sa',password='password',database='TestDB') cur = mssqlConnection.cursor() query="Select ID, Term from TestTable where ID = 204;" cur.execute(query) row = cur.fetchone() results = [] while row is not None: term = unicode(row[1], "cp850") print ord(row[1][4]) print ord(term[4]) print term results.append(term) row = cur.fetchone() cur.close() mssqlConnection.close() print results --- The values outputed were, in pyscripter: None cp1252 ascii 135 231 França [uFran\xe7a'] and in the command line cp850 cp1252 ascii 216 207 FranÏa [u'Fran\xcfa'] regards, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Hi Martin, > One would have to ask the authors of pymssql, or Microsoft, > why that happens; alternatively, you have to run pymssql > in a debugger to find out yourself. Tried running pymssql in a debugger, but I felt a bit lost. There are too many things I would need to understand about pymssql first. Meanwhile, I got to some very interesting conclusions. Remember the "ANSI-to-OEM conversion" option you mentioned before? I began reading some docs about it and this description turned up: "The ANSI to OEM conversion translates data coming back from SQL Server into the local code page used by your client." which seemed exactly what I don't want.. so I turned it to "OFF" (by using the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\DB-Lib\AutoAnsiToOem) and everything started working the way I was originally expecting! I think that the best way to actually solve this will be to override AutoAnsiToOem (instead of using the registry setting) from within pymssql, or find a way to specify what the "local code page" should be. If not, I will have to have the pain of verifying this setting in every system where the code I'm developing will be deployed. Which reminds me... that this setting doesn't exist on non-windows environments (ie, no registry on Linux) so I'm not sure how will it all work there. Anyone with experience in using DB-Library that can confirm how it works (namely, on linux)? (but perhaps this is outside the scope of this newsgroup.. ) I got in touch with Andrzej Kukula, the current developer of pymssql, who has also been very helpful, and knows what we've been discussing over here. thanks for all the help, Filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Dennis Lee Bieber wrote: > The setting most likely has to be made on the machine running the > server -- and M$ SQL Server doesn't exist on Linux either > > If the conversion was being done by some client library on Windows, > then again, since that library probably doesn't exist on Linux, the > conversion probably is not done. yes, it's something done on the client side. And I think DB-Library does exist on Linux because Pymssql depends on it and Pymssql is cross-platform: "pymssql 0.7.4 has been tested on FreeBSD 6.0, NetBSD 2.1, 3.0, Linux with kernel 2.6, and Windows XP. It should also run on other platforms." -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Dennis Lee Bieber wrote: > If I interpret a short Google search, DB-Library might date back to > the original Sybase core from which M$ SQL Server was spawned. M$'s site > recommends /not/ using DB-Library but to use ODBC/OLEDB methods instead > -- something about ODBC being extensible. Could be confusing if both > Sybase and M$ SQL Server were on the same machine... > > http://www.cs.sfu.ca/CourseCentral/Software/Sybase/DB-LIBRARY/DB-LIBRARY.html > > Technical details reference Sybase, but the wordy stuff is "SQL > Server" and "Transact-SQL". The only reason I still think Pymssql (and therefore, DB-Library) might be the best option is that, it is the only one I know that is both cross-platform and free - as in beer and as in freedom. (check, in this thread, a previous message by Tim Golden) I searched a bit if there are any OLEDB based python libs and found this one: http://pyoledb.datadmin.com/ I'm still not sure if it's cross-platform or not, but It does have a commercial license, so it's not my first choice for now. -- http://mail.python.org/mailman/listinfo/python-list
Re: handling unicode data
Frank Millman wrote: > Filipe wrote: > Try out the suggestions and let us know what happened. I for one will > be very interested. The last version of ODBTPAPI is 0.1-alpha, last updated 2004-09-25. Which is a bit scary... I might try it just the same though. -- http://mail.python.org/mailman/listinfo/python-list
Decoding group4 tiff with PIL
Is decoding group4 tiff possible with Python Imaging Library? I'm trying to convert some image files (several formats, among which, G4 tiffs) to PNG but everything I read about it seems to indicate PIL doesn't support group4 encoding, and won't support it in the short term. yet, I found something[1] that seems like some sort of plugin for PIL and that does look promising, although it seems to be in a bit of an alpha state. However, I don't know where to start.. Should I copy libtiff.py to some location? Also found an alternative[2] to PIL, but I'd rather have PIL doing everything, no matter the format of the file read, if possible. Haven't tried this alternative yet. [1] http://effbot.python-hosting.com/file/stuff/sandbox/pil/libtiff.py [2] http://www.haynold.com/software_projects/2004/pytiff/ Any thoughts? Cheers, Filipe -- http://mail.python.org/mailman/listinfo/python-list
|help| python 3.12
I'm with a problem I'm doing a program in python, it sends the
following error message:
File "C:/Documents and Settings/Filipe Vinicius/Desktop/Filipe/Cefet/
LP/Python/trab.sistema.academico/sistemaacademico.2010.5.23.c.py",
line 40, in administrador
lp = pickle.load(f)
File "D:\Arquivos De Programa\Python31\lib\pickle.py", line 1365, in
load
encoding=encoding, errors=errors).load()
EOFError
What you need to do to repair this error? in my program at the part
where the error is find is that:
def administrador():
f = open('professores.dat', 'rb')
lp = pickle.load(f)
f.close()
...
Note: i had already imported the pikcle library
THX
--
http://mail.python.org/mailman/listinfo/python-list
sqlite3.OperationalError: Could not decode to UTF-8 column
Hi
I'm trying to use sqlite with python 2.5 but I have this problem:
Traceback (most recent call last):
File "converter.py", line 13, in
c_old.execute('select id_aluno, nome from aluno')
sqlite3.OperationalError: Could not decode to UTF-8 column 'nome' with
text 'Ana Margarida Fernandes Gonçalves de Sá'
The database was created with another program and all data is in
database is in latin1.
Thanks,
Filipe Sousa
--
http://mail.python.org/mailman/listinfo/python-list
Re: sqlite3.OperationalError: Could not decode to UTF-8 column
Carsten Haese wrote: > On Wed, 2007-09-05 at 16:40 +0100, Filipe Sousa wrote: > > Try setting > > conn.text_factory = str > > where conn is the name of your sqlite3 connection object. See > http://docs.python.org/lib/sqlite3-Connection-Objects.html for more > information. > > HTH, > Thanks! It works :) -- http://mail.python.org/mailman/listinfo/python-list
parser recommendation
I have a project that uses a proprietary format and I've been using regex to extract information from it. I haven't hit any roadblocks yet, but I'd like to use a parsing library rather than maintain my own code base of complicated regex's. I've been intrigued by the parsers available in python, which may add some much needed flexibility. I've briefly looked at PLY and pyparsing. There are several others, but too many to enumerate. My understanding is that PLY (although more difficult to use) has much more flexibility than pyparsing. I'm basically looking to make an informed choice. Not just for this project, but for the long haul. I'm not afraid of using a difficult (to use or learn) parser either if it buys me something like portability (with other languages) or flexibility). I've been to a few websites that enumerate the parsers, but not all that very helpful when it came to comparisons... http://nedbatchelder.com/text/python-parsers.html http://www.python.org/community/sigs/retired/parser-sig/towards-standard/ I'm not looking to start a flame war... I'd just like some honest opinions.. ;) thanks, filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: parser recommendation
On Tue, Jun 3, 2008 at 10:41 AM, Paul McGuire <[EMAIL PROTECTED]> wrote: > If you learn both, you may find that pyparsing is a good way to > quickly prototype a particular parsing problem, which you can then > convert to PLY for performance if necessary. The pyparsing prototype > will be an efficient way to work out what the grammar "kinks" are, so > that when you get around to PLY-ifying it, you already have a clear > picture of what the parser needs to do. > Thanks (both Paul and Kay) for responding. I'm still looking at Trail in EasyExtend and pyparsing is very nicely objected oriented but PLY does seems to have the speed advantage, so I'm leaning towards PLY But I do have more questions... when reading the ply.py header (in 2.5) I found the following paragraph... # The current implementation is only somewhat object-oriented. The # LR parser itself is defined in terms of an object (which allows multiple # parsers to co-exist). However, most of the variables used during table # construction are defined in terms of global variables. Users shouldn't # notice unless they are trying to define multiple parsers at the same # time using threads (in which case they should have their head examined). Now, I'm invariably going to have to use threads... I'm not exactly sure what the author is alluding to, but my guess is that to overcome this limitation I need to acquire a thread lock first before "defining/creating" a parser object before I can use it? Has anyone ran into this issue? This would definitely be a showstopper (for PLY anyway), if I couldn't create multiple parsers because of threads. I'm not saying I need more than one, I'm just not comfortable with that limitation. I have a feeling I'm just misunderstanding since it doesn't seem to hold you back from creating multiple parsers under a single process. filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: parser recommendation
> On Jun 3, 12:34 pm, "Filipe Fernandes" <[EMAIL PROTECTED]> wrote: >> On Tue, Jun 3, 2008 at 10:41 AM, Paul McGuire <[EMAIL PROTECTED]> wrote: >> But I do have more questions... when reading the ply.py header (in >> 2.5) I found the following paragraph... >> >> # The current implementation is only somewhat object-oriented. The >> # LR parser itself is defined in terms of an object (which allows multiple >> # parsers to co-exist). However, most of the variables used during table >> # construction are defined in terms of global variables. Users shouldn't >> # notice unless they are trying to define multiple parsers at the same >> # time using threads (in which case they should have their head examined). >> >> Now, I'm invariably going to have to use threads... I'm not exactly >> sure what the author is alluding to, but my guess is that to overcome >> this limitation I need to acquire a thread lock first before >> "defining/creating" a parser object before I can use it? >> >> Has anyone ran into this issue? This would definitely be a >> showstopper (for PLY anyway), if I couldn't create multiple parsers >> because of threads. I'm not saying I need more than one, I'm just not >> comfortable with that limitation. >> On Tue, Jun 3, 2008 at 1:53 PM, Kay Schluehr <[EMAIL PROTECTED]> wrote: > Nope. It just says that the parser-table construction itself relies on > global state. But you will most likely build your parser offline in a > separate run. Thanks Kay for the context.., I misunderstood completely, but your last sentence coupled with a few running examples, cleared things right up... On Tue, Jun 3, 2008 at 4:36 PM, Paul McGuire <[EMAIL PROTECTED]> wrote: > You can use pyparsing from any thread, and you can create multiple > parsers each running in a separate thread, but you cannot concurrently > use one parser from two different threads. Some users work around > this by instantiating a separate parser per thread using pickle to > quickly construct the parser at thread start time. I didn't know that pyparsing wasn't thread safe. I kind of just assumed because of it's OO approach. Thanks for the work around. I haven't given up on pyparsing, although I'm now heavily leaning towards PLY as an end solution since lex and yacc parsing is available on other platforms as well. Thanks Kay and Paul for the advice... I'm still using the first two I started looking at, but I'm much for confident in the choices made. filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: re.search much slower then grep on some regular expressions
On Fri, Jul 4, 2008 at 8:36 AM, Peter Otten <[EMAIL PROTECTED]> wrote:
> Henning_Thornblad wrote:
>
>> What can be the cause of the large difference between re.search and
>> grep?
>
> grep uses a smarter algorithm ;)
>
>> This script takes about 5 min to run on my computer:
>> #!/usr/bin/env python
>> import re
>>
>> row=""
>> for a in range(156000):
>> row+="a"
>> print re.search('[^ "=]*/',row)
>>
>>
>> While doing a simple grep:
>> grep '[^ "=]*/' input (input contains 156.000 a in
>> one row)
>> doesn't even take a second.
>>
>> Is this a bug in python?
>
> You could call this a performance bug, but it's not common enough in real
> code to get the necessary brain cycles from the core developers.
> So you can either write a patch yourself or use a workaround.
>
> re.search('[^ "=]*/', row) if "/" in row else None
>
> might be good enough.
>
Wow... I'm rather surprised at how slow this is... using re.match
yields much quicker results, but of course it's not quite the same as
re.search
Incidentally, if you add the '/' to "row" at the end of the string,
re.search returns instantly with a match object.
@ Peter
I'm not versed enough in regex to tell if this is a bug or not
(although I suspect it is), but why would you say this particular
regex isn't common enough in real code?
filipe
--
http://mail.python.org/mailman/listinfo/python-list
Retrieving int from hex in a file.
Hi.
I have to open a binary file from an old computer and recover the
information stored (or at least try to). I use:
f=open('file.bin','rb')
a=f.read()
f.close()
a in now a string full of hex representations in the form:
a[6]='\x14'
a[7]='\x20'
I would like to convert these hex representations to int, but this
(the most obvious way) doesn't seem to be working
>>> q=a[6]
>>> q
'\x14'
>>> int(q,16)
Traceback (most recent call last):
File "", line 1, in ?
ValueError: invalid literal for int():
>>>
How can I do this?
Thanks
--
http://mail.python.org/mailman/listinfo/python-list
Re: Retrieving int from hex in a file.
> > Use ord(q) > > py> help(ord) > Help on built-in function ord in module __builtin__: > > ord(...) > ord(c) -> integer > > Return the integer ordinal of a one-character string. > > py> > > -- > Gabriel Genellina Thank you Gabriel. It fit's my purpose. -- http://mail.python.org/mailman/listinfo/python-list
simple GUI for my application?
Hi, I'm really struggling to find the best GUI to make a simple application. I'm doing a program to load all the ini files in the current folder, or the folder that the user chooses and list the specifics entries in it. So, the program would be like this: Som tabs here like: ( Load | Edit | Options) In the [ Load ] tab: A folder tree in the left, and two labels or edit boxes showing some specific entries in the ini file. In the [ Edit ] tab: really straight-forward, Edit boxes of the entries so the user can edit in the [ options ] tab: More edits to specifie the default folder, etc. Basically I will use a lot of edit boxes and some tabs, and a folder tree, any tips so I can search in the right place? -- http://mail.python.org/mailman/listinfo/python-list
alternative to JoinableQueue's please
I'm currently using the multiprocessing package and I'm hugely impressed at its simplicity (thanks very much Jesse Noller). Although, it seems that there's a bug in JoinableQueue's which renders using it pointless over a regular Queue as per Issue 4660 http://bugs.python.org/issue4660 To re-iterate... task_done() which is to be called for each get() throws the exception: ValueError: task_done() called too many times every once in a while. The reasons for this are outlined by Brian in the issue above, but no confirmation has been provided. The reasons for using JoinableQueue I think are obvious. I want to block the main processing using queue.join() until the tasks that have been placed on the queue have been finished by the worker processes. I can't be the only one experiencing this (besides Brian)... are there others who ran into this? Are there work arounds (besides a home-brewed one) ? filipe -- http://mail.python.org/mailman/listinfo/python-list
Re: alternative to JoinableQueue's please
Raymond Hettinger wrote:
> [Filipe Fernandes]
>> The reasons for using JoinableQueue I think are obvious. I want to
>> block the main processing using queue.join() until the tasks that have
>> been placed on the queue have been finished by the worker processes.
>>
>> I can't be the only one experiencing this (besides Brian)... are there
>> others who ran into this? Are there work arounds (besides a
>> home-brewed one) ?
>
> Before Queue.task_done() and Queue.task_join() were added, other
> idioms were used.
>
> One technique is to use a second queue to track incomplete tasks.
>
> # adding work
> unfinished_tasks.put(None)
> q.put(task)
>
>
> # doing work
> t = q.get()
> f(t)
> unfinished_tasks.get()
>
>
> # waiting for unfinished tasks to complete
> while unfinished_tasks.qsize():
> sleep(0.1)
Thanks Raymond... yours is by far is the simplest and should have been an
obvious solution. I didn't want to stray too far from what I had and this
fits the bill.
In case others are curious...
I had looked at using the example in
http://docs.python.org/library/multiprocessing.html#using-a-remote-manager
to use the traditional Queue from module Queue which includes the join and
task_done method calls. But creating a server process/thread just for that
is rather over-kill.
I also looked at using process pools asynchronously and waiting for the
iterators to come back to determine if jobs were completed.
But your solution is the simplest to implement. And easy
enough to create a composite class containing the two queues to simulate
the real one (although I have not tried the following, I'm not in the
office right now).
class JoinableQueue(object):
def __init__(*args, **kwargs):
self.__tasks = Queue()
self.__queue = Queue(*args, **kwargs)
def put(self, *args, **kwargs):
self.__tasks.put(None)
self.__queue.put(*args, **kwargs)
def get(self, *args, **kwargs):
return self.__queue.get(*args, **kwargs)
def task_done():
try:
self.__tasks.get(False)
except Queue.Empty:
raise ValueError('task_done called too many times')
def join():
while not self.__tasks.empty():
sleep(0.1)
[Add methods to simulate as required]
filipe
ps: Thanks Raymond for the quick reply... and I feel rather apologetic for
having bothered the list with this :S
--
http://mail.python.org/mailman/listinfo/python-list
