Sure, here's a bit more info.
The external data is generated by a script and it describes a
catalogue of lot items for an auction site I'm building. The format
includes a lot number, a brief description of the lot for sale, and an
estimate for the item. Each lot is separated in the file by a '$' with
some whitespace. Here's a snippet:
$
292 A collection of wine bottle and trinket boxes
Est. 30-60
$
293 A paper maché letter rack with painted foliate decoration and a
C19th papier mache side chair and one other (a/f)
Est. 20-30
$
294 A wall mirror with bevelled plate within gilt frame
Est. 40-60
I've got a regular expression to extract out all the bits I need from
the external file:
lot = re.compile(r'\s*(?P<lot_number>\d*) (?P<description>.*)\s*Est. (?
P<min_estimate>\d*)-(?P<max_estimate>\d*)')
This information is extracted when this file is submitted to an 'add
new catalogue' form in Django's admin interface:
class CatalogueAdmin(admin.ModelAdmin):
# ...
def save_model(self, request, obj, form, change):
"""
Check for an attached data file, if instance is being created,
also
creates models within data file.
"""
obj.save()
if not change and form.cleaned_data['data']:
# Creating a new catalogue
handle_data_upload(form.cleaned_data['data'], obj)
And here's that handle_data_upload function (it's passed the uploaded
file object):
def handle_data_upload(f, cat):
"""
Creates and Adds lots to catalogue.
"""
lot = re.compile(r'\s*(?P<lot_number>\d*) (?P<description>.*)
\s*Est. (?P<min_estimate>\d*)-(?P<max_estimate>\d*)')
iterator = lot.finditer(f.read())
f.close()
for item in iterator:
if not item.group('description') == "end":
Lot.objects.create(
lot_number=int(item.group('lot_number')),
description=item.group('description').strip(),
min_estimate=Decimal(item.group('min_estimate')),
max_estimate=Decimal(item.group('max_estimate')),
catalogue=cat
)
Again, this all seems to work fine until Django come across the "é" in
the external data when it decides to throw this error:
Environment:
Request Method: POST
Request URL: http://192.168.0.2:8000/admin/catalogue/catalogue/add/
Django Version: 1.1 pre-alpha SVN-9646
Python Version: 2.5.1
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'auction.catalogue',
'auction.mailouts',
'auction.users',
'auction.bidding',
'django.contrib.flatpages',
'profiles',
'registration',
'django_extensions',
'tinymce',
'auction.lot-alerts']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware')
Traceback:
File "/Library/Python/2.5/site-packages/django/core/handlers/base.py"
in get_response
86. response = callback(request, *callback_args,
**callback_kwargs)
File "/Library/Python/2.5/site-packages/django/contrib/admin/sites.py"
in root
157. return self.model_page(request, *url.split('/',
2))
File "/Library/Python/2.5/site-packages/django/views/decorators/
cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "/Library/Python/2.5/site-packages/django/contrib/admin/sites.py"
in model_page
176. return admin_obj(request, rest_of_url)
File "/Library/Python/2.5/site-packages/django/contrib/admin/
options.py" in __call__
191. return self.add_view(request)
File "/Library/Python/2.5/site-packages/django/db/transaction.py" in
_commit_on_success
238. res = func(*args, **kw)
File "/Library/Python/2.5/site-packages/django/contrib/admin/
options.py" in add_view
494. self.save_model(request, new_object, form,
change=False)
File "/Library/Python/2.5/site-packages/auction/catalogue/admin.py" in
save_model
34. handle_data_upload(form.cleaned_data['data'], obj)
File "/Library/Python/2.5/site-packages/auction/catalogue/utils.py" in
handle_data_upload
27. catalogue=cat
File "/Library/Python/2.5/site-packages/django/db/models/manager.py"
in create
99. return self.get_query_set().create(**kwargs)
File "/Library/Python/2.5/site-packages/django/db/models/query.py" in
create
319. obj.save(force_insert=True)
File "/Library/Python/2.5/site-packages/auction/catalogue/models.py"
in save
170. super(Lot, self).save(kwargs)
File "/Library/Python/2.5/site-packages/django/db/models/base.py" in
save
328. self.save_base(force_insert=force_insert,
force_update=force_update)
File "/Library/Python/2.5/site-packages/django/db/models/base.py" in
save_base
400. result = manager._insert(values,
return_id=update_pk)
File "/Library/Python/2.5/site-packages/django/db/models/manager.py"
in _insert
138. return insert_query(self.model, values, **kwargs)
File "/Library/Python/2.5/site-packages/django/db/models/query.py" in
insert_query
894. return query.execute_sql(return_id)
File "/Library/Python/2.5/site-packages/django/db/models/sql/
subqueries.py" in execute_sql
309. cursor = super(InsertQuery, self).execute_sql(None)
File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py"
in execute_sql
1756. cursor.execute(sql, params)
File "/Library/Python/2.5/site-packages/django/db/backends/util.py" in
execute
22. sql = self.db.ops.last_executed_query(self.cursor,
sql, params)
File "/Library/Python/2.5/site-packages/django/db/backends/
__init__.py" in last_executed_query
177. u_params = tuple([to_unicode(val) for val in
params])
File "/Library/Python/2.5/site-packages/django/db/backends/
__init__.py" in <lambda>
175. to_unicode = lambda s: force_unicode(s,
strings_only=True)
File "/Library/Python/2.5/site-packages/django/utils/encoding.py" in
force_unicode
70. raise DjangoUnicodeDecodeError(s, *e.args)
Exception Type: DjangoUnicodeDecodeError at /admin/catalogue/catalogue/
add/
Exception Value: 'utf8' codec can't decode bytes in position 12-14:
invalid data. You passed in 'A paper mach\xe9 letter rack with painted
foliate decoration and a C19th papier mache side chair and one other
(a/f)' (<type 'str'>)
I hope that clears a few things up.
Is this an admin thing? (http://www.factory-h.com/blog/?p=56)
RM
On Feb 8, 12:00 am, Karen Tracey <[email protected]> wrote:
> On Sat, Feb 7, 2009 at 11:56 AM, redmonkey
> <[email protected]>wrote:
>
>
>
> > Hey everyone,
>
> > I'm trying to create django model instances from data stored in a flat
> > text file but I get `force_unicode` errors when the script comes
> > across one of the data items containing the "é" character.
>
> > Can anyone explain to me what this problem is and now I can fix it? I
> > can't really get my head around unicode, ascii and UTF-8 stuff.
>
> If you expect anyone on the list to help, you really need to share some
> snippets of the code you are using to read the data from the file and create
> Django model instances, plus the full traceback you get when it runs into
> trouble. Django certainly handles unicode data in models, so there's
> something specific about what you are doing that is causing a problem.
>
> Karen
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
