Re: Are there use cases for storing null bytes in CharField/TextField?

[Tim Graham]

A reply from Luke came only to me:


Is it even possible for null bytes to be copied into a text field and for 
them to be submitted? The original report uses Javascript to get them in 
there - https://code.djangoproject.com/ticket/28201

These pages suggests it is going to be really hard for users to be entering 
0x00 accidentally:

- 
http://stackoverflow.com/questions/6961208/how-to-input-a-null-character-into-a-web-form

- 
https://superuser.com/questions/946533/is-there-any-way-to-copy-null-bytes-ascii-0x00-to-the-clipboard-on-windows
If we are talking about this kind of rare condition, I think a slightly 
obscure error message is fine. Defaulting to stripping any data would be a 
bad idea IMO, for the case where a backend is currently handling and 
storing 0x00 chars we should try hard not to break that.

Luke


I guess we'll proceed with the form field / serialization enhancements.

On Friday, May 19, 2017 at 9:35:28 AM UTC-4, Tim Graham wrote:
>
> If CharField/TextField have a form validation error if null bytes are in 
> the input, are users going to be able to understand that error and fix it? 
> I'm not sure if it's a probable case, but I'm thinking of a non-technical 
> user who copy/pastes some text that includes a null byte.
>
> Perhaps a " strip_null_bytes" model field option that defaults to True 
> would be reasonable. That could be passed to the form field to toggle where 
> or not that validation happens. Actually, three possible behaviors might be 
> needed: silently strip null bytes, allow null bytes (an invalid option when 
> using PostgreSQL), prohibit null bytes.
>
> On Tuesday, May 16, 2017 at 5:11:38 AM UTC-4, Jani Tiainen wrote:
>>
>> Hi,
>>
>> I would guess that one could use null byte to denote "empty field" in 
>> Oracle for example. (I recall seeing such a convention in one of our 
>> non-django apps). And that's to overcome limitation that Oracle doesn't 
>> have real concept of empty string so we stored single null byte to mark 
>> that. 
>>
>>
>> On 15.05.2017 18:54, Tim Graham wrote:
>>
>> Does anyone know of a use case for using null bytes in 
>> CharField/TextField?
>>
>> psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL 
>> (0x00) characters.") when trying to save null bytes [0] and this 
>> exception is unhandled in Django which allow malicious form submissions to 
>> crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are 
>> silently truncated by PostgreSQL. Other databases that I tested (SQLite, 
>> MySQL, Oracle) allow saving null bytes. This creates possible 
>> cross-database compatibility problems when moving data from those databases 
>> to PostgreSQL, e.g.[2].
>>
>> I propose to have CharField and TextField strip null bytes from the value 
>> either a) only on PostgreSQL or b) on all databases. Please indicate your 
>> preference or suggest another solution.
>>
>> [0] https://github.com/psycopg/psycopg2/issues/420
>> [1] https://code.djangoproject.com/ticket/28201 - Saving a 
>> Char/TextField with psycopg2 2.7+ raises ValueError: A string literal 
>> cannot contain NUL (0x00) characters is unhandled
>> [2] https://code.djangoproject.com/ticket/28117 - loaddata raises 
>> ValueError with psycopg2 backend when data contains null bytes
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>> To post to this group, send email to django-d...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> -- 
>> Jani Tiainen
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/35ac6812-e02e-47fc-a3f0-e615e59c4de2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Integrate dj-database-url into Django

[Shai Berger]

If I understand things correctly, registration of new url-schemas to enable 
something like dj-database-url to support 3rd parties should be a trivial 
matter; just add an entry to some dictionary, with the schema as key, and the 
dotted-path of the backend as value.

IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
engineering for this. Just do the registration from the settings file, before 
invoking the url-parsing mechanism:

from django.database_url import register_backend, configure_db

register_backend('pyodbc-azure', 'sql_server.pyodbc')

DATABASES = {'default' : configure_db() }

The only barrier I see is namespacing -- the functions mentioned above cannot 
live in django.conf (would cause a circular import if imported from the 
settings module) nor django.db (which shouldn't be imported too early), and 
that's why I wrote `django.database_url` above.

What am I missing?