Re: Integrate dj-database-url into Django

[Tom Forbes]

> I'm wary of possible security ramifications: if we do this, changing a
configuration value will import an arbitrary module, which could make it
easier to run arbitrary code in some scenarios. I don't have a clear threat
model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model.
Django does import the settings based on an environment variable, but it's
relative and if you can use that to do anything nasty then you're most
likely already through the airtight hatch (so to speak). However importing
potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third
party database provider has to be in the `INSTALLED_APPS`? That could
provide some form of whitelisting so not any old module is imported. Not
sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd
party backends could specify a name which then has a fixed & verified
import path.

This seems like it could get complex, and be quite unlike anything else in
Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello,
>
> I'm wondering what the exact definition of the URL format is. Is it
> specified somewhere? Or is it just:
>
> [engine]://[username]:[password]@[host]:[port]/[name]
>
> where we create arbitrary [engine] values in an ad-hoc fashion?
>
> On 24 May 2017, at 21:21, Tom Forbes  wrote:
>
> My two cents: connection strings/database URI's are a feature I've sorely
> missed in Django.
>
> Built-in functionality to convert environment variables like
> DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant
> DATABASE setting would make some deployment situations a lot simpler.
> Currently, unless you use dj-database-uri you have to define a bunch of
> ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary
> together yourself.
>
>
> Fully agreed. While relatively minor, it's an annoyance.
>
> How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?
>
>
> I don't think it's reasonable to cram them in a URL.
>
> dj-database-url allows passing options as extra keyword arguments. Other
> values should be explicitly added in the settings module, by updating the
> dict generated from the URL.
>
> To help support third part backends: perhaps the scheme portion of the URI
> could be either a relative import from django.db.backends or an absolute
> import to a third party library? It seems URI schemes can have dots and
> underscores in them, so they can be python package paths.
>
> I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but
> sqlserver_ado://xyz would resolve to the third party django-mssql engine
> via an absolute import.
>
>
> I'm wary of possible security ramifications: if we do this, changing a
> configuration value will import an arbitrary module, which could make it
> easier to run arbitrary code in some scenarios. I don't have a clear threat
> model in mind here, though.
>
> I'd rather specify the database engine explicitly when calling
> dj-database-url if it's a third-party engine. There's an open question
> about what to do with the [engine] part of the URL in that case. Ignoring
> it entirely is the easiest.
>
> Best regards,
>
> --
> Aymeric.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/A0870D90-75DE-496B-A838-
> AC7D7488A67D%40polytechnique.org
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJO%3DGXGqV%3DeAbDzSr0_WWsDi%2Bv%2B-G7XhC%3DwnWvK1KXuukA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Integrate dj-database-url into Django

[Tom Forbes]

Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary
module you give it. If we accept that then I think we are accepting the
risk of imports via an attacker controlling environment variables whilst
Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes  wrote:

> > I'm wary of possible security ramifications: if we do this, changing a
> configuration value will import an arbitrary module, which could make it
> easier to run arbitrary code in some scenarios. I don't have a clear threat
> model in mind here, though.
>
> Good point, it's not wise to enable this even without a clear threat
> model. Django does import the settings based on an environment variable,
> but it's relative and if you can use that to do anything nasty then you're
> most likely already through the airtight hatch (so to speak). However
> importing potentially global modules could be bad news.
>
> Ignoring it is always an option, but could we not specify that the third
> party database provider has to be in the `INSTALLED_APPS`? That could
> provide some form of whitelisting so not any old module is imported. Not
> sure about any issues that may arise from this though.
>
> > One possibility would be to use entrypoints in setuptools, this way 3rd
> party backends could specify a name which then has a fixed & verified
> import path.
>
> This seems like it could get complex, and be quite unlike anything else in
> Django.
>
> Perhaps just supporting this for first-party database backends is easiest?
>
> On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin  polytechnique.org> wrote:
>
>> Hello,
>>
>> I'm wondering what the exact definition of the URL format is. Is it
>> specified somewhere? Or is it just:
>>
>> [engine]://[username]:[password]@[host]:[port]/[name]
>>
>> where we create arbitrary [engine] values in an ad-hoc fashion?
>>
>> On 24 May 2017, at 21:21, Tom Forbes  wrote:
>>
>> My two cents: connection strings/database URI's are a feature I've sorely
>> missed in Django.
>>
>> Built-in functionality to convert environment variables like
>> DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant
>> DATABASE setting would make some deployment situations a lot simpler.
>> Currently, unless you use dj-database-uri you have to define a bunch of
>> ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary
>> together yourself.
>>
>>
>> Fully agreed. While relatively minor, it's an annoyance.
>>
>> How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?
>>
>>
>> I don't think it's reasonable to cram them in a URL.
>>
>> dj-database-url allows passing options as extra keyword arguments. Other
>> values should be explicitly added in the settings module, by updating the
>> dict generated from the URL.
>>
>> To help support third part backends: perhaps the scheme portion of the
>> URI could be either a relative import from django.db.backends or an
>> absolute import to a third party library? It seems URI schemes can have
>> dots and underscores in them, so they can be python package paths.
>>
>> I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but
>> sqlserver_ado://xyz would resolve to the third party django-mssql engine
>> via an absolute import.
>>
>>
>> I'm wary of possible security ramifications: if we do this, changing a
>> configuration value will import an arbitrary module, which could make it
>> easier to run arbitrary code in some scenarios. I don't have a clear threat
>> model in mind here, though.
>>
>> I'd rather specify the database engine explicitly when calling
>> dj-database-url if it's a third-party engine. There's an open question
>> about what to do with the [engine] part of the URL in that case. Ignoring
>> it entirely is the easiest.
>>
>> Best regards,
>>
>> --
>> Aymeric.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%
>> 40polytechnique.org
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at h

Re: Integrate dj-database-url into Django

[Tim Allen]

I've recently been introduced to `django-environ`, a similar library that 
has additional features to DB connect URLs that we may want to consider: 
https://github.com/joke2k/django-environ

It has the same issue with third party DB engines; for example, I recently 
issued a PR to include `pyodbc` as an option. It has some nice newcomer 
friendly features, such as a cleaner way of determining BASE_URL / 
SITE_ROOT. Have a look at the URL for a good "before" and "after" settings 
file example.

Having full support for .env files would make Django projects easier to 
bring in line with 12-factor.

Regards,

Tim

On Saturday, May 27, 2017 at 3:52:23 PM UTC-4, Tom Forbes wrote:
>
> Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary 
> module you give it. If we accept that then I think we are accepting the 
> risk of imports via an attacker controlling environment variables whilst 
> Django starts up?
>
> On Sat, May 27, 2017 at 8:49 PM, Tom Forbes 
> > wrote:
>
>> > I'm wary of possible security ramifications: if we do this, changing a 
>> configuration value will import an arbitrary module, which could make it 
>> easier to run arbitrary code in some scenarios. I don't have a clear threat 
>> model in mind here, though.
>>
>> Good point, it's not wise to enable this even without a clear threat 
>> model. Django does import the settings based on an environment variable, 
>> but it's relative and if you can use that to do anything nasty then you're 
>> most likely already through the airtight hatch (so to speak). However 
>> importing potentially global modules could be bad news.
>>
>> Ignoring it is always an option, but could we not specify that the third 
>> party database provider has to be in the `INSTALLED_APPS`? That could 
>> provide some form of whitelisting so not any old module is imported. Not 
>> sure about any issues that may arise from this though.
>>
>> > One possibility would be to use entrypoints in setuptools, this way 3rd 
>> party backends could specify a name which then has a fixed & verified 
>> import path.
>>
>> This seems like it could get complex, and be quite unlike anything else 
>> in Django.
>>
>> Perhaps just supporting this for first-party database backends is easiest?
>>
>> On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <
>> aymeric@polytechnique.org > wrote:
>>
>>> Hello,
>>>
>>> I'm wondering what the exact definition of the URL format is. Is it 
>>> specified somewhere? Or is it just:
>>>
>>> [engine]://[username]:[password]@[host]:[port]/[name]
>>>
>>> where we create arbitrary [engine] values in an ad-hoc fashion?
>>>
>>> On 24 May 2017, at 21:21, Tom Forbes > 
>>> wrote:
>>>
>>> My two cents: connection strings/database URI's are a feature I've 
>>> sorely missed in Django.
>>>
>>> Built-in functionality to convert environment variables like 
>>> DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant 
>>> DATABASE setting would make some deployment situations a lot simpler. 
>>> Currently, unless you use dj-database-uri you have to define a bunch of 
>>> ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary 
>>> together yourself.
>>>
>>>
>>> Fully agreed. While relatively minor, it's an annoyance.
>>>
>>> How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?
>>>
>>>
>>> I don't think it's reasonable to cram them in a URL.
>>>
>>> dj-database-url allows passing options as extra keyword arguments. Other 
>>> values should be explicitly added in the settings module, by updating the 
>>> dict generated from the URL.
>>>
>>> To help support third part backends: perhaps the scheme portion of the 
>>> URI could be either a relative import from django.db.backends or an 
>>> absolute import to a third party library? It seems URI schemes can have 
>>> dots and underscores in them, so they can be python package paths.
>>>
>>> I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but 
>>> sqlserver_ado://xyz would resolve to the third party django-mssql engine 
>>> via an absolute import.
>>>
>>>
>>> I'm wary of possible security ramifications: if we do this, changing a 
>>> configuration value will import an arbitrary module, which could make it 
>>> easier to run arbitrary code in some scenarios. I don't have a clear threat 
>>> model in mind here, though.
>>>
>>> I'd rather specify the database engine explicitly when calling 
>>> dj-database-url if it's a third-party engine. There's an open question 
>>> about what to do with the [engine] part of the URL in that case. Ignoring 
>>> it entirely is the easiest.
>>>
>>> Best regards,
>>>
>>> -- 
>>> Aymeric.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@google

Fellow Report - May 27, 2017

[Tim Graham]

Triaged

---

https://code.djangoproject.com/ticket/28243 - Renaming Base model in 
multi-table inheritance raises django.core.exceptions.FieldError: 
Auto-generated field clashes (duplicate)

https://code.djangoproject.com/ticket/28244 - Exceptions derived from 
BaseException instead of Exception are not caught by 
convert_exception_to_response() (wontfix)

https://code.djangoproject.com/ticket/28238 - Changelist save method 
causing MemoryError (needsinfo)

https://code.djangoproject.com/ticket/28239 - incorrect 
django.template.Context documentation since 1.10 (fixed)

https://code.djangoproject.com/ticket/28240 - django.db.models.query.values 
not acting as documented (needsinfo)
https://code.djangoproject.com/ticket/28247 - Don't require last_login 
property on custom user models (duplicate)

 

Reviewed/committed

--

https://github.com/django/django/pull/8333 - Fixed #28015 -- Added 
makemessages --add-location option.

https://github.com/django/django/pull/8533 - Refs #28207 -- Fixed 
contrib.auth.authenticate() if 'backend' is in the credentials.

https://github.com/django/django/pull/8373 - Fixed #28112 --  Added 
'time_zone' in GeoIP2.city() dict.

https://github.com/django/django/pull/8532 - Fixed #28230 -- Allowed 
DjangoJsonEncoder to serialize CallableBool.

https://github.com/django/django/pull/8471 - Fixed #27859 -- Ignored 
db_index for TextField/BinaryField on Oracle and MySQL.

https://github.com/django/django/pull/8155 - Refs #27804 -- Used subTest() 
in several tests.

https://github.com/django/django/pull/8515 - Fixed docs build with Sphinx 
1.6.

https://github.com/django/django/pull/8543 - Simplified schema.tests with 
assertForeignKeyExists()/assertForeignKeyNotExists().

https://github.com/django/django/pull/8545 - Fixed #28224 -- Tested for 
SuspiciousOperation subclasses in Django's tests.

https://github.com/django/django/pull/8517 - Fixed #28211 -- Prevented 
ORing an empty Q() from reducing query join efficiency.

https://github.com/django/django/pull/8295 - Fixed #28017 -- Allowed 
customizing PasswordResetTokenGenerator's secret.

https://github.com/django/django/pull/8255 - Fixed #27978 -- Allowed 
loaddata to read data from stdin.

https://github.com/django/django/pull/8326 - Fixed #27881 -- Added 
diffsettings --output option.

https://github.com/django/django/pull/8163 - Fixed #27922 -- Added 
ErrorDict.get_json_data().

https://github.com/django/django/pull/7799 - Fixed #25006 -- Allowed custom 
time shortcuts in admin's time picker.

https://github.com/django/django/pull/8518 - Fixed #28222 -- Allowed 
settable properties in QuerySet.update_or_create()/get_or_create() defaults.

https://github.com/django/django/pull/8527 - Fixed #28226 -- Replaced use 
of str.join() with concatenation.
https://github.com/django/django/pull/8560 - Fixed #28248 -- Fixed password 
reset tokens being valid for 1 day longer than PASSWORD_RESET_TIMEOUT_DAYS.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4eb2ac07-ff9d-4ffa-a430-9723170ac60c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.