Re: Requiring sqlparse for sqlite introspection

[Adam Johnson]

I'm fine with adding it as a dependency, my experience has been that it's a
stable, well-maintained package over the past few years I've used it.

On Sun, 28 Oct 2018 at 05:00, charettes  wrote:

> After a bit of work to minimize the cases where sqlparse would be a
> required at runtime for SQLite to AddConstraint/RemoveConstraint operations
> [0] I came to the conclusion that it would make more sense to make sqlparse
> an hard dependency of Django 2.2.
>
> The two reasons backing this conclusions are
>
> 1. Given we run the suite with sqlparse installed on CI it will be really
> hard to prevent inadvertently breaking the promise of a soft dependency on
> sqlparse for Meta.constraints only. I guess we could have a daily CI job
> but that would quickly get out of hand once we have to perform backport and
> such.
>
> 2. There's a few instances of fragile regex parsing that could be made
> more reliable if sqlparse was an hard dependency for SQLite. Two examples
> are
>  -
> https://github.com/django/django/blob/f892781b957f674806a227a10c58768f66a48c07/django/db/backends/sqlite3/introspection.py#L90-L146
>  -
> https://github.com/django/django/blob/f892781b957f674806a227a10c58768f66a48c07/django/db/backends/sqlite3/schema.py#L91-L127
>
> Given the transition to require pytz in Django 1.11 went smoothly and our
> needs for sqlparse are similar to provide a solid experience on SQLite I'd
> be +1 on requiring it from Django 2.2 to a point where the lowest version
> of SQLite we support has better introspection capabilities.
>
> Simon
>
> [0] https://github.com/django/django/pull/10572
>
> Le mercredi 10 octobre 2018 14:53:53 UTC-4, Tim Graham a écrit :
>>
>> sqlparse is already installed as part of Django's tests. The question is
>> whether sqlparse should be mandatory for SQLite users (i.e. when getting
>> started with a new project an error message will say, "You must install
>> sqlparse to use SQLite" (I don't think there's a way to install it
>> automatically unless we install it for all users, regardless of database)
>> or if we should try to make it optional and only raise an error if a
>> project's migrations require it.
>>
>> The question is "what percentage of SQLite projects are going to end up
>> needing sqlparse based on their migrations? If it's high enough, it might
>> make for a better user experience to have users install it when they're
>> getting started instead of a lazy error when get_constraints() is
>> called."
>>
>> On Monday, October 8, 2018 at 3:22:32 AM UTC-4, Andrew Godwin wrote:
>>>
>>> Adding sqlparse into the introspection code for SQLite specifically
>>> means it's going to be a runtime dependency for anything to do with
>>> migrations.
>>>
>>> I'm alright having that be the case, but I do think we should make sure
>>> the tests run with SQLite as otherwise a large section of the most
>>> complicated code in migrations won't be tested properly.
>>>
>>> Andrew
>>>
>>>
>>> On Mon, 8 Oct 2018, 00:59 Ian Foote,  wrote:
>>>
 Hi all,

 On my pull request (https://github.com/django/django/pull/10406)
 refactoring how Django creates database constraints I introduced a
 dependency on sqlparse in the sqlite introspection code. This allows Django
 to correctly read information about constraints on sqlite, particularly the
 name.

 When reviewing (
 https://github.com/django/django/pull/10406#issuecomment-424542217)
 Tim Graham raised the question of whether we should make sqlparse a
 mandatory requirement for the sqlite tests or if we should skip those tests
 that require it. In later discussion (
 https://github.com/django/django/pull/10406#issuecomment-427362983),
 Tim raised the point that the introspection code is used by migrations.

 I'm not clear myself what the best route forwards here is, so I'm
 bringing it to Django Developers for discussion (as Tim suggested).

 Thanks,
 Ian

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-developers/CAFv-zfKadOeWit8M6GMmx4H2ChUCU6u%3DscHX8F7oBKJkHRbuVg%40mail.gmail.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiv

Re: Standalone is_safe_url() function

[Adam Johnson]

>
> I needed that functionality on another project that doesn't use Django at
> all.
>

If this was my own project, I would have installed Django and imported the
function. Afaict it doesn't depend on settings or any other setup so it
should work from an import. The only concern would be size of site-packages
but that's rarely an issue with web projects. Was there something else that
stopped this?

- If so, should Django possibly depend on that package by itself? Given how
> often Django had security releases because of issues in `is_safe_url()`
> releasing a smaller package and not the full Django package could possibly
> be beneficial.


This seems like the start of a broader request for unbundling other things
from Django - if we unbundle this one function, shouldn't we npm-style
unbundle "all the things"? (e.g. cached-property has been
'unbundled'/forked by PyDanny, Django could start depending on that
version...). But unbundling increases maintenance work, testing complexity,
and the number of things users need to understand.

Also, I tried to look up documentation for `is_safe_url()` and failed. Is
> there any particular reason why it's not documented? You can see seven
> other things in
> https://docs.djangoproject.com/en/2.1/ref/utils/#module-django.utils.http but
> not a single mention of is_safe_url...
>

Being undocumented, is_safe_url isn't part of Django's public API. It could
probably be documented now at least, given its utility outside of the
places Django uses it.

On Wed, 10 Oct 2018 at 18:09, ivan via Django developers (Contributions to
Django itself)  wrote:

> Hi Markus,
>
> Well, adding new requirements for every function might be not a scalable
> approach, but if some stuff in django important for security was separated
> into a sub-project it could allow more people to patch their projects more
> easily. From what I've seen in various projects I worked in, far too many
> folks don't update django (and other deps) often enough because their fear
> that something will break (and it often does when custom code depending on
> django internals is involved), keeping some parts of the django codebase in
> a separate but official package could make it easier for many projects to
> update just it without worrying, provided that package's api is absolutely
> stable and backward-compatible. People who run unsupported versions of
> django (I don't approve but it's a reality we have to deal with) could also
> benefit from it as they can update the package regardless of which version
> of django itself they use.
>
> The major downsides, it seems to me, are that people might get even
> sloppier with updating django if they think that security sub-package is
> enough to stay safe (it should be made clear that it's not) and that it
> adds an additional dependency (but I think it's ok as long as it's just one
> and its purpose and contents is obvious). Also it would be hard to decide
> what to leave in django and what to separate.
>
> tl;dr what I'm saying is I'm not against moving it from django to a
> separate package django will depend on, but I'd prefer it having a broader
> goal than one function, however important.
>
> Also, I tried to look up documentation for `is_safe_url()` and failed. Is
> there any particular reason why it's not documented? You can see seven
> other things in
> https://docs.djangoproject.com/en/2.1/ref/utils/#module-django.utils.http
> but not a single mention of is_safe_url...
>
> Ivan.
>
> On Wednesday, October 10, 2018 at 1:06:46 PM UTC+3, Markus Holtermann
> wrote:
>>
>> Hi all,
>>
>> Django provides a function `django.utils.is_safe_url()` to ensure that a
>> given URL (absolute or relative) is safe to redirect to. I needed that
>> functionality on another project that doesn't use Django at all. I thus
>> built a standalone is-safe-url Python package that can be installed from
>> PyPI and exposes exactly that functionality:
>>
>>   $ pip install is-safe-url
>>   Collecting is-safe-url
>> Downloading https://files.pythonhosted.org/packages/7a/c3
>>  
>> /40c363bc4c3d0ddcda3489239ba64752b8c18cb6493e058f8f1b73154925/is_safe_url-1.0-py3-none-any.whl
>>
>>   Installing collected packages: is-safe-url
>>   Successfully installed is-safe-url-1.0
>>
>> The code is available on GitLab: https://gitlab.com/MarkusH/is_safe_url
>>
>> I'd love to get some feedback on a couple of things:
>>
>> - As Django is published under the BSD-3 clause license, the standalone
>> package is published under the same license. I'd love some feedback if the
>> package adheres to the required references and naming of the source.
>>
>> - I added a note that security issues should be reported privately to the
>> Django security team at secu...@djangoproject.com or me personally (I'm
>> a member of the security team and could forward the report accordingly).
>> Are there suggestions how the statement in the README could be made more
>> clear?
>>
>> - The package is available for Python 2.7, 3.4, 3.5,

need help to get started

[Harsh Sahu]

Hello There!
This is Harsh. I'm pursuing master's degree in Computer Science. I'm a 
intermediate python person and learning full stack 
development(Flask/Django). Also I'm new to open source and would like to 
make my contributions to Django. Please help me to get started .
Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a2f571d4-1d3c-4f06-9a28-a8bbe45ac445%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: need help to get started

[Tim Graham]

Hi Harsh, please read 
https://docs.djangoproject.com/en/dev/internals/contributing/new-contributors/.

On Sunday, October 28, 2018 at 9:42:07 AM UTC-4, Harsh Sahu wrote:
>
> Hello There!
> This is Harsh. I'm pursuing master's degree in Computer Science. I'm a 
> intermediate python person and learning full stack 
> development(Flask/Django). Also I'm new to open source and would like to 
> make my contributions to Django. Please help me to get started .
> Thanks in advance.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6eedbfca-6ec8-489d-b52d-b6cbca3fbfa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: #29752 Adding a ALLOWED_HOSTS_IGNORABLE_URLS setting

[Peter Baumgartner]

I implemented something for this in the django-alive package via a
middleware that will bypass the host checking:

https://github.com/lincolnloop/django-alive/#disabling-allowed_hosts-for-healthchecks
https://github.com/lincolnloop/django-alive/blob/master/django_alive/middleware.py

On Fri, Sep 14, 2018 at 3:18 PM Adam Johnson  wrote:

> The snippet Matt posted is the same technique I've used for ages, albeit
> using the ec2-metadata  library. I
> think it's perfectly fine as-is, the Host header EC2 uses is actually
> predictable as the EC2 Private IP. I don't think Django needs another
> setting that disables a security feature and could be open to
> misconfiguration.
>
> On Fri, 14 Sep 2018 at 20:29, Mattia Procopio  wrote:
>
>> What I usually do is rewriting the Host value at webserver level using
>> one of the allowed when receiving healthchecks from a load balancer. This
>> is not optimal and having a whitelist for some uris to allow requests
>> without a valid host could make this specific thing easier
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/e51aa4d8-d263-4448-ab3c-d0717035fbcb%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0_uL%2B7APa%3DwgvU_GZaqO8fXDJOWAFKf0jGGB1pMVs2kg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAAoxf4v_eJ-%3D%3Dpd-ZA42PLkmN2Gq_QNeHz_D%3DsunZZexVqdt5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

[Josh Smeaton]

I strongly dislike Trac in nearly every way. It's hard to search and the 
filters are next to useless, and the categorisation features we use are not 
very useful. I believe the better way to search Trac is to use google and 
site:code.djangoproject.com which is a red flag itself.

On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>
> How much of this would you attribute to the current ticketing system 
> itself, rather than tickets being tagged appropriately?
>
> I know when I started contributing I found trac to be pretty intimidating 
> in terms of complexity, especially the search. I still prefer to use the 
> 'Search Trac' field in the root code.djangoproject.com page than fiddle 
> with the myriad of options and drop downs in the browse tickets section.
>
> If we think of getting new people onboard as a conversion funnel we need 
> to stop dropoff as much as possible, and that extends to the UI of the 
> ticket tracker as well I believe.
>
> Tom
>
> On Fri, 26 Oct 2018, 22:43 Ian Foote, > 
> wrote:
>
>> Hi Carlton,
>>
>> I've had similar thoughts sitting in the back of my mind for at least a 
>> couple of months, so thank you for sharing this. I agree that finding 
>> tickets is one of the big problems here, both for new contributors and for 
>> sprint leaders. At Pycon UK I took on the role of sprint leader along with 
>> Adam Johnson and directing people to appropriate tickets was a definite 
>> difficulty. I was also unaware of the django core mentorship list and will 
>> be joining that soon. I'm willing to spend some time mentoring a small 
>> number of people, life permitting.
>>
>> Ian
>>
>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson > > wrote:
>>
>>> Hi All. 
>>>
>>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if you 
>>> organised that! Hi! if we met and chatted.) 
>>> I gave a talk ("Your web framework needs you!") inspired by the 
>>> discussion on the 
>>>  
>>> DSF list and the proposal to dissolve Django Core 
>>> . (Can’t see the DSF list? Join 
>>> the DSF 
>>> 
>>> .)
>>> I was asking for more participation in general, and participation that 
>>> is more representative of the wider Django community in particular.
>>>
>>> There was lots of good input from many people, including (but not, at 
>>> all, limited to) representatives of groups such Pyladies, DjangoGirls, and 
>>> so on. 
>>>
>>>
>>> The recurring themes seem to me to fit into three categories:
>>>
>>>1. The importance of *mentoring*. 
>>>2. The difficulty of *finding tickets*. 
>>>3. The importance of *sprints*. 
>>>
>>> The rest here is a summary of that. Hopefully it’s useful. 
>>>
>>> Mentoring
>>>
>>> For whatever reasons, the exiting *Contributing How-To* 
>>>  doesn’t 
>>> lead to contributions from a demographic that matches the wider Django 
>>> Community. 
>>>
>>> The point that came up again and again about this was that *mentoring* 
>>> is one of the best (perhaps the best?) tool in helping to change this. 
>>>
>>> Django Core Mentorship
>>>
>>> We don’t have an official mentoring programme but we do have the 
>>> django-core-mentorship 
>>> list . 
>>>
>>> This must be about the best-kept secret in the Django world: it’s gets 
>>> ≈0 traffic, but I told everybody at DjangoCon about it, and that they 
>>> should use it. 
>>>
>>> If you are not on django-core-mentorship, and you’re willing to help 
>>> prospective contributors, please sign-up. I’m hoping we can drive some 
>>> traffic to it. 
>>>
>>> Maybe there’s call for something more formal, but at least until DCM is 
>>> actually being used, that seems (to me) like something we can postpone. 
>>>
>>> Finding Tickets
>>>
>>> The next thing was that there’s not enough guidance on what to work on. 
>>>
>>> The guidance is to look for *Easy Pickings*. There are ≈1300 accepted 
>>> open tickets in TRAC. 13 of these are marked *Easy Pickings*. 
>>>
>>> That’s not enough. I think we’re too tight with it (or need another 
>>> grade). 
>>>
>>> There are *many* tickets which aren’t super hard: I put it that, most 
>>> of our community solve harder problems every day *using Django* than 
>>> most tickets require. 
>>>
>>> Yes, they still require time, love, energy, etc — and maybe some 
>>> mentoring — but it’s not primary research, in the main.
>>>
>>> I talked to people who had (at the conference) got the test suite 
>>> running and such, but been overawed by the (for want of a better phrase) 
>>> *sheer 
>>> face* of issue tracker. 
>>>
>>> We would do well to invite people better here. (I don’t have instant 
>>> solutions.) 
>>>
>>> Sprints
>>>
>>> I’m not historically a Django-sprinter.

python-memcached seems to be unmaintained

[Adrian Turjak]

Through some of my last few projects using Django and Memcached I kept
running into the problem that python-memcached appears to no longer be
maintained[1], and even before that the release frequency was getting
quite low.

For Django pylibmc is an alternative, but having to rely on an
underlying C library is a bit annoying and the library does act a little
differently.

Give the state of python-memcached I think Django should support
pymemcache[2]. It is actively maintained by Pintrest, with appropriate
opensource licensing, is fully python, and appears to be a great library.

There appear to have been attempts by thirdparties to make a Django
cache backend for it, but none of those are maintained. Bringing the
support into core, and replacing python-memcached with pymemcache may be
a better good long term solution than continuing to support and
recommend an unmaintained library when there is a suitable alternative.

I opened this ticket here to track that, and the recommendation was to
bring it (rightly so) to the mailing list:
https://code.djangoproject.com/ticket/29887

I'm also a member of the OpenStack community and I've talked to devs who
were interested in taking over python-memcached as per the discussion in
the linked issue, but it looks like OpenStack's oslo.cache is also
switching to pymemcache at this stage because it offers them a great
alternative.

Any thoughts?


1: https://github.com/linsomniac/python-memcached/issues/95
2: https://github.com/pinterest/pymemcache

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3ad5467c-8073-7273-ae5b-00f5dd713efa%40catalyst.net.nz.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

[Dan Davis]

Trac can be made easier to search with Apache Solr 
- 
https://www.pycon.it/conference/talks/full-text-search-for-trac-with-apache-solr

On Sunday, October 28, 2018 at 6:04:12 PM UTC-4, Josh Smeaton wrote:
>
> I strongly dislike Trac in nearly every way. It's hard to search and the 
> filters are next to useless, and the categorisation features we use are not 
> very useful. I believe the better way to search Trac is to use google and 
> site:code.djangoproject.com which is a red flag itself.
>
> On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>>
>> How much of this would you attribute to the current ticketing system 
>> itself, rather than tickets being tagged appropriately?
>>
>> I know when I started contributing I found trac to be pretty intimidating 
>> in terms of complexity, especially the search. I still prefer to use the 
>> 'Search Trac' field in the root code.djangoproject.com page than fiddle 
>> with the myriad of options and drop downs in the browse tickets section.
>>
>> If we think of getting new people onboard as a conversion funnel we need 
>> to stop dropoff as much as possible, and that extends to the UI of the 
>> ticket tracker as well I believe.
>>
>> Tom
>>
>> On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:
>>
>>> Hi Carlton,
>>>
>>> I've had similar thoughts sitting in the back of my mind for at least a 
>>> couple of months, so thank you for sharing this. I agree that finding 
>>> tickets is one of the big problems here, both for new contributors and for 
>>> sprint leaders. At Pycon UK I took on the role of sprint leader along with 
>>> Adam Johnson and directing people to appropriate tickets was a definite 
>>> difficulty. I was also unaware of the django core mentorship list and will 
>>> be joining that soon. I'm willing to spend some time mentoring a small 
>>> number of people, life permitting.
>>>
>>> Ian
>>>
>>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson  
>>> wrote:
>>>
 Hi All. 

 OK, so last week I was at DjangoCon US in San Diego. (Thank you if you 
 organised that! Hi! if we met and chatted.) 
 I gave a talk ("Your web framework needs you!") inspired by the 
 discussion on the 
  
 DSF list and the proposal to dissolve Django Core 
 . (Can’t see the DSF list? Join 
 the DSF 
 
 .)
 I was asking for more participation in general, and participation that 
 is more representative of the wider Django community in particular.

 There was lots of good input from many people, including (but not, at 
 all, limited to) representatives of groups such Pyladies, DjangoGirls, and 
 so on. 


 The recurring themes seem to me to fit into three categories:

1. The importance of *mentoring*. 
2. The difficulty of *finding tickets*. 
3. The importance of *sprints*. 

 The rest here is a summary of that. Hopefully it’s useful. 

 Mentoring

 For whatever reasons, the exiting *Contributing How-To* 
  
 doesn’t lead to contributions from a demographic that matches the wider 
 Django Community. 

 The point that came up again and again about this was that *mentoring* 
 is one of the best (perhaps the best?) tool in helping to change this. 

 Django Core Mentorship

 We don’t have an official mentoring programme but we do have the 
 django-core-mentorship 
 list . 

 This must be about the best-kept secret in the Django world: it’s gets 
 ≈0 traffic, but I told everybody at DjangoCon about it, and that they 
 should use it. 

 If you are not on django-core-mentorship, and you’re willing to help 
 prospective contributors, please sign-up. I’m hoping we can drive some 
 traffic to it. 

 Maybe there’s call for something more formal, but at least until DCM is 
 actually being used, that seems (to me) like something we can postpone. 

 Finding Tickets

 The next thing was that there’s not enough guidance on what to work on. 

 The guidance is to look for *Easy Pickings*. There are ≈1300 accepted 
 open tickets in TRAC. 13 of these are marked *Easy Pickings*. 

 That’s not enough. I think we’re too tight with it (or need another 
 grade). 

 There are *many* tickets which aren’t super hard: I put it that, most 
 of our community solve harder problems every day *using Django* than 
 most tickets require. 

 Yes, they still require time, love, energy, etc — and maybe some 
 mentoring — but it’s not primary research, in the main.

 I talked to people who