Re: GSoC 2020 Proposal for ModelStates in Migrations

[Sanskar Jaiswal]

Hey Simon,

While I certainly see the benefits of this, I still fail to understand how
this would help ModelState get access to related db_types, (the problem
that the mapping in my proposal aims to solve). Am I missing something
here? Some more clarification/explanation would be highly helpful.

Kind regards
Sanskar

On Fri, Mar 27, 2020 at 8:11 AM charettes  wrote:

> Thanks for the examples Sanskar.
>
> I think the mapping should be kept ProjetState instances and invalidated
> when state alterations are performed. In an ideal world I think that
> ProjectState should have methods mapping to all Operation subclasses
> state_forwards and these state_forwards methods should only proxy
> ProjectState methods.
>
> The CreateModel.state_forwards method is a good example of that[0]; it
> calls into ProjectState.add_model where all the storage and cache
> invalidation logic is encapsulated[1]. If ProjectState had methods such as
> rename_model, alter_field, and such it would make it easily testable in
> isolation and allow it to maintain it's related object cache, or mapping as
> you call it, more easily.
>
> Simon
>
> [0]
> https://github.com/django/django/blob/421622548060499881df9966b7a352bce63901cd/django/db/migrations/operations/models.py#L80-L87
> [1]
> https://github.com/django/django/blob/421622548060499881df9966b7a352bce63901cd/django/db/migrations/state.py#L91-L95
>
> Le jeudi 26 mars 2020 16:01:05 UTC-4, Sanskar Jaiswal a écrit :
>>
>> Hi Simon
>>
>> Taking advice into account, I have made some more changes to my proposal
>> .
>> Please check it out and criticize it wherever you think I could improve it
>> as always. :)
>>
>> Also, could you kindly give some kind of idea as to when the new mappings
>> should be populated? I think it makes sense to do so whenever we run
>> makemigrations. Do you have a better idea?
>>
>> Thanks!
>> Kind Regards
>> Sanskar
>>
>> On Thu, Mar 26, 2020 at 6:05 AM charettes  wrote:
>>
>>> Thanks Sanskar, this iteration is much better.
>>>
>>> Now that you have a bit more details about your proposed changes
>>> breakdown I'd give a bit more examples of what a particular operation
>>> currently looks like now and you plan to change it. For example, you could
>>> describe how an AlterField operation currently alter the state/database
>>> forwards with the involved function calls and how you plan to changes
>>> things.
>>>
>>> Detailing a bit more how the related lookups cache invalidation would be
>>> busted and populated when a specific operation is performed would go a long
>>> way to support your proposal.
>>>
>>> To summarize, a few examples of what you have in mind and how things
>>> would interact together would be appreciated. This is a quite a complex
>>> problem to solve so a solid initial plan and understanding of the problem
>>> will make a huge difference down the road.
>>>
>>> Cheers,
>>> Simon
>>>
>>> Le mercredi 25 mars 2020 19:10:40 UTC-4, Sanskar Jaiswal a écrit :

 Hey Simon

 I have made some changes to the proposal. I kindly request you to take
 a look at it and give any feedback you can :)

 https://gist.github.com/aryan9600/b1c2eaf445006c17e02e7677cf1098d5

 Kind Regards
 Sanskar



 On Wed, 25 Mar 2020 at 23:57, charettes  wrote:

> Are you thinking about the Operation.state_forwards logic? I guess it
> would make it easier to test state alterations in isolation but I don't
> think it's necessary for this already large project.
>
> Cheers,
> Simon
>
> Le mercredi 25 mars 2020 06:05:22 UTC-4, Sanskar Jaiswal a écrit :
>>
>> Hey Simon
>>
>> Just so that I am more clear, is it favourable to move all logic that
>> ModelOperations and FieldOperations implement right now to ProjectState 
>> and
>> ModelState?
>>
>> Kind regards
>> Sanskar
>>
>> On Tue, 24 Mar 2020 at 04:20, Sanskar Jaiswal 
>> wrote:
>>
>>> Hey Simon,
>>>
>>> Thank you for your feedback! It helps a lot. I shall look into your
>>> doubts right now, and edit my proposal accordingly.
>>>
>>> Kind regards
>>>
>>> Sanskar
>>>
>>> On Tue, Mar 24, 2020 at 3:37 AM charettes  wrote:
>>>
 Hello Sanskar,

 Thank you for your submission, from a quick look it seems to be
 heading in the right direction.

 Would it be possible to break the large overview section into more
 granular sections where you describe how you roughly plan to tackle 
 each of
 the point mention?

 I personally have doubts about your desire to make minimal change
 to the schema editor and offload the logic to database_forwards 
 instead of
 the other way around. That doesn't seem to be in line with a future
 deprecation of support for direct model passing. I'd m

GSOC Student

[Edidiong Etuk]

Hi, I'm Edidiong, I'd like to be engaged with the GraphQL ORM wrapper, 
either as a student or just freelancing. How do I begin?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57245b50-e0af-4b11-a07c-e5fb4b4148b8%40googlegroups.com.

Re: GSoC Mentors

[Antje Kazimiers]

Hi Ramya,

To start contributing to Django, first I would look at the list of “Easy 
tickets” which are still unassigned:

https://code.djangoproject.com/query?status=!closed&easy=1&stage=Accepted&desc=1&order=changetime
 


There is just one right now:

https://code.djangoproject.com/ticket/30864 


Then you should try building the Django code base and run the tests, how, is 
described here:

https://github.com/carltongibson/dcus2019sprints 


Since this ticket is about documentation, we probably also have to build the 
code base of the Django website, so

https://github.com/django/djangoproject.com 


I’m kind of at the same stage, so if you want to reach out to me directly via 
email we can go through those steps together. We also should assign the ticket 
to one of us once we know we start working on it.

Best,
Antje

> On 26. Mar 2020, at 20:57, ramya kurapati  
> wrote:
> 
> I want to take on a project and start contributing my part to django but I 
> have no idea where to start. If anyone can help me, that would be great...
> thanks a lot.
> 
> On Wednesday, March 25, 2020 at 2:13:49 PM UTC+5:30, Carlton Gibson wrote:
> Hi all. 
> 
> We're reaching the end of the application period for GSoC. 
> 
> In order to know how many students we might accept, we need to know how many 
> prospective mentors we have. 
> 
> This falls into two kinds of job: 
> 
> 1. General project management help: communicating with students to help them 
> set a schedule and a rhythm, and make sure they're able to make progress. 
> (The hope is they're self motivated but...) 
> 
> 2. More technical input. For this I'm thinking particularly about the 
> migrations framwork (Markus, Simon, Shai, Andrew, ...) but experienced hands 
> with knowledge of the internals: your input would be invaluable here. 
> 
> The forum seems to be working well, and it's a good format for this kind of 
> thing, so I want to aim to focus the discussion there. 
> 
> I'd like to share the mentoring as a group. We'd need to assign specific 
> mentors but I don't see why it can't be a group thing. 
> 
> If you could hang out a bit on the forum (mainly for task 1) and/or offer 
> technical input, which you might be doing already/anyway (for task 2) then 
> that would be a super contribution. 
> 
> Please let me know if you'd be willing to help mentor. 
> 
> Thanks
> 
> Kind Regards,
> 
> Carlton
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/92972211-e4c4-41e5-b7bd-9cb3115a0322%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/375F2F15-3C72-403E-A26E-33A6D93D9B47%40gmail.com.

GSoC proposal research - type hinting

[Kacper Szmigiel]

Hello everyone!

My name is Kacper, I'm 22 y.o. CS student from Lodz, Poland. I've started 
researching some nice feature for my GSoC proposal, and I have a few 
questions to you, dear Community :)
Unfortunately, due to this whole mess caused by covid-19 I had like no time 
to sink into Django codebase and figure some things out by myself (none of 
polish Universities were ready to switch to e-learning). I have a few years 
of experience in using Django for my commercial projects, but I had no 
occasion to contribute to the framework yet.

What I would like to implement is type hinting, as suggested here: 
https://code.djangoproject.com/ticket/29299

So, my first question is: what do you think about this one? Does it make 
sense at all? I suppose such feature could be really useful for many of 
developers (
https://www.willmcgugan.com/blog/tech/post/adding-type-hints-to-the-django-orm/
).

As I mentioned, I had no time to read the code and understand how Django 
works. Do you have any tips on where should I start my technical research 
in reference to this feature? I know I have not much time to prepare my 
proposal, but I will do my best during this weekend and send it to get some 
feedback.

Thanks in advance, any tips and tricks welcome!

Kindest regards,
Kacper Szmigiel

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/61592cd2-87b5-4d6b-a2f8-52a3dbf3fa21%40googlegroups.com.

Re: GSoC proposal research - type hinting

[Tom Forbes]

As an avid user of type hints I think it would be fantastic. I believe with the 
upcoming release we only target Python versions that support type hinting, 
which was one of the issues holding us back before.

I would recommend choosing a specific part of Django and investigating adding 
type hints. Django is quite big, and I think most of the value for type hints 
comes from user-facing components rather than some internals.

I would start looking at:
1. The ORM
2. Class based views
3. The Request/Response objects

And see if you think it’s viable to add a decent amount of type hints to any of 
these. There is some “magic” in each of these which might be hard to express 
with type hints, but it would be valuable to know how far we could get.

Tom

> On 27 Mar 2020, at 23:07, Kacper Szmigiel  wrote:
> 
> 
> Hello everyone!
> 
> My name is Kacper, I'm 22 y.o. CS student from Lodz, Poland. I've started 
> researching some nice feature for my GSoC proposal, and I have a few 
> questions to you, dear Community :)
> Unfortunately, due to this whole mess caused by covid-19 I had like no time 
> to sink into Django codebase and figure some things out by myself (none of 
> polish Universities were ready to switch to e-learning). I have a few years 
> of experience in using Django for my commercial projects, but I had no 
> occasion to contribute to the framework yet.
> 
> What I would like to implement is type hinting, as suggested here: 
> https://code.djangoproject.com/ticket/29299
> 
> So, my first question is: what do you think about this one? Does it make 
> sense at all? I suppose such feature could be really useful for many of 
> developers 
> (https://www.willmcgugan.com/blog/tech/post/adding-type-hints-to-the-django-orm/).
> 
> As I mentioned, I had no time to read the code and understand how Django 
> works. Do you have any tips on where should I start my technical research in 
> reference to this feature? I know I have not much time to prepare my 
> proposal, but I will do my best during this weekend and send it to get some 
> feedback.
> 
> Thanks in advance, any tips and tricks welcome!
> 
> Kindest regards,
> Kacper Szmigiel
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/61592cd2-87b5-4d6b-a2f8-52a3dbf3fa21%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/FAFB6CCC-0A8E-4272-AE8D-F808B285981E%40tomforb.es.

Re: GSoC proposal research - type hinting

[charettes]

You might also be interested in this Django Enhancement Proposal to add 
type hints to Django

https://github.com/django/deps/pull/65

Cheers,
Simon

Le vendredi 27 mars 2020 19:08:07 UTC-4, Kacper Szmigiel a écrit :
>
> Hello everyone!
>
> My name is Kacper, I'm 22 y.o. CS student from Lodz, Poland. I've started 
> researching some nice feature for my GSoC proposal, and I have a few 
> questions to you, dear Community :)
> Unfortunately, due to this whole mess caused by covid-19 I had like no 
> time to sink into Django codebase and figure some things out by myself 
> (none of polish Universities were ready to switch to e-learning). I have a 
> few years of experience in using Django for my commercial projects, but I 
> had no occasion to contribute to the framework yet.
>
> What I would like to implement is type hinting, as suggested here: 
> https://code.djangoproject.com/ticket/29299
>
> So, my first question is: what do you think about this one? Does it make 
> sense at all? I suppose such feature could be really useful for many of 
> developers (
> https://www.willmcgugan.com/blog/tech/post/adding-type-hints-to-the-django-orm/
> ).
>
> As I mentioned, I had no time to read the code and understand how Django 
> works. Do you have any tips on where should I start my technical research 
> in reference to this feature? I know I have not much time to prepare my 
> proposal, but I will do my best during this weekend and send it to get some 
> feedback.
>
> Thanks in advance, any tips and tricks welcome!
>
> Kindest regards,
> Kacper Szmigiel
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/159ae7fa-f694-4ff5-929d-2c4ea55c0773%40googlegroups.com.

Re: GSoC Proposal to extend the parallel test runner to Windows and macOS and to support Oracle parallel test running

[Ahmad A. Hussein]

Apologies for the late response. I've had to attend to personal matters 
concerning the current crisis we're all facing. All is well though

I should have posted a more detailed post from the get-go. I apologize for 
the lack of clarity as well.

Last week, I initially did exactly what you suggested. I called 
django.setup() in each child process during worker initialization. This 
fixed app registry issues but like you said it isn't enough for testing. 
Test apps and test models were missing and caused tons of errors. Later, I 
read through runtests.py and saw the setup method there; it was exactly 
what I needed since it searched for setup the correct template backend, 
searched for test apps and added them to installed apps, and called 
django.setup(). I passed that method along with its initial arguments into 
the runner so I could call it during worker initialization. That fixed all 
errors related to state initialization. I do wonder though if any 
meaningful time could be saved if we use a cache for setup instead of 
having to call the setup for each process.

The last glaring hole was correct database connections. I had a naming 
mismatch with Postgres and I ended up fixing that through prepending 
"test_" in each cloned database's name during worker initialization. In 
case of the start method being fork, we can safely ignore that step and 
it'll work fine. 

Currently what I'm figuring out is getting a SQL dump to change SQLite's 
cloning method and implementing an Oracle cloning method. I'm searching 
through Django's documentation and internal code to see if there is a 
ready-made SQL dump method for SQLite and if not I'll search for it in a 
third-party library, and if I still don't find it, I'll start thinking 
about a ground-up implementation using the ORM. 

As for progress on the Oracle cloning method, I'm consulting Oracle 
documentation right now to see if anything has changed in the last 5 years. 
If I don't find anything interesting, I'll start toying around with Shai 
Berger's ideas to see what works and what's performance-costly.

Lastly, testing does seem to be an enigma to think about right now. I've 
thought about tests for both SQLite's and Oracle's cloning method, but I 
can't imagine anything else.

If you have any pointers, suggestions or feedback, I'd love to hear it! And 
thank you for your help so far.


Regards,
Ahmad


On Thursday, March 26, 2020 at 10:39:28 AM UTC+2, Aymeric Augustin wrote:
>
> Hello Ahmad,
>
> I believe there's interest for supporting parallel test runs on Oracle, if 
> only to make Django's own test suite faster.
>
> I'm not very familiar with spawn vs. fork. As far as I understand, spawn 
> starts a new process, so you'll have to redo some initialization in the 
> child processes. In a regular application, this would mean calling 
> `django.setup()`. In Django's own test suite, it might be different; I 
> don't know. Try it and see what breaks, maybe?
>
> Hope this helps :-)
>
> -- 
> Aymeric.
>
>
>
> On 23 Mar 2020, at 20:22, Ahmad A. Hussein  > wrote:
>
> Django's parallel test runner works through forking processes, making it 
> incompatible on Windows by default and incompatible in macOS due to a 
> recent update. Windows and macOS both support spawn and have it enabled by 
> default. Databases are cloned for each worker.
>
> To switch from fork to spawn, state setup will be handled by spawned 
> processes instead of through inheritance via fork. Worker’s connections to 
> databases can still be done through get_test_db_clone_settings which 
> changes the names of databases assigned to a specific worker_id; however, 
> SQLite's cloning method is incompatible with spawn. 
>
>
> SQLite’s cloning method relies on it both being in-memory and fork as when 
> we fork the main process, we retain SQLite's in-memory database in the 
> child process. The solution is to get a SQL dump of the database and throw 
> it into the target cloned databases. This is also the established standard 
> in handling MySQL’s cloning process. Both Postgresql's and MySQL's cloning 
> methods are independent of fork or spawn and won't require any modification.
>
> Oracle has not been implemented in the parallel test runner originally 
> even on Linux and I propose to extend support to Oracle as well in my 
> proposal. I want to confirm if there is significant support behind this as 
> a feature or not before I commit to writing a specification, but as a 
> summary it is definitely possible as the work collaborated on by Aymeric 
> Augustin and Shai Berger show that cloning CAN be done through multiple 
> ideas. The reason why it's a headache is that Oracle does not support 
> separate databases under a single user- unlike our other supported 
> databases, so we can't clone databases without introducing another user. 
> Some methods may also need to be rewritten to accommodate for the Oracle 
> backend, but that isn't an issue. I've glossed over writing out a schedule 
> or a m