Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-24 Thread Shai Berger
On Wednesday 23 March 2016 11:50:59 Hugo Chargois wrote: > > ... MySQL's REPEATABLE READ may have > it flaws, it may cause repeating bugs because that level is a bit awry, > with its reads and writes that don't work the same, but all in all, it IS a > higher isolation level than READ COMMITTED. It

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-24 Thread Shai Berger
While there are some different opinions expressed here (mine included), the rough consensus seems to be to apply the for_update() patch to 1.8.x (it is neither needed nor applicable to 1.9.x) and change the transaction isolation level for 1.10 and going forward. To the best of my knowledge, nob

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-24 Thread Ben Cail
What's the current status on this? Is the .for_update() change mentioned in the bug report the best way to go? Is anyone working on a PR? On 03/18/2016 05:15 PM, Karen Tracey wrote: This is the 2nd major issue I can recall caused by MySQL default of REPEATABLE READ transaction isolation level.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-23 Thread Hugo Chargois
Le mardi 22 mars 2016 23:33:10 UTC+1, Shai Berger a écrit : > It is Django's job to try, as best as it can, to fulfill these expectations. How could I disagree? Of course, if there is one single sensible, obvious default, that would help 99.9 % of users, like your webserver analogy, it should

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-22 Thread Shai Berger
On Tuesday 22 March 2016 11:57:03 Hugo Chargois wrote: > > I'm unsure that your relentless rant against MySQL is useful in this topic. I'm sorry if anyone sees it as a rant. It is intended as a technical argument that MySql's REPEATABLE READ is unsuitable for use as a default by Django. > > Yo

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-22 Thread Hugo Chargois
Le mardi 22 mars 2016 00:11:31 UTC+1, Shai Berger a écrit : > I disagree. The ORM cannot keep you safe against MySql's REPEATABLE READ. >> Incidentally, to prove my point, >> this has been changed in Django 1.9 and data-loss doesn't happen anymore, >> in that same default isolation level. >>

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Shai Berger
Hi, On Monday 21 March 2016 13:01:27 hcharg...@medici.tv wrote: > > I agree with Karen that documenting that some functions don't work is not > helping. Especially since this is Django's fault, not MySQL's. I mean, > sure, MySQL has some very weird isolation modes, I definitely agree with > Shai

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Aymeric Augustin
On 21 Mar 2016, at 15:02, Hugo Chargois wrote: > > But here, we have "two non-conflicting updates (no-ops, even) causes data to > be lost". I bet no one would call this safe. Yes, this one was clearly a bug (and, if I understand correctly, it was resolved by a not-directly-related change, orig

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Hugo Chargois
Le lundi 21 mars 2016 13:35:19 UTC+1, Aymeric Augustin a écrit : > > > The problem is to determine what “safe” means here :-) I’m afraid this > won’t be possible in general (at least not in a remotely efficient fashion) > because Django inevitably depends on the guarantees of the underlying > da

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Aymeric Augustin
Hello, > On 21 Mar 2016, at 12:01, hcharg...@medici.tv wrote: > > In my opinion, the fix should not be to set a different default isolation > level, as that could trigger other problems that may be very hard to find. I > was lucky to find this one. I think that Django should find sequences of

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Cristiano Coelho
Shai Berger, this explanation is pure gold! Definetely better than MySQL's one. Now I may agree on that changing the isolation level should be probably left to a major release, or either have a huge warning on the patch notes. I personally can't think of any project I have done that this would

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread hchargois
Hi, I'm the ticket OP. As a user, I definitely expect that my ORM won't eat my records when I save something, even with concurrency. Furthermore, I expect it to work with the default configuration of the backend, or if I have to change it, the ORM should refuse to work until the configuration

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Aymeric Augustin
We’ve had, and known about, such problems for so long that I’m also inclined not to make that change in already released versions. -- Aymeric. > On 21 Mar 2016, at 12:33, Anssi Kääriäinen wrote: > > This being MySQL I wouldn't be surprised if changing the isolation > level would introduce new

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Anssi Kääriäinen
This being MySQL I wouldn't be surprised if changing the isolation level would introduce new problems. Also, user code might rely on Django using repeatable read. If we introduce the change in to stable releases, we risk breaking sites that work perfectly well currently. To me this is against our b

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Shai Berger
My recommendation to backport is based on the observation that the peculiar REPEATABLE READ behavior is highly conductive to data loss in the presence of concurrency, combined with a sense that it is not very well known; I find it much more likely that the change will fix broken code than break

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Anssi Kääriäinen
I'm strongly -1 on changing the default isolation level in a minor release. We can recommend users switch the level and complain loudly if they don't. But just changing the isolation level has potential for breaking working code. - Anssi On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger wrote: > Fir

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Aymeric Augustin
On 21 Mar 2016, at 01:25, Cristiano Coelho wrote: > > What performance changes can you expect doing this change? It is probably > that default on MySQL for a good reason. Barring implementation weirdness, reducing the isolation level is supposed to improve performance (if it makes a difference

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-21 Thread Shai Berger
First of all, I would like to say that I strongly support the move to READ COMITTED, including backporting it to 1.8.x. But we also need to explain: REPEATABLE READ is a higher transaction isolation level than READ COMMITTED. If you have problematic code, it should lead to more deadlocks and/or

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-20 Thread Cristiano Coelho
What performance changes can you expect doing this change? It is probably that default on MySQL for a good reason. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop re

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-20 Thread Curtis Maloney
+1 for me, too... this aligns with "safe/secure by default". -- C On 21/03/16 09:13, Aymeric Augustin wrote: I agree with Karen. -- Aymeric. On 18 Mar 2016, at 22:15, Karen Tracey mailto:kmtra...@gmail.com>> wrote: This is the 2nd major issue I can recall caused by MySQL default of REPEATA

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-20 Thread Aymeric Augustin
I agree with Karen. -- Aymeric. > On 18 Mar 2016, at 22:15, Karen Tracey wrote: > > This is the 2nd major issue I can recall caused by MySQL default of > REPEATABLE READ transaction isolation level. I think Django should simply > switch itself to a default of using READ COMMITTED, consistent

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-19 Thread Karen Tracey
This is the 2nd major issue I can recall caused by MySQL default of REPEATABLE READ transaction isolation level. I think Django should simply switch itself to a default of using READ COMMITTED, consistent with (all?) other supported database backends, and document how, if a user really really wants

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-14 Thread Cristiano Coelho
I can't tell if it is a bug on MySQL or not, but I did understand the same as you (the first example with Session A and Session B makes it more clear) so I can't quite understand how did the poster get that issue. I would like a similar example as the one in the docs, but with a delete in the mi

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-14 Thread Shai Berger
Hi, I just commented on the ticket, but wanted to clarify a few things here: On Tuesday 15 March 2016 00:48:02 Cristiano Coelho wrote: > The django-admin interface is quite bad at handling concurrent > modifications, this is one problem that might not happen on other backends > and is quite criti

Re: MySQL data loss possibility with concurrent ManyToManyField saves

2016-03-14 Thread Cristiano Coelho
The django-admin interface is quite bad at handling concurrent modifications, this is one problem that might not happen on other backends and is quite critical, but other issues (that ain't critical like data loss but might cause unexpected errors) like two people viewing a model with some inli