Re: Configurable safety options for high performance Django systems

2014-11-23 13:27 GMT+01:00 Shai Berger : > Hi Rick, > > On Sunday 23 November 2014 14:11:13 Rick van Hattem wrote: > > > > So please, can anyone give a good argument as to why any sane person > would > > have a problem with a huge default limit which will kill the performance > of > > your site an

Re: Configurable safety options for high performance Django systems

2014-11-20 8:30 GMT+01:00 Schmitt, Christian : > Nope. a large OFFSET of N will read through N rows, regardless index >> coverage. see >> http://www.postgresql.org/docs/9.1/static/queries-limit.html >> > > That's simple not true. > If you define a Order By with a well indexes query, the database w

Re: Configurable safety options for high performance Django systems

ore complex and django > is already really complex inside the core orm. > Let's first understand what's needed, than we can decide if it has a story inside the django core > > > 2014-11-19 13:59 GMT+01:00 Marco Paolini : > >> also, the offset + limit pagination stra

Re: Configurable safety options for high performance Django systems

also, the offset + limit pagination strategy of django paginator is sub-optimal as it has N complexity: doing SELECT * FROM auth_user LIMIT 100 offset 100 causes a 10-long table scan 2014-11-19 13:56 GMT+01:00 Marco Paolini : > > > 2014-11-19 13:50 GMT+01:00 Rick v

Re: Configurable safety options for high performance Django systems

2014-11-19 13:50 GMT+01:00 Rick van Hattem : > Definitely agree on this, silently altering a query's limit is probably > not the way to go. Raising an exception in case of no limit and lots of > results could be useful. > > For the sake of keeping the discussion useful: > - Let's say you have a ta

Re: Websockets... again

What if we do it with asyncio? 2014-10-28 22:47 GMT+01:00 Aymeric Augustin < aymeric.augus...@polytechnique.org>: > No, there isn’t. > > I assume that “including in core” means at least “making usable in > combination with WSGI and with the ORM”. > > Even if we disregard for a minute the fact tha

Re: postgresql range types v2

On 18/01/2013 12:55, Anssi Kääriäinen wrote: On 18 tammi, 13:00, Florian Apolloner wrote: Hi, On Thursday, January 17, 2013 11:08:01 PM UTC+1, Russell Keith-Magee wrote: So - while I'm not sure there's a place for this in core (unless you can demonstrate how to implement range types on oth

Re: postgresql range types v2

On 17/01/2013 22:08, Russell Keith-Magee wrote: On Mon, Dec 31, 2012 at 5:56 PM, Marco Paolini mailto:markopaol...@gmail.com>> wrote: Hi all, sorry for the noise, forget my previous mail as it was pointing to the wrong commit, here's the good one: https:/

Re: postgresql range types v2

On Thursday, January 17, 2013, Simon Litchfield wrote: > Also, did you see psycopg2.extras.DateTimeRange? > > No, I missed that one ! Thanks I 'll see if it can be used somehow, but since the range types have to be used by the core django code, I doubt they can be imported from a third party libr

Re: postgresql range types v2

On Thursday, January 17, 2013, Simon Litchfield wrote: > Marco, this is great. Thanks, did you give it a try? > > I wonder if it would be possible to add range fields without modifying > django? > Very difficult, there are many small changes scattered around the core django ORM and driver code

postgresql range types v2

Hi all, sorry for the noise, forget my previous mail as it was pointing to the wrong commit, here's the good one: https://github.com/mpaolini/django/commit/b754abdeab204949510500ccb1b845b7ad143542 copying here the rest of the original mail: postgresql since version 9.2 added support for range

postgresql range types

Hi all, postgresql since version 9.2 added support for range types [1] they have a nice set of specialized operators like "overlaps", "left of", etc... [2] So I decided to work on a reference implementation for Django even if it looks like psycopg2 does not fully support yet these data types [3

Re: URL dispatcher slow?

On 11/10/2012 15:53, Tom Evans wrote: On Wed, Oct 10, 2012 at 7:52 AM, Moonlight wrote: Here is an article comparing various URL dispatchers: http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html What cause django URL dispatcher that much... slow? Now that I've looked in de

Re: queryset caching note in docs

On 04/11/2011 03:05, Anssi Kääriäinen wrote: On Nov 4, 3:38 am, Marco Paolini wrote: Postgresql: .chunked(): 26716.0kB .iterator(): 46652.0kB what if you use .chunked().iterator() ? Quick test shows that the actual memory used by the queryset is around 1.2Mb. Using smaller fetch size than

Re: queryset caching note in docs

On 04/11/2011 01:50, Anssi Kääriäinen wrote: On Nov 4, 1:20 am, Marco Paolini wrote: time to write some patches, now! Here is a proof of concept for one way to achieve chunked reads when using PyscoPG2. This lacks tests and documentation. I think the approach is sane, though. It allows

Re: queryset caching note in docs

On 04/11/2011 01:50, Anssi Kääriäinen wrote: On Nov 4, 1:20 am, Marco Paolini wrote: time to write some patches, now! Here is a proof of concept for one way to achieve chunked reads when using PyscoPG2. This lacks tests and documentation. I think the approach is sane, though. It allows

Re: queryset caching note in docs

... The SQLite3 shared cache mode seems to suffer from the same problem than mysql: """ At any one time, a single table may have any number of active read- locks or a single active write lock. To read data a table, a connection must first obtain a read-lock. To write to a table, a connection mus

Re: queryset caching note in docs

Now, calling the .iterator() directly is not safe on SQLite3. If you do updates to objects not seen by the iterator yet, you will see those changes. On MySQL, all the results are fetched into Python in one go, and the only saving is from not populating the _results_cache. I guess Oracle will just

Re: queryset caching note in docs

On 02/11/2011 17:33, Tom Evans wrote: On Wed, Nov 2, 2011 at 4:22 PM, Marco Paolini wrote: On 02/11/2011 17:12, Tom Evans wrote: If you do a database query that quickly returns a lot of rows from the database, and each row returned from the database requires long processing in django, and you

Re: queryset caching note in docs

On 02/11/2011 17:12, Tom Evans wrote: On Wed, Nov 2, 2011 at 11:28 AM, Marco Paolini wrote: mysql can do chunked row fetching from server, but only one row at a time curs = connection.cursor(CursorUseResultMixIn) curs.fetchmany(100) # fetches 100 rows, one by one Marco The downsides to

Re: queryset caching note in docs

On 02/11/2011 15:18, Anssi Kääriäinen wrote: On 11/02/2011 01:36 PM, Marco Paolini wrote: maybe we could implement something like: for obj in qs.all().chunked(100): pass .chunked() will automatically issue LIMITed SELECTs that should work with all backends I don't think that will

Re: queryset caching note in docs

On 02/11/2011 14:36, Ian Kelly wrote: On Wed, Nov 2, 2011 at 5:05 AM, Anssi Kääriäinen wrote: For PostgreSQL this would be a nice feature. Any idea what MySQL and Oracle do currently? If I'm following the thread correctly, the oracle backend already does chunked reads. The default chunk siz

Re: queryset caching note in docs

On 02/11/2011 12:05, Anssi Kääriäinen wrote: On 11/02/2011 12:47 PM, Marco Paolini wrote: if that option is true, sqlite shoud open one connection per cursor and psycopg2 should use named cursors The sqlite behavior leads to some problems with transaction management - different connections

Re: queryset caching note in docs

On 02/11/2011 12:05, Anssi Kääriäinen wrote: On 11/02/2011 12:47 PM, Marco Paolini wrote: if that option is true, sqlite shoud open one connection per cursor and psycopg2 should use named cursors The sqlite behavior leads to some problems with transaction management - different connections

Re: queryset caching note in docs

On 02/11/2011 12:05, Anssi Kääriäinen wrote: On 11/02/2011 12:47 PM, Marco Paolini wrote: if that option is true, sqlite shoud open one connection per cursor and psycopg2 should use named cursors The sqlite behavior leads to some problems with transaction management - different connections

Re: queryset caching note in docs

On 02/11/2011 10:10, Luke Plant wrote: On 02/11/11 08:48, Marco Paolini wrote: thanks for pointing that to me, do you see this as an issue to be fixed? If there is some interest, I might give it a try. Maybe it's not fixable, at least I can investigate a bit Apparently, the pro

Re: queryset caching note in docs

On 02/11/2011 09:43, Luke Plant wrote: On 02/11/11 00:41, Marco Paolini wrote: so if you do this: for obj in Entry.objects.all(): pass django does this: - creates a cursor - then calls fetchmany(100) until ALL rows are fetched - creates a list containing ALL fetched rows - passes

Re: queryset caching note in docs

On 28/10/2011 15:55, Tom Evans wrote: On Fri, Oct 28, 2011 at 1:05 PM, Marco Paolini wrote: it's a bit more complex: there are basically two phases: 1) row fetching from db using cursor.fetchmany() 2) model instance creation in queryset both are delayed as much as possible (que

Re: queryset caching note in docs

On 28/10/2011 12:58, Tom Evans wrote: On Fri, Oct 28, 2011 at 11:23 AM, Marco Paolini wrote: Hi all, I'd like to add a small note of warning about queryset caching in dos, in topics/db/queries.txt "Caching and QuerySets" section, something like: Keep in mind when looping thr

queryset caching note in docs

Hi all, I'd like to add a small note of warning about queryset caching in dos, in topics/db/queries.txt "Caching and QuerySets" section, something like: Keep in mind when looping through a queryset and altering data that might be returned by the same queryset, that instances might not be "fresh"

Re: DecimalField model validation

On 07/10/2011 06:29, Tai Lee wrote: Why is ROUND_HALF_EVEN superior? Perhaps for statistics, where rounding all halves up would skew results, but I guess not for most other cases. If the default rounding behaviour produces different results than a regular calculator, common spreadsheet and accou

Re: DecimalField model validation

On 06/10/2011 02:45, Paul McMillan wrote: .. (A) silent rounding issue: when a decimal value is saved in db its decimal digits exceeding decimal_places are rounded off using .quantize(). Rounding defaults to ROUND_HALF_EVEN (python default). There is no mention of this behavior in

DecimalField model validation

Hi, I would like to share some thoughts regarding django.db.models.DecimalField: .. (A) silent rounding issue: when a decimal value is saved in db its decimal digits exceeding decimal_places are rounded off using .quantize(). Rounding defaults to ROUND_HALF_EVEN (python default).

Re: URL-building with site-framework

On 29/09/2011 12:12, momo2k wrote: Hello, this is my first post here and I hope this is the right place for discussing ideas django-features before reporting a ticket. you don't need to open a ticket: #8995 covers this issue already I know that searching tickets is not the most thrilling activ

Re: Thoughts on solution to forward references in MySQL (#3615)

Karen Tracey ha scritto: On Tue, Jun 28, 2011 at 8:44 AM, Jim Dalton > wrote: I have not had time to try out the patch, but did look at it. Doesn't the base implementation of disable_foreign_key_checks need to return False instead of just passing? The r

Re: Small decision needed on #15025

bpeschier ha scritto: what if you add an alters_data class attribute to your ModelForm subclass? class MyForm(ModelForm): alters_data = True this way __call__ is not going to get called... No, but it will be replaced with settings.TEMPLATE_STRING_IF_INVALID. See: http://code.djan

Re: Small decision needed on #15025

bpeschier ha scritto: I am agreeing with the changes as made in [14992] as they make things more consistent. The snag I run into was a flexible filter for making ModelForm-instances which took a form-class as an argument via a Variable. Since classes are callable (their constructor), suddenly I h

Re: Small decision needed on #15025

Don Spaulding ha scritto: On Jan 12, 4:23 pm, Tai Lee wrote: On Jan 13, 5:06 am, Don Spaulding wrote: I will say that the debug template has been "relying on the bug" since 2005. By my estimation, that makes nearly every template ever created a candidate for relying on the old beh

Re: Feedback required: #14799 -- Problem with setting up test databases

Russell Keith-Magee ha scritto: On Tue, Dec 7, 2010 at 11:21 PM, mpaolini wrote: Maybe unrelated... have you had a look at #14662? It's related, but in the sense that it's the manual manifestation of what #14799 needed to correct. The contenttype and auth post_syncdb handlers ignore

call_command retval

I think call_command should return something significant to let the caller know if the command was successful or not. Another issue ralated to this is: having an error retcode when calling management commands from commandline The problem now is that for instance loaddata swallows all exception