Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)
On 29 syys, 07:45, Tai Lee wrote: > Docs have been added (patch should be complete now) and pull request has > been opened, at Anssi's request. > > https://groups.google.com/d/topic/django-developers/RrNPfuJxnlM/discu... To me it seems the use of streaming responses in static file serving will cause problems for upgraders. If using 3rd party middleware accessing .content, then the following will happen: - The static files will now automatically use streaming responses - The middleware will be applied to the static file responses, too - There is no .content for streaming responses -> problems The problem stems from using the new streaming response by default in Django. If it wasn't used, then there would be no problems for upgrades. I think we have to put in place something like BackwardsCompatStreamingResponse - on access of .content raise a deprecation warning, but still allow .content to work. This could be used for two releases in static file serving. Alternate approach is to tell upgraders to wrap non-working middleware in a subclass skipping the middleware for streaming responses. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
Hi, Sorry about the delay in writing this -- it was in my mind, but I forgot to let it out... On Sunday 23 September 2012, Anssi Kääriäinen wrote: > On 22 syys, 23:34, Shai Berger wrote: > > > > [...] I will also include the extra setting in > > the "options" dictionary (choosing if expressions should be treated as > > decimals or floats). I think this is then an added feature, and needs to > > get in before the feature freeze. Otherwise, just fixing the performance > > can be treated as a bug in be included later. Is this guess correct? > > There isn't much time to do feature additions. It is possible to get > the feature into 1.5, but you should not expect it to happen. As for > the feature itself, I am not yet sure if it is a good idea or not. The > basic problem is granularity: wouldn't you want to do this per query, > or even per expression basis? > No, I don't; the issue is cross-database compatibility. We have raw SQL statements calculating some expressions, and then use the returned values in calculations in Python -- adding and muliplying them with floats. This worked fine with MySql and the 3rd-party SQL Server backends, but blew up on Oracle because Decimals and floats don't mix very well: >>> from decimal import Decimal as D >>> D(1) + 1.0 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'Decimal' and 'float' >>> 1.0 + D(1) Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'float' and 'Decimal' I could, of course, solve it on a query-by-query basis, adding a cast to float that is a nop on other backends. That seems a little backward to me, and an unfair requirements of pluggable apps. Similarly, any per-query or per-expression API would be added only when using Oracle, and at best be a NO-OP on other engines -- which means it would only be added by Oracle users, with the same consequences. It seems much better to me to make it possible to run Oracle in a mode that makes expressions in raw sql have semantics that is compatible with other backends. (would filing a bug with the above arguments make it more likely to include the feature in 1.5?) Thanks, Shai. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
On 29 syys, 21:51, Shai Berger wrote:
> On Sunday 23 September 2012, Anssi Kääriäinen wrote:> On 22 syys, 23:34, Shai
> Berger wrote:
> > There isn't much time to do feature additions. It is possible to get
> > the feature into 1.5, but you should not expect it to happen. As for
> > the feature itself, I am not yet sure if it is a good idea or not. The
> > basic problem is granularity: wouldn't you want to do this per query,
> > or even per expression basis?
>
> No, I don't; the issue is cross-database compatibility. We have raw SQL
> statements calculating some expressions, and then use the returned values in
> calculations in Python -- adding and muliplying them with floats. This worked
> fine with MySql and the 3rd-party SQL Server backends, but blew up on Oracle
> because Decimals and floats don't mix very well:
The problem seems to be that on Oracle, 1.1 is NUMERIC, and that is
something which _is_ a decimal in Python. We should not change that.
To me it seems the answer is to use to_binary_float/to_binary_double
in raw SQL queries. For example:
cur = connection.cursor()
cur.execute("select count(*) from tests_place")
print cur.fetchone()[0]
vals = None
def do_query():
global vals
cur.execute("select 1.1 from tests_place")
vals = [val[0] for val in cur.fetchall()]
t = Timer('do_query()', 'from __main__ import do_query')
print t.timeit(number=20)
print type(vals[0])
OUT:
2000
0.719314813614
Replace the query with "select to_binary_double(1.1) from
tests_place":
OUT:
2000
0.337587118149
Based on the above I am -1 on having a flag that interprets NUMBER
fields as floats. There is already an Oracle data type to use for
this.
I tested the pull request 393 (the speedup to ._rowfactory). Above
test case, and the decimal case runs in around one second, the float
case in 0.1 seconds. So, there is a regressions for decimals. On the
other hand floats are way faster. Will investigate...
As for new features for 1.5: there is roughly a day before feature
freeze. Getting any new ticket into 1.5 is really unlikely at this
point, existing pull requests have priority.
- Anssi
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Need help with writing a Pontoon hook for Django apps
Hi all, I have lately worked on a hook to enable a Django website to use Mozilla's Pontoon service. https://github.com/rtnpro/django-pontoon-hook What this app tries to achieve is to wrap marked up strings anywhere in a Django project's source code (templates, python files: forms, models, views), with some special character sequence, say, ''. This will help Pontoon to identify the marked up strings and mark them as editable. I tried to monkey patch 'django.utils.translation._trans' at https://github.com/rtnpro/django-pontoon-hook/blob/master/pontoon_hook/middleware.py#L88 so that all calls to the needed gettext methods will wrap the source string in the above mentioned way. However, this approach is not at all clean and reliable. I was discussing about this with Jacob (@jacobian) at Pycon India, 2012, and we found that it'd be cool if Django allowed implementing custom i18n backends, just like it allows implementing custom test runners. Could you please help in implementing django-pontoon-hook in a better way? Regards, Ratnadeep -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/Qz77ucuYwAoJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
Hi, and thanks for your attention. I know you must be extremely busy with other issues of the feature freeze. I just want to make some things clear: > Based on the above I am -1 on having a flag that interprets NUMBER > fields as floats. There is already an Oracle data type to use for > this. > The flag does not affect the interpretation of NUMBER columns with scale and precision. True, in its current form, it turns bare NUMBER columns (decimal- precision floating point) to floats. But these are columns that are not generated by Django's built-in fields; if you have them, you are doing something Oracle-specific. The flag is mostly intended to improve cross-database compatibility; other than that, it offers a very clear trade-off and it is off by default. It really shouldn't hit anybody by surprise. > > > I am not yet sure if it is a good idea or not. The > > > basic problem is granularity: wouldn't you want to do this per query, > > > or even per expression basis? > > > > No, I don't; the issue is cross-database compatibility. > > > The problem seems to be that on Oracle, 1.1 is NUMERIC, and that is > something which _is_ a decimal in Python. We should not change that. > > To me it seems the answer is to use to_binary_float/to_binary_double > in raw SQL queries. For example: > But this is an Oracle-specific solution. Exactly the thing I'm trying to avoid. > > I tested the pull request 393 (the speedup to ._rowfactory). Above > test case, and the decimal case runs in around one second, the float > case in 0.1 seconds. So, there is a regressions for decimals. On the > other hand floats are way faster. Will investigate... > I confirm your findings: Running with different parameters (10,000 rows) I found Decimals are ~1.8x slower, floats are ~3 times faster, and significantly, ints are ~2 times faster. I suspect in the typical Django application, most numbers are ints. I investigated a little myself, and found the reason to be the redundant lookup in _decimal_or_int(): It was referring to Decimal as "decimal.Decimal". I moved that to a default argument, and got performance to be ~40% better than trunk for decimals too. The pull request is already updated. > As for new features for 1.5: there is roughly a day before feature > freeze. Getting any new ticket into 1.5 is really unlikely at this > point, existing pull requests have priority. > FWIW, the pull request exists (402). Thanks for your help, Shai. Shai. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
On 29 syys, 23:25, Anssi Kääriäinen wrote:
> I tested the pull request 393 (the speedup to ._rowfactory). Above
> test case, and the decimal case runs in around one second, the float
> case in 0.1 seconds. So, there is a regressions for decimals. On the
> other hand floats are way faster. Will investigate...
So, while investigating this I found out the reason for the
performance regression is calling _decimal_or_int for each row. As far
as I can tell there is no alternative to doing a call for each row.
The next idea was to cache the type of the column so that conversions
would be faster. This leads to an interesting test case:
cur.execute("select case when dbms_random.random > 0.5 then 0.1
else 0 end from testtable")
Guess what is the type of the column? It is both Decimal and int,
alternating based on the random result.
This seems like surprising behaviour to me. The problem is in
cx_Oracle, not Django code. In raw cx_Oracle queries you get ints and
floats mixed. I don't believe it is OK to return different types for
the same column in one resultset. I will report to cx_Oracle mailing
list and see if they have anything to say about this. DBAPI2 doesn't
offer any clear answer to this, all numeric types are put under same
type, NUMBER, no matter if they are ints, floats or Decimals.
To me it seems we can't do anything to this in Django. We simply don't
have enough information available in the outputtypehandler to guess
the type correctly.
- Anssi
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
On 30 syys, 01:24, Shai Berger wrote: > > The problem seems to be that on Oracle, 1.1 is NUMERIC, and that is > > something which _is_ a decimal in Python. We should not change that. > > > To me it seems the answer is to use to_binary_float/to_binary_double > > in raw SQL queries. For example: > > But this is an Oracle-specific solution. Exactly the thing I'm trying to > avoid. I just don't like a global flag. You could easily then have some code that requires numbers as decimals, and some other code that requires numbers as floats. A global flag is a no-go in this situation. > > I tested the pull request 393 (the speedup to ._rowfactory). Above > > test case, and the decimal case runs in around one second, the float > > case in 0.1 seconds. So, there is a regressions for decimals. On the > > other hand floats are way faster. Will investigate... > > I confirm your findings: Running with different parameters (10,000 rows) I > found > Decimals are ~1.8x slower, floats are ~3 times faster, and significantly, ints > are ~2 times faster. I suspect in the typical Django application, most numbers > are ints. > > I investigated a little myself, and found the reason to be the redundant > lookup in _decimal_or_int(): It was referring to Decimal as "decimal.Decimal". > I moved that to a default argument, and got performance to be ~40% better than > trunk for decimals too. The pull request is already updated. This looks very good. > > As for new features for 1.5: there is roughly a day before feature > > freeze. Getting any new ticket into 1.5 is really unlikely at this > > point, existing pull requests have priority. > > FWIW, the pull request exists (402). I walked into this one... But, sorry, I will not have time for this pull request for 1.5. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
On Sunday 30 September 2012, Anssi Kääriäinen wrote: > > I walked into this one... But, sorry, I will not have time for this > pull request for 1.5. > Ok, I can't say you didn't warn me this would be the case. Thanks a lot for the time you did spend on it, Shai. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Oracle backend vs. numbers
On Sunday 30 September 2012, Anssi Kääriäinen wrote:
>
> cur.execute("select case when dbms_random.random > 0.5 then 0.1
> else 0 end from testtable")
>
> Guess what is the type of the column? It is both Decimal and int,
> alternating based on the random result.
>
> This seems like surprising behaviour to me. The problem is in
> cx_Oracle, not Django code. In raw cx_Oracle queries you get ints and
> floats mixed.
I tried this query in sqlplus, and the response looks like mixed ints and
floats there too. I don't know a way to get a value's datatype inside Oracle,
so it's hard to be sure, but it could be that cx_Oracle is only reflecting
Oracle with this.
> I don't believe it is OK to return different types for
> the same column in one resultset. I will report to cx_Oracle mailing
> list and see if they have anything to say about this.
I'm curious as well.
Shai.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
