Cursor closing or reuse: am I missing something?
My testing of remote mssql is proceeding nicely, or was until a few days ago. After much debugging it looks like my database proxy server may be running out of worker threads: it uses one for each cursor. My traces show that, when running the test suite in django-mssql (using ./manage.py test) cursors are neither re-used, nor closed. My verbose trace shows, for each new test class, a connection is created, a cursor is created, one SQL statement is run, another cursor is created, another SQL statement run, etc., until the test completes. Then the connection is closed, which (inside my adapter) triggers the closing of all of those cursors, sometimes a dozen or more. I assumed that my adapter was at fault, and have gone looking for where, in other adapters, they check for re-use of the cursor, or close it. I have not been able to find anything. The idea of re-using connections in 1.6 causes me to _really_ worry. Is this behaviour expected but only happens in testing? Surely a production server does not do this. Can some kind person point me to some back end example code where in says "close the cursor here" or "find the already open cursor"? A module name and line number (or text search reference) would be wonderful. I am starting to panic. -- Vernon Cole -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: django and paramstyle: what's the actual story?
Shai:
I think that you are showing how rotten this whole "paramstyle" mess is:
the thing you are describing is, IIUC, "pyformat" paramstyle. "named" uses
a ":name" SQL statement syntax, and expects a mapping of parameters. My
understanding was that Oracle expected that one. I am only an egg.
Here's my understanding:
'qmark' uses '?' inside SQL statements, and expects parameters in a
sequence, to be applied in order.
'format' uses '%s' inside SQL statements, and expects parameters in a
sequence, to be applied in order.
'numeric' uses ":n" (where 'n' is an integer), and expects parameters in a
sequence, to be applied by index number 'n'.
'named' uses ":name" inside SQL statements, and expects parameters in a
mapping, to be applied by name.
'pyformat' uses "%(name)s", and expects parameters in a mapping.
The intent of PEP-249 is that the ORM should look at the value of
adapter.paramstyle and use that format, whatever it happens to be.
Clearly, that is not happening. Should it be put on the To-do list for
some future version of django, at least to supporting the two expected
future winners in the parmstyle competition ('named', 'qmark')?
--
Vernon Cole
On Thursday, May 16, 2013 7:49:47 PM UTC+1, Shai Berger wrote:
>
> On Thursday 16 May 2013, VernonCole wrote:
> > I noticed in a recent post that there is an outstanding patch for Oracle
> > for support of 'named' paramstyle.
> >
>
> As the author of that patch, I should probably clarify that it is a
> "format"
> style -- %(name)s -- not a "named" -- :name -- style.
>
> AFAIK, that style is currenly supported by all core django backends (well,
> except Oracle, of course); I know our code which uses it has been
> successfully
> run against SQLite and Postgres, and I suspect django-pyodbc also supports
> it.
> It has its problems, both with respect to Python string interpolation and
> to
> SQL's pattern operator, but AFAIK it is the most common in use.
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Cursor closing or reuse: am I missing something?
Hi Vernon, On 17 mai 2013, at 13:26, VernonCole wrote: > After much debugging it looks like my database proxy server may be running > out of worker threads: it uses one for each cursor. With DB-API adapters, the connections are expected to be expensive and cursors cheap. (If this sounds obscure, I recommend re-reading PEP 249.) It appears that creating a cursor is expensive in your setup, and that's the problem. Roughly, your proxy server can use one thread per connection, but not per cursor. It's hard to give accurate advice here; overall I think that's why you're running into trouble. > My traces show that, when running the test suite in django-mssql (using > ./manage.py test) cursors are neither re-used, nor closed. Django can create a lot of cursors and that isn't expected to be a problem. Django should close these cursors and I'm not convinced it always does. If you could file a bug report with a bit more information (X leaks an unclosed cursor) that would help. > My verbose trace shows, for each new test class, a connection is created, a > cursor is created, one SQL statement is run, another cursor is created, > another SQL statement run, etc., until the test completes. Then the > connection is closed, which (inside my adapter) triggers the closing of all > of those cursors, sometimes a dozen or more. That's very, very weird. The test suite is supposed to keep a single connection opened during the entire test run — that's required to make it work with an in-memory SQLite database (ignoring multi-db tests). > The idea of re-using connections in 1.6 causes me to _really_ worry. Is this > behaviour expected but only happens in testing? Fortunately I don't think persistent connections are related to the problems you're seeing. They're about keeping connections, while your problems seem to revolve around cursors. -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Tool for Google Summer of Code: Flower Dev Center
Hello GSoC Mentors, As you are participating to Google Summer of Code, I'm wondering if you could consider using Flower Dev Center [1] while working with students. Flower Dev Center is an online platform for UML modeling diagramming, with a strong focus on code synchronization, integration with dev tools (Git, SVN, etc) and real time collaboration on diagrams (and a little bit on code as well). We think Flower Dev Center can be helpful for both: mentors and students during Google Summer of Code. We have created an article on this topic (i.e. Flower Dev Center + GSoC): [2], [3]. If you have a couple of spare minutes, could you tell us if you would like to use Flower Dev Center? And/or raise topics that you think are important (based on previous GSoC participations) to be supported by Flower Dev Center? REMARK1: Right now, Flower Dev Center is closed source and offered for free to open-source projects. Starting with next version we'll begin to open its API and source code, so that anyone could write extension plugins, etc. REMARK2: The next version of Flower Dev Center, the 2.0.0 planned for June/July 2013, has major new features, that we did not demonstrate yet and that will improve even more the collaboration between developers. REMARK3: Flower Dev Center doesn't currently support Python programming language. But if you are interested in using Flower Dev Center, we'll prioritize it's implementation and we'll work hard to do it ASAP, with Flower Dev Center 2.0.0. REMARK4: Flower Dev Center will support programming languages that are not object oriented, starting with 2.0.0 Thank you in advance! Best regards, Mircea @ The Flower Platform Team. [1] - Flower Dev Center web site - http://www.flower-platform.com [2] - Video “Flower Dev Center + Google Summer of Code” - http://learn-discuss.flower-platform.com/flower_dev_center/videos/flower_dev_center_and_google_summer_of_code_2013_the_video [3] - Text version with same content as the above video - http://learn-discuss.flower-platform.com/flower_dev_center/tutorials/flower_dev_center_and_google_summer_of_code_2013 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Tool for Google Summer of Code: Flower Dev Center
django-developers has over 7500 subscribers. Your message is relevant to at most 4 of them. Unless it was plain spam. In either case, that's not cool, avoid that in the future. -- Aymeric. On 16 mai 2013, at 11:51, Flower Platform Team wrote: > Hello GSoC Mentors, > > As you are participating to Google Summer of Code, I'm wondering if you could > consider using Flower Dev Center [1] while working with students. > > Flower Dev Center is an online platform for UML modeling diagramming, with a > strong focus on code synchronization, integration with dev tools (Git, SVN, > etc) and real time collaboration on diagrams (and a little bit on code as > well). > > We think Flower Dev Center can be helpful for both: mentors and students > during Google Summer of Code. We have created an article on this topic (i.e. > Flower Dev Center + GSoC): [2], [3]. > > If you have a couple of spare minutes, could you tell us if you would like to > use Flower Dev Center? And/or raise topics that you think are important > (based on previous GSoC participations) to be supported by Flower Dev Center? > > REMARK1: Right now, Flower Dev Center is closed source and offered for free > to open-source projects. Starting with next version we'll begin to open its > API and source code, so that anyone could write extension plugins, etc. > > REMARK2: The next version of Flower Dev Center, the 2.0.0 planned for > June/July 2013, has major new features, that we did not demonstrate yet and > that will improve even more the collaboration between developers. > > REMARK3: Flower Dev Center doesn't currently support Python programming > language. But if you are interested in using Flower Dev Center, we'll > prioritize it's implementation and we'll work hard to do it ASAP, with Flower > Dev Center 2.0.0. > > REMARK4: Flower Dev Center will support programming languages that are not > object oriented, starting with 2.0.0 > > Thank you in advance! > > Best regards, > Mircea @ The Flower Platform Team. > > [1] - Flower Dev Center web site - http://www.flower-platform.com > [2] - Video “Flower Dev Center + Google Summer of Code” - > http://learn-discuss.flower-platform.com/flower_dev_center/videos/flower_dev_center_and_google_summer_of_code_2013_the_video > [3] - Text version with same content as the above video - > http://learn-discuss.flower-platform.com/flower_dev_center/tutorials/flower_dev_center_and_google_summer_of_code_2013 > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cursor closing or reuse: am I missing something?
Right now Django absolutely does not close all cursors, if you look at django/db/models/sql/compiler.py def execute_sql (sorry, going from memory here) you'll see this. It'd be good if it did. There used to be some crazy stuff around reading chunks and emitting those, now that I believe Anssi has removed those it should be more straightforward to close cursors there. Alex On Fri, May 17, 2013 at 5:23 AM, Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > Hi Vernon, > > On 17 mai 2013, at 13:26, VernonCole wrote: > > > After much debugging it looks like my database proxy server may be > running out of worker threads: it uses one for each cursor. > > > With DB-API adapters, the connections are expected to be expensive and > cursors cheap. (If this sounds obscure, I recommend re-reading PEP 249.) > > It appears that creating a cursor is expensive in your setup, and that's > the problem. Roughly, your proxy server can use one thread per connection, > but not per cursor. > > It's hard to give accurate advice here; overall I think that's why you're > running into trouble. > > > My traces show that, when running the test suite in django-mssql (using > ./manage.py test) cursors are neither re-used, nor closed. > > > Django can create a lot of cursors and that isn't expected to be a > problem. Django should close these cursors and I'm not convinced it always > does. If you could file a bug report with a bit more information (X leaks > an unclosed cursor) that would help. > > > My verbose trace shows, for each new test class, a connection is > created, a cursor is created, one SQL statement is run, another cursor is > created, another SQL statement run, etc., until the test completes. Then > the connection is closed, which (inside my adapter) triggers the closing of > all of those cursors, sometimes a dozen or more. > > That's very, very weird. The test suite is supposed to keep a single > connection opened during the entire test run — that's required to make it > work with an in-memory SQLite database (ignoring multi-db tests). > > > The idea of re-using connections in 1.6 causes me to _really_ worry. Is > this behaviour expected but only happens in testing? > > Fortunately I don't think persistent connections are related to the > problems you're seeing. They're about keeping connections, while your > problems seem to revolve around cursors. > > -- > Aymeric. > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers?hl=en > . > For more options, visit https://groups.google.com/groups/opt_out. > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Cursor closing or reuse: am I missing something?
On 17 touko, 16:47, Alex Gaynor wrote: > Right now Django absolutely does not close all cursors, if you look at > django/db/models/sql/compiler.py def execute_sql (sorry, going from memory > here) you'll see this. It'd be good if it did. There used to be some crazy > stuff around reading chunks and emitting those, now that I believe Anssi > has removed those it should be more straightforward to close cursors there. Minor correction: I haven't yet. There is a ready patch but I haven't committed anything as I have been distracted by other things lately. Luckily this spring's distractions are over tomorrow morning... - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Proposal: Redefine specific {% block %} in an intermediate template
Hi Emil, On May 16, 2013, at 5:21 PM, Emil Stenström wrote: > Any feedback on how I was thinking? Does it make sense? > > Based on the feedback so far I gather that changing the block tag is a bad > idea. I'd love to continue working on this, because I've felt this need in > lots of different projects. A new proposal could be in terms of a new pair of > tags, maybe "layout" and "content". But before going further I would love > some thoughts on the area, and if you think it's a worthwhile problem to > solve. My feeling is that the existing tools are adequate, using wrapper blocks as I outlined earlier in the thread, so I don't see this as a problem that needs solving in the core template language. If you're thinking of using new tag names anyway, you could certainly experiment with options in a third-party project and bring it back for consideration in the future if it's demonstrated usefulness and wide use. Carl -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
