Problems with DatabaseCreation, table names, and index names

[Andrew Godwin]

Hi all,

I'm posting this here, not as a ticket, as I'm not entirely sure it's 
Django's problem, and so would like some input*.


[Base]DatabaseCreation has a method sql_indexes_for_field, which, 
handily, returns the SQL indexes for a field. Less usefully, both 
PostgreSQL and MySQL have limits on the length of table/index names (63 
and 50 bytes by default, respectively).


Firstly, this means that if you have a very long model (table) name 
(say, 64 characters), Django's syncdb thinks it's not in the database 
(since your database has truncated the table name, and they now don't 
match), and tries to add it every time you run syncdb. This is an issue 
to itself, and should probably have either a more useful error or be 
fixed, although it's not easy to detect this has happened.



Secondly, if you have an indexed field on that same model, or two 
indexed fields with common prefixes on a model with a slightly shorter 
name, the method will happily spit out SQL with index names that are 
identical to the table/other index name when truncated to 63 characters, 
meaning the indices aren't added.


The reason I came across this is because a South user has filed a 
similar bug against South that I traced back to this behaviour (South 
uses sql_indexes_for_field, as it means special GeoDjango indexes get 
created as well). I have, however, replicated the bug with plain syncdb, 
using the following Django model:


class 
VeryLongModelNameThatHappensToBeOverSixtyThreeCharsLongEventually(models.Model):


my_slightly_less_long_field_name = models.IntegerField(db_index=True)
my_slightly_more_long_field_name = models.IntegerField(db_index=True)

The question is, is it even worth fixing? I'm tempted to conclude that 
you're limited to shorter model/field names by your database (or use 
db_table/db_column), but there's also the possibility of that method 
using a much shorter way of generating the names (i.e. using the first 
40 bytes or so, then appending an 8-byte hash of the rest). I can easily 
write and submit a patch to change the function to use this behaviour - 
we used to do it in South, before I started using the ``creation`` 
methods - but it's catering to such a small audience I'm not sure it's 
worth it, and documenting this behaviour might be a better solution.


Andrew

* Providing it doesn't significantly impact fixing of actual Django 
bugs, that affect more than 0.01% of the users.


--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Problems with DatabaseCreation, table names, and index names

[Andrew Godwin]

On 11/03/10 01:05, Russell Keith-Magee wrote:

We have an incentive to fix it too -- #12977 points out that Django's
own test suite runs foul of this problem.
   


Ah, that's not good.


Just documenting the problem isn't really a solution, IMHO. MySQL is
the fly in the ointment here, because it doesn't raise errors for a
long index name: it raises a warning and truncates the name.


From my experiments, PostgreSQL is even worse; it will silently 
truncate the name without even raising a warning. You only know it's 
happened when you get to "relation already exists".



  If you
have two similarly named long fields... hilarity ensues. At the very
least, we should be raising validation errors. Preferably, this would
be at a consistent length (rather than a backend dependent length) so
that you don't accidentally write a model that is fine under Postgres,
but can't be synchronized under MySQL.

However, I'm not sure whether truncation+hashing would improve things.
Hashing has the advantage of allowing any name you want, but it does
result in some less-than-pretty database names. It also means
introducing a change in naming policy, and putting on my "hey, I used
to run a migration project too" hat -- changes in naming policy in the
base project were really annoying to track and handle.
   


I don't particularly care about naming policy; South looks at the 
database's meta info to work out what constraints there are on any given 
column, it doesn't try and work out possible names, and anyone that does 
otherwise should really steal my code!



If we do go with the validation approach, I would make one additional
suggestion. You can manually specify database table names, but you
can't (currently) manually specify index names. Given that long index
names are the bigger problem (since they are composed from database
table and column names), we might also need to introduce a syntax to
allow manual index naming. For example, we could allow
db_index='foobar' to manually specify an index name; db_index=True
would continue to use the default generated name.
   


That sounds like a good idea to me; the amount of people this affects is 
so small, adding extra parameters probably won't be annoying for most 
people.



As for timing: I don't think this is critical enough for 1.2, but it's
something we should do and probably could go immediately on the 1.3
milestone.
 


I agree as well; I'll be happy to write the patch when we get around to 
1.3 coding time.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: Data importation class

[Andrew Godwin]

I feel the need to wade in here, since this is vaguely my area.

On 25/03/10 17:47, subs...@gmail.com wrote:

The last bit sounds a bit nebulous. You could optimise it by not
including any empty files, or be a bit more specific about what the
empty files are meant to represent. :)
 

startapp, startproject, et al.
   


I see where you're coming from here; in the final proposal, though, 
you'd want to follow Malcom's advice and have an actual use case for 
each file. I'd say you probably only want one, or even none - if your 
approach is so complicated that even simple use cases need lots of 
files, you're unlikely to get much traction.



South has a target use case of relatively simple changes to schema,
and assisting teams maintain synchronicity. However, it isn't long
before you're really pushing the limits of South. Take for example one
legacy model which needs to be split into two or three current models:
South has no answer for this as you may only use defaults for creation
of fields (in this case, foreign keys, or potentially OneToOnes, and
then what? What if my defaults are based on other values in that
record? Already, I'm on my own). By and large, South is an immaculate
tool for tracking changes during development.
   


South does have an answer to this - you create the columns as nullable, 
add in the data, and then alter them back to non-nullable. That's the 
only way a database is going to let you add a column; they need either a 
global default or NOT NULL (there are some cases you can do it, but 
they're really not appropriate for most people).


In fact, I'd say this fits perfectly into the South model; one migration 
to make the two/three new tables, one that moves all the data around 
using the ORM (something Django developers know, and mostly love), and 
one to delete the old table. If you only use --auto then yes, it's only 
good at tracking small changes, but the rest of the power is right 
there, you just have to actually write code.




In theory, yes. But in practice, I've found the shortest way around
the mountain is to get it into "SQL-enough" format manually. As for
Python, sure, but the more you write this monolithic script the more
you realize you're conducting a lot of repetitive work, the mechanics
of which are generally re-usable but an implementation which is
completely nuanced to your current task. If data importation is
something you do a lot, you've probably got a file somewhere holding
piecemeal bits that are hopefully vaguely useful to the next project,
all the while not being able to fight off the feeling that this
general task mirrors a lot of what goes on in forms with
clean_somefield() and clean().

   


Actually, most of my data import scripts are while loops over 
cursor.fetchall(), which just use the ORM to put in new data. With 
MultiDB, I probably won't even need the cursor part, I can just loop 
over the legacy model and insert into the new one.


While it might be nice to make this more generic, each of those while 
loops has slightly different bodies - correcting coordinates here, 
fixing postcodes there - and the generic bits only take up one or two 
lines each time.


I'd really like to see a more compact or succinct way of doing this, but 
I'm dubious as to how flexible it would be. I'm more than happy to be 
proven wrong, however.



I'm puzzled by this conclusion. The 'system administration
functionality' isn't in any way different to what you'd find in all
kinds of projects--South included. I'm not even sure what to say about
the 'everything else in your computing life' statement, but I will
assume good faith and that you're not alleging I'm presenting this as
some kind of crutch for more correct methods. As for detecting 'what
is thrashing', there are only so many tasks that can be conducted in
the business of moving large gobs of data, some of these tasks (often)
bring CPU to 100% and (hopefully never) bring free memory to %0.
Things that do one or the other are treated like bugs or avoided.
   


I read your initial proposal here as "code things in a sensible way", 
not "actively monitor performance and correct on the fly". Using 
pagination and making sure there's no memory leaks in the code's loops 
is a great idea, attempting to self-optimise at runtime probably isn't.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Opinions sought on m2m signal ordering

[Andrew Godwin]

On 27/03/10 17:08, Russell Keith-Magee wrote:

There are 5 options I can see.

Option 1: Do nothing. #13087 describes a use case we don't want to
support, so we ignore it.
   


I think it should be supported; it seems like a reasonable suggestion, 
and I can see reasons for implementing things that maintain M2M 
relationships in this way.



Option 2: We add a "cleared" signal that occurs after the clear
actually occurs. This solves the use case for #13087, but only adds 1
signal.

Option 3: We modify the existing signals so we have a pre-post pair
for every signal. This maintains the analog with pre/post save, and
gives the most control. For example, on Alex Gaynor has suggested to
me that some people might want to use a pre-add signal rather than a
post-add signal for cache invalidation since there is a marginally
lower chance of getting a race condition. However, signals aren't free
-- an unattached signal is roughly equivalent to the overhead of a
function call.

Option 4: (1), but also move add and remove to be *pre* signals, to
alleviate Alex's concern from (3)

Option 5: (2), but also move add and remove to be *pre* signals, to
alleviate Alex's concern from (3)
   


If the overhead isn't too great I'd personally prefer (3), as it's the 
cleanest solution, and the one that seems most obvious (consistency 
makes me feel all warm and fuzzy inside). However, since clear() isn't 
the same as the other two, (2) also seems reasonable enough to me.


The only use-cases I can think of where you'd have to use a *pre* signal 
for add or remove would be to stop them happening (i.e. enforce 
constraints at the model level), and this seems like the Wrong Way to do 
things. On the contrary, there are more reasons to have them there 
*post* action, as you can then safely modify the relationship again/do 
some raw database queries on the data/use the relationship to do 
traversal or iteration over the contents (although one might argue 
that's also a case for having a pre-remove).


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Opinions sought on m2m signal ordering

[Andrew Godwin]

On 28/03/10 00:01, Russell Keith-Magee wrote:

Cache invalidation is a reasonably compelling case for pre-signals; if
you invalidate a cache on the post-signal, there is a small window
between having modified the m2m and the cache being flushed. In that
window, any operation hitting the cache will see the m2m relations
still existing, but any operation that actually hits the database will
disagree. It's a really small edge case, but that's the class of
Heisenbug that is a pain to find in the wild.
   


Surely there's a Heisenbug for the opposite case, though; you invalidate 
the cache, someone else hits a page, the M2M gets re-cached in its 
current (still original) state, and then finally the M2M relation change 
goes through. You now have an incorrect cache that's not going to be 
invalidated any time soon.



However, as you note, any raw SQL activity will probably need to
happen in the post signal, so that's probably enough of a reason to
include pre and post signals.
   


I'm strongly *for* including pre- and post- for all three; it just makes 
sense to include these as options, considering they are elsewhere, and 
I'm not convinced that the hit of an empty signal is that significant 
(there's a function call, an assignment, and an if statement); as a 
percentage of runtime, it seems very small (someone feel free to prove 
me wrong, though!)


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Proposal: Schema migration/evolution backend

[Andrew Godwin]

Hi all,

As perhaps was inevitable, I'm proposing implementing part of a schema 
migration backend in Django. I'm not sure if this is a 1.3 thing (it may 
well be 1.4, perhaps with some implemented in time for 1.3 but not 
exposed), but it's something I'd like to get started in this release 
cycle. To make it clear, I'm happy to make all these changes - I'm alre


Firstly, let me make it clear that this is not a proposal to merge South 
into core. Quite the opposite, in fact, the idea is to keep the option 
open to have different migration frontends available and in fact make 
the implementation of them much easier.


Secondly, this particular proposal is something that me and Russ have 
preliminarily agreed on here at the sprints - however, I'd really like 
people to suggest changes and things we may have missed. Implementing 
this essentially consists of drawing a line of how much of a migrations 
framework we'll implement in Django, and I'm only mostly sure we have it 
in the right place.


The first part of the proposal is pretty uncontroversial, and it's to 
implement schema-changing operations on the backends. Specifically, the 
proposed new operations are:


 - add_table
 - delete_table
 - rename_table
 - add_column
 - rename_column
 - alter_column
 - delete_column
 - add_primary_key
 - delete_primary_key
 - add_unique
 - delete_unique
 - add_index
 - delete_index

Some of these operations are already mostly implemented (add_table, 
add_index, etc.) in backends' creation modules, but they'll need a bit 
of rearranging and separating into a full public API. I also plan to 
modify them to take model names and field names, instead of table names 
and column names, so the API is exclusively using the Django model layer 
to represent changes (there's a possibility that some changes make sense 
for schemaless databases as well, specifically renames, so it's best not 
to tie it directly to relational databases).


(Additionally, this means that if someone has specified the table name 
or column name directly using something like Meta: db_table then we'll 
need to have those as either extra arguments to the function or as 
marked strings - e.g mark_raw('auth_users'))


I expect this will take a while and be quite fiddly, but we have the 
codebases of django-evolution and South to draw on for the modification 
code, so there's not much new discovery and backend-specific bugfixing 
to be done.


Additionally, unlike in current South, backends will always be able to 
generate SQL for operations, but it won't necessarily be runnable 
(things like index names can only be resolved at runtime, so you'd get 
code like "DROP INDEX <> ON users;". We feel this 
is a pretty good tradeoff between being able to actually work with 
things like index names (they're basically nondeterministic) while also 
satisfying people who like to read the SQL that's going to be run.


The second part is to implement migration tracking, dependency resolving 
and basic running into Django. There will be a core contract migrations 
have to follow:


 - Migrations are per-application
 - Applications have a directory (usually appname/migrations/, but 
configurable via a setting like SOUTH_MIGRATION_MODULES does now), which 
contains zero or more .sql and .py files
 - Inside an application, migrations are implicitly ordered by name (by 
string sort, so "0001_initial" is before "0002_second", and "alpha" is 
before "beta", but "11_foo" is not before "2_bar").
 - Migrations are uniquely identified by the combination of their app 
label and their migration name.
 - There will be a table, probably "migration_history" or similar, 
which records which migrations have been applied, and when.
 - Django will ship with a "migrate" command, which will work out what 
migrations to run, and run them. There will be an automatic mode which 
runs dependencies, and a manual mode where you say if you'd like to run 
each migration (and ones that are missing dependencies it tells you 
about, but you're not allowed to run).


As for the migration files themselves, the idea is to provide a very 
basic interface that means that apps (and Django itself, potentially) 
can ship migrations that have no dependencies, but that still allows 
third-party tools like South to exist that will provide ORM access and 
autogeneration.


.py migration files will be a normal Python module, and should have a 
"migrate" callable, which will get called with three arguments (a 
connection/operations instance, much like 'south.db.db', reverse, a 
boolean saying if the migration should run backwards, which will be 
entirely optional and some migrations will just raise an error, and 
dry_run, which indicates if the migration should just run through and 
check there's no obvious calling errors, which is useful for catching 
errors on MySQL before the SQL gets sent to the database).


The files can also optionally have a __depends__ variable in scope, 
which should be an iterable of (app

Re: Proposal: Schema migration/evolution backend

[Andrew Godwin]

On 28/05/10 13:06, Andrew Godwin wrote:

Hi all,

As perhaps was inevitable, I'm proposing implementing part of a schema 
migration backend in Django. I'm not sure if this is a 1.3 thing (it 
may well be 1.4, perhaps with some implemented in time for 1.3 but not 
exposed), but it's something I'd like to get started in this release 
cycle. To make it clear, I'm happy to make all these changes - I'm alre


For the record, this was supposed to end "I'm already pretty familiar 
with both the model layer and migrations, and I feel like using up all 
my free time."


I'll hit send less quickly next time.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Decision required: PostgreSQL minimum versions

[Andrew Godwin]

On 09/06/2010 12:59, Russell Keith-Magee wrote:

Hi all,

While we support PostgreSQL, our documentation doesn't actually
specify a minimum supported version. We have a couple of features that
are no-ops for versions prior to 8.2 (savepoints and database
autocommit), but we don't actually document a minimum required
version.

We have a specified minimum of SQLite 3.3.6 and Oracle 9i (10g for
certain features); we have a vague suggestion that MySQL 4.1 is the
minimum supported version (but we don't rule out support for 3.23 and
4.0); but what is our policy with PostgreSQL? Do we support PostgreSQL
7.4?

PostgreSQL 7.4 was released in November 2005, and will be end-of-lifed
(along with PostgreSQL 8.0) in July this year. Our usual yardstick of
slow updates, RHEL4, shipped with PostgreSQL 7.4. RHEL5 shipped with
PostgreSQL 8.1.

Why am I asking? Because of r13328. In that changeset, I added a fix
for #8901 using a database function that is available in PostgreSQL
8.0, but not in PostgreSQL 7.4.

So - I'm looking for community opinion on how to deal with this. I can
see (at least) three options:

  1) Rollback the changeset, and find a PostgreSQL 7.4-compatible way
of solving the problem. Continue to support PostgreSQL 7.4, and
formally document this fact.

  2) Add documentation for 1.3 that imposes a PostgreSQL 8.0 minimum;
rollback r13328, wait until the 1.3 branch is forked, and reapply to
that branch. In other words treat #8901 as a feature, rather than a
bugfix, and introduce the Python 8.0 minimum as a new restriction for
1.3, much in the same way that we dropped support for Python 2.3 in
Django 1.2.

  3) Retroactively modify the documentation saying Django 1.2 required
PostgreSQL 8.0 as a minimum. This treats the absence of a documented
minimum required version as a bug, and addresses the bug by picking a
minimum supported version that. r13328 stays as is.

Our general policy for deprecating old dependencies is something we
need to have a bigger discussion about -- especially as it relates to
Python itself -- but this PostgreSQL issue is immediate and pressing.
I certainly hope that any reasoning behind the decision we make for
PostgreSQL could be generalized to answer the same question for any
other dependency.

Opinions?

Yours,
Russ Magee %-)
   


So, my opnion on this is that we should drop 7.4 support, but I'm not 
sure if it should be done in 1.3 or 1.2.


Firstly, from what I can find 
(http://www.postgresql.org/docs/7.4/static/release-7-4.html), PostgreSQL 
7.4 was released in 2003, not 2005, which puts it in the same 
depreciation area as Python 2.3 (also 2003). RHEL4 itself was released 
in 2005.


Given than 1.2 has dropped support for Python 2.3, and thus already 
won't work on RHEL4, I see no real reason we should keep supporting 7.4, 
especially if it's being EOL'd soon.


As for whether to apply the fix to 1.2 or 1.3, 1.3 seems a safer option, 
but if there's no outcry from 7.4 users I'd be tempted to document 1.2 
as being PostgreSQL 8.0 and up (after all, best to bundle one 
RHEL4-breaking change with another). Given that it's a database, though, 
people might want to keep running their old RHEL4 database boxes and 
only upgrade their frontend servers, so it's definitely not easy to choose.


Still, I think option 1 is just going to hold Django back - there's a 
few things they fixed in 8.0, most notably (for me) the implementation 
of information_schema, which any PostgreSQL schema-altering backend is 
going to want.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Imports in the tutorial

[Andrew Godwin]

Hi all,

I noticed today that the tutorial still does imports like "from 
mysite.polls.models import Poll", and URLs like "(r'^polls/$', 
'mysite.polls.views.index')".


At least in the places and projects I've worked with, the standard has 
been not to use the project name as part of the import, for various reasons:


 - It makes apps more reuseable and not tied to the one project
 - It means you can't run two copies of the same application if they're 
direct neighbours (as, to get the imports running, you need to add the 
directory the project is contained in to the Python path, and thus you 
can't rename the directory from its import name).


I realise both of these can be worked around (and I do a lot), but it 
would seem common sense to start encouraging people to do imports like 
"from polls.models import Poll" from day one. There might well be 
something I'm missing - please say - but this has bugged me for ages.


(The previous threads I can find on this topic seems to end pretty 
inconclusively, and were in 2007 and 2009, so I feel partially justified 
in bringing this up again.)


I'm happy to write a patch for the docs to change it to match should 
people agree - I just feel like I'm missing something here, as it's not 
been done yet.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Imports in the tutorial

[Andrew Godwin]

On 11/06/2010 03:28, Peter Baumgartner wrote:

In my experience, almost every project has domain-specific
applications that don't get reused. If you have a reusable app, you
bundle it separately (like South).
   


I entirely agree, but there's also a lot of domain-specific apps people 
make that get used in more than one project. If you're in the business 
of writing (say) CMS-backed sites, you're going to have a lot of apps 
matching that pattern, and as far as I'm aware, there's no downside to 
switching away from having project names in imports.



Why do you need to change the Python path at all? Just drop your
project on the existing Python path and import from there. This is
easy enough with a symlink or better yet give it a setup.py and
install it like you would any other Python package.
   


But that doesn't work. Say I have my nice shiny project "andrewblog". I 
want to deploy a staging version and a live version of this on my server.


If I had local imports, I could just put both projects on the python 
path / install them / egg-link them, or whatever. But because they use 
module names for imports, if I install one site as a package 
"andrewblog" and the other as "andrewblog_staging", say, staging is 
actually going to be running off of the live models, URLs, etc, since it 
has code that goes "from andrewblog.posts.models import Post".



I agree that it is confusing that it will work either way, but I'd
argue in favor of using the project name everywhere. It is explicit
instead of mucking around with the Python path behind the scenes.
   


Explicitness is good. I like being explicit, that's one of the reasons I 
love Python. But I'd argue that having to separate installations of the 
same app because they _have_ to have the same package name involves more 
python path mucking around than just installing them on the global path 
with different names.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Imports in the tutorial

[Andrew Godwin]

On 11/06/10 17:00, Peter Baumgartner wrote:
> Why do your two sites need to share the same python path? Virtualenv
> solves this problem quite gracefully.
>   

They don't have to, but bear in mind that:

 a) The tutorial is aimed at people new to Django, and often new to
Python. Virtualenv isn't even on their radar at this point.
 b) Bugs from this issue can be really tricky to track down, and I've
seen it happen several times on both developer workstations and servers.

I'm not saying that removing them is entirely a win, but for the target
audience of that document, I really think it's appropriate to teach it
from the start. I've seen new Django developers make the mistake of
importing from the project name countless times in
supposedly-reuseable-internally apps, and I've spent many hours going
through with grep and fixing it.

The way I see it, if we remove them, the worst bug that could happen is
if someone manages to get two different project directories on their
python path at the same time (thus the app namespace clashes).

Compared to the alternative, and the unlikeliness of this scenario, I
really feel it's the best move, unless we try to teach new users about
virtualenv, pip, dependency files, and the whole stack that a lot of
more advanced Django developers use. We certainly need to think about
teaching that stuff - it came up as a topic during djangocon.eu, but I
don't think any consensus was reached - but you need to walk before you
can run.

Andrew


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Imports in the tutorial

[Andrew Godwin]

On 11/06/2010 17:38, Russell Keith-Magee wrote:

You're not missing anything specific -- it's really just a matter of
time. Good documentation take time to write; doubly so for good
tutorials.

The issue you raise - that the current tutorial is exclusively "app
inside project" has been raised as a ticket more times than I care to
count. I would certainly welcome anybody that wants to fix it so that
it never gets raised again :-)

It's not as simple as just rewriting the existing 4 steps of the
tutorial, though. As Peter points out, there is a legitimate use case
for "project-specific apps" -- as a way of namespacing apps that truly
are project specific.

Where the current tutorial fails is that it doesn't take the next step
by and demonstrate how (and when) an app can (or should) be broken out
from the project structure. The simple step of making the 'poll' a
truly reusable app, and explaining why this is a good idea, would be a
great tutorial 5 IMHO. Part of this tutorial may be to point out
exactly how unnecessary the 'project' directory really is in the
purist sense -- at the end of the day, all you really need is a
settings file and a bunch of apps in your PYTHONPATH.
   


Right, I remember the whole "Part 5" and virtualenv thing being 
discussed before. If we can come to some consensus on what should go in 
there, I'm happy to write more tutorial parts/edit the current ones and 
have someone who doesn't, write, like they, speak, go over it in an 
editorial role.


Problem is there's several things that could go in a part 5 (in addition 
to all of those there currently), like:


 - Making your apps reuseable (so things like removing project name 
imports, adding a setup.py, general python packaging theory)
 - Using virtualenv to run multiple copies of apps/projects/Django 
side-by-side on a development box
 - Basic deployment onto servers (this is kind of covered elsewhere, so 
perhaps just a pointer, or refine those docs with more common pitfalls)


There's also the risk of guiding people down too narrow a path. I'm 
really not very good at deciding the order and layout of documentation, 
just at generating it in large amounts (see the mass of mostly 
unnavigable words that is the South docs).


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Imports in the tutorial

[Andrew Godwin]

On 12/06/2010 01:03, Russell Keith-Magee wrote:

What - very very quickly? I don't see the problem :-)
   


Well, that wasn't quite the quality I was going for, but you never know.


Problem is there's several things that could go in a part 5 (in addition to all 
of those there currently), like:

  - Making your apps reuseable (so things like removing project name imports, 
adding a setup.py, general python packaging theory)
  - Using virtualenv to run multiple copies of apps/projects/Django 
side-by-side on a development box
  - Basic deployment onto servers (this is kind of covered elsewhere, so 
perhaps just a pointer, or refine those docs with more common pitfalls)
 

That sounds like three potential tutorials to me. Better get your
pencil nice and sharp :-)
   
Honestly - getting a good first draft is the hardest part. Working out

a good example, working out a seizable order for presentation within
he tutorial, and getting even a rough first cut at the text is the
biggest hurdle. Once we have that, it's relatively simple to knock a
draft into shape and fit it into a bigger picture


Right, but I really think reuseability, packaging and virtualenv is a 
combination that could be worked into one tutorial part (they tend to be 
quite long, but I'm not sure if that's an overwhelmingly good thing).


If nobody has any particular suggestions, I'll try to cook up a very 
basic outline/first draft this week, and see what people think. I'm 
unfortunately not that much of a heavy virtualenv user, so I might just 
take a stab at the first part, but I feel like there really does need to 
be a bit more tutorial after all these years. I find it even funnier 
that the section I'm proposing to add isn't one of those in the "Coming 
Soon" section.


Andrew


--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Imports in the tutorial

[Andrew Godwin]

On 15/06/2010 20:52, Brian Luft wrote:

Short of new tutorials, I think the best service we could do for the
(beginner) community is to at least call out a sidebar that provides
some brief context around whether or not to namespace by project.
Since this is a matter of personal preference the responsible thing is
for the documentation to make people aware of the choices and what the
associated trade-offs are. It seems like this could be done in a few
paragraphs and would serve to concisely indicate that:

* there are different ways to do it
* each has its (subjective) pros/cons
* Django isn't endorsing a particular style but has chosen one for the
sake of convenience in the tutorial
   


This I like - there's only two ways, so it won't be very long, and if I 
do get a part 5 out we can say "part 5 shows you the other option in 
more detail".


When I get around to my first attempt at part 5, I'll include an 
interjection somewhere in the other parts in the patch, and people can 
see if they feel like it's not too much of a distraction.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Changing the role of the Technical Board

[Andrew Godwin]

Hi everyone,

I want to start a conversation about the Technical Board and its role in 
Django, and how I'd like to change it, including its name.

Since its inception, the Technical Board has effectively only functioned as a 
backstop vote for large features that require DEPs, of which there have been 
only two in the last five years. I believe it can be much more useful than 
this, while still not dominating Django's direction.

I would like to initially suggest the following high-level changes, that I will 
start formalising into a process DEP if we reach broad agreement:

*Eligibility requirements are loosened and made less code-oriented*

Anyone who demonstrates a "decent" history of contributing to Django in any 
fashion, and who does not have any conflicts of interest, and clearly wants to 
work in the best interests of the framework, would be eligible. (Yes, this is 
going to be fun to define in a DEP).

The current eligibility requirements include the need to have participated in 
discussions about the direction or development of Django in the last two years, 
which given the relative lack of big discussions in the last two years makes it 
rather problematic.

*A more active role is taken in suggesting features*

A regular "call for big ideas" is taken by the Board, and a clear set of "these 
are the technical/design/process/etc. ideas that we'd like to do" is published. 
This list would be, among other things, useful to raise money for grants to 
work directly on those features.

DEP 10 already mostly provides for this - it says that the Board should "put 
out calls for proposals and ideas for the future technical direction of 
Django", so this would more be us just Doing The Thing rather than changing too 
many rules.

*The name is changed to "Steering Council"* (like Python)

I believe this is a more accurate reflection of what the group *should* be 
doing and a reflection that you should not need to be directly coding Django 
every day to participate - and names have meaning, and thus power.

*The overall election process, other powers, and current members remain the 
same*

I think this part works relatively well. I did consider if we should allow 
simultaneous Technical Board/Steering Council membership *and* DSF Board 
membership, as this is currently banned, but I think the reasons behind this 
still hold for now.

This of course would be a "major change" and trigger a lot of procedure, 
notably a supermajority vote of the current Board, and a DSF Board vote, but I 
do believe it is in the best interests of the framework.

I am very interested in any feedback on these suggested ideas, including any 
additional changes you think might be appropriate that I have not covered here.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4d7a9a73-f231-4f01-93c9-59ba32b67a99%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

These are some great points, James - let me try to tackle them roughly in order.

Proposing features - this is already in DEP 10, so I more just want to get that 
aspect of the Board actually going (and, as a side-effect, have something to 
aid fundraising). I am talking with the current Board separately on an internal 
thread, where the current stance (not everyone has responded) is that I am 
personally happy to take on all the work here for now - but I want to make sure 
it's not just me in the long run, be that merely proving that the idea works or 
attracting board members who want to specifically mediate such discussions and 
interaction with the wider community.

Engagement - It's not about "lack of engagement", and I think any issues there 
are deeper problems with OSS communities and the fact that we have to learn to 
sustainably work with the people we have rather than throwing everything at 
trying to recruit fresh, new people. I have ideas around this topic 
specifically, but they will not be solved in terms of the Board alone.

Loosening eligibility - If you're up for it I would very much value your help 
here in terms of refining wording once I have a first draft. My initial 
direction was to still require the 18 month history of contributions, but widen 
it from "technical" to more kinds of work (obviously the discussion part is in 
there too, but I think in general we can do a bit more of an OR rather than an 
AND on the current requirements, keeping a minimum time of contribution to 
prevent bad actors)

Serving on the DSF Board - you are of course right, I misread the DEP last week.

Overall, if all we do is change the name and start actually doing calls for 
features as outlined in DEP 10, I'll honestly be happy - but I think, given the 
most recent TB election was uncontested and several long-time Django 
contributors have told me they'd be more willing to join a TB that was less 
strictly technical-all-the-time, that it makes sense for us to also look at 
those requirements.

Andrew

On Mon, Oct 24, 2022, at 2:54 PM, James Bennett wrote:
> Something I note here is that it's presenting a solution, but not clearly (at 
> least, from my reading) presenting the problem to be solved.
> 
> Is it a lack of people proposing features? If so, I'm not sure this helps -- 
> it would, to me, suggest that only members of the Technical Board/Steering 
> Council/whatever-it's called are supposed to do that, since it's in their job 
> description. Would people then expect to, or be expected to, run for a seat 
> in order to contribute something?
> 
> Is it a more general lack of engagement? If so, I'm still not sure how this 
> helps -- the idea of DEP 10 was to make it *easier* for people to step up and 
> get involved, since it got rid of the idea of the "core team" with their 
> special privileges, but I don't think any form of technical governance 
> actually solves engagement issues. At best it can make engagement-specific 
> efforts easier, but I don't see how re-centralizing authority (or creating 
> the impression of it) would achieve that.
> 
> Is it to make fundraising easier? That sounds again like something that 
> technical governance really can't do on its own -- it needs to involve the 
> DSF Board, and there are reasons why the DSF was historically wary about 
> doing targeted fundraising for specific features in Django.
> 
> Loosening eligibility is fine, though I agree it's going to be very difficult 
> to write down in an enforceable way -- the DEP 10 language and process was 
> intended primarily to prevent trolls and other bad-faith actors from being 
> able to run effectively for the Technical Board, and there's a balance where 
> the more you loosen it up, the more you also open the door for those kinds of 
> people.
> 
> Also, regarding the multiple roles restriction: it currently is allowed for a 
> single person to simultaneously be on both the Technical Board and the DSF 
> Board, and there are even procedures in DEP 10 for things like mandatory 
> recusal for DSF Board votes and actions that affect the Technical Board. 
> What's not allowed is simultaneously being a Merger and on the Technical 
> Board.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg-mWWK0%2Bzkvi%3DCWu0e%3DbX-rOsLq4gHHdBuKQ8UL_8pRSg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe

Re: Changing the role of the Technical Board

[Andrew Godwin]

> Has that not been occurring? Because if it hasn't, then we have a major 
> problem, and I don't see how the current proposal would resolve it.

It has not. While I cannot speak for the other members of the Board, I got 
burnt out in 2019, and then the pandemic began, and so it has not really been 
something I've pushed for in the past three years (and I believe I was one of 
the drivers of getting that in DEP 10 in the first place). We never really got 
into the rhythm of doing it, and I think a lot of us got busy or burnt out.

My personal belief is that if something is not working, it is time for a 
re-analysis of the situation and a change, even if what's written down in the 
rules is fine on its face.

This proposal is part of a larger set of community changes I would like to 
consider - I am, for obvious reasons, not blasting the community with them all 
at once, but I do believe that a strong, visible set of leaders, with a clearly 
outlined set of principles and shepherding a community-guided vision, is the 
most important thing to establish - both in terms of the Technical Board and 
the DSF Board.

I have a much longer blog post in me about my evolving belief in the need for 
visible, servant leaders in OSS communities rather than trying to embrace a 
flat hierarchy with mechanical checks and balances - but that is for another 
day.

Andrew

On Mon, Oct 24, 2022, at 4:26 PM, James Bennett wrote:
> On Mon, Oct 24, 2022 at 2:24 PM Andrew Godwin  wrote:
>> __
>> Proposing features - this is already in DEP 10, so I more just want to get 
>> that aspect of the Board actually going (and, as a side-effect, have 
>> something to aid fundraising). I am talking with the current Board 
>> separately on an internal thread, where the current stance (not everyone has 
>> responded) is that I am personally happy to take on all the work here for 
>> now - but I want to make sure it's not just me in the long run, be that 
>> merely proving that the idea works or attracting board members who want to 
>> specifically mediate such discussions and interaction with the wider 
>> community.
> 
> I admit I haven't been following Django development all that closely since I 
> mic-dropped after DEP 10 passed, but this is worrying, because canvassing for 
> feature proposals is not an optional thing -- DEP 10 *requires* the Technical 
> Board to do this at least once per feature release of Django. Has that not 
> been occurring? Because if it hasn't, then we have a major problem, and I 
> don't see how the current proposal would resolve it.
> 
>> Overall, if all we do is change the name and start actually doing calls for 
>> features as outlined in DEP 10, I'll honestly be happy - but I think, given 
>> the most recent TB election was uncontested and several long-time Django 
>> contributors have told me they'd be more willing to join a TB that was less 
>> strictly technical-all-the-time, that it makes sense for us to also look at 
>> those requirements.
>>> 
> I have concerns about this because what it really feels like is a first step 
> toward merging the Technical Board and the DSF Board into a single unified 
> governance board of everything to do with Django, and I think the split 
> between governance of Django-the-codebase and 
> Django-the-everything-the-DSF-does is a useful and important one (not least 
> for legal reasons). 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg93CvLgVU567MsvqKZAiWVBxxm1-L_YuEUhjr_JRySBwg%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CAL13Cg93CvLgVU567MsvqKZAiWVBxxm1-L_YuEUhjr_JRySBwg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6196974e-a1a5-4585-87ab-02f79509%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

On Tue, Oct 25, 2022, at 12:12 AM, James Bennett wrote:
> 
> My first reaction to this is: if having a DEP that says the Technical Board 
> is supposed to take the lead in gathering feature proposals didn't get them 
> to do it, it doesn't feel like another DEP saying they're responsible for 
> that is going to fix it.

I agree. Me proposing the DEP changes is mostly merely formalising some other 
changes I want to catalyse here - the DEP is an outcome, not the start, here.

> Getting burned out or overcommitted is a thing that happens, and a thing that 
> was anticipated in drafting the governance -- DEP 10 has a procedure for it!
> 
> Why did no member of the Technical Board do that?

Again, speaking for myself - because we were doing almost all the functions of 
the Board except for the feature canvassing. I also saw the relative lack of 
candidates for Board elections and essentially thought "better burnt-out me 
than literally nobody".

> And from the sound of what you're saying in this thread, the Technical Board 
> is mostly communicating with itself about this, in private, when the 
> direction of Django is supposed to be worked out publicly and transparently. 
> That's why we shut down the old private communication spaces for the former 
> "core team" after DEP 10 was adopted.

That is not the case - this thread is the majority of the discussion. I opened 
a thread on the TB mailing list in case people there wanted to give specific, 
blunt feedback about how they would feel about being affected by this as 
current Board members, but it has only a couple of replies, and nothing that 
hasn't been discussed here.

> So forgive me for being blunt, but: if the Technical Board is not following 
> the governance we have, I think replacing the Technical Board's current 
> membership should be higher on the list of remedies than replacing the 
> governance.

Forgive me for being blunt but... that seems like a really bad idea? The 
current board ran uncontested, so if all five of us step down I suspect there 
may not *be* a Board at the end of the day. The DSF Board's current 
difficulties finding candidates I think reinforces that fact - one of the 
changes I was considering bringing in here was "what if we have to reduce the 
TB to 4 or 3 people in the near future".

Again, I'm not saying "we should write a new DEP and *that'll* fix it", I'm 
trying to come from a position of working out what we can and should be 
*doing*, and then ensuring our rules match that.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e04f4e94-7466-4d58-b0d8-21a5e4ba3faa%40app.fastmail.com.

Re: Changing the role of the Technical Board

[Andrew Godwin]

I agree the Technical Board has not followed the letter of DEP 10, and I think 
the things you have highlighted are all valid failings, but I want to focus on 
- what should we do to remedy them?

Given the lack of candidates we already have, if we ditch the current Board and 
try to elect a new one that has to do all these things, it's likely we will 
fail to have a valid election, and from what I recall, there is no provision 
for that in DEP 10.

We could consider changing to a board size of three to overcome this, but that 
would increase the workload even further.  We could hope that five incredibly 
willing people are waiting in the wings, but I doubt it.

At this point, it is my view that it is our job to govern with the people we 
have, and the time and energy they can provide, and that's my intention with 
these suggested changes.

I honestly think you and I both want the Board to do the same rough role - my 
current draft "extension DEP" on top of DEP 10 is literally just "change the 
name and tweak the eligibility clauses", it's not like we're going to throw it 
out. I just want to come at this a bit more incrementally with our current set 
of people, and throw the net a bit wider so we do have more of a chance of 
finding five people with the energy to run the tasks assigned to the Board with 
the vision we initially had.

At the end of the day, my feeling is that inaction is not the right path - we 
need to enact some sort of change. I'm more than willing to hear alternative 
suggestions for what that change should be (though as outlined previously, I 
really don't think that change should be "remove the entire current Board for 
underperformance and have another election").

Andrew

On Wed, Oct 26, 2022, at 12:23 PM, James Bennett wrote:
> I'm going to avoid trying to get too much into point-by-point back-and-forth 
> argument here, and try to present the larger picture.
> 
> The Technical Board has multiple active responsibilities under DEP 10. Let's 
> look at some of them:
> 
> 1. Canvas for feature proposals once per feature release of Django. This also 
> comes with a responsibility to ensure the proposals are archived in a useful 
> way -- we used to have wiki pages in Trac for this, but DEP 10 doesn't 
> require any specific mechanism. This is supposed to happen within one week 
> after feature freeze of each feature release.
> 
> 2. Set the release schedule for each feature release of Django. This is 
> supposed to happen within one week after the prior feature release coming out.
> 
> 3. Maintain the teams of Mergers and Releasers, including by nominating and 
> confirming new members of those teams when the Technical Board considers it 
> necessary (or when the current roster falls below the DEP 10 quorum).
> 
> 4. Vote, as requested, on DEPs.
> 
> 5. Vote, as requested, on any technical decision that fails to resolve 
> through normal public discussion.
> 
> (1) has not been happening. It's possible that (2) has been happening and 
> just hasn't been that formally published, but I suspect it's been the Fellows 
> who've been doing that one. For 3-5, things have gone... not so great.
> 
> It took multiple attempts to get the Technical Board to vote on the first 
> Merger nomination (Claude), and if we're being pedantic it still wasn't quite 
> done correctly because his appointment as a Merger should have been temporary 
> and auto-expired after the first Technical Board election unless the new 
> Board voted to confirm him permanently.
> 
> The first time the Technical Board voted on a DEP was the proposed DEP 484 
> for type hints. The Technical Board's discussion and voting on it apparently 
> took place entirely in private and the result was communicated publicly via 
> Carlton copy/pasting the reply he got from them. And we lucked out that the 
> Board didn't accept the DEP, because it was at a time when an election was 
> pending and thus they had no power to accept a DEP (similar to the Merger 
> nomination -- once an election triggers, the Board cannot accept DEPs and can 
> only make temporary appointments of Mergers/Releasers).
> 
> On (5) the only explicit request I can find in this mailing list's archive is 
> one from Carlton to make a decision on ticket 31747. Although several 
> Technical Board members posted opinions in response, I'm not finding any 
> statements of their votes on the matter.
> 
> If grant a half point for the Merger nomination (though the Board really 
> ought to hold a public vote to properly permanently confirm Claude), and 
> maybe another half point for the fact that release schedules have been 
> getting set (though, I suspect, not explicitly by the Board), that gets us to 
> 1 out of a possible 5. This is not a great track record.
> 
> Before we propose a different setup, I think we need to figure out what went 
> wrong here. It already seems like some members of the Technical Board may not 
> have really been familiar with the responsibilitie

Re: Changing the role of the Technical Board

[Andrew Godwin]

My intention is indeed for us to run a new Technical Board election come the 
end of 4.2, with much better and more explicit communication about what will be 
expected of the new members, and a larger candidate pool to pull from to 
hopefully make that work.

I will be posting my actual proposed DEP shortly so it's more clear exactly 
what I want to change at a written-rules level - I suspect feedback on a more 
concrete proposal will help us talk about it more clearly.

Andrew

On Wed, Oct 26, 2022, at 4:55 PM, James Bennett wrote:
> On Wed, Oct 26, 2022 at 12:02 PM Andrew Godwin  wrote:
>> At this point, it is my view that it is our job to govern with the people we 
>> have, and the time and energy they can provide, and that's my intention with 
>> these suggested changes.
> 
> If the problem in front of us is that the Technical Board isn't up to the 
> level of activity DEP 10 asks of them, and you believe no substitute group of 
> members exists that would be up to it either, I'm not sure I see how your 
> proposals are going to fix that, especially since you seem to eventually want 
> to create *more* responsibility for active intervention and leadership.
>> 
> 
>> At the end of the day, my feeling is that inaction is not the right path - 
>> we need to enact some sort of change. I'm more than willing to hear 
>> alternative suggestions for what that change should be (though as outlined 
>> previously, I really don't think that change should be "remove the entire 
>> current Board for underperformance and have another election").
> 
> A Technical Board election will automatically trigger at the release of 
> Django 4.2, which is not *that* far off. But the Technical Board could always 
> trigger an election any time they want to.
> 
> I personally would still like to understand how the current Technical Board 
> came to what seems to be such a misunderstanding of how the governance was 
> supposed to work. And while there have been explanations presented for why 
> the Technical Board didn't communicate the problems it was having, they also 
> come across as worrying -- we're all adults here, and we need to be able to 
> trust each other to speak up when there's a problem. How did we go, 
> apparently, multiple years with the Technical Board not carrying out their 
> responsibilities and also nobody saying anything about it?
> 
> Until we understand that I don't think we should be trying to change the 
> governance again.
> 
> And I still would like to see DEP 10 actually tried out. Maybe it involves 
> electing a Technical Board with much more explicit up-front communication of 
> expectations, since it seems a lot of the current members were unaware of the 
> Technical Board's actual responsibilities. Maybe it involves someone just 
> constantly poking them with reminders. Maybe something else. But it feels 
> wrong on many levels to start moving on from DEP 10 when it's becoming 
> increasingly clear that DEP 10 was never really *tried*.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAL13Cg_%3DbczbtMP5B6avbNTLi%3DLf1SU_Z%3Dchz6j0u5L1MPMbzw%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CAL13Cg_%3DbczbtMP5B6avbNTLi%3DLf1SU_Z%3Dchz6j0u5L1MPMbzw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/88084dd9-a2e9-4271-9160-79bb463a8adc%40app.fastmail.com.

Draft Steering Council DEP

[Andrew Godwin]

Hi all,

As a followup to my previous post about potential changes to the Technical 
Board - for which I thank you all for the feedback - I have taken the process 
to the next step and written a draft DEP:

https://github.com/django/deps/pull/75/files

(If you wish to see the DEP with styling, it can be viewed at 
https://github.com/andrewgodwin/deps/blob/steering/draft/0012-steering-council.rst)

I could just merge this into the DEPs repo as a draft given our process, but I 
would like to invite both review of the DEP styling and layout (to ensure it's 
ready to merge in that regard) as well as initial feedback on its content as 
well.

I have copied in the DSF Members mailing list as it is a governance-related 
DEP, but if we could keep all discussion on the thread in the Django Developers 
mailing list, as per DEP 0001, that would be great.

Thanks,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4e6893a8-8e22-45a3-96f6-d56d69db7b43%40app.fastmail.com.

Re: Proposal for Django Core Sprints

[Andrew Godwin]

Hi Paolo,

I do like the overall idea - a few thoughts below.

My first concern for this, which somewhat echoes James, is that trying to 
organise an additional in-person event that a large number of contributors are 
expected to go to is difficult. Funding considerations are one concern - we 
would need to make sure everyone whose companies did not cover them to go had 
their trip paid for - but time away from home is also something that we should 
be considerate of, be it for family or for anything else.

That, too, brings an interesting challenge - who do you invite (and presumably 
help pay for)? We no longer have a core team, and I don't think it's worth 
singling the Technical Board out in this regard - the Fellows are an obvious 
requirement if they can make it, but past that, it's a hard metric to gauge.

I do like your point about disconnecting it from needing to give a talk, though 
- conferences and sprints are certainly Different Moods for me, and it can be 
draining to be sociable for three days and *then* have to go into sprints and 
be even more so. I don't know if there's a useful resolution to this specific 
problem.

My second concern is almost the opposite of that, ironically, which is that I 
do believe there are some things that benefit from synchronous, in-person 
communication. A single week at the most recent DjangoCon has done wonders for 
me in that regard, and while I do believe it should be a hybrid event, I also 
think it should *not* be a remote-only event. While it is a personal opinion, I 
find in-person sprinting significantly easier (due to the nature of the work as 
opposed to purely coding) and would probably find it hard to attend a 
remote-only sprint.

The final question I would raise is that of location. There are two elements 
here - where in the world do you hold it, and what the actual venue is like.

There is no single location in the world you will ever find where everyone can 
attend - even with very patient Australians who are far too willing to sit on 
planes for ages. It's likely you'd have to geolocate in either the Americas or 
in Europe, and if you additionally want to host it in a location that is safe 
for all attendees, that cuts down the number of US states and European 
countries even more.

And then, the venue. In the modern world, I ideally want an *outdoors* sprint 
venue, so that we're not all stuck inside a room hoping the masks work well 
enough. That places limitations on climate and season and also availability of 
power - so many that it starts to look like an impossible task to satisfy this 
along with all the above.

I don't write this as a means of discouragement - I think there's a lot to be 
said for your idea, and even though I've outlined a lot of potential flaws, I 
think a flawed event in this fashion would be better than nothing happening at 
all! I just want to highlight all the talking points we're going to need to 
really think about and have answers to if we take this forward.

(I didn't even discuss how we might fund this, which is also a conversation to 
have, but waving our hands in the air and going "sponsorship" is enough for me 
to start with)

Andrew

On Wed, Oct 26, 2022, at 4:01 PM, Paolo Melchiorre wrote:
> Hi everyone,
> 
> Following Andrew Godwin's example, I too share with you a proposal
> that I made during DjangoCon US 2022 and that I have already shared
> with some of you.
> 
> Inspired by an article by Will Vincent, I wrote my proposal for Django
> Core Sprints, and its genesis, in an article and would also like to
> have the opinions of the Django community members interested in the
> idea:
> https://www.paulox.net/2022/10/26/about-my-proposal-for-the-django-core-sprints/
> 
> Ciao,
> Paolo
> -- 
> Paolo Melchiorre
> 
> https://www.paulox.net
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAKFO%2Bx6FAf-M14fKx5QS1K8rru2L3mOiR1R%3Dn6FzDk%3DrnaUq3A%40mail.gmail.com.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ebd86080-6735-45f7-8dd7-3ddc23c4a44d%40app.fastmail.com.

Re: Draft Steering Council DEP

[Andrew Godwin]

On Sun, Oct 30, 2022, at 10:42 PM, James Bennett wrote:
> On Wed, Oct 26, 2022 at 4:34 PM Andrew Godwin  wrote:
>> __
>> 
>> I have copied in the DSF Members mailing list as it is a governance-related 
>> DEP, but if we could keep all discussion on the thread in the Django 
>> Developers mailing list, as per DEP 0001, that would be great.
> 
> My main concern remains the thing I've been saying repeatedly in the other 
> thread: how does this actually solve a problem that Django is facing right 
> now?

By widening the set of people who can run for the Board/Council.

> 
> We've established that the Technical Board was not carrying out its duties. 
> There's also been a claim advanced that probably there is not a replacement 
> slate of board members who would step up and have the capacity to do what DEP 
> 10 asks.
> 
> So, either this new DEP is intended to be a slight clarification of the 
> Technical Board's role, in which case I don't see how saying "same as before, 
> but this time we actually expect you to do the thing" solves the issue. Or 
> it's intended to be a first step toward a more active group, in which case I 
> don't see how it can succeed given that a lower required level of activity 
> has already failed.

It is intended to allow for a more active group by giving us a wider set of 
candidates who can run and thus a higher chance of people being on the 
Board/Council who want to be that active.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/347c6859-5315-45ff-80e6-5480b99e95a7%40app.fastmail.com.

Re: Draft Steering Council DEP

[Andrew Godwin]

On Tue, Nov 1, 2022, at 6:54 AM, C. Kirby wrote:
> Having run the elections for the current technical board I agree with 
> Andrew's assessment that a more open requirement to run is a good idea. It 
> may create a bit more work on candidate verification for the DSF Board and 
> Fellows, but anything that can work to encourage new blood in the 
> "Leadership" of Django and the DSF has a +1 from me.
> 
> I don't have a strong feeling on renaming of the board, but as such don't 
> really see it as necessary.

It is not an absolute requirement from me, but I think it's a better 
description of what it should do and also helps us distinguish it from the 
"Technical Team" a bit more, so I will keep it in the DEP unless there's strong 
objection.

> 1. There are several of timelines and triggers listed in DEP 10. A section 
> that lays them out explicitly with references back to the details could be 
> immensely useful. A flow chart perhaps - what are all the things that happen 
> when the final release of a major version occurs. A list of trigger actions 
> would be useful as well - Fewer Than 3 remaining members of the technical 
> board - elections, Fewer than 3 mergers - select new Merger, etc. A TL;DR; 
> for the Technical Board on the "day to day" working of the board.

Agreed, and I think this is something we should do either way - if this DEP 
goes in, doubly so, as we want to have a single place to reference things 
rather than multiple DEPs.

> 
> 2. Probably controversial - an enforcement mechanism. The DSF Board has a 
> regulatory requirement to meet and follow the bylaws as a registered 
> non-profit. We can be subject to lawsuit if we don't. The Technical Board has 
> no such liability, except for the, perhaps stronger, moral liability to the 
> community. To be clear I am not suggesting a legal enforcement mechanism, but 
> perhaps a community one. I dearly hope that it would never be needed, but not 
> having one at all seems an oversight. Something along the lines of:
> "DEP 10 enforcement: Any DFS individual member may make a public statement of 
> no-confidence in the technical board by identifying a material breach of DEP 
> 10. Upon seconding by another individual member of the DSF the DSF Board 
> SHALL no later than the next scheduled board meeting evaluate the merits of 
> the statement of no-confidence. If the statement is found to be accurate and 
> correct the Board shall inform the Technical Board of the breach and provide 
> 2 weeks to rectify said breach. If the Technical Board fails to rectify the 
> breach in the time allotted Technical Board elections SHALL be triggered and 
> current members of the Technical Board shall be barred from running in the 
> no-confidence election"

Interesting - having the DSF board moderate that makes it more agreeable to me, 
though if we are going to introduce this we should _also_ introduce wording for 
what happens if we fail to elect a Board, as this makes it much more likely 
(barring the entire previous board from running).

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/46c7e2a6-78be-48f9-95f9-66b891d1d5d3%40app.fastmail.com.

Re: Proposal: Clarify individual members page

[Andrew Godwin]

Just want to pop in and say these are great ideas - feel free to copy me in 
on any PR if you want extra opinions!

On Tuesday, November 8, 2022 at 8:26:28 AM UTC-7 Carlton Gibson wrote:

> Great, Thanks Andrew. No urgency 😊
>
> On Tue, 8 Nov 2022 at 16:16, Andrew Mshar  wrote:
>
>> Will do, Carlton.
>>
>> Tim and Cory, thanks for the suggestions. I'll incorporate those in the 
>> PR and post here when it's ready. Probably not today, but I should be able 
>> to open it before the end of the week.
>>
>> Thanks,
>> Andrew
>>
>> On Tuesday, November 8, 2022 at 10:10:51 AM UTC-5 carlton...@gmail.com 
>> wrote:
>>
>>> Hey Andrew. 
>>>
>>> I had thought this was a Flatpage (stored in the database) but it's not. 
>>> The source is here: 
>>> https://github.com/django/djangoproject.com/blob/main/djangoproject/templates/members/individualmember_list.html
>>> If you wanted to open a PR suggesting your changes, that would be 
>>> amazing 🤩
>>>
>>> Thanks. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>> On Mon, 7 Nov 2022 at 19:51, Tim Allen  
>>> wrote:
>>>
 I'm of the opinion that if you care enough about Django to investigate 
 becoming a member of the DSF, that's enough of a qualification - it is 
 just 
 challenging to formalize that into proper text for the website. Maybe two 
 changes to encourage people to join:

- We could tweak *"Running Django-related events or user groups"  *
to *"Attending or organizing Django-related events or user groups"*.
- Add a sentence to the end of the first stanza: "The following are 
Individual Members of the Django Software Foundation. The DSF appoints 
individual Members in recognition of their service to the Django 
 community. 
If you would like to join the DSF, we welcome you. Please feel free to 
self-nominate for membership."

 Regards,

 Tim

 On Monday, November 7, 2022 at 11:12:41 AM UTC-5 cory...@gmail.com 
 wrote:

> Hey Andrew,
>
> Thanks for drafting this language and I think it looks great. As 
> someone who only recently applied after hearing it discussed on an 
> episode 
> of Django Chat[1], I'm all for the goals of making it more encouraging 
> and 
> accessible and think this is a great step in that direction.
>
> Here are a few minor thoughts to specific bits:
>
> Service to the Django community takes many forms. Here are some 
>> examples (non-exhaustive) of categories of work performed by members:
>>
>
> "performed by members" is a little ambiguous as to whether it means 
> "this is how we evaluate applicants" vs "this is what you'll do if part 
> of 
> the DSF". Since I think the intention is the former it might make sense 
> to 
> change to something like:
>
> *Service to the Django community takes many forms. Here are some 
> (non-exhaustive) examples of the categories of work that might qualify as 
> "service":*
>
> Borrowed the list of categories from Andrew Godwin's DEP for the 
>> update to the technical board. Per Tim's recommendation, do we want to 
>> include anything about the review process?
>>
>
> When I applied I didn't (and still don't, really) have any visibility 
> into the process, so it wasn't a deterrent for me, personally, but I 
> think 
> having information certainly wouldn't hurt. My two cents would be good to 
> put something in, but not necessarily if it slows down/stalls this change 
> if for whatever reason that isn't super easy, since I think this 
> represents 
> an improvement on its own.
>
> Also, I'm a little unsure about that last bit about applying, but I 
>> wanted to put something encouraging to folks to apply. Happy to reword 
>> that 
>> if someone has a better suggestion. I'd prefer that to having a full 
>> rubric 
>> for membership on this page, primarily because I think it would be very 
>> difficult to nail that down because the work that folks perform can be 
>> so 
>> disparate (must have run X django meetups, or triaged Y tickets). 
>>
>
> Definitely agree a rubric would cause more problems than it would help 
> at this stage. The goals of rubrics in terms of increasing objectivity 
> and 
> reducing bias are great, but as applied to the already-squishy definition 
> of "service to the community" it doesn't seem like a good fit here.
>
> Finally, this is wildly out of scope, but it may make sense to (either 
> here or separately) attempt to create a bit more content about what it 
> means to be an individual member of the DSF. That information is also 
> somewhat lacking, and having it somewhere may encourage more people to 
> apply. One possibility could be to link to one of the recent conference 
> talks[2][3] on the DSF. But wouldn't want that dis

Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Hi all,

As it seems discussion on DEP 12 has reached its end, and we received generally 
positive feedback, I am requesting a vote from the technical board on the 
following:

"Shall we accept DEP 12 and send it to the DSF Board for further approval?"

Note that as this is a governance change, it requires a 4/5 majority from the 
Technical Board, after which it will go to the DSF Board for a vote, and after 
which it *may* go to the membership for a vote if the two Boards believe that 
it should (you can read more about these requirements in DEP 10).

You can see the text of the DEP here: 
https://github.com/django/deps/blob/main/draft/0012-steering-council.rst

My vote is +1, as I am the author of the DEP and believe it is in the best 
interests of the longevity of the Django project and sustainable governance.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d1ba4e9-6eca-41a8-aa4b-64d9ee2310fb%40app.fastmail.com.

Re: Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Yes, I agree we can use the forum in future since it's less tied to Google.

Provided the current +5 vote carries through to the end of the voting period, I 
will be suggesting that the Technical Board triggers the DEP 10 mechanism where 
we move this to the membership for a vote once the DSF Board has performed 
their vote, as it only seems appropriate.

Andrew

On Mon, Nov 28, 2022, at 9:30 PM, 'Adam Johnson' via Django developers  
(Contributions to Django itself) wrote:
> +1 from me
> 
> And +1 to using the forum in future
> 
> On Tue, 29 Nov 2022 at 00:23, charettes  wrote:
>> +1 from me as well.
>> 
>> Le lundi 28 novembre 2022 à 08:32:01 UTC-5, carlton...@gmail.com a écrit :
>>> Hi All. 
>>> 
>>> Tom Forbes is currently unable to post to the Google Group here, due to 
>>> Currently Undiagnosed Google Account Weirdness™. 
>>> 
>>> For the record he's asked me to say that his vote is +1 in support of the 
>>> change, but I feel we should probably move voting to the forum because of 
>>> this. 
>>> 
>>> 
>>> Kind Regards,
>>> 
>>> Carlton
>>> 
>>> 
>>> On Monday, 28 November 2022 at 09:52:35 UTC+1 f.apo...@gmail.com wrote:
 Hi,
 
 +1 from me, but I'd like to ask the wider community (ie DSF members) 
 whether they support this change.
 
 While there has been some opposition on whether a change likes this will 
 actually change things, I think that given the overall good reception of 
 the proposal it is at least worth to try it.
 
 Cheers,
 Florian
 On Friday, November 25, 2022 at 10:04:36 AM UTC+1 Mariusz Felisiak wrote:
> Thanks!
> 
> Reminder: according to DEP 10 voting will end on *December 2nd, 2022* 
> *AoE* (members of the Technical Board may change their votes at any time 
> prior to closing of the final voting period).
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com.

Re: Technical Board vote on DEP 0012: The Steering Council

[Andrew Godwin]

Hi all,

I am pleased to announce that this vote has passed with 5 Board members voting 
+1 and no other votes.

I will now proceed to notify the DSF Board that they must hold a vote, as is 
required by DEP 10. Either they or I will update the list with information once 
that vote has been concluded and a result reached.

Andrew

On Wed, Nov 30, 2022, at 10:44 AM, Andrew Godwin wrote:
> Yes, I agree we can use the forum in future since it's less tied to Google.
> 
> Provided the current +5 vote carries through to the end of the voting period, 
> I will be suggesting that the Technical Board triggers the DEP 10 mechanism 
> where we move this to the membership for a vote once the DSF Board has 
> performed their vote, as it only seems appropriate.
> 
> Andrew
> 
> On Mon, Nov 28, 2022, at 9:30 PM, 'Adam Johnson' via Django developers  
> (Contributions to Django itself) wrote:
>> +1 from me
>> 
>> And +1 to using the forum in future
>> 
>> On Tue, 29 Nov 2022 at 00:23, charettes  wrote:
>>> +1 from me as well.
>>> 
>>> Le lundi 28 novembre 2022 à 08:32:01 UTC-5, carlton...@gmail.com a écrit :
>>>> Hi All. 
>>>> 
>>>> Tom Forbes is currently unable to post to the Google Group here, due to 
>>>> Currently Undiagnosed Google Account Weirdness™. 
>>>> 
>>>> For the record he's asked me to say that his vote is +1 in support of the 
>>>> change, but I feel we should probably move voting to the forum because of 
>>>> this. 
>>>> 
>>>> 
>>>> Kind Regards,
>>>> 
>>>> Carlton
>>>> 
>>>> 
>>>> On Monday, 28 November 2022 at 09:52:35 UTC+1 f.apo...@gmail.com wrote:
>>>>> Hi,
>>>>> 
>>>>> +1 from me, but I'd like to ask the wider community (ie DSF members) 
>>>>> whether they support this change.
>>>>> 
>>>>> While there has been some opposition on whether a change likes this will 
>>>>> actually change things, I think that given the overall good reception of 
>>>>> the proposal it is at least worth to try it.
>>>>> 
>>>>> Cheers,
>>>>> Florian
>>>>> On Friday, November 25, 2022 at 10:04:36 AM UTC+1 Mariusz Felisiak wrote:
>>>>>> Thanks!
>>>>>> 
>>>>>> Reminder: according to DEP 10 voting will end on *December 2nd, 2022* 
>>>>>> *AoE* (members of the Technical Board may change their votes at any time 
>>>>>> prior to closing of the final voting period).
>>> 
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/django-developers/378f606b-5d70-4013-a8d7-e8abb605f178n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/CAMyDDM1dGUTiKTezaYDbqF0PYv1%3DEf%2Ba_VqT1J%2ByP%2BbgbdePZQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/12cfb130-17d0-4f42-9d58-631c927f473f%40app.fastmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1d72d359-eff3-4b7d-90e7-2d971445b618%40app.fastmail.com.

Re: Can we move the activity on this list to the Forum now?

[Andrew Godwin]

I did some investigation of moving django-users and django-developers to 
the Forum right after DjangoCon; I wanted to see if we could import all the 
old posts too, which we probably could, but I'm not entirely sure of the 
utility of that.

I will say that the forum is a lot easier to moderate - the ability to 
moderate after a post has gone out, rather than gating all posts behind 
approval if they're untrusted, is a big step in itself, not to mention the 
ability to remove sensitive or offensive content once it's posted.

Andrew

On Monday, November 28, 2022 at 10:01:17 PM UTC-7 m...@kye.id.au wrote:

> IMO django-announce and django-updates serve a very different purpose and 
> I would be against moving them if it were suggested.
>
> I am incredibly strongly in favour of moving django-developers and 
> django-users to the forums. IMO being able to more easily trap people 
> misusing this list as a tech support channel is itself reason enough to 
> move. Beyond that, I’d argue that the plentiful UX issues with Google 
> Groups, and mailing lists in general, certainly don’t do the community any 
> favours in terms of getting more people on board. 
>
> Kye
> On 28 Nov 2022 at 11:40 PM +0800, 'Tobias McNulty' via Django developers 
> (Contributions to Django itself) , wrote:
>
> As someone who only just joined the forum -- I'm +1: 
>
>- The forum has seen great adoption from what I can tell (nearly half 
>the number of posts as django-developers during the same time period, not 
>bad given the mailing list's head start in subscribers).
>- It seems beneficial to house future conversations in a single place, 
>e.g., so members don't need to subscribe to both the mailing list and 
> forum 
>to get the full picture of current active development, set up two 
> different 
>sets of mail filters to tag things appropriately, etc... 
>
> Would the plan be to switch django-users as well? I think similar 
> arguments could be made for consolidating those...
>
> (On the other hand, I see little value in switching django-announce and 
> django-updates, but I'm not necessarily opposed to it either, especially if 
> there's a way to import the subscribers to those lists...)
>
> Cheers,
>
>
> *Tobias McNulty*Chief Executive Officer
>
> tob...@caktusgroup.com
> www.caktusgroup.com
>
>
> On Mon, Nov 28, 2022 at 9:05 AM Carlton Gibson  
> wrote:
>
>> Hey Roger,  
>>
>> Indeed it does. You can set up Email Mode (that may not be the actual 
>> name) and it’ll work just like a mailing list. 
>>
>> You can also subscribe to just a particular category, so the Internals 
>> one would map to the discussion on this list. 
>>
>>
>>
>> On Monday, 28 November 2022, Roger Gammans  
>> wrote:
>>
>>> Hi
>>>
>>> I can't speak for others, but I personally STRONGLY value the fact that 
>>> this discussion happens in my inbox, not on yet another website. 
>>>
>>> But perhaps the forum still supports this reading mode?
>>>
>>> On Mon, 2022-11-28 at 05:38 -0800, Carlton Gibson wrote:
>>>
>>> Hi all.  
>>>
>>> Given the issues with Tom's access to the mailing list here, and the 
>>> fact that the Forum has been active for a few years now, and is a great 
>>> success, I'd like to revisit whether we can move on-mass (all few of us :) 
>>> over there? 
>>>
>>> We'd enjoy the benefits of a much nicer system. We'd not have issues 
>>> such as the current, and we'd be one less item in the pocket of a 
>>> mega-corp. (You can rank those as you will :) 
>>>
>>> Initially when this can up (a long time ago) Andrew and Tom discussed 
>>> whether we could import the history here into the forum. I think that's 
>>> unnecessary. We can still access the history here (until such a day as 
>>> Google takes it away) at worst -- but, rather, if we can get an archive we 
>>> could import it into read-only Datasette instance[0] — and that would 
>>> likely be good enough. 
>>>
>>> Can we move now? 
>>>
>>> Thanks. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>>
>>> [0]: I'd happily do this. 
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/101f4e6d-9b83-47ab-bb1b-b571402e037dn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msg

DEP 12 (Steering Council) Fully Adopted

[Andrew Godwin]

Hi everyone,

I am pleased to report that we've completed the extended approval process for 
DEP 12, which was needed since it was a governance change. Both the Technical 
Board and the DSF Board voted to not require a vote from the membership on the 
change, so it is now officially adopted and I will begin implementation 
immediately (which is mostly going to be renaming things).

Thanks everyone for your comments and help refining this DEP! You'll be hearing 
more from myself or one of the other Steering Council members soon about our 
first proper "call for feature proposals", as it'll be timed right after the 
4.2 feature freeze in January.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89537c96-377d-4e86-8a8e-5ef738576ea2%40app.fastmail.com.

Re: Can we move the activity on this list to the Forum now?

[Andrew Godwin]

We should at least update those Trac and Triage Workflow docs to point to both, 
maybe with the Forum first?

Andrew

On Thu, Jan 19, 2023, at 12:30 AM, Carlton Gibson wrote:
> I'm trying to begin new conversations there where I can. 
> 
> The main issue is that we're still pointing people here from Trac and the 
> Triage Workflow docs — so if there's a rough "Yeah, let's do it" we can 
> adjust practice there. 
> 
> I expect there will always be the "How do I start?" posts. The Forum at least 
> has a pinned post for that. ... 
> 
> On Thu, 19 Jan 2023 at 01:04, 'Kye Russell' via Django developers 
> (Contributions to Django itself)  wrote:
>> Hi all,
>> 
>> I find that the signal-to-noise ratio on this mailing list is (by my 
>> determination) quite bad around this time of year.
>> 
>> Is a move to the forum still on the cards? 
>> 
>> Kye
>> On 6 Dec 2022 at 7:16 AM +0800, Andrew Godwin , wrote:
>> 
>>> I did some investigation of moving django-users and django-developers to 
>>> the Forum right after DjangoCon; I wanted to see if we could import all the 
>>> old posts too, which we probably could, but I'm not entirely sure of the 
>>> utility of that. 
>>> 
>>> I will say that the forum is a lot easier to moderate - the ability to 
>>> moderate after a post has gone out, rather than gating all posts behind 
>>> approval if they're untrusted, is a big step in itself, not to mention the 
>>> ability to remove sensitive or offensive content once it's posted.
>>> 
>>> Andrew
>>> 
>>> On Monday, November 28, 2022 at 10:01:17 PM UTC-7 m...@kye.id.au wrote:
>>>> IMO django-announce and django-updates serve a very different purpose and 
>>>> I would be against moving them if it were suggested.
>>>> 
>>>> I am incredibly strongly in favour of moving django-developers and 
>>>> django-users to the forums. IMO being able to more easily trap people 
>>>> misusing this list as a tech support channel is itself reason enough to 
>>>> move. Beyond that, I’d argue that the plentiful UX issues with Google 
>>>> Groups, and mailing lists in general, certainly don’t do the community any 
>>>> favours in terms of getting more people on board. 
>>>> 
>>>> Kye
>>>> 
>>>> On 28 Nov 2022 at 11:40 PM +0800, 'Tobias McNulty' via Django developers 
>>>> (Contributions to Django itself) , wrote:
>>>>> As someone who only just joined the forum -- I'm +1: 
>>>>>  * The forum has seen great adoption from what I can tell (nearly half 
>>>>> the number of posts as django-developers during the same time period, not 
>>>>> bad given the mailing list's head start in subscribers).
>>>>>  * It seems beneficial to house future conversations in a single place, 
>>>>> e.g., so members don't need to subscribe to both the mailing list and 
>>>>> forum to get the full picture of current active development, set up two 
>>>>> different sets of mail filters to tag things appropriately, etc...
>>>>> Would the plan be to switch django-users as well? I think similar 
>>>>> arguments could be made for consolidating those...
>>>>> 
>>>>> (On the other hand, I see little value in switching django-announce and 
>>>>> django-updates, but I'm not necessarily opposed to it either, especially 
>>>>> if there's a way to import the subscribers to those lists...)
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> *Tobias McNulty
*Chief Executive Officer
>>>>> 
>>>>> 
>>>>> tob...@caktusgroup.com
>>>>> www.caktusgroup.com
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Nov 28, 2022 at 9:05 AM Carlton Gibson  
>>>>> wrote:
>>>>>> Hey Roger,  
>>>>>> 
>>>>>> Indeed it does. You can set up Email Mode (that may not be the actual 
>>>>>> name) and it’ll work just like a mailing list. 
>>>>>> 
>>>>>> You can also subscribe to just a particular category, so the Internals 
>>>>>> one would map to the discussion on this list. 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Monday, 28 November 2022, Roger Gammans  
>>>>>> wrote:
>>

GSOC 2012

[Andrew Godwin]

Hello everyone,

As some of you have already noticed, we've been accepted into GSOC 2012 
and we're now starting to discuss ideas with students. The actual 
application period doesn't open until the 26th March, but we'd encourage 
you to start discussing applications on this mailing list before then so 
we can help you get them refined.


Please try and put [GSOC] before any posts you make so we know what 
you're after, and so we can make sure to give you timely responses.


Some key points to remember are:

 - We're looking for detailed proposals - we have an application 
template on the GSOC page at 
http://www.google-melange.com/gsoc/org/google/gsoc2012/django that 
details what you should be including.


 - While we'll happily look over a few drafts of your proposal (in 
fact, it's likely that you will have at least a couple), we won't put up 
with 10 or 20 revisions each with tiny changes. GSOC requires the 
ability to get things done by yourself, and it's best to start 
demonstrating that now. We'll also expect that you have previous 
experience working with Open Source projects or other related


 - The absolute deadline for proposals is 7pm UTC on April 6th, but by 
that point we'd expect that you've already discussed things with us. If 
you just submit an application without prior discussion with the 
django-developers list, it's not likely to get accepted.


Our current ideas page is at 
https://code.djangoproject.com/wiki/SummerOfCode2012, but you're not 
limited to just the ideas on that page, so feel free to propose other ideas.


Looking forward to seeing your applications!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC 2012] Schema Alteration API proposal

[Andrew Godwin]

On 19/03/12 11:08, Jonathan French wrote:

On 18 March 2012 23:33, Russell Keith-Magee mailto:russ...@keith-magee.com>> wrote:

 > 2. An inspection tool that generates the appropriate python code
after
 >inspecting models and current state of database.

The current consensus is that this shouldn't be Django's domain --
at least, not in the first instance. It might be appropriate to
expose an API to extract the current model state in a Pythonic form,
but a fully-fledged, user accessible "tool".


Is there a writeup anywhere of why this is the consensus? AFAICT it
looks like Django already provides half of this in the form of
DatabaseIntrospection, that e.g. South actually uses, which generates a
model class from the current state of the database. Doing the diff as
well doesn't seem like much of a stretch, and might make it more likely
for third party custom fields to be made migrateable, if the interface
for doing so is in Django core.


No writeup that I know of - however, the main part of the work here 
would be the "model differencing" code, which means creating a versioned 
ORM, being able to load and save model definitions to some kind of 
format, and the actual difference-creating code, which is all too much 
to stick into Django.


I've long maintained that I want South to become just that automatic 
differencing code, and to just move the actual database API across; this 
is mostly because I see there being scope for other kinds of migration 
systems apart from the kind South is (for example, a very declarative 
one, whose model states are retrieved using the combination of all 
migrations, rather than a lump on the bottom of the last one).


As for your proposal, Kushagra, Russ has said most of the points I would 
have thought of and a few more - I'd recommend a good look into previous 
discussions on this mailing list for most of the current views on how we 
want the schema alteration API to work.


I would, however, definitely recommend not touching the Oracle or MSSQL 
backends - three is already a lot of work, and they're harder databases 
to get a hold of for testing.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration API proposal

[Andrew Godwin]

On 19/03/12 20:33, Kushagra Sinha wrote:

Andrew's thread[1] also mentions  - "backends will always be able to
generate SQL for operations, but it won't necessarily be runnable
(things like index names can only be resolved at runtime, so you'd get
code like "DROP INDEX <> ON users;"."

[1]
https://groups.google.com/forum/?fromgroups#!topic/django-developers/usFXJvpelmI

Am I correct to assume that the plan is to allow migration files in
python as well as pseudo-SQL like above?
In that case, I think will concentrate on just the core part of
migrations API and nothing else as far as GSoC is concerned.


The actual migration file loading/running system was never intended to 
be part of the GSOC (nor my port when I was planning it) - the idea was 
to get just the database API in for a release, allowing South to lose 
all that code, then work on a migration file/running/dependency API for 
the next one.


There's a lot more potential bikeshedding and design issues with writing 
a migration-running API, so that's one of the reasons it's separated 
out. I'd highly recommend focusing just on the database API - what's in 
South currently can't be ported straight across, and it needs quite a 
bit of cleanup (especially in the SQLite code), so it's still a decent 
amount of work.




Another query:
Andrew's thread above also mentioned:
Some of these operations are already mostly implemented (add_table,
add_index, etc.) in backends' creation modules, but they'll need a bit
of rearranging and separating into a full public API. I also plan to
modify them to take model names and field names, instead of table names
and column names, so the API is exclusively using the Django model layer
to represent changes (there's a possibility that some changes make sense
for schemaless databases as well, specifically renames, so it's best not
to tie it directly to relational databases).

As it happens xtrqt last year had implemented, documented and tested the
migrations API for at least SQLite[2]. However he used explicit table
and column names in his API methods. Obviously he put the task of table
name translation on the API caller. Is there any consensus on the API
design regarding this point?


I feel that table names should definitely be explicit, as models expose 
their expected name. Column names are harder if you accept Django fields 
as valid inputs for the type - South currently uses implicit column 
names in add_table (i.e. _id gets auto-added) and explicit elsewhere. 
I'd rather it was explicit everywhere, and the field's column 
information was ignored in the schema alteration API (it's the migration 
runner's job to feed it the right stuff, I'd say).


Don't rely too heavily on xtrqt's work from last year - the work that 
was done was basically a copy of South's code into a new module with a 
few minor changes. We're looking for a more extensive, clean move, with 
some partial rewriting - in particular, I'd like to make the dry_run 
system more sensible (or possibly drop it in favour of having a SQL 
generation mode that would serve a similar purpose).


SQL generation itself would be a nice feature - it's not always possible 
(delete_index, delete_unique), but something that tries where it can, 
and puts comments/invalid SQL where it can't, would be nice.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Improved Error Reporting in Django - GSoC

[Andrew Godwin]

On 19/03/12 18:12, Sachin Gupta wrote:

Hi,

My name is Sachin Gupta and I am student of Indian Institute of
Technology Roorkee, in my final year of undergraduate course in Computer
Science. I have been developing a project entirely based on Django for
the last 4 months. Here is the link  and
it has been absolutely fun using Django. But I have had my share of
trouble with the error reporting so I wanted to contribute to improve
it. In some of the cases the error reports came up were so unrelated it
took me some time to eventually figure it out.

I would like to list a few of them here:

1) One of the very basic one is that in some cases where we may have an
indentation error, it is not reported which file has that error. It just
throws

indentation error in views.py

For a new developer it may appear that the error is in the views.py file
in the root directory(if one such exists) but the error may actually be
in any of the views.py file of the apps.

2) Another error that bugged me for a long time was related with
comment_was_posted signal. I made a mistake in the listener function for
the comment_was_posted signal So whenever I posted a comment, I got the
error

|IntegrityError  at/comments/post/|

The integrity error was at the listener of the signal, but the url does
not depict that.

3) Another point (it may be intentional) is that if some attribute is
missing in the Queryset or the attribute name is wrongly written (in the
then template) a TemplateError is not raised. It just fails silently.
Personally, I feel that it should throw some error, otherwise it can
lead to some bugs that might be difficult to discover.

These are some of the errors that, if reported would have made things
easier for me. Although I have used django extensively, I am not quite
familiar with the error reporting facility. I have forked the git
repository on github and will go through it in the next few days.

Since this is also listed as a topic for GSoC 2012, I am interested in
applying for that. Meanwhile it would be great if I can be suggested
some pointers.

Cheers

Sachin Gupta


Hi Sachin,

A few comments from myself:

 - The indentation error issue is relatively minor, I'd say - that's a 
core Python error, and relatively uncommon (especially if you use a 
linter, which I always recommend to people)


 - Signals generally can be confusing as they make the control flow 
jump around - however, if you're planning to make it more obvious what's 
happened, we'd like to see a clear description of what you're going to 
improve. The current error page will show you the signal handler towards 
the bottom of the traceback, after all.


 - Silent template failure is a deliberate design decision in Django, 
though it was made many years ago - the idea was that a template editor 
couldn't write a template that crashed the site. If you're planning to 
change that, you'll need to have good reasoning why and backwards 
compatability worked out for all the applications that rely on this 
behaviour (some people rely on it and don't realise it - you'll break a 
lot of sites if you just change the errors in there to be page-breaking).


I'd recommend you have a look through the 
https://code.djangoproject.com/wiki/BetterErrorMessages page in 
particular and look at some of the ideas in there - as each particular 
area is quite small in scope, we'd be looking for a decent number of 
error fixes to form a GSOC proposal that would be considered enough 
work. There's some more meaty errors to fix in django.db, for a start.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Improved Error Reporting in Django - GSoC

[Andrew Godwin]

On 20/03/12 07:18, Sachin Gupta wrote:


Could you guide me what number of error fixes would be good work for
GSOC proposal. Also if there are any class of Django errors that are of
greater concern (like django.db)


There's no clearly defined "number" that would be good - we'll be 
looking at your schedule that's on the proposal, instead, to make sure 
that you've scheduled the fixes with the right amount of time and that 
you've got enough work.


As for which ones are of greater concern, that's a difficult question to 
answer - everyone has their own "favourite" (my particular nemesis is 
the Postgres transaction-is-aborted one). I'd suggest looking through 
django-users, StackOverflow, etc., to get an idea of what's tripping 
people up the most.



For the error message

OperationalError: Unable to close due to unfinalised statements

The context is not correct. I tried using a file that does not have write 
permission for sqlite3, but the error that came up is not what is listed above. 
The error message was quite appropriate in this case

django.db.utils.DatabaseError: attempt to write a readonly database

So is there something which I am not getting correctly?


I'm not sure - I've never had that one myself. It might have only been 
with a previous version of the sqlite library or Python binding - have a 
look through the history on the wiki page to see when it was put in 
there, and by whom.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration API proposal

[Andrew Godwin]

On 21/03/12 13:27, Kushagra Sinha wrote:

One more thing:
The current creation API in django has methods like "sql_create_model"
which basically return sql and it is the caller's responsibility to
either call cursor.execute on it (syncdb) or output the sql itself (sql).

South's (and xtrqt's) design is to have functions like "create_table"
which execute the sql themselves. Makes little difference to commands
like syncdb when they will be rewritten but commands like sql will have
to so something like start_transaction, get_sql, rollback_transaction
which is kinda hackish according to me.

What should be the design of the new migrations api?


Choosing one of these options and justifying it is a decent part of the 
application :)


I'd suggest not having multiple functions to do the same thing, but 
rather a modification of the South API to allow it to output SQL 
(probably replacing the dry-run mode). It won't be possible for all the 
South functions, but crucially it will be possible for the functions in 
.creation you'd be replacing.


Of course, this method carries with it a high testing cost as you'll 
have to essentially remove .creation, migrate all the core code over to 
the new stuff, and add a new shim back into .creation for the bits you 
replaced. You'll also have to consider what happens to apps like 
GeoDjango throughout all this.


You may want to opt for the simpler method and just get .alteration 
working for now and mirror some of the methods. Just make sure that 
whatever you choose, it fits into the schedule, and you can justify it.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Improved Error Reporting in Django - GSoC

[Andrew Godwin]

On 20/03/12 20:33, Sachin Gupta wrote:

It seems most of the errors on this page
https://code.djangoproject.com/wiki/BetterErrorMessages are very old. It
states that if the attribute enctype="multitype/form-data" is not sent
then the following error comes up

TypeError at ...
string indices must be integers

However if this is the case then I am getting a form error, saying that
the file field is required. If the field attribute null is set True then
no error comes up at all.

Would it be a good idea to change this from a 'form' error to a
different kind of error that would state that the enctype attribute is
not set for the given form?


Well, is that error possible to detect sensibly? These tiny questions 
aren't what you should be asking now - we really want to see a 
well-rounded application with a list of what you plan to tackle (it 
doesn't have to be precise, just state the main problem areas), and a 
demonstration of some possible ways you might solve it.


Attempting to pick the errors and the correct solutions now is too 
fragile - if it turns out those errors are hard to fix, or the potential 
error you want to raise isn't working for whatever reason (perhaps it 
interrupts the call stack of something else), you'll need to work around it.


We want to see the ability to think independently in an application - 
your mentor isn't there to continually answer questions for you, they're 
there to check up on your progress now and again and answer the 
occasional design decision/gnarly core Django question. The ability to 
work independently, make good decisions and research/justify them well 
is a big part of GSOC, and something we're looking for.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

[GSOC Announce] Student application period open

[Andrew Godwin]

Just a quick note to everyone to say that the student application period 
for GSOC is now open, and closes on Friday, 6th April.


As I've mentioned on this list previously, please discuss your 
applications with us first so we can give you some feedback rather than 
just submitting them directly - there's much more detail in my previous 
post about the GSOC process.


Thanks,
Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

[GSOC Announce] One day remaining for applications

[Andrew Godwin]

Hi everyone,

Just a quick reminder that there's only one day left for GSOC 
applications - the deadline that they must be filed on the GSOC website 
by is April 6th, 19:00 UTC (roughly 30 hours from now).


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC 2012] Schema Alteration API proposal

[Andrew Godwin]

Just thought I'd chime in now I've had a chance to look over the current 
proposal (I looked at the current one you have in the GSOC system):


 - When you describe feeding things in from local_fields, are you 
referring to that being the method by which you're planning to implement 
things like syncdb?


 - I'd like to see a bit more detail about how you plan to test the 
code - specifically, there are some backend-specific tests you may need, 
as well as some detailed introspection in order to make sure things have 
applied correctly.


- Russ is correct about your models approach - as I've said before in 
other places, the models API in Django is not designed with models as 
moveable, dynamic objects. South has one approach to these sorts of 
tests, but I'd love to see a cleaner suggestion.


- There's been some discussion on south-users about the benefits of a 
column-based alteration API versus a field/model-based alteration API - 
why have you picked a column-based one? If you plan to continue using 
Django fields as type information (as South does), what potential issues 
do you see there?


- Some more detail on your background would be nice - what's your 
specific experience with the 3 main databases you'll be handling 
(postgres, mysql, sqlite)? What was a "high voltage database migration"?


Sorry for the late feedback, I've been far too busy.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC 2012] Schema Alteration API proposal

[Andrew Godwin]

On 06/04/12 06:34, j4nu5 wrote:

Actually I am not planning to mess with syncdb and other management
commands. I will only refactor django.db.backends creation functions
like sql_create_model etc. to use the new API. Behaviour and functionality
will be the same after refactor, so management commands like syncdb
will not notice a difference.


Alright, that's at least going to leave things in a good working state, 
then.



Currently, I can only think of things like the unique index on SQLite and
oddities in MySQL mostly again from South's test suite, I will give another
update before today's deadline.


There's a few other ones that South handles - like booleans in SQLite - 
but a look through the codebase would hopefully give you hints to most 
of those.



Are you referring to the fake orm? Well if you are satisfied with my above
explanation, there would be no need for it, since we will be using django's
orm.


Well, the "fake ORM" is exactly what you described above - models loaded 
and then cleared from the app cache. I'm not saying it's a bad thing - 
it beats what South had before (nothing) - but there could be alternatives.



Well you said it yourself above that "the models API in Django is not
designed with models asmoveable, dynamic objects". That is why I used
a column-based approach. The advantage will be felt in live migrations.
As for using Django fields for type information, I frankly cannot think of a
major valid negative point for now, I will revert later today.



If you plan to continue using
Django fields as type information (as South does), what potential issues
do you see there?

The only issue I can think of is the case of custom fields created by the user.


That's one big issue; one of South's biggest issues today is custom 
fields, though that's arguably more the serialisation side of them. 
Still, I'd at least like to see how you would want something like, say, 
GeoDjango to fit in, even though this GSOC wouldn't cover it - it has a 
lot of custom creation code, and alteration types that differ from 
creation types (much like SERIAL in postgres, which you _will_ have to 
address) and room would have to be made for these kinds of problems.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

GSOC 2012 Projects

[Andrew Godwin]

Hi everyone,

I'm pleased to announce that Django has accepted two proposals for this 
year's Google Summer of Code:


Customizable serialization, from Piotr Grabowski
http://www.google-melange.com/gsoc/project/google/gsoc2012/grap/15001

Security Enhancements, from Rohan Jain
http://www.google-melange.com/gsoc/project/google/gsoc2012/crodjer/24002

They should both be making a post here in the next week introducing 
themselves and their projects - please feed back to them with your 
suggestions and ideas.


Thanks also to everyone who helped discuss the proposals this year.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Proposed Field API additions

[Andrew Godwin]

Hi everyone,

As part of my planning for adding schema alteration/migrations into
Django proper, I need to make a few changes to Fields to allow for
better serialisation of model definitions (pretty much a requirement for
any change-detecting migrations system).

In particular, I propose:

 - Requiring that all fields expose a method which says how to
reconstruct them.

Essentially, it returns the positional and keyword arguments you would
have to pass into the constructor to make the field again (as there's no
way to get them directly). For those familiar with south_field_triple,
it would be a bit like that.

There will have to be some restrictions on what can be returned as valid
values - no values that are pre-made anonymous functions, for example -
but that can be cleared up later.


 - Requiring all fields to be accessible by only their app label and
class name/other unique name.

This means either having to register custom fields (like admin classes,
for example), or requiring fields to live in a fields.py or fields
package (like models and models.py). This is to provide for a
less-fragile way of referring to them than their full module path (which
might change based on project name or package location).

Neither of these two options is perfect - registration means a little
more "boilerplate" code, while requiring them to be in a certain module
is going to hurt apps that don't have them there already. For that
reason, I prefer the registration approach - it will only be one extra
line per field, it's a pattern already used for admin classes and
filters/tags, and it will allow for field classes to be renamed while
keeping the same registered name (if someone wants).

I'd appreciate feedback on these general ideas - a more concrete API
proposal will come later along with details about how I plan to approach
the rest of the problem, but this is one of the few direct changes to
Django's core and so needs dicussion first.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 07/06/12 20:14, Alex Gaynor wrote:
>
>
> On Thu, Jun 7, 2012 at 12:17 PM, Andrew Godwin  <mailto:and...@aeracode.org>> wrote:
>
>
> In particular, I propose:
>
>  - Requiring that all fields expose a method which says how to
> reconstruct them.
>
> Essentially, it returns the positional and keyword arguments you would
> have to pass into the constructor to make the field again (as
> there's no
> way to get them directly). For those familiar with south_field_triple,
> it would be a bit like that.
>
>
> This sounds similar to pickling's __getinitargs__, is there anyway we
> can reuse some of:
> http://docs.python.org/library/pickle.html#object.__getinitargs__

Hmm, it's close, but by not allowing keywords it's going to be very
unsightly for something with optional arguments. Additionally, we're
going to need to restrict the output from any function like this to
values we can serialise as text (basically, repr plus a few custom
handlers for things like model instances), and getinitargs doesn't
enforce that either.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 07/06/12 21:56, Anssi Kääriäinen wrote:
> Is the reason for this to be able to track changes to field by
> checking if its init arguments have changed? Why is it not possible to
> track changes by checking the SQL output the field will generate
> instead? This is guaranteed to be a string, and should be readily
> available for any field. I am just trying to get up to speed here... 

No, it is so a model can be instantiated for any point in time
(essentially a versioned ORM). This is needed so that any custom
python/ORM code in migrations see the models as they were when they were
created, and thus won't get weirded out by the columns not matching.

There was a decent explanation about why and how models are resurrected
like this on south-users ages ago, but I think that post may have been
lost to the voids of time.

(Also worth pointing out: fields don't have just one SQL definition,
they can have many - not only for different database backends, but also
different ones for creation-with-the-table and being-added-later).

> As an idea, why not mimic the models.Model.__new__. You could have
> Field.__new__ which does the registration. A field could have
> app_label and "field name" as Meta (or just base level arguments). If
> these are not supplied, they will be generated by the __new__ from the
> field's class.__name__ and module. Once backwards compatibility period
> is over the meta variables are required.
>
> Any field which is never imported will never get registered, but I
> don't see that as a problem - if the field is not in use in the
> project, why should it need to be registered?

As some of the other posts argue, that's also quite fragile - I'm not
sure there's a guarantee all field classes will be unique (especially as
fields have been around for ages). It also needs metaclasses - you can't
do it in Field.__new__, you need to do it in the metaclasses' __new__,
and if there's one thing I want to avoid it's adding more metaclasses.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 08/06/12 18:16, Michael Manfre wrote:
> 
> 
> On Thursday, June 7, 2012 4:16:12 PM UTC-4, Alex Ogier wrote:
> 
> This isn't particularly robust. The SQL string generated by a
> particular backend isn't considered part of any API, and might change
> formatting or semantics on minor updates. Some SQL is quite
> complicated and dubiously related to the particular field, such as
> extra INDEX or UNIQUE constraint statements. Certain backends elide
> some arguments, performing constraint checking in the application (the
> sqlite backend for example just drops many constraints). In general,
> you want the migrations to be DB-agnostic, and checking SQL output
> loses this feature.
> 
> 
> While the entire raw SQL generated by a field may be too unstable to
> inspect, the underlying SQL shouldn't be completely ignored. At the very
> least, the datatype needs to be inspected when freezing to detect if the
> rug has been pulled out from under the ORM.
> 
> A real world use case. Django-mssql uses the datetime datatype for
> datetime fields. Several years ago, Microsoft made the recommendation
> that all new work should use datetime2. A future version of django-mssql
> will switch to using datetime2. It would be great if the proposed
> migration framework handled that sort of switch, or at the very least
> displayed a warning about the need for manual user intervention.
> 
> The same type of problem could be encountered if a database backend
> becomes unsupported and users are forced to switch to another backend
> where the maintainer follows a different set of "best practices". There
> are many ways of storing the same data that are all completely valid.
> The most likely problem field will be IPAddressField due to varchar(15)
> vs. unsigned int.

I'm not sure it's going to be sensible to have the migration framework
change things when the apps themselves have not changed, only the
backend - Django already does a reasonable job of abstracting types, and
that will need to be slightly improved but not radically changed for
this work.

The situation where someone changes a database backend completely and
points it at the exact same database is going to be rare - far more
common (I suspect) is someone changing database server and database
backend in sync, and for that you DO want to ignore the SQL completely -
if I go from PostgreSQL to MySQL (heaven forbid) I don't want South
thinking it should still try and use an IP column type.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 08/06/12 16:16, Anssi Kääriäinen wrote:
> I did a little digging into South code, and I think I now understand
> the needs. Basically, when you migrate a model, you might need to read
> the database data by using the old model definitions. You can't use
> the currently installed model definition because it does not match the
> database state. So, you read in by using the unfreezed old model
> definition, do needed changes to data, do actual schema migrations,
> then continue to next freezed state, do changes to data, migrate
> schema and so on. Even if you don't need the ORM for data migrations,
> you might need the old field definition: how to generate a column's
> SQL for a field in multidb friendly way if you don't have the field
> available?

Precisely.

> The problem with the above approach is that if for example a field
> __init__ definition is changed (or the user deletes unneeded field
> code) there goes your ability to do migrations. The migrations do not
> contain all the needed data to do the migrations. You should have the
> fields available for your whole migration history, and the fields
> should not change in ways that break the unfreeze. This is also the
> reason why identifying fields uniquely is so important - to unfreeze,
> you need the original field class. If you can't get the original field
> definition back, your migration is nearly worthless.
> 
> The dependency on having the field code available is by design. This
> is somewhat fragile no matter what you do, because you might need
> historical code to be able to run your migrations. But, it seems there
> is no way around this if you want to have multidb capabilities for
> custom fields, or any kind of versioned ORM capability available.

Right. There are obviously limits to any approach, as (as I've seen
people do) you can generate models dynamically at runtime as the
application is loaded - that said, migrations are not going to be a
required feature, and so some limits on how you're allowed to arrange
your models seems sensible.

> 
> I favor the explicit registrations API with free format strings. The
> idea would be that the field author will do:
> migrations.register('mypackage:somefield', SomeField)
> If he needs to later do incompatible changes to the field, he could
> change the registration to
> migrations.register('mypackage:somefield:v2', SomeField)
> Likely field versioning isn't commonly needed. At some point old
> migrations lose their value, and incompatible changes in fields aren't
> that common to begin with. Still, if the registration string is just a
> string, then the ability to have the ':v2' in there comes for free.
> For example, Django's XMLField could be resurrected for migrations by
> downloading its code, and registering it to 'django:xmlfield' (or
> whatever the key was previously).

Yes, I'd say versioning is unnecessary, as you can just use a different
name - the new migrations stuff will also include a much easier way to
delete older migrations from the history and thus wipe out-of-date
references, while still keeping things the same in the database.

> 
> The __init__ args and kwargs needed for unfreeze should be asked
> directly from the Field, just as Andrew suggested in his original
> mail. If a field is registered to migrations, then it will need to
> provide "get init arguments" method.
> 
> I think I am finally up to speed with this discussion... In short: +1
> for field registering, -0 for automatic fields.py introspection, +1
> for "get init arguments".
> 
> To me it seems there would be room for different kind of migrations,
> too. For example, just record the raw SQL needed. This is the method I
> am using (manually) currently. This is totally multidb unfriendly, but
> it doesn't matter for some use cases. It seems South migrations files
> could be used in this way, too, but there isn't support for recording
> raw SQL changesets between current database state, and current
> models.py.

There's no way of doing a raw SQL method without extra code as Django
will never emit ALTER TABLE statements by itself (otherwise, we could
just capture it from the cursor during syncdb or something).

The new format will include provision for just having a big block of raw
SQL, though - so if someone wants to fall back to doing that it's pretty
easy.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 08/06/12 16:45, Jacob Kaplan-Moss wrote:
> Hi Andrew --
> 
> Generally I'm +1, and I think I see the point pretty clearly. Just a
> couple of questions:
> 
> On Thu, Jun 7, 2012 at 7:17 PM, Andrew Godwin  wrote:
>>  - Requiring that all fields expose a method which says how to
>> reconstruct them.
>>
>> Essentially, it returns the positional and keyword arguments you would
>> have to pass into the constructor to make the field again (as there's no
>> way to get them directly). For those familiar with south_field_triple,
>> it would be a bit like that.
>>
>> There will have to be some restrictions on what can be returned as valid
>> values - no values that are pre-made anonymous functions, for example -
>> but that can be cleared up later.
> 
> I'm not sure I follow this limitation -- you're not saying that things
> like `datetime.now` would be verboten, right? At least for defaults,
> callables are pretty common.

No, that'd be allowed (and already is in South) - it's just edge cases
like, for example, def'ing a function inside the method that returns the
arguments and then using that as a default - there's no way to serialise
that other than by using its code object.

The basic restriction will be that you're allowed callables, but they
must be importable by another module and that they're available as long
as the migration they're in exists. That second point will be helped by
a much easier way of collapsing old migrations and removing out-of-date
references.

> 
>>
>>  - Requiring all fields to be accessible by only their app label and
>> class name/other unique name.
>>
>> This means either having to register custom fields (like admin classes,
>> for example), or requiring fields to live in a fields.py or fields
>> package (like models and models.py). This is to provide for a
>> less-fragile way of referring to them than their full module path (which
>> might change based on project name or package location).
> 
> Can't this be done by auto-discovering subclasses of Field
> (Field.__subclasses__)?

It could - however, that would mean that we'd have to enforce unique
naming rules for fields inside an app (not really a problem), and that
people making fields inside a function or other weird situation might
have those fields detected (could cause some weird issues).

The other problem with that is the inability to change a field's class
name without breaking old migrations (whereas with a template-tag-like
registering approach you could optionally provide a different name to
register it as, thus allowing it to exist under two names).

Still, it's a hard balance to strike, really - registering is still a
little clunky, and we can't (yet) rely on things like class decorators
to make it nicer.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 08/06/12 17:42, Carl Meyer wrote:
> Hi Andrew,
> 
> On Thursday, June 7, 2012 11:17:51 AM UTC-6, Andrew Godwin wrote:
> 
>  - Requiring that all fields expose a method which says how to
> reconstruct them.
> 
> Essentially, it returns the positional and keyword arguments you would
> have to pass into the constructor to make the field again (as
> there's no
> way to get them directly). For those familiar with south_field_triple,
> it would be a bit like that.
> 
> There will have to be some restrictions on what can be returned as
> valid
> values - no values that are pre-made anonymous functions, for example -
> but that can be cleared up later.
> 
> 
> If you're allowing named callables, but not anonymous ones, that implies
> that you'll be looking up the named ones by import path? Does this not
> present the same issue as with Fields - if the callable moves, the
> migration breaks? Or am I misunderstanding?

See my reply to Jacob about the same situation - yes, that's the case,
but it's mitigated somewhat by an easy way of collapsing old migrations
(with bad callable references) down. This proposed one would be just
like how fields currently work in South.

> 
>  - Requiring all fields to be accessible by only their app label and
> class name/other unique name.
> 
> This means either having to register custom fields (like admin classes,
> for example), or requiring fields to live in a fields.py or fields
> package (like models and models.py). This is to provide for a
> less-fragile way of referring to them than their full module path
> (which
> might change based on project name or package location).
> 
> Neither of these two options is perfect - registration means a little
> more "boilerplate" code, while requiring them to be in a certain module
> is going to hurt apps that don't have them there already. For that
> reason, I prefer the registration approach - it will only be one extra
> line per field, it's a pattern already used for admin classes and
> filters/tags, and it will allow for field classes to be renamed while
> keeping the same registered name (if someone wants).
> 
> 
> Yuck. I am not at all convinced that this cure isn't worth than the
> disease. In every case where Django has introduced flattened
> pseudo-namespaces in place of Python's existing namespace system, I
> think it's come back to bite us later.
> 
> How bad is it really to do nothing here? If I understand correctly, it
> would simply mean that you have to keep a Field class always importable
> from the same location, or else manually fix the import location in
> older migrations. Frankly, given how rarely I've seen this issue in
> practice with South, I think this limitation is perfectly fine, and
> much, much, better than introducing a whole new registration machinery
> and flattened pseudo-namespace for Fields.

You're entirely correct that that's the restriction, and given that I'm
imposing precisely the same restrictions on default callables, you may
well be right here - after all, "Namespaces are one honking great idea
-- let's do more of those!".

I think, unless anyone has an objection I can't think of, we should do
what you suggest, and just use full paths; after all, third-party apps
will have to keep aliases to fields in the right place so imports don't
break - registration would just be giving them one more thing to add an
alias to.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 08/06/12 20:01, Anssi Kääriäinen wrote:
> On 8 kesä, 19:42, Carl Meyer  wrote:
>> Yuck. I am not at all convinced that this cure isn't worth than the
>> disease. In every case where Django has introduced flattened
>> pseudo-namespaces in place of Python's existing namespace system, I think
>> it's come back to bite us later.
>>
>> How bad is it really to do nothing here? If I understand correctly, it
>> would simply mean that you have to keep a Field class always importable
>> from the same location, or else manually fix the import location in older
>> migrations. Frankly, given how rarely I've seen this issue in practice with
>> South, I think this limitation is perfectly fine, and much, much, better
>> than introducing a whole new registration machinery and flattened
>> pseudo-namespace for Fields.
> 
> Why not just import the needed fields explicitly in "needed_fields.py"
> as part of the migrations - the file would be autogenerated by the
> migrations machinery.
>
> This isn't a big change to what currently happens in South - only
> change is that the import is done explicitly in a .py file, instead of
> writing the import path in the freeze file. But, this way it would be
> pretty obvious that the migrations need the fields in their original
> position. If something is deleted or moved you will get an import
> error. The user can fix it manually. You would still have problems if
> a field is updated in incompatible ways, but I don't see any way
> around that issue except of freezing the whole Python path for each
> migration.

Eugh, autogenerated python files full of imports? Plus, you'd
potentially need more than one (for different migration versions), and I
don't see what improvement this is over full paths, which is what Luke
proposed, what South currently does, and it works reasonably well.

Remember, if a path does get hilariously outdated then there'll be a new
option to collapse old migrations into one new initial migration, wiping
out the outdated references in the process.

> 
> I am afraid the unfreeze can't be made totally robust by better core
> support. There simply can be no guarantees of having the code
> dependencies around.
> 

Oh, I entirely agree, I'm just trying to improve it. There's going to be
no solution that works 100%, and part of using any migrations library
will be a documented set of limitations like this. However, they'll work
for 99% of cases, and I think that's a big improvement.

> I would like there to be alternative solution for those who want to
> record just the upgrade SQL. It doesn't need freezed model states at
> all, and is thus more robust. If there is core migrations support,
> then there should be option for this, even if all the SQL needs to be
> hand written.

That'll definitely be there - South has never needed the frozen ORM, and
this won't either. If you want, you can just use sql: and depend:
stanzas in the new migration files, and write everything manually - and
you'll still get the benefits, like state tracking, transaction
management around DDL and dependency resoution. Complete raw SQL support
was one of the main points on my design notes for this.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 10/06/12 21:54, Anssi Kääriäinen wrote:
> I agree the needed_fields.py idea was bad. One part of the idea was
> that you would get immediately broken migrations if you remove a
> field. Do all the migrations break if you have a broken field
> reference, or only when you actually use a migration referencing the
> field?

In the new system all the migrations in the same app would immediately
break (and it would print out a nice explanation showing exactly where
the issue was) - that's a side-effect of deducing the current model
state from the previous migrations, and probably better than only
failing at some mysterious future point.

> Is there anything else tied to the fields identifier than just how to
> get the field class back when unfreezing? The rules in South seem to
> reference the module path in some way which is why I am asking.

Well, the module path is in there so you know where to get the field
from - as Luke suggested, that's probably just as good a way of doing it
as registration, and involves not introducing another namespace.

South's rules also include provision for saying which fields are safe or
unsafe to freeze - that's only because it has to enact this system via
entirely third-party means. Once it's a part of the Field API, checking
if it's there becomes a lot simpler.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposed Field API additions

[Andrew Godwin]

On 11/06/12 10:27, Anssi Kääriäinen wrote:

> All of the above sounds good - my main worry was that if you subclass
> a field, then it will not get a rule match as the module path prefix
> will be different than the parent field's. I don't know if this is an
> issue even in South... But if the rules are part of the field API,
> then there is no worry. Are the rules even needed after the
> introduction of "get_init_args()"?.

Nope, the whole point of introducing this stuff is to get rid of the
rules. That will make me a happy man.

> The only possible problem with using the module path as the identifier
> is the inability for the user to replace removed fields with copied
> backwards compatibility code. As this is probably a non-issue (and
> possible to work around) I will surprise you all and not complain
> about this :)

I think that the case of a field being removed completely is rare enough
that this is acceptable (having to use a workaround). Most times a field
becomes unimportable it will likely have moved, in which case a good
number of apps would keep the old import working (as code probably
depends on it).

But yes, I'll stop now before you complain again :)

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: QuerySet refactoring

[Andrew Godwin]

On 13/06/12 17:15, Luke Plant wrote:

> I think this is a very necessary piece of work. The problem with that
> layer of code is that it is very difficult to really grok and therefore
> to review patches. It would take almost as much effort to do a review of
> a substantial patch as the patch itself. You have to really understand
> what Query is doing, and it is a huge class, that takes hours to even
> vaguely get the hang of and get in your head.
> 
> I would like to be able to help, and if I dropped everything else I
> might otherwise do on Django, that should be possible. I think with two
> pairs of eyes on these changes, and with a good test suite, we should be
> reasonably safe, but getting more review than that would be hard.

I'm happy to look over any patches as well - while I generally know more
about the actual backends rather than Query, the more eyes the better on
this kind of stuff, I think.

> I do actually have another proposal, that has been a growing draft email
> on my computer for a few weeks: that we rewrite our backend using
> SQLAlchemy Core (not the ORM). I've used SQLAlchemy a bit, and I'm very
> confident we could use it profitably. While the ORM works significantly
> differently to ours, the lower layer (Core - the SQL Expression
> Language) can indeed build queries that can do all the joins etc that we
> would need - it's a full wrapper around SQL. Some of the complexity in
> our code comes from not having this wrapper.
> 
> SQLAlchemy is a really excellent library, and I fear that long term if
> we don't switch to it, we will just produce an ad-hoc, informally
> specified, buggy, slow implementation of half of SQLAlchemy.

I'd love to see the full proposal for this - it's been a while since
I've heard rumblings about SQLAlchemy, and I've forgotten what the major
objections/drawbacks were.

It would certainly save me some work if I could rely on it for some DDL
code generation, though because of the way models are defined in terms
of Fields that's unlikely to be possible soon without some serious
mapping and wrapping.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test runner with search

[Andrew Godwin]

On 14/06/12 11:51, Anssi Kääriäinen wrote:
> On 14 kesä, 13:35, Chris Wilson  wrote:
>> I've made some improvements (in my view) to the DjangoTestSuiteRunner. I
>> got tired of having to remember my test class names and of typing so much:
>>
>>   ./manage.py test binder.BinderTest.test_can_create_users
>>
>> This new version searches for tests with the given name in all
>> INSTALLED_APPS, so you can just type this:
>>
>>   ./manage.py test test_can_create_users
> 
> A big +1 to easier running of single tests. I haven't checked the
> implementation / API details at all. But I do know I would have
> immediate use for this feature in running Django's tests.

+1 from me as well - the test runner you linked to looks reasonably
sane, only a few small things that I'm questioning (like why types is
imported inside the for loop).

> Another feature which I would find useful would be "--log-failed-
> cases=somefile". Then you could re-run the failed tests using:
> 
> ./manage.py test < somefile
> 
> I would find this very useful in fixing regressions - run the full
> test suite - fix regressions - try to run the failed cases - fix more
> - run the failed cases again - when done rerun full test suite.
> 
> Anybody else for this feature?

Interesting idea - I can see that being useful, too. Presumably we're
just talking an output with one test function reference per line?

Not sure about having to read from stdin - I can see that potentially
interfering with things like pdb.set_trace() - but there's always the
option of adding a command-line flag too.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: auth_permission column lengths

[Andrew Godwin]

On 19/06/12 15:25, Stephan Jaensch wrote:
> Hi Florian,
> 
> Am 19.06.2012 um 16:12 schrieb Florian Apolloner:
> 
>> Django itself can't change that currently since there is no
>> support for schema alteration in Core. Once we get that we can
>> tackle issues like that and increase to a sensible limit. (both
>> name and codename might cause problems…).
> 
> How is this a factor if the limit exists only at the database level
> and is not enforced or considered in Django code? Django code
> already creates values that are too long for the fields. You would
> just eliminate this bug on new installations. In the case of
> usernames and email addresses, the limit was enforced in Django
> code, so increasing the field length could break existing
> installations since we can't do automatic schema migrations. But in
> this case, things already break since Django does not care about
> the field length.

I agree, this is one of the few cases where it's possible to get away
without a schema migration, as the behaviour if we made this change
would be exactly the same on old installations and it would be fixed
on newer ones.

Of course, this just leads to a higher limit after which you have
values which are too long, but if your class names are getting over
200 characters long I suggest you have other issues.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: auth_permission column lengths

[Andrew Godwin]

On 19/06/12 16:13, Greg Aker wrote:
> Florian:
> 
> I don't think waiting for migrations in the Django core is totally
> necessary to fix a bug like this (or others that might be similar).
>  With proper documentation in the release/upgrade notes, I think it's
> completely reasonable to expect someone working with Django to be able
> to run a manual SQL query to alter those columns.
> 
> If this is a core philosophy not to ask users to run manual queries on
> updates, is starting with a patch to enforce limits here a good thing?

It's messy to ask people to manually run SQL queries to change this
stuff in general - we'd have to provide four or five different queries,
and the operation isn't even possible on SQLite (you'd have to make a
new table with the right schema and copy things over). I'm working
studiously on getting schema migration stuff in, it'll just take a
release or two till it's ready.

As for the interim solution, I replied to Stephan's post about that - I
think just increasing max_length will work well enough for now in this
case, as it's one of the few situations where the schema is slightly
decoupled from the models.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Plans for "forms"

[Andrew Godwin]

On 21/06/12 11:58, Klaas van Schelven wrote:

Hi all,

I'm not entirely sure about the forum to ask this question: the
motivation is "django-users" like; but since it's about a roadmap /
the future I suppose only developers will know the answer.

I vaguely remember there being mention of big plans to do a big
refactoring of Django's forms/modelforms/formsets functionalities at
DjangoCon EU 2011. I'm not entirely sure what these plans entailed, if
there were ever such plans or if they've already been implemented. If
anything is surely to come up for Django 1.5 we'll postpone some
refactorings in our own code untill we know what the new canonical way
of doing this is.


Hi Klaas,

I suspect you're thinking of the form templating refactor (making all 
the form widgets use templates that can be overridden etc.) - this was 
done for GSOC last year but unfortunately it's blocked on us improving 
the template engine - it's too slow in the current one to be workable.


There was also a GSOC project that same year to make the templating 
faster, but that unfortunately didn't result in something we could use.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal: use SQLAlchemy Core for query generation

[Andrew Godwin]

On 30/06/12 15:22, Luke Plant wrote:
> Hi all,
> 
> A good while back I put forward the idea of using SQLAlchemy Core in
> Django [1]. Having had more experience working with SQLAlchemy, I'm
> putting that idea forward as a formal proposal, as I mentioned in a more
> recent thread here.
> 
> Apologies in advance for the length! I've included a few 'TL;DR'
> summaries and headings in the different sections which you might want to
> scan first.
> 
> === Proposal ===
> (snip)

Nice to see this after you mentioned it a couple of weeks ago, Luke! I'm
very slightly in favour of the whole idea, but I'm skeptical about the
amount of work this will require in the DB backend, and what that means
for the ORM's large number of little tricks that people are used to
using. It's going to be a big change for them.

One area I'm particularly interested in (naturally) is the DDL
generation stuff - I'm in the middle of writing new full DDL (i.e.
ALTER, DROP) backends for Django, and obviously that comes with its own
set of DB-specific changes and workarounds.

It looks like SQLAlchemy Core would help that aspect, but only slightly
- most of the code is in mapping operations on models to operations on
tables, and I don't think SQLAlchemy Core doesn't have any support for
mutating SQLite (for that we'd need to turn to SQLAlchemy-Migrate).

Basically, what I'm saying is that while I'd jump behind it if it would
save me this work in the short term, I don't think it will - otherwise,
I'd propose we start the gradual introduction of it in the new DDL
modules, and get the ball rolling. However, it looks like it would end
up resulting in far more work than sticking with what's there (mostly
due to having to fiddle with things like how fields specify types and
relationships and tables work).

Still, I'm +0 on the whole idea.

Andrew


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal: use SQLAlchemy Core for query generation

[Andrew Godwin]

> I am pretty sure SQLAlchemy-Migrate has fallen out of favor. You should
> check Alembic, http://alembic.readthedocs.org/en/latest/. 

Ah, interesting. Alembic doesn't appear to support fully mutating SQLite
databases, which is the really gnarly part of the South code I'd love to
outsource. Looks like there's still plenty of work to be done no matter
what happens.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django should load custom SQL when testing

[Andrew Godwin]

On 26/07/12 19:33, Anssi Kääriäinen wrote:

On 26 heinä, 19:35, Andrei Antoukh  wrote:

Having the hooks as a method or function, I find it much more interesting
than having them in files. Above that allows logic in these methods, which
with flat files is not possible.

I also like raw SQL in Python hooks better. An example why this is
preferred is that we will likely have settings configurable db_schema
one day in the future. How to use the correct schema in raw SQL files
when you don't control the db_schema? This could happen for example in
3rd party apps.

While I am likely making this feature request more complex than it
needs to be, I do think the interaction with migrations is an
interesting aspect.

For normal model definition changes the migrations framework spots the
changes, and you get the migrations for free. Thus, you need to define
the changes in just one place.

For raw SQL it is impossible to automatically create migrations based
on changes in the SQL. Maybe the answer to this is that you have to
define the migrations separately, and the current state separately.
Unfortunately this could lead to situations where you test on
different schema than what your production setup will be after
migrations. Thus, it would be nice to define the changes just once to
ensure you can't have state errors between production and testing.
And, as you can't do the "define just once" in the raw SQL hooks, it
must be the migrations which contain the just once setup.


Chiming in here on the migration front, my proposal for the next-gen 
migration solution that'll potentially land in Django includes support 
for just running chunks of raw SQL, so there's potential to auto-include 
those in an initial migration.


However, as you quite rightly point out, detecting changes is a pain. 
Still, South sends the post_syncdb signal after model creation anyway, 
so there's potential here for people who are lazy and don't want to port 
their SQL across to migrations to have it Just Work. (and then Just Fail 
when they want to change it). It should be good enough until I get the 
migration stuff in, I'd say.


Andrew

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC Check-in: Security Enhancements

[Andrew Godwin]

Thanks for your work during the GSOC, Rohan - don't worry about not
achieving everything, it looks like there's still some useful code there!

Hopefully we can get some of the code merged, especially centralised
tokenisation if it's so near completion, as it looks like a nice bit of
cleanup code!

Andrew

On Mon, Aug 20, 2012 at 2:49 PM, Rohan Jain  wrote:

> Hi,
>
> Today is the 'pencils down' date for this GSoC project. Past 4 months
> have been a great learning experience from me. Just being in the
> context of security side of the web has been really beneficial. Moving
> around in a very well written code base is also delightful.
> Meanwhile, I did get to work on multiple sections of the code base.
> But compared to the expectations at the initial phase of the program,
> I myself am a little disappointed about the output I was able to
> generate out of the time - specially regarding CSRF.
> This can be attributed to my being unable to give consistent attention
> to the project through the duration and to solicit feedback. Also, the
> project comprised of multiple discreet tasks, so not enough
> consideration was there for each task.
>
> This is a overview of various complete/incomplete topics I worked on.
>
> Centralized Tokenization:
> This was about integrating a central tokenization api throughout the
> django project. I tracked this at [centralized-tokenization][0] branch
> of my fork on github. This is near completion, and also has a
> documentation[1] of the API with basic usage examples.
>
> Improvements to sessions:
> My first task, related to some improvements to the sessions framework.
> Includes signing and session cleanup based on the engine. Tracked at
> [sessions-improvements][2] branch.
>
> CSRF Enhancements:
> It occupied most of my time through the project. I tried multiple ways
> to solve the issues under this. Tracked at [csrf-enhancements][3].
> First up, I started with using the origin header for doing a CSRF
> check. One implementation with some tests is at
> [csrf-origin-checking][4]. Recently I started on the idea of
> implementing moduler checkers for each kind of CSRF check, but
> haven't got anything useful out of it yet. While progressing, it
> seemed like I was virtually writing a middleware per checker, so now I
> have moved on to attempt on CSRF cookie store. Basically something
> which should seamlessly switch Session/Cookie for storage based on the
> availability of session.
>
> I'll try to hang around the django bug tracker and submit some
> patches.
>
> --
> Thanks
> Rohan Jain
>
> [0]: https://github.com/crodjer/django/tree/centralized-tokenization
> [1]:
> https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt
> [2]: https://github.com/crodjer/django/tree/sessions-improvements
> [3]: https://github.com/crodjer/django/tree/csrf-enhancements
> [4]: https://github.com/crodjer/django/tree/csrf-origin-checking
> [5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers
>
> On 01:57 +0530 /  7 Aug, Rohan Jain wrote:
> > Hi,
> >
> > Sorry for the delay in getting back. I was meanwhile working on
> > centralized tokenization for few days, while still trying to figure
> > something better for CSRF.
> >
> > On 03:52 -0400 / 25 Jul, Alex Ogier wrote:
> > > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain 
> wrote:
> > > >
> > > > I had one more idea, "Pluggable CSRF checkers".
> > > >
> > > > Currently, the CSRF middleware has two kinds of checks, referer (for
> > > > https) and secret validation token (common). These with origin header
> > > > based checker (if we add it) come in conditional blocks, making
> > > > switching them difficult. So what I propose to do is decouple their
> > > > logic from CSRF middleware and each of them provide a checker. It
> goes
> > > > like this:
> > > >
> > > > A setting for configuring global CSRF checkers:
> > > >
> > > > CSRF_CHECKERS = {
> > > > 'django.middleware.csrf.checkers.OriginChecker',
> > > > # This one can be strict for https and lax for http
> > > > 'django.middleware.csrf.checkers.RefererChecker',
> > > > # contrib.sessions could provide a csrf checker maintained
> > > > # with sessions. This stores the token in session data.
> > > > 'django.contrib.sessions.csrf_checkers.SessionChecker'
> > > > }
> > > >
> > >
> > > I don't think this is a good idea. If you enumerate security features
> > > in settings.py, then later additions won't be picked up by default. If
> > > Django add a new CSRF checking mechanism, we want everybody to take
> > > advantage of it with no modifications.
> > >
> > > Ordinarily I agree with you, explicit is better than implicit.
> > > However, in the case of security features, I think this is inverted:
> > > Django sites should be implicitly enrolled in all security mechanisms
> > > if possible, and should be able to explicitly opt out if necessary.
> > > Almost everyone should be using every single protecti

Re: Backporting some Python 3 support features to 1.4.x?

[Andrew Godwin]

I'm definitely +1 on this - I have a few codebases I want to start
converting but also want to keep running on 1.4, and the patch looks
sensible to me. There is precedent for this, and even if there wasn't, this
is a nice way to get the migration cycle started.

Andrew

On Thu, Sep 6, 2012 at 3:05 PM, Donald Stufft wrote:

>  Just as an additional aside, the apps can also depend on the actual six
> library itself instead of Django's embedded version (It could be an
> optional dependency on Django < 1.5). The major things I think would be
> anything Django specific that don't come from six.
>
> On Thursday, September 6, 2012 at 2:09 PM, Aymeric Augustin wrote:
>
> Hello,
>
> Several people have started porting their apps for Python 3, but they're
> running into trouble when they try to support both Django 1.4 and
> master. They end up with regrettable patterns such as:
>
> try:
> from django.utils.six import PY3
> except ImportError:
> PY3 = False
>
> Authors of pluggable apps generally chose to support the current and the
> previous version of Django. Thus their users can alternatively upgrade
> Django and their apps, which makes upgrades less scary.
>
> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5 +
> 1.6 — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
>
> We generally don't add features to minor releases, but there's a precedent
> (csrf_noop), and the patch is as harmless as they come (see attachment).
>
> What do you think?
>
> --
> Aymeric.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
> Attachments:
>  - python3-forwards-compat-for-django-14.diff
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Andrew Godwin]

That's an incredible speedup - I've had a quick look over the patch, and it
looks to be doing all the right things, so I'd definitely be behind merging
this in. Have you tried running the test runner over some third-party apps'
tests to make sure it works? I suspect South's/migrations' might get weird,
but then I do nasty things and that's my problem to fix.

Andrew

On Thu, Sep 6, 2012 at 4:35 PM, Anssi Kääriäinen wrote:

> On 6 syys, 19:17, Anssi Kääriäinen  wrote:
> > I now have a pretty good WIP approach of tracking changes in testing.
> > The changes can be found from here: [https://github.com/akaariai/
> > django/tree/fast_tests_merged]. The approach relies on existing
> > signals + a new "model_changed" signal, which is used to signal
> > bulk_create and .update() changes, and could be used for raw SQL
> > changes, too.
> >
> > The results are promising:
> >   - PostgreSQL with selenium tests, master 33m4s, patched: 9m51s
> >   - SQLite, no selenium tests, master: 7m35s, patched: 3m6s.
>
> I did a little bit more tuning, and now the runtime for SQLite is
> 2m34s (of which 30s is spent in time.sleep()). I can't see any more
> easy targets for performance improvements. The only thing that comes
> to mind is making fixture_loading use bulk_insert, but that has some
> problems (signals and multitable models aren't well supported by
> bulk_create).
>
> The changes done compared to the last try:
> - Made validation skip already validated models
> - Made contenttype fetching in permission creation more efficient
> - Skip permission creation when there is nothing to create
>
> I also tested MySQL and Oracle, MySQL was around 9m30s without
> selenium tests and Oracle ran in 16m, no selenium tests. On Oracle the
> database creation takes 7m, actually running the tests 9m.
>
> For those interested there is a pyprofile at:
> http://users.tkk.fi/~akaariai/sqlite.pyprofile.gz
>
> As said, there doesn't seem to be anything too easy to squash any
> more.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Schema Alteration - Review needed!

[Andrew Godwin]

Hi all,

I have now, I believe, got a working, feature-complete schema alteration
branch ready to go. It has full support for PostgreSQL, MySQL, and SQLite,
and follows the rough design principles I emailed the list about a while
back.

It's fully tested, but no documentation yet - I'm not sure what level of
documentation we should give this, but it is my intention for it to be an
API usable by third-party apps. In the meantime, the tests file should give
you a reasonable idea of the way the API can be used (if you ignore the
slightly cheap way to make new model classes - 'migrations' has a much
nicer one of those).

It should be Python 3 compatible, too - I've checked the major pain points
and run the test suite under Python 3, and it all passes.

The branch is schema-alteration in https://github.com/andrewgodwin/django
Handy link to the diff-with-master view:
https://github.com/andrewgodwin/django/compare/schema-alteration

Feedback is very much welcomed - I want to get this merged in before the
feature freeze.

There's doubtless some polishing and bugfixing that will be needed before
final release, but I'll be keeping on top of that (by developing an app
that uses it, partially) and making sure that 1.5 ships with something
that's decently solid, and with an Oracle backend if I can get it working
on my machine.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration - Review needed!

[Andrew Godwin]

>
>
> Can you put this in a pull-request so I can comment inline?
>
>
I can indeed: https://github.com/django/django/pull/376

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration - Review needed!

[Andrew Godwin]

An update from discussions with Alex and Anssi - I'm going to modify things
a little so we don't have a Borg-pattern AppCache (i.e. you can instantiate
it multiple times and get different caches), which should solve most of the
problems currently caused by app cache state fiddling. Should take a day or
two.

I can help with Oracle problems if you encounter a such thing since that's
> my main backend I work with. (And I have to deal with all the pain it
> causes)


Thanks Jani - that would be much appreciated. It should be possible to get
a decent way just from the Oracle module, I'll let you know when I get
around to it.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration - Review needed!

[Andrew Godwin]

Alright, I've now removed my AppCache state handling and replaced it with
swappable AppCaches - surprisingly, doing this hasn't broken anything, but
it did need an AppCacheWrapper to stop references going wrong.

The commits are here:

https://github.com/andrewgodwin/django/commit/dbc17d035b255a4da977251fe399f5c80cffeecd
https://github.com/andrewgodwin/django/commit/49d1e6b0e20a363cbf9b105e8e6d3fc5fc1cad2f

The SQLite test suite all passes after these changes, so this is looking
good. Thoughts?

Andrew


On Tue, Sep 18, 2012 at 4:49 PM, Andrew Godwin  wrote:

> An update from discussions with Alex and Anssi - I'm going to modify
> things a little so we don't have a Borg-pattern AppCache (i.e. you can
> instantiate it multiple times and get different caches), which should solve
> most of the problems currently caused by app cache state fiddling. Should
> take a day or two.
>
> I can help with Oracle problems if you encounter a such thing since that's
>> my main backend I work with. (And I have to deal with all the pain it
>> causes)
>
>
> Thanks Jani - that would be much appreciated. It should be possible to get
> a decent way just from the Oracle module, I'll let you know when I get
> around to it.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Schema Alteration update

[Andrew Godwin]

So, the patch [1] is looking alright, but after some consideration I think
it's going to be best to leave this until just after the 1.5 branch has
happened and then merge it in as part of the 1.6 cycle.

My reasoning is thus:

 - The whole point of getting something into 1.5 was so I could build
migrations as a third-party app. Since I now plan to make it more tightly
integrated, that's no longer useful.
 - The current patch is a bit ugly with the way it deals with AppCache -
discussion has led to a better, less hacky solution of implementing models
in a way where they can register into a different appcache.
 - It makes more sense to have the schema alteration, Field API changes and
migration runners all come out at once as otherwise it's not really a
user-facing solution.

Thus, I'd like to wait until 1.5 is branched off, and then merge in an
improved patch which has better support for making models that don't
register themselves, following that up at some point in the 1.6 cycle with
patches that implement a new field description API (so fields can be
serialised in a more human-readable way than pickle) and then finally the
migration runner/dependency solver, meaning 1.6 would ship with a full
migrations system in place.

Input on this plan is welcome - I just didn't want to rush something in
when it can be more polished and useful next release! It also means that
this can hopefully be integrated more smoothly with app-loading when that
lands.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration update

[Andrew Godwin]

Yeah, I think I mentioned it a couple of times at DjangoCon but perhaps not
loudly enough - Jacob and I had a talk at DjangoCon EU where he said he
wanted it all in core, and I tend to agree.

Preston has had a look at what I'm doing/planning with AppCache and
apparently it'll be compatable with what's in app-loading with a little
work, which is reassuring.

Only remaining question is whether migration-runner (the smarts) should be
in contrib or core. I know Alex believes core; I'm split, seeing as some
people will want to turn it off (so contrib), but it overrides core
commands like syncdb (so core would be cleaner).

Andrew

On Fri, Sep 28, 2012 at 3:22 AM, Russell Keith-Magee <
russ...@keith-magee.com> wrote:

> On Fri, Sep 28, 2012 at 8:29 AM, Jacob Kaplan-Moss 
> wrote:
> > On Fri, Sep 28, 2012 at 4:55 AM, Russell Keith-Magee
> >  wrote:
> >> Have I missed part of the discussion here? At DjangoCon, South was
> >> still going to exist (as the "smarts" part of the problem) -- has this
> >> changed?
> >
> > Obviously nothing's really decided, but I've been asking Andrew to
> > push for getting full-blown solution into core. I just don't see the
> > point in beating around the bush any more. The lack of a built-in
> > schema migration tool in Django is a major wart, so let's just fix it.
>
> No disagreement from me that this is a big wart that needs to be
> addressed. I just missed the memo about us merging the smarts bit into
> trunk.
>
> Russ %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Schema Alteration update

[Andrew Godwin]

I certainly don't want to tread on anyone's toes - the idea will be that,
like in South currently, migrations will be enabled/disabled on a per-app
basis, so if you don't want them they won't muck stuff up. Alternatively,
we could let the other apps override syncdb.

I'm hoping, in fact, that adding the database backends into core will make
anyone else who is doing custom migration stuff's lives easier - especially
if it's something highly custom internal to a company where you don't have
the time or team to do that stuff properly.

Andrew

On Thu, Oct 11, 2012 at 11:21 AM, Luke Plant  wrote:

> On 28/09/12 08:41, Andrew Godwin wrote:
> > Yeah, I think I mentioned it a couple of times at DjangoCon but perhaps
> > not loudly enough - Jacob and I had a talk at DjangoCon EU where he said
> > he wanted it all in core, and I tend to agree.
> >
> > Preston has had a look at what I'm doing/planning with AppCache and
> > apparently it'll be compatable with what's in app-loading with a little
> > work, which is reassuring.
> >
> > Only remaining question is whether migration-runner (the smarts) should
> > be in contrib or core. I know Alex believes core; I'm split, seeing as
> > some people will want to turn it off (so contrib), but it overrides core
> > commands like syncdb (so core would be cleaner).
>
> I'm very happy with South being in core/contrib, but I am aware of other
> people who use different solutions. It would be nice if we didn't make
> life hard for them. I don't know what that means in practice, but if we
> got feedback from the developers of those other solutions it would be
> good. We might need something like a 'syncdb --ignore-migrations' option.
>
> Luke
>
>
> --
> "Because Your lovingkindness is better than life,
>  My lips shall praise You."  (Ps 63:3)
>
> Luke Plant || http://lukeplant.me.uk/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

GSoC 2013 - New and improved!

[Andrew Godwin]

I'm very pleased to announce that Django is part of Google Summer of Code
once again - and that this year, we're widening the scope of the kinds of
projects we'll be accepting.

In past years, we've only accepted projects working on Django itself, and
while this has resulted in some very useful pieces of functionality, we've
also found that the complexity of the challenges in the core parts of
Django can turn away enthusiastic students, or just isn't a fit for some
ideas.

Thus, this year, we're accepting projects from one of three categories:

 - Work on Django itself - such as the ORM, forms, etc. This is what we've
traditionally accepted GSoC entries in.
 - Work on tools to support Django or its community - the dashboard (
https://dashboard.djangoproject.com/) is a good example of an existing tool
that would have fitted into this category.
 - Work on libraries that supplement or add new features to Django to aid
development - South and Django Debug Toolbar are good examples of existing
projects that would have fitted here.

We're not looking for people to work on existing third-party libraries - we
aren't able to guarantee commit access to them. We may allow an exception
if a maintainer of the library in question agrees to help mentor beforehand.

The broadening in scope is to allow people to work on new ideas to help
Django development and developers without tying you down to having to
implement it in the core codebase (and thus ruling out some projects that
might otherwise be useful).

We're still going to be strict with what we accept - you'll need to provide
a strong use case for your idea and show that it would be useful to a
majority of developers or significantly improve the development of Django
itself.

We're not looking for small groups of incremental updates - like "improve
Django's Trac" - nor are we looking for impossible tasks, like "replace
Trac with this brand new issue tracker I'm writing". What you propose
should be a single project, achievable within the time period of GSoC, and
something the core developers can help mentor you on.

We're also not looking for sites or projects that are merely written in
Django - this GSoC is not for you to propose your new forum hosting site or
amazing Django-based blogging engine.

The GSOC wiki page for this year has some ideas:
https://code.djangoproject.com/wiki/SummerOfCode2013 - though they are
mostly for working on the core of Django itself at the moment. We'll add
some more ideas as we think of them, but you do not need to submit from
that list. Anything that fits the guidelines is fine.

If you have an idea you want to run past us to see if it would be viable
before the application process opens on the 22nd April, then post here
(django-developers) with [GSoC 2013] at the start of the subject. Myself or
another member of core who's helping with GSoC will pitch in and discuss
the idea with you.

The official application period opens on April 22nd - I'll post nearer the
time with detailed application instructions.

This is the first year we're trying this widening of scope and we hope
it'll invite some exciting new applications. Many of Django's best tools
live outside of core and it would be a shame to be unable to help.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSoC 2013] Improved error reporting

[Andrew Godwin]

Hi Damian,

We don't generally accept GSOC projects that are just a bit grab bag of
problems - this is looking a little bit like that. I'd like to see a better
breakdown of what kind of time each ticket would take and what your
planning schedule would be - in particular, I'd like to make sure you have
roughly 12 full weeks worth of work, as some of these tickets are quite
easy to fix.

It would also be nice to see you pick fewer issues but ones which were more
difficult - a past security GSOC just focused on three or four main,
difficult problems, and while not all got done, it developed in a
reasonable way.

Andrew


On Sat, Apr 20, 2013 at 11:00 PM, Damian Skrodzki wrote:

> *
>
> Hi again.
>
>
> Let’s try analyse the topic Improved error handling a little deeper.
>
>
> I’ve found dozen opened issues related to bad error handling (these are
> rather *not* from Better Error Messages list). In general most of the bugs
> demand different approach for each one.
>
> 1. django admin
>
> https://code.djangoproject.com/ticket/9373
>
> https://code.djangoproject.com/ticket/14642 - maybe patch from last
> comment will be enough here
>
> https://code.djangoproject.com/ticket/14408
>
> https://code.djangoproject.com/ticket/10919 - probably limiting number of
> displaying modified objects would fix the problem
>
> https://code.djangoproject.com/ticket/17417 -  it would worth to track
> and fix or raise suitable error
>
>
> 2. templates:
>
> https://code.djangoproject.com/ticket/18995
>
> 3. django.db
>
>
> https://code.djangoproject.com/wiki/BetterErrorMessages?version=48#django.db- 
> quite complex problem with high severity
>
> https://code.djangoproject.com/ticket/13776 - [related to above] this
> could be fixed after some core dev will make decision how this code should
> behave (if it’s worth to pass this task from someone that will understand
> the problem to someone else)
>
> 4. various:
>
> https://code.djangoproject.com/ticket/20250 - probably NoneType should be
> detected in django.db.models.sql.where.as_sql(self, qn, connection) already
> or even before. If we don’t want to prevent this value to be None or it’s
> not easily possible, the solution would be to catch None in last function
> raising some error. Error message could contain tip saying what might solve
> this problem.
>
> https://code.djangoproject.com/ticket/16840 - need to raise an error for
> unexisting fields
>
> https://code.djangoproject.com/ticket/19560 - consider if the warning
> should be changed into error here
>
> https://code.djangoproject.com/ticket/15063 - I don’t fully understand
> the comment  2 but possibly I could fix it
>
> 5. group of tickets “approved” by core developers to produce more verbose
> message :
>
> https://code.djangoproject.com/ticket/18959
>
> https://code.djangoproject.com/ticket/18866
>
> https://code.djangoproject.com/ticket/14343
>
> https://code.djangoproject.com/ticket/12756
>
> https://code.djangoproject.com/ticket/8760
>
>
> Plan what to do:
>
> Fixing most of errors should consist of following steps:
>
> 1. Analyse where and why error occurs.
>
> 2. Try to reproduce the bug (the best way would be creating a test if it’s
> not provided)
>
> 3. Now take one of actions:
>
>  a. Just fix error message
>
>  b. Consult issue with core developer and fix the problem raising error
> in more proper place or way
>
> 4. Create a fix for ticket and submit a patch to review
>
>
> Schedule
>
>
> I thought about splitting the time for few-days intervals (it might be
> weeks) and assigning that bugs to them. The most important problems should
> be resolved first. As I said: If i take an easier set of issues - it should
> be possible to fix a dozen of them. If I’ll take more complex tickets that
> demand to take some decisions and medium reworks - I should take just few.
> Taking few sounds more reasonable for me.
>
>
> Committing
>
>
> I’m going to focus on a bunch of small problems so here I propose to do a
> commit (pull request) per problem. This would be easier to review by other
> developers and easier to link with problem. Additionally if some of the
> fixes will need additional changes, it won’t have any influence on the
> others.
>
>
> Questions
>
> 1. Do you guys know some other “known” errors which are confusing to users?
>
> 2. Could this task include some things like:
>
> https://code.djangoproject.com/ticket/18171 - some steps have been taken
> but it is still unfixed (and related to raised exception)
>
> https://code.djangoproject.com/ticket/17015 - another issue that requires
> bigger patch and finally causes confusing error message.
>
> or I rather shouldn’t touch errors like this?
>
>
> What next?
>
>
> I think finally I should take few or dozen of problems. My proposals here
> are weekly covering with your exemplary areas except admin area. Am I still
> on a correct track? There are few tickets related to django admin, maybe
> focusing on them will be a good direction. Although there are very few
> ope

Re: [GSoC 2013] Composite fields: first draft of the proposal

[Andrew Godwin]

Hi Michal,

This looks like a good starting point for a proposal (not to mention that
we don't doubt that you know about the problem area here!) - a few comments:

 - I'd like a bit more detail on some of your timeline points, in
particular what the introspection parts and primary key updates are
(personal bias, I suspect)
 - I'd like more details on what's been holding it back since GSOC 2011,
and how this GSOC is going to help serve those

Still, I like the general idea. You've mentioned tests and docs too. And I
really, really want to land this feature!

Andrew


On Fri, Apr 19, 2013 at 10:17 PM, Michal Petrucha wrote:

> Hello,
>
> as I promised a week ago, I'm posting a draft of my proposal. It is
> far from complete, specifically, it does not contain technical details
> on how things are going to be implemented. I'll try to fill these in
> as soon as possible, although I wrote more on most of them in my
> previous emails to this list. The most important thing is that it
> contains a timeline and a description of possible fallbacks.
>
> The proposal is accessible as a public gist [1] where it will be
> updated in the future, I'm posting the full text here as well for
> convenience.
>
> Have a nice weekend.
>
> Michal
>
> [1] https://gist.github.com/konk/5408673
>
>
> Composite Fields, vol. 2
> 
>
> Aim of this project
> ===
>
> Since I started working on this two years ago, I managed (with the
> help of a few people) to get most of the hard work done. The biggest
> part that I didn't get around to implementing was support in the ORM
> for multi-column joins. This has, however, been implemented recently
> by Jeremy Tiller and Anssi, which means there are only a few things
> left to be done.
>
> First of all, the code sitting in my github repo is badly out of date,
> which means it needs to be updated to the current state of master.
> While I'm at it, I also want to clean up the revision history to get
> it as close to a state where it could be just reviewed and merged
> directly as possible.
>
> Beside this, there are only the juicier features left that we
> initially wanted to leave unsupported and get back to them later.
> Those are the following (in no particular order):
>
> - ``GenericForeignKey`` support for ``CompositeField``
> - more intelligent handling of ``__in`` lookups
> - database instrospection and ``inspectdb``
> - detection of a change of the primary key in model instances
>
> Timeline
> 
>
> Our exam period starts on May 20. and ends on June 28., which means
> that I can't guarantee I will be able to fully dedicate myself to the
> project for the first two weeks, however, if nothing goes wrong, I
> should be able to pass all exams before June 17.
>
> Also, I intend to go to EuroPython, which means the first week of July
> won't be as fruitful as the others, but otherwise I'm ready to work
> full time on the project.
>
> Week  1 (Jun 17. - Jun 23.)
> ~~
> - porting the required virtual field changes, like a ``VirtualField``
>   class and ``django.db.models.options`` changes
> - revisiting the documentation I wrote two years ago to reflect the
>   evolution this project has gone through
>
> Week  2 (Jun 24. - Jun 30.)
> ~~
> - porting the virtual ``ForeignKey`` patchset on top of master to get
>   most of the functionality right
>
> Week  3 (Jul  1. - Jul  7.)
> ~~
> - EuroPython 2013, Florence
> - I intend to spend the full two days of sprints working on this but I
>   can't promise much more during this week.
> - running through the tests and fixing those that need updating,
>   ironing out the remaining wrinkles
>
> Week  4 (Jul  8. - Jul 14.)
> ~~
> - basic ``CompositeField`` functionality, ``CompositeValue`` and
>   descriptor protocol
>
> Week  5 (Jul 15. - Jul 21.)
> ~~
> - ``db_index``, ``unique`` and ``primary_key`` for ``CompositeField``
> - backend-dependent SQL for ``__in`` lookups on ``CompositeField``
>
> Week  6 (Jul 22. - Jul 28.)
> ~~
> - the few patches required to make admin work with composite primary
>   keys
>
> Week  7 (Jul 29. - Aug  4.)
> ~~
> - composite ``ForeignKey``: basic functionality, descriptors, database
>   JOINs
>
> Week  8 (Aug  5. - Aug 11.)
> ~~
> - composite ``ForeignKey``: customization of auxiliary fields,
>   adapting ``ModelForms`` and the admin in case it is necessary
>
> Week  9 (Aug 12. - Aug 18.)
> ~~
> - ``GenericForeignKey`` and ``GenericRelation`` support
>
> Week 10 (Aug 19. - Aug 25.)
> ~~
> - database introspection: create composite fields where necessary
>
> Week 11 (Aug 26. - Sep  1.)
> ~~
> - database introspection: composite ``ForeignKey``
>
> Week 12 (Sep  2. - Sep  8.)
> ~~
> - better handling o

GSOC: Deadline soon!

[Andrew Godwin]

Hi everyone,

It's great to see all the GSOC proposals on the list and the feedback being
useful, but I feel I should remind you all that the deadline is this
Friday, May 3rd, at 7pm UTC (so 11am if you're on the west coast of the US,
for example).

Melange, the GSOC software, can be a little hard to understand sometimes so
please try and get an inital copy of your application in with plenty of
time to spare - you can always save new versions right up to the deadline.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSoC 2013] contribute to django-deployer for deploying django to PaaS easily

[Andrew Godwin]

I feel like the deployment problem is one that cannot be solved in a mere
12 weeks and I'm not sure django-deployer is the right approach to
encourage for this - it sits at the wrong level of abstraction and looks
pretty fragile, and I'd be hesitant to put any sort of official emphasis on
it without a lot of my qualms being answered. In particular, it appears to
be sitting at the lowest-common-denominator level of the various feature
sets and looks pretty immature.

I'm not sure I could accept this without a much clearer idea of what's
going to happen and a very clear focus on what section of our user base it
will help. Deployment isn't something there can be a single solution (or
even pattern) for, and I think encouraging that could be a bad idea.

Andrew


On Wed, May 1, 2013 at 8:20 PM, Nate Aune  wrote:

> Hi Russell and Florian,
>
> A bit late to the party, but hopefully there's still time for this GSoC
> proposal to be considered. I'm the maintainer of 
> django-deployer.
> It was initiated at the DjangoCon 2012 sprint and recently revisited at the
> PyCon 2013 sprint. The overarching goal of django-deployer is to make it
> easier to get Django deployed. So far the focus has been on deploying to
> PaaS providers, because they require little to no sysadmin skills, and have
> the added benefit of being free for small projects.
>
> django-deployer is the sister project PaaS Cheatsheet, a larger
> documentation project that I've started recently. PaaS 
> Cheatsheetis a guide for how to get a 
> Django/Python deployed on various PaaS
> providers.
>
> The way django-deployer works is by asking a series of questions about
> your Django project, and then generates a generic deploy.yml file that
> captures all of your project's requirements. When you choose a particular
> PaaS provider, django-deployer then translates the requirements defined in
> the deploy.yml file and generates configuration files for that specific
> PaaS provider.
>
> I met Colin (a student in Taiwan) at the PyCon sprint, and he was
> responsible for single-handedly adding Google App Engine support to
> django-deployer. He brought considerable expertise to our sprint team, and
> shipped working code within a matter of hours. Since returning from PyCon,
> we've been working remotely together, and he has continued to be a
> dedicated contributor to the project. I have utmost confidence in his
> abilities to add even more cool features and improve code quality of the
> django-deployer tool if this project is accepted into GSoC.
>
> English is a second language for Colin, so he may have had some
> difficulties in defending his proposal, so please allow me to step in and
> clarify a few things.
>
>
>> Firstly, I'm not familiar with Django-deploy; but from a quick inspection
>> of the project page it doesn't appear that anyone in the Django core team
>> is on the committee list. In order for you to have this proposal accepted
>> as part of the GSoC, you'll need to secure agreement from the project
>> maintainers of Django-deploy that they want the help your offering, that
>> your project proposal is sound, and that they're willing to commit to
>> applying any patches you produce.
>
>
> As stated above, I'm willing to work closely with Colin to meet the
> objectives described in the proposal.
>
> Secondly, you haven't provided any timelines for your proposal. My
>> immediate reaction is that you're proposing a lot of work for a student to
>> complete in 12 weeks. For example, writing a fully tested and documented
>> database backend strikes me as a 12 week project in itself - yet this is
>> apparently just one line item in one part of your project proposal.
>>
>
> I think you may have misread that part of proposal. He's not proposing to
> write a database backend.  There is already one provided by the Google
> App Engine 
> SDK,
> so we're simply incorporating that into the configuration scripts (i.e.
> adding a settings_appengine.py that substitutes the DATABASE setting
> value).
>
>>
>> Lastly, your project proposal is very light on details. You're proposing
>> a lot of ideas, but you haven't really specified anything beyond "I'm going
>> to do it". Are there APIs that need to be specified? If, in the case of
>> something like a database backend, the APi is already specified, are there
>> going to be any complications in satisfying the API? We need a lot more
>> detail before we will be able to judge if you understand the project your
>> proposing, or if you are just proposing a bunch of ideas but haven't given
>> those ideas any more thought than "this sounds interesting"
>>
>
> I think the draft proposal was intended to get initial feedback and was
> intentially light on details. Colin and I can work on specifying more
> details on the actual deliverables within the next couple of days. We've
> a

Re: Django 1.6 release timeline

[Andrew Godwin]

I'm happy with this - I doubt schema alteration will be mergeable by the
15th (though I'm trying to get something together for DjangoCon EU) and I
want to encourage smaller release cycles. And the new transaction stuff is
great, damnit, we should ship that.

Andrew


On Tue, Apr 30, 2013 at 4:45 PM, Carl Meyer  wrote:

> Hi Shai,
>
> On 04/30/2013 04:38 PM, Shai Berger wrote:
> > On Tuesday 30 April 2013, Jacob Kaplan-Moss wrote:
> >> Hi folks -
> >>
> >> Unless there are strong objections, here's what I'm thinking for the
> >> Django 1.6 release timeline:
> >>
> >> Alpha: May 16
> >> Beta: June 20
> >> RC: Aug 1
> >> Final: as early as Aug 8, or later if more RCs are needed.
> >>
> > I see one issue with this: According to current procedures, if this
> timeline
> > is followed, support for 1.4 will be dropped less than 6 months after the
> > release of 1.5. At least for some of us (which, as I mentioned earlier
> on the
> > list, only moved to 1.4 when the 1.5 release forced us to), this may be
> a bit
> > of a problem.
>
> Yes, thanks for bringing this up. The core team discussed this at PyCon
> and I believe we had general agreement that if we achieve faster
> releases we'll need to extend support longer, at least for certain
> "long-term support" releases (a category which would probably be
> retroactively applied to 1.4). We haven't hammered out the details of
> the new policy yet, but I think it's safe to say that if we do release
> 1.6 on the proposed schedule, we won't drop support for 1.4 at that time.
>
> Carl
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSoC 2013] contribute to django-deployer for deploying django to PaaS easily

[Andrew Godwin]

Hi Nate,

I guess I'm personally just burnt by the PaaS model in general - it's a
very limited-scope way of modelling applications, I've never had it work
for anything serious, and in particular every attempt I've seen at
something that targets multiple PaaS hosts ultimately ends up only having
the core feature set of the lowest common denominator. A lot of small
projects are transitioning from PaaS to more "traditional" hosting, and I'm
not sure if this would help there.

Though it's too late for this GSOC, I'd rather like to see work on the
parts of Django that actually do affect deploy-ability for everyone, like
database configuration through a standard set of environment variables and
sorting out settings and how Django expects to find production vs.
development settings. I know that's a terrible argument, akin to the "why
are all the physicists colliding particles when they should be curing
cancer?!", but I do generally feel a little that tailoring to just a
certain specific set of PaaS providers is kind of papering over the problem.

Anyway, if the proposal is good and you're willing to mentor, I don't see
why we shouldn't accept it - you have commit access to the target project,
after all, and it'll set a good precedent - and it's worth seeing what
issues come up, because just knowing what some of them are is useful data.

Sorry about the slightly haphazard way I've handled this so far this year,
I'm still trying to work out where to draw the boundaries with the new
rules.

Andrew



On Fri, May 3, 2013 at 8:57 AM, Nate Aune  wrote:

>
> On Thursday, May 2, 2013 4:11:35 AM UTC-4, Andrew Godwin wrote:
>>
>> I feel like the deployment problem is one that cannot be solved in a mere
>> 12 weeks and I'm not sure django-deployer is the right approach to
>> encourage for this - it sits at the wrong level of abstraction and looks
>> pretty fragile, and I'd be hesitant to put any sort of official emphasis on
>> it without a lot of my qualms being answered.
>>
>
> Could you elaborate on what your qualms are? I won't claim that it's a
> work of art and I know there is lots of room for improvement, but please
> remember that django-deployer is the result of two days of work, and it
> already deploys Django projects to Dotcloud and Google App Engine with
> almost minimal effort on the part of the user. Imagine what we can do in 12
> weeks!
>
>
>> In particular, it appears to be sitting at the lowest-common-denominator
>> level of the various feature sets and looks pretty immature.
>>
>
> The initial goal of django-deployer was to reduce the barrier to entry for
> someone new to Django who wants to quickly and easily get their Django
> project running up on one of the PaaS providers.
> So we started out with a minimum feature set which seems to cover the
> non-production deployment needs for 80-90% of Django projects.
>
> We purposely omitted support for services such as Memcached, Celery,
> email, cron jobs, etc. because we were trying to limit the scope at the
> sprint and get a basic version working first. Now that the basic version is
> working, we can look into adding support for these other services that
> you'd most likely find in a production environment.
>
> I'm not sure I could accept this without a much clearer idea of what's
>> going to happen and a very clear focus on what section of our user base it
>> will help. Deployment isn't something there can be a single solution (or
>> even pattern) for, and I think encouraging that could be a bad idea.
>>
>
> Agreed. We're not attempting to make a one-size-fits-all solution, and
> django-deployer will never claim to serve that purpose. But for
> bootstrapping your Django project so that you can minimize the amount of
> time it takes to try deploying your project to various PaaS providers, and
> for demonstrating best practices when readying your project to be deployed,
> I think it has a place in the Django ecosystem.
>
> I've looked at all the getting started docs from the various PaaS
> providers, and many of them leave a lot to be desired. Many developers that
> I've spoken with who've tried PaaS have thrown up their hands in
> frustration when things don't work as advertised. This is largely the
> impetus for starting the django-deployer project, to try to streamline this
> process for those new to PaaS and Django deployment.
>
> Nate
>
>
>> On Wed, May 1, 2013 at 8:20 PM, Nate Aune  wrote:
>>
>>> Hi Russell and Florian,
>>>
>>> A bit late to the party, but hopefully there's still time for this GSoC
>>> proposal to be considered. I&#

Re: test discovery

[Andrew Godwin]

Just want to say that I'm happy with a "fast transition".

Is there a possibility we can detect the case where the tests might be
broken (how might they be?) and print a helpful error message?

Andrew


On Thu, May 9, 2013 at 11:51 AM, Russell Keith-Magee <
russ...@keith-magee.com> wrote:

>
> On Thu, May 9, 2013 at 5:00 AM, Carl Meyer  wrote:
>
>> Hi all,
>>
>> Preston Timmons and I have been working the last several weeks to get
>> the test discovery branch (#17365) ready for merge, and I think we now
>> have a pull request ready for consideration:
>> https://github.com/django/django/pull/1050
>>
>> Brief summary of the features, changes, and open questions:
>>
>> * You can now place tests in any file matching the pattern 'test*.py',
>> anywhere in your codebase, rather than only in tests.py and models.py
>> modules of an INSTALLED_APP. The filename pattern is customizable via
>> the --pattern argument to "manage.py test".
>>
>> * When you run "manage.py test" with no arguments, tests are discovered
>> and run in all subdirectories (recursively) of the current working
>> directory. (This means that django.contrib and other third-party app
>> tests are no longer run by default).
>>
>> * When you pass arguments to "manage.py test", they are now full Python
>> dotted module paths. So if you have a "myproject.myapp" app, and its
>> tests.py contains "SomeTestCase", you would now run that single test
>> case via "manage.py myproject.myapp.tests.SomeTestCase" rather than
>> "manage.py myapp.SomeTestCase". This is longer, but allows more control
>> when an app's tests are split into multiple files, and allows for tests
>> to be placed anywhere you like.
>>
>> * Doctests are no longer discovered by default; you will need to
>> explicitly integrate them with your test suite as recommended in the
>> Python docs: http://docs.python.org/2/library/doctest.html#unittest-api
>>
>> * Tests in models.py and tests/__init__.py will no longer be discovered
>> (as those don't match the default filename pattern).
>>
>> * The old test runner, and Django's extensions to the doctest module,
>> are deprecated and will be removed in Django 1.8; they could of course
>> be packaged separately if some people would like to continue using them.
>>
>>
> Great work Carl and Preston! (And everyone else who had a hand in the
> patch - I know this has been kicking around for a while now)
>
>
>> Open question: how to handle the transition?
>>
>> Jacob has suggested that back-compat breaks in test-running are not as
>> serious as in production code, and that we should just switch to the new
>> test runner by default in Django 1.6. This is what the pull request
>> currently does. This will mean that some people's test suites will
>> likely be broken when they upgrade to 1.6. They would have two options,
>> both documented in the release notes: they can update their test suite
>> to be discovery-compatible immediately, or they can just set TEST_RUNNER
>> to point to the old runner and get back the old behavior, which they can
>> keep using until Django 1.8 (or longer, if they package the old runner
>> externally).
>>
>> The alternative would be to keep the old test runner as the default in
>> global_settings.py until it is removed in 1.8, and add the new runner as
>> the TEST_RUNNER in the startproject-generated settings.py. This would
>> provide a gentler transition for upgrading projects. On the other hand,
>> we just recently got the startproject settings.py all cleaned-up,
>> slimmed-down, and newbie-friendly, so it would make me sad to start
>> polluting it again with stuff that new projects generally shouldn't care
>> about, for solely legacy reasons.
>>
>
> I agree with Jacob's tests aren't production sensitive, so making a fast
> transition to the new test discovery has less impact.
>
> However, we know from experience that it doesn't matter how much we flag
> this change in the release notes, they won't be read, and *someone* is
> going to get stung. In an ideal world, it would be good to be able to warn
> people who upgrade that their test suite may not be run as expected.
>
> We've had a proposal kicking around for a while to add a management
> command that does an "upgrade check". I'm wondering if this might be an
> opportune time to dig into this some more.
>
> To be clear, this isn't something I consider to be a merge blocker. I'd be
> happy to see 1.6 released with a fast transition to the new test runner. It
> would just be nice icing on the cake if we can get an upgrade validator
> included in the same release.
>
> Yours,
> Russ Magee %-)
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl

Re: DB Migrations: Proposed fix for "Index name collisions after objects are renamed then re-created"

[Andrew Godwin]

Just chiming in on this; I sat down with Tom to look at this during the
sprint, and while we could go to a large amount of effort to try and rename
indexes on every rename of a model/field, the problem is solved much more
easily by introducing a random element to index names.

This does, of course, make the databases that migrations creates have
subtly different schemas; this doesn't matter to Django, as we always look
up indexes based on characteristics rather than names, but if you have any
examples of where predictable names do matter, please share them!

(also, I had an idea on the way back that perhaps we could derive the
random element from the migration name, making it different across
different migrations so they don't clash but still the same across runs,
but I'm not sure how you'd access that from within SchemaEditor)

Andrew

On Thu, Nov 20, 2014 at 1:56 AM, tomv  wrote:

> Hi all,
>
> At the Django under the hood sprint I picked up #23577
> <https://code.djangoproject.com/ticket/23577> which says, in the general
> case:
>
>1. For any database object when django creates an associated index
>2. If you rename that object
>3. Then re-create another with the original's name
>4. You get a collision in the name of the associated index
>
> This applies in the simplest case to renaming any field with
> db_index=True, then re-creating another in it's place. But also renaming a
> model containing a ForeignKey or indexed field, then re-creating the model.
> (All via db migrations)
>
> The instinctive desire is to rename indexes when renaming objects whose
> name was used in the index creation. But speaking to Andrew Godwin, he
> feels this would be quite a large challenge, across all our supported
> backends.
>
> The alternative he suggested was to add a random element to all index
> names the schema editor creates. This post is seeking to validate that
> proposal.
>
> Note: it's only the names of indexes at issue here, databases keep
> everything pointing at the right objects. There's also no issue with
> Django needing to access an index via a predictable name, as there's
> introspection which grabs indexes according to the objects they work on.
>
> Bonus question: given the scenario above, can we come up with a source of
> index naming that differs after the "parent" object is renamed/re-created,
> but which remains deterministic? Something like pulling in a hash of the
> whole app model state. Unfortunately, as we aim to output migrations to
> sql, we can't have any introspection at migration application time, so
> for example no checking for name collisions.
>
> Cheers,
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/99246d04-b205-42a6-a389-a0370bea3cd8%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/99246d04-b205-42a6-a389-a0370bea3cd8%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urW_cCq_rdvbvck6Jf96fRuPFULR-aBkv6q0w7cNARD5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Infinite loop in migration code

[Andrew Godwin]

Hi Luke,

That was already a fix for infinite looping on the previous iteration that
I committed at the DUTH sprints, but your fix looks more understandable and
cleaner, so I say commit it for sure.

As for backporting - I think we should, as this is potentially a crash bug
(though not a data loss bug) and it's a pretty small thing to backport
anyway.

Andrew


On Tue, Nov 25, 2014 at 8:23 AM, Markus Holtermann  wrote:

> Hey Luke,
>
> It would be interesting to see why A.1 and B.1 depend on each other. If
> there are e.g. FK constraints pointing to models in the other app the
> autodetector should end up with e.g. A.1 <-- B.1 <-- B.2 <-- A.2 (or
> optimized A.1 <-- B.1 <-- A.2), in which case you wouldn't end up with a
> cycle. C.1 would then depend on B.2 (B.1 respectively in the optimized
> graph).
>
> /Markus
>
>
> On Tuesday, November 25, 2014 4:49:22 PM UTC+1, Luke Plant wrote:
>>
>>
>> I came across a bug with an infinite loop in migration dependency
>> searching code. This is fixed here:
>>
>> https://github.com/django/django/commit/ff3d746e8d8e8fbe6de287bd0f4c3a
>> 9fa23c18dc
>>
>> (another person reviewing it would be good, though I think it is
>> correct).
>>
>> My question is, should we backport this to 1.7.x? For me, the bug
>> manifested itself with migrations that were automatically created by
>> Django itself, in a project with apps A, B, and C:
>>
>> App B depends on app A
>> App A depends on app B (it didn't initially, but does now)
>> App C depends on one/both of them.
>>
>> After doing makemigrations for A and B, makemigrations for C then went
>> into an infinite loop.
>>
>> So this is not a really obscure case, and could affect a fair number of
>> people attempting to upgrade to Django 1.7, as I was.
>>
>> Regards,
>>
>> Luke
>>
>>
>> --
>> "We may not return the affection of those who like us, but we
>> always respect their good judgement." -- Libbie Fudim
>>
>> Luke Plant || http://lukeplant.me.uk/
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5e547550-1002-4a6b-82f0-a8dc7e3ed2b0%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uq3zkayXL8%3D1QtGpY8hZcYwC64VRXL%2BEXPC6c9nH%3Dawwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding model managers to migrations

[Andrew Godwin]

As I've said before, I like this idea, and the opt-in thing is even better
and gets rid of a lot of the compatibility headaches I was worried about.

The review looks good overall - have left a couple of notes, only really
the one change I want to see (the operational dependency stuff).

Andrew

On Fri, Dec 5, 2014 at 10:28 AM, Carl Meyer  wrote:

> Hi Markus,
>
> On 12/05/2014 11:22 AM, Markus Holtermann wrote:
> > Ticket: https://code.djangoproject.com/ticket/23822
> > Pull Request: https://github.com/django/django/pull/3687
> >
> > Hey folks,
> >
> > As of now migrations don't expose the model managers to models. Thus it
> is
> > "impossible" to use e.g. `create_superuser` in a `RunPython` operation. I
> > opened ticket #23822 to keep track of that issue. I propose a solution as
> > implemented in pull request 3687: managers get an optional class
> attribute
> > `use_in_migrations` that, if True, will mark that manager to be added to
> > migrations by the migration framework. Managers that are part of
> migrations
> > need to stay around for as long as a migration references the managers.
> > This is the same as with custom model fields or functions passed to field
> > attributes (e.g. `upload_to`).
> >
> > To keep track of the ordering of the managers (and know which one is
> really
> > the default), I merged the `Model._meta.abstract_managers` and
> > `.concrete_managers` into a single list and added properties to return
> the
> > respective subsets instead.
> >
> > The `managers` attribute on `CreateModel` will be empty without enabling
> at
> > lease one manager. The normal manager (`models.Manager`) will (almost)
> > never be serialized. Except for the situation where a model's default
> > manager is a manager without migration support, but a custom manager is.
> In
> > that case a default manager `objects = models.Manager()` will be added
> > explicitly to prevent the non-default manager to become a default
> manager.
> >
> > I added migration support for all contrib managers (where appropriate),
> e.g
> > `UserManager` instead of `BaseUserManager`.
> >
> > I'd like to get a thorough review before merging, especially regarding
> the
> > `ModelState.render()` and `from_model()` functions. I think there are
> some
> > improvements possible I haven't though about.
>
> I think the feature makes sense, as an opt-in.
>
> I've added it to my review list, I'll try to get to it soon.
>
> Thanks!
>
> Carl
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5481F965.3010600%40oddbird.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urbKFNTYtT8rJSxEFE9qAjQ-1DD3dLqdTOtr2JYCz3eVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Experimental APIs DEP

[Andrew Godwin]

Hi everyone,

One of the results of discussions at Django Under The Hood was support for
the idea of marking APIs "experimental" - that is, document them and
include them in mainline Django releases, but away from the standard Django
deprecation cycle while still not hiding them under the "Internal" moniker
(these would be APIs for public consumption).

Of particular interest here is the issues we faced with some of the
migration APIs during the 1.7 release; 1.7 might have been able to release
faster if we could have released early with some parts of migrations and
the schema editor marked as "experimental" and iterated on it during the
main release cycle, and communicated with third-party database backends
more effectively and given them documentation and release notes.

There's an extra consideration here if we should extend the proposed part
of the release notes listing experimental changes to also include internal
changes, and possibly a separate discussion to be had about marking some of
the Django database API as stable (or experimental for a release or two,
then upgrade them).

The pull request for the DEP is here: https://github.com/django/deps/pull/10

(If you're not familiar with the updated DEP process, you should read the
new DEP 1 at
https://github.com/django/deps/blob/master/final/0001-dep-process.rst -
but, quickly, the pull request is for both corrections to the layout and
spelling and content suggestions, though this thread is the best place for
long-form discussion).

Discussion welcome! This is by no means a final proposal, just a first
draft.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqkeuZ1GMWdYhN_X9nvmKLXz61CTdqWzdef8OK6%2BRha3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Experimental APIs DEP

[Andrew Godwin]

My notes from the meeting say "experimental API language", so I may have
taken an adjective too literally when I made this.

Nevertheless, the key thing _I_ want to see is for us to commit to putting
release notes out for some of Django's APIs that aren't necessarily
considered stable. The 1.7 database API stuff was the main catalyst for
this; those are changes and APIs we should have documented, and fall
somewhere above "internal" (as we expect people to build third-party
database backends), but not tie ourselves into a 3-release deprecation
cycle for.

How about we change the label from "Experimental" to something like
"Unstable", but keep the same provisions - documentation called out as
"unstable feature", separate section in the release notes, etc. That
establishes a clear level between "internal and we don't care about it" and
"stable and publicly documented". The alternative is to change the DEP to
just say we're going to start putting release notes up for certain internal
APIs, and then somehow list the ones we will somewhere (probably on the API
stability page).

Andrew

On Sat, Dec 6, 2014 at 1:02 PM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> On 6 déc. 2014, at 10:05, Carl Meyer  wrote:
>
> > As the DEP notes, our backwards-compatibility policy already includes a
> > longstanding carve-out for anything documented within the "internals"
> > section of the docs. We haven't made much use of this for documenting
> > actual internal APIs (most of that section of the docs is about
> > contribution process), but we could.
>
> Like Carl, I believe “internal” is an established concept that fits our
> needs
> better than “experimental”.
>
> To me “experimental” sounds much like an ill-defined gray zone between
> private and public.
>
> --
> Aymeric.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/A3D3D45A-19CE-44D4-82DA-84AA45EDA586%40polytechnique.org
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqV%2BYbu%3DQgCy3W5AC3aFDX_0W4YzUQLT22RH3h01KLLvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Bypassing 1.7+ migrations

[Andrew Godwin]

Hi Marcin,

If you're using an external tool to manage schemas of models, just set
managed=False on the models and Django will stop trying to change their
schemas (including stopping making migrations for them).

Andrew

On Tue, Dec 16, 2014 at 1:11 AM, Marcin Nowak 
wrote:
>
> Hello!
>
> I'm using Django as a part of bigger stack where I'm using external tool
> to manage database migrations.
> After migration to Django 1.7 I'm encouraged at every runserver command to
> execute some migrations, which may blow my databases.
>
> I want to bypass Django migrations to prevent any schema modification. I
> can bypass migrate*  commands by overriding them in my custom app, but this
> is a little tricky and does not cover all cases. Using managed=False is not
> a solution, because none of reusable apps have models defined as unmanaged
> (same for django.contrib, especially auth and sessions).
> That's why I need to bypass migrations globally.
>
> Have you ever considered possibility to allow disabling it? Maybe via some
> setting to use kind a DummyMigrationExecutor instead of MigrationExecutor?
>
> Cheers,
> Marcin
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a27293a1-4212-4a2a-ae44-4720fc674162%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1up6t0%3DST6FCFBFrv3FyU6L1V%2BqXhEk43NhMcjJjv-SHiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Bypassing 1.7+ migrations

[Andrew Godwin]

Hi Marcin,

You can't bypass migrations as much as you can't bypass syncdb; that is,
the commands are separate and you can choose not to run them (or just
delete the migration files). As far as I can tell your only issue is the
runserver warning that you have unmigrated changes, which there's currently
no plan to be able to disable.

Also bear in mind that, by doing your own schema management for the contrib
apps, you're kind of implicitly forking the schema; Django 1.8 will ship
with some migrations for these apps, and you're going to have to make sure
you manage those yourself and keep the database in sync with the code
(though of course the release notes will mention this kind of thing).

If you want to stub out migration operations, you can just make a custom
database backend that inherits from the one you use and define a new
SchemaEditor subclass with an execute() function that does nothing; that
should accomplish what you want. Alternatively, you could wrap all
migration operations into the State side of SeparateDatabaseAndState, but
that's a bit cumbersome.

Andrew

On Tue, Dec 16, 2014 at 4:42 AM, Marcin Nowak 
wrote:
>
> Hi Andrew,
>
> Thanks for a reply. But as I've mentioned above I can't set managed=False
> on models delivered from external apps like django.contrib.auth,
> django.contrib.sessions, and many more. Forking them is not an option.
>
> I think that bypassing migrations should be possible via some setting.
> That's why I wrote to developers forum directly.
> Please think about that.
>
> Marcin
>
> On Tuesday, December 16, 2014 1:34:17 PM UTC+1, Andrew Godwin wrote:
>>
>> Hi Marcin,
>>
>> If you're using an external tool to manage schemas of models, just set
>> managed=False on the models and Django will stop trying to change their
>> schemas (including stopping making migrations for them).
>>
>> Andrew
>>
>> On Tue, Dec 16, 2014 at 1:11 AM, Marcin Nowak 
>> wrote:
>>>
>>> Hello!
>>>
>>> I'm using Django as a part of bigger stack where I'm using external tool
>>> to manage database migrations.
>>> After migration to Django 1.7 I'm encouraged at every runserver command
>>> to execute some migrations, which may blow my databases.
>>>
>>> I want to bypass Django migrations to prevent any schema modification. I
>>> can bypass migrate*  commands by overriding them in my custom app, but this
>>> is a little tricky and does not cover all cases. Using managed=False is not
>>> a solution, because none of reusable apps have models defined as unmanaged
>>> (same for django.contrib, especially auth and sessions).
>>> That's why I need to bypass migrations globally.
>>>
>>> Have you ever considered possibility to allow disabling it? Maybe via
>>> some setting to use kind a DummyMigrationExecutor instead of
>>> MigrationExecutor?
>>>
>>> Cheers,
>>> Marcin
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/django-developers/a27293a1-4212-4a2a-ae44-
>>> 4720fc674162%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-developers/a27293a1-4212-4a2a-ae44-4720fc674162%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d8226a81-9dfe-4019-94ce-98beb6e1bf04%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/d8226a81-9dfe-4019-94ce-98beb6e1bf04%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.

Re: Migrations in Django 1.7 make unit testing models harder

[Andrew Godwin]

I agree that migrations are slower than syncdb - that's perhaps the only
thing worse about them - but the reason we plan to deprecate the other
methods is code simplicity; migrations does not share almost any code with
the old DatabaseCreation backends, and if we don't deprecate it we're going
to end up maintaining two creation backends for every database driver,
which isn't going to go well.

There's perhaps something to be said for an option where tests make an
in-memory set of migrations from the autodetector and an empty state and
run them immediately - somewhat replicating the syncdb process while still
using the same code paths - but I haven't had the time to investigate how
well this would work yet (there are some migration decisions that would
need defaults inserted).

I think the end result would be an alternative test runner that you could
switch to if you wanted this behaviour (and a mixin with the actual logic
or something similar so it's easy to incorporate into other test runners).

Andrew

On Wed, Dec 17, 2014 at 6:59 PM,  wrote:
>
> At the risk of reviving an old topic, I wanted to add one significant
> point in favor of (and mitigation for) running tests with migrations
> disabled: Speed.
>
> Creating a test DB in sqlite for our project (~100 database tables) takes
> approximately 1-2 minutes on most developer machines. 1-2 minutes of idle
> time to run any test was just unacceptable so we disabled migrations by
> setting fake migrations in MIGRATION_MODULES and brought the test DB
> creation time down to about 5 seconds (!!).
>
> However the risk of committing invalid code because someone forgot to
> makemigrations is real. We've addressed it by only overriding migrations on
> our local test settings and still having migrations run on our CI server.
> We have our CI server use settings.test, while developers running tests on
> their local machine use settings.test_local which is just:
>
> from settings.test import *
>
> MIGRATION_MODULES = ((app, '%s.fake_migrations' % app) for app in
> INSTALLED_APPS)
>
>
> This seems to get us the best of both worlds. I was surprised to read
> through this thread and not see other mentions of the performance
> implications of running migrations on every test run. I'd be curious to
> hear if this has been a bottleneck for other teams.
>
> -Chris
>
> On Tuesday, March 25, 2014 10:21:51 AM UTC-7, Bernie Sumption wrote:
>
>> Hi Django devs,
>>
>> I've just started a new project in 1.7b, and the development experience
>> working with unit tests on models is more complicated than it was in 1.6.
>> It's all down to how the throwaway test databases are created. In 1.6, the
>> create_test_db function "Creates a new test database and runs syncdb
>> against it." In 1.7, it runs "migrate".
>>
>> While generally speaking, migrate is the new syncdb, this behaviour is
>> not ideal for tests. In 1.6 "syncdb" created a database reflecting the
>> current state of the models in models.py. "migrate" creates a database
>> reflecting the state of the models at the last time makemigrations was run.
>> If you're doing TDD and constantly making small changes to your models then
>> runnning unit tests, you have to run makemigrations before each test run to
>> get your tests to work. You therefore end up with many tiny migration files
>> representing the minute-by-minute history of development.
>>
>> I came up with a pretty effective workaround that is working for me, but
>> I thought I'd post this here as others are sure to encounter this issue,
>> and I think that it makes more sense for the behaviour produced by this
>> workaround to be the default for running tests.
>>
>> If makemigrations has not yet been run, the "migrate" command treats an
>> app as unmigrated, and creates tables directly from the models just like
>> syncdb did in 1.6. I defined a new settings module just for unit tests
>> called "settings_test.py", which imports * from the main settings module
>> and adds this line:
>>
>> MIGRATION_MODULES = {"myapp": "myapp.migrations_not_used_in_tests"}
>>
>> Then I run tests like this:
>>
>> DJANGO_SETTINGS_MODULE="myapp.settings_test" python manage.py test
>>
>> This fools migrate into thinking that the app is unmigrated, and so every
>> time a test database is created it reflects the current structure of
>> models.py.
>>
>> So my feature request is as follows:
>>
>> If the new behaviour is by design and considered desirable, then it is a
>> big change from the previous version and should be prominently documented
>> in the migrations and testing pages, along with the workaround. I'm happy
>> to write this documentation if that's the way you want to go.
>>
>> However, if the new behaviour is not by design but just a by-product of
>> the new migrations feature, I suggest making the workaround the default
>> behaviour. I don't (yet!) know enough about Django internals to volunteer
>> for this however.
>>
>> Thanks for your time,
>>
>> Bernie :o)
>>
>>  --
> You rece

Re: Migrations in Django 1.7 make unit testing models harder

[Andrew Godwin]

Hi Tim,

I can have a look, but I can't be certain about hitting any deadlines. I do
want to get that deprecation in, though...

Did you want it with a view to us being able to drop that in for tests
rather than making migrations for every test app, I presume?

Andrew

On Fri, Dec 19, 2014 at 3:06 PM, Tim Graham  wrote:
>
> Andrew, I've thought of something similar to the in-memory migrations idea
> you've proposed. It would be great not to have to add and maintain
> migrations for all of the apps in Django's test suite. Do you think you
> might be able to investigate this solution in the next month or so before
> 1.8 alpha? I think we need a solution in 1.8 if we are to complete #22340 - 
> Legacy
> Table Creation Methods Not Properly Deprecated (otherwise, we can again
> postpone that deprecation).
>
> On Friday, December 19, 2014 8:17:05 AM UTC-5, Andrew Godwin wrote:
>>
>> I agree that migrations are slower than syncdb - that's perhaps the only
>> thing worse about them - but the reason we plan to deprecate the other
>> methods is code simplicity; migrations does not share almost any code with
>> the old DatabaseCreation backends, and if we don't deprecate it we're going
>> to end up maintaining two creation backends for every database driver,
>> which isn't going to go well.
>>
>> There's perhaps something to be said for an option where tests make an
>> in-memory set of migrations from the autodetector and an empty state and
>> run them immediately - somewhat replicating the syncdb process while still
>> using the same code paths - but I haven't had the time to investigate how
>> well this would work yet (there are some migration decisions that would
>> need defaults inserted).
>>
>> I think the end result would be an alternative test runner that you could
>> switch to if you wanted this behaviour (and a mixin with the actual logic
>> or something similar so it's easy to incorporate into other test runners).
>>
>> Andrew
>>
>> On Wed, Dec 17, 2014 at 6:59 PM,  wrote:
>>
>>> At the risk of reviving an old topic, I wanted to add one significant
>>> point in favor of (and mitigation for) running tests with migrations
>>> disabled: Speed.
>>>
>>> Creating a test DB in sqlite for our project (~100 database tables)
>>> takes approximately 1-2 minutes on most developer machines. 1-2 minutes
>>> of idle time to run any test was just unacceptable so we disabled
>>> migrations by setting fake migrations in MIGRATION_MODULES and brought the
>>> test DB creation time down to about 5 seconds (!!).
>>>
>>> However the risk of committing invalid code because someone forgot to
>>> makemigrations is real. We've addressed it by only overriding migrations on
>>> our local test settings and still having migrations run on our CI server.
>>> We have our CI server use settings.test, while developers running tests on
>>> their local machine use settings.test_local which is just:
>>>
>>> from settings.test import *
>>>
>>> MIGRATION_MODULES = ((app, '%s.fake_migrations' % app) for app in
>>> INSTALLED_APPS)
>>>
>>>
>>> This seems to get us the best of both worlds. I was surprised to read
>>> through this thread and not see other mentions of the performance
>>> implications of running migrations on every test run. I'd be curious to
>>> hear if this has been a bottleneck for other teams.
>>>
>>> -Chris
>>>
>>> On Tuesday, March 25, 2014 10:21:51 AM UTC-7, Bernie Sumption wrote:
>>>
>>>> Hi Django devs,
>>>>
>>>> I've just started a new project in 1.7b, and the development experience
>>>> working with unit tests on models is more complicated than it was in 1.6.
>>>> It's all down to how the throwaway test databases are created. In 1.6, the
>>>> create_test_db function "Creates a new test database and runs syncdb
>>>> against it." In 1.7, it runs "migrate".
>>>>
>>>> While generally speaking, migrate is the new syncdb, this behaviour is
>>>> not ideal for tests. In 1.6 "syncdb" created a database reflecting the
>>>> current state of the models in models.py. "migrate" creates a database
>>>> reflecting the state of the models at the last time makemigrations was run.
>>>> If you're doing TDD and constantly making small changes to your models then
>>>> runnning unit tests, you have to run makemig

Re: Migrations in Django 1.7 make unit testing models harder

[Andrew Godwin]

Clause, I believe that line is to allow people to run makemigrations when
AUTH_USER_MODEL points to an app that doesn't yet have migrations; before,
it would fail hard, as the AUTH_USER_MODEL was not in the migrate state and
so nothing could use it, and you couldn't run makemigrations to create it
as it would error out.

I'd test variations of that to see if your patch still works; if not, we'll
need a workaround again.

Andrew

On Sat, Dec 20, 2014 at 10:47 AM, Claude Paroz  wrote:
>
> On Friday, December 19, 2014 6:30:32 PM UTC+1, Tim Graham wrote:
>>
>> Yes. Claude has worked on the deprecation in https://github.com/django/
>> django/pull/3220 but it requires adding more migrations to our test
>> suite and he noted that each migration we add to Django's test suite adds
>> up to ~3.5 seconds to the run time of the test suite. He also worked on
>> some speed ups to mitigate this in https://github.com/django/
>> django/pull/3484 but there are some unsolved issues.
>>
>
> I think that the first mentioned patch I worked on (PR 3220) shows that it
> should be possible to use the new schema infrastructure without requiring
> migrations, at least for apps that don't depend on other migrated apps.
> Tell me if I miss anything.
>
> But surely, I'd really like to solve first the speed issue of migrations
> (#23745, PR 3484). Chris, if you have the opportunity to test that patch
> with your database, it would be great. The way forward for this patch is to
> first test the "if app_label == settings.AUTH_USER_MODEL and
> ignore_swappable:" line which is currently not tested and is probably not
> addressed by my patch. Andrew, do you remember why you introduced that part
> of the code?
>
> Claude
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1f5f37b2-5309-4af6-8b3c-aa76eee3bde1%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urVDjwidnzk-%3DrEhvjYCtivcVCrtgpdserQ6B36zMdrxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Keeping apps without migrations?

[Andrew Godwin]

My main argument for removing them entirely was the dependency issues
between apps with and without migrations. Having syncdb use SchemaEditor is
a big step and one I'm happy we've got to, but the only advantage of
keeping syncdb is for the test suite and I'd rather we approach that more
as "migrations made and run at runtime as a special case" rather than
"syncdb".

If nothing else, I'd like to see the end-developer-facing parts, like the
syncdb command itself, gone.

Andrew

On Sun, Jan 18, 2015 at 1:43 PM, Claude Paroz  wrote:

> Tim recently did a fabulous job of removing deprecated code for the
> future 1.9 on master. Thanks for that.
>
> However, one thing he removed was support for apps without migrations.
>
> https://github.com/django/django/commit/7e8cf74dc74539f40f4cea53c1e8bba82791fcb6
>
> Considering that we have to keep internal support for Django's own test
> suite anyway, I wonder if we should remove that support at all for
> "normal" projects. I think one of Andrew's motivation was not to have to
> keep two schema editing code bases. But now that the old syncdb also
> uses the new schema editor, I think that this argument doesn't stand.
>
> So what about keeping support for apps without migrations in the longer
> term. Of course, the fact that those apps cannot depend on migrated apps
> limits its use, but I think that for quick prototyping and initial
> developement, migrations could be more of a hindrance. Would you see
> major drawbacks with keeping this support?
>
> Opinions welcome.
>
> Claude
> --
> www.2xlibre.net
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1421617394.5741.4.camel%40doulos
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urpQkj2mov0QGpepbHqc-2mRKvLXGNK%2B1A%2B%3Df%2BMBZNoSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Must a Django Database Support Migrations?

[Andrew Godwin]

Hi Ivan,

I'm not sure what you're asking here - are you asking to have a way to not
have Django create the migrations recording table? I was under the
impression that it was only created when migrate was run (at least, that
was my original intention) so if you're managing your own schema just don't
run migrate. Is there something else that's not working right, or is that
being made too optimistically?

Andrew

On Tue, Jan 20, 2015 at 2:44 PM, Ivan VenOsdel  wrote:

> From Andrew: "The only extra thing it would achieve is not having Django
> record migrations in the django_migrations table"
>
> The Django Users thread on how to keep this table from being created
> seemed to result in the 'solution' being either to stay with 1.6 or comment
> the relevant lines in the Django code base. Is there really no other way?
>
> I love the new migrations facilities in Django 1.7 and was a contributor
> to the kickstarter but please keep in mind that many legacy DB projects are
> not avoiding migrations because they want to. IMO it's usually because they
> are forced to for some (usually political) reason where they don't have
> control over the schema. Forcing a perpetually empty django_migrations
> table pretty much locks out those users.
>
> I see that people are worried about encouraging the non-use of migrations
> but might I suggest following the Zen of Python and making migrations the
> "one obvious way to do it" and turning them off the not obvious way.
>
> Ivan
>
> On Wednesday, January 22, 2014 at 5:40:35 AM UTC-6, Andrew Godwin wrote:
>>
>> My thoughts in brief on this:
>>
>>  - Database backends don't support migrations - they support schema
>> alteration via SchemaEditor. This could be used separately from migrations
>> if something wants it, and is meant to be an API on its own, so the backend
>> is not the place to say if you want migrations or not.
>>
>>  - There is some merit to the ability to turn off migrations on a
>> per-backend basis, via a DATABASES setting, but bear in mind that routers
>> already let you do this (with the allow_migrate method). The only extra
>> thing it would achieve is not having Django record migrations in the
>> django_migrations table, but it also looks like it would be useful for
>> tests if you hadn't written that part yet.
>>
>>  - I feel like a DB backend should at least implement SchemaEditor even
>> if it returns NotImplementedError for most of the endpoints; even in the
>> weirdest relational system, you can still manage create/delete model
>> without too much difficulty.
>>
>>  - Bear in mind that the plan is to remove DatabaseCreation entirely in
>> favour of SchemaEditor in a few releases' time (and backends are more than
>> welcome to make DatabaseCreation use SchemaEditor behind the scenes if they
>> want), so at that point if you don't implement it the backend is only ever
>> good for querying, which to me feels like an incomplete backend.
>>
>>  - I'm not sure what the future of ./manage.py sqlall is, but it's going
>> to have to be ported to SchemaEditor in future anyway, so it's only useful
>> in the transition.
>>
>> Looking at the discussion, I think the best thing to do is:
>>
>>  - Make the schema and migrations test skip if connection.schema_editor()
>> raises a NotImplementedError (not too hard, we can implement
>> connection.has_schema_editor as a boolean to switch on)
>>  - Provide some way to skip the "creating models" part of test setup, so
>> SchemaEditor is never needed during it
>>
>> I still think all the current third-party backends should be able to
>> provide a partial SchemaEditor implementation though, as at minimum they
>> all have the DatabaseCreation code in place to make new models. Bear in
>> mind that the ./manage.py sqlmigrate command lets you turn migrations into
>> SQL scripts to send to your DBA (and many DBAs appreciate having some SQL
>> they can work from - I know ours do), so having the ability to make that
>> SQL is useful even if Django never runs it.
>>
>> Andrew
>>
>>
>> On Wed, Jan 22, 2014 at 10:10 AM, Shai Berger  wrote:
>>
>>> On Wednesday 22 January 2014 16:26:50 Russell Keith-Magee wrote:
>>> > On Wed, Jan 22, 2014 at 3:59 PM, Shai Berger 
>>> wrote:
>>> > >
>>> > > B) Allow the test suite to run on an existing schema. The Oracle
>>> backend
>>> > > already does this (badly) -- its six specific TEST_* parameters are
>>> > > awkwardly na

Re: Must a Django Database Support Migrations?

[Andrew Godwin]

Aha! Then, I would suggest redesigning MigrationRecorder to only make the
table when an actual recording operation is done, and have it swallow the
table not existing as "no migrations applied" the rest of the time, if
people think that seems sensible.

Andrew

On Thu, Jan 22, 2015 at 10:44 AM, Markus Holtermann <
i...@markusholtermann.eu> wrote:

> Hey,
>
> as soon as the MigrationRecorder is used there is a call to
> "ensure_schema" that forces the creation of the migration table. The
> runserver command (among others?) checks for unapplied migrations and thus
> creates the migration table.
>
> /Markus
>
> On Wednesday, January 21, 2015 at 12:36:47 AM UTC+1, Andrew Godwin wrote:
>>
>> Hi Ivan,
>>
>> I'm not sure what you're asking here - are you asking to have a way to
>> not have Django create the migrations recording table? I was under the
>> impression that it was only created when migrate was run (at least, that
>> was my original intention) so if you're managing your own schema just don't
>> run migrate. Is there something else that's not working right, or is that
>> being made too optimistically?
>>
>> Andrew
>>
>> On Tue, Jan 20, 2015 at 2:44 PM, Ivan VenOsdel  wrote:
>>
>>> From Andrew: "The only extra thing it would achieve is not having Django
>>> record migrations in the django_migrations table"
>>>
>>> The Django Users thread on how to keep this table from being created
>>> seemed to result in the 'solution' being either to stay with 1.6 or comment
>>> the relevant lines in the Django code base. Is there really no other way?
>>>
>>> I love the new migrations facilities in Django 1.7 and was a contributor
>>> to the kickstarter but please keep in mind that many legacy DB projects are
>>> not avoiding migrations because they want to. IMO it's usually because they
>>> are forced to for some (usually political) reason where they don't have
>>> control over the schema. Forcing a perpetually empty django_migrations
>>> table pretty much locks out those users.
>>>
>>> I see that people are worried about encouraging the non-use of
>>> migrations but might I suggest following the Zen of Python and making
>>> migrations the "one obvious way to do it" and turning them off the not
>>> obvious way.
>>>
>>> Ivan
>>>
>>> On Wednesday, January 22, 2014 at 5:40:35 AM UTC-6, Andrew Godwin wrote:
>>>>
>>>> My thoughts in brief on this:
>>>>
>>>>  - Database backends don't support migrations - they support schema
>>>> alteration via SchemaEditor. This could be used separately from migrations
>>>> if something wants it, and is meant to be an API on its own, so the backend
>>>> is not the place to say if you want migrations or not.
>>>>
>>>>  - There is some merit to the ability to turn off migrations on a
>>>> per-backend basis, via a DATABASES setting, but bear in mind that routers
>>>> already let you do this (with the allow_migrate method). The only extra
>>>> thing it would achieve is not having Django record migrations in the
>>>> django_migrations table, but it also looks like it would be useful for
>>>> tests if you hadn't written that part yet.
>>>>
>>>>  - I feel like a DB backend should at least implement SchemaEditor even
>>>> if it returns NotImplementedError for most of the endpoints; even in the
>>>> weirdest relational system, you can still manage create/delete model
>>>> without too much difficulty.
>>>>
>>>>  - Bear in mind that the plan is to remove DatabaseCreation entirely in
>>>> favour of SchemaEditor in a few releases' time (and backends are more than
>>>> welcome to make DatabaseCreation use SchemaEditor behind the scenes if they
>>>> want), so at that point if you don't implement it the backend is only ever
>>>> good for querying, which to me feels like an incomplete backend.
>>>>
>>>>  - I'm not sure what the future of ./manage.py sqlall is, but it's
>>>> going to have to be ported to SchemaEditor in future anyway, so it's only
>>>> useful in the transition.
>>>>
>>>> Looking at the discussion, I think the best thing to do is:
>>>>
>>>>  - Make the schema and migrations test skip if
>>>> connection.schema_editor() raises a NotImplementedError (not too hard,

Re: Possible bug: makemigrations caching prior migrations after deletion

[Andrew Godwin]

Indeed, Django can make many migrations for an initial set if it needs them
to de-circularise dependencies (e.g. two models with foreign keys pointing
at each other - it splits one of their FKs into a second migration).

Andrew

On Sun, Feb 8, 2015 at 11:17 PM, Curtis Maloney 
wrote:

> Could you provide details about what sort of field you added, please?
>
> I have seen cases where migrations will create two separate migrations for
> an initial.
>
> --
> Curtis
>
> On 9 February 2015 at 10:11, Yo-Yo Ma  wrote:
>
>> Using Python 3.4.2 and Django @stable/1.8.x (installed just moments
>> before this writing), I created a model with a "name" field, then created a
>> migration for it. Moments after, I added an "age" field to the model,
>> deleted the 0001_initial.py migration, deleted __pycache__ directories in
>> my entire project, deleted any *.py[cod] files in my project, then ran
>> makemigrations again (the goal being to create a fresh 0001 migration with
>> the latest fields again, since there is no SQLite3 database file (I
>> verified this as well)). When I do this, Django creates the same 0001
>> migration it did the first time, then creates a 0002 migration to add the
>> new field. Firstly, how is this even possible? Second, is this a known
>> issue in 1.8? I've never experienced this bug before, and it's always been
>> my M.O. on a new project to recreate a 0001 migration when I'm first
>> working on the models (to avoid having 30 migrations before the app is even
>> on a dev server).
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/bfb1f2cd-b2be-4a16-a882-ecc6695865c9%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAG_XiSD%3DG4tR6fE4aF6xdgdEX6%3D0gRRCTW8%3DyiS8g9KBrBz7pA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upLKUR4G2eKMCegBii72LWjB%3DvXoNe9VR-e1FC8SQWHWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Bytecode in Migrations

[Andrew Godwin]

Your problem is likely that the pyc file for the new initial migration
somehow seems newer than the py file and so Python is using it over the new
source file.

I'm not sure how this happens if it's a new migration, but I've seen it
happening when switching git branches before. We have pyc files turned off
in dev to stop this.

Andrew
On 11 Feb 2015 21:05, "abhillman"  wrote:

> I have gotten bitten by lingering bytecode in migrations on several
> occasions. Steps to reproduce are a little bit complex, but here is an
> rough example:
>
> Local Box:
> — have *.pyc rule in .gitignore
> — create some migrations
> — commit them
>
> Server Box:
> — pull repository
> — execute migrations
>
> Local Box:
> — remove migrations
> — commit
> — create new initial migration
>
> Server Box:
> — pull repository
> — remove migrations history from db
> — run fake initial migration — get an error here
>
> At this point, the server box has a migrations directory that looks
> something like this (bytecode is still around because it was in the
> gitignore):
> — migrations
> — __init__.py
> — 0001_initial.py
> — 0001_initial.pyc
> — 0002_second.pyc
> — 0003_third.pyc
>
> When running "python manage migrate" the bytecode appears to be
> referenced, which often causes errors when running the migrations. When
> using git, for example "git clean -df" clears out the problem. What I am
> wondering is if it might make sense to make a deliberate attempt to ignore
> bytecode without accompanying *.py files. This appears to be an issue
> because of the way that migrations dynamically import python code, but I am
> not sure. Perhaps the problem is more subtle as I am not deeply familiar
> with the way that migrations work.
>
> aryeh
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1dbf1434-b24a-47c9-b45b-ae04530d7867%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo%3D3M5iD58f-NGOYub7v8_yWs3Tp%3DXrBxvO0N66xMEPsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Signature of the allow_migrate database router.

[Andrew Godwin]

I am fine with the proposed change to allow better routing of
RunSQL/RunPython (though in the RunPython case people can easily do routing
inside the python code provided anyway, as you can see the DB aliases
there).

Aymeric: The problem with your proposal is that there's no easy way to get
the versioned model that the migration is using from just (app_label,
model_name); we'd need to pass the Apps instance in there as well. At least
if the whole model instance is around it'll be the correctly-versioned one
from the right point in time (I'm not sure if that makes a big difference
to most routers but it will stop lookups failing for models that have since
been deleted, for example)

Andrew

On Tue, Feb 17, 2015 at 12:58 PM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> On 17 févr. 2015, at 11:06, Loïc Bistuer  wrote:
>
> > The proposed fix for the release blocker
> https://code.djangoproject.com/ticket/24351 suggests changing the
> signature of the `allow_migrate` database router.
> >
> > From:
> > def allow_migrate(self, db, model):
> >
> > To:
> > def allow_migrate(self, db, app_label, model, **hints):
> >
> > We need to make a design decision.
>
> It appears that we have the change the signature of this function one way
> or another and that inspect.getargspec is our only option to preserve
> compatibility.
>
> Did you consider the following signature?
> def allow_migrate(self, db, app_label, model_name=None, **hints):
>
> While it’s a slightly larger change, it has a several advantages:
> - passing (app_label, model_name) as strings is an established convention
> in many other APIs.
> - it removes the possibility that app_label != model._meta.app_label —
> which doesn’t make much sense anyway.
> - our examples only ever use app_label — and keep repeating
> model._meta.app_label to get it.
>
> If the model class is needed, it can be fetched with
> apps.get_model(app_label, model_name).
>
> --
> Aymeric.
>
> PS: we chose to make 1.8 the next LTS instead of 1.7 precisely so we could
> make such adjustments.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/B42BA226-16FE-43D5-BE12-8FFD3DCD3094%40polytechnique.org
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo2tSTOUdZy0nAAftcEXGGdcZmezWCThkM9ePYe3_brFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Andrew Godwin]

I must also point out that the sqlmigrate command will still get you SQL
from migrations, though it sadly lacks the ability to take a range of
migrations, optimise them down, and output the SQL for _that_ - that's
probably the best thing for us to aim towards, and will almost entirely
recreate the functionality of sqlall without actually being wrong (sqlall
doesn't understand things like data migrations or custom SQL sections of
migrations, and so will sadly be wrong in if you have anything like a
moderately complex migration set).

The plus side is that it would be possible to do this sort of thing as a
third-party command/backport if we wanted to have it usable on 1.8 and even
1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it
in).

Andrew

On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee <
russ...@keith-magee.com> wrote:

>
> On Mon, Mar 30, 2015 at 1:36 AM, Pkl  wrote:
>
>> Hello all,
>>
>> I've noticed that all sql* commands that were in django core management
>> have disappeared from the latest sources.
>>
>> However, even though Django now has builtin south-style migrations now,
>> these *sql** commands (especially sql-all) were priceless to debug
>> differences between the models and the actual DB schema.
>> Especially as long as some django apps haven't caught up with all these
>> recent changes - I myself ended up with missing tables after scrupulously
>> following upgrade instructions, with "manage.py migrate" saying "all is
>> fine".
>>
>> *What is the new way to dump the sql schema of currently installed django
>> appz ?* It'd maybe be worth that I provide a doc patch to inform users
>> about it.
>> *If there is none, is there an agreement to restore at least sqlall and
>> sqlclear ?*
>>
>
> No, there isn't :-)
>
> Also what was the reasoning behind, on the first, place, blocking access
>> to these utilities in django1.7 (they seemed to work, when I forced them),
>> and now removing them ? A simple warning about the new "migrations" stuffs
>> (or a "--force" argument) would most probably have sufficed to prevent
>> improper usage of these sql* commands, while letting django users getting
>> the info they need to move forward, wouldn't it ?
>>
>
> Right now, there are two (almost) completely independent implementations
> of SQL table generation logic:
>
>  * The legacy version, used by syncdb, sqlall, et al
>
>  * The new version, used by migrate.
>
> In the past, there was a good reason for sqlall to exist - you needed to
> be able to run syncdb and create tables. The only time you ever called
> "CREATE TABLE" was when a table was fresh, and after that, you never issued
> any command related to the table schema. It didn't hurt anyone to allow
> sqlall to generate "what would the CREATE TABLE be if we had clean
> databases", and it was at least partially useful as a migration assistant,
> so the sqlall functionality was available.
>
> In the era of migrations, there's still a need to do a CREATE TABLE, but
> the table creation logic is a lot more complex, because it's not *just* a
> CREATE TABLE that is needed; all the ALTER TABLE statements are needed to
> add/remove columns as well.
>
> It would probably be *possible* to refactor manage.py sqlall to use the
> new table creation logic; but the question would then be "why do you want
> it"? manage.py migration *should* be handling all your table creation and
> migration functionality, so the need to manually diff a schema shouldn't
> exist. If you've hit problems with "manage.py migrate" *not* identifying
> migrations, then that's a much bigger problem, and one that needs to be
> reported as a bug - because it's a huge bug. Anything you can do to narrow
> down the situation that led to the problems you've observed would be very
> helpful.
>
> Yours,
> Russ Magee %-)
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAJxq84_Ww_ZcuhEx8MmfcxwY%3DGpMzuPGh1maGxBDJFCikUO5MA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.c

Re: Fate of sql* management commands

[Andrew Godwin]

 table creation and migration functionality, so the need to manually diff a
> schema shouldn't exist*
>
> My answer is : for the same reason that I need all other debug tools
> available, from django-debug-toolbar to lowest-level cpython debuggers. ^^
>
> I see tons of ways how stuffs could go wrong, from an accidental
> interruption of migrations (DDL instructions often having the flaw of being
> non-rollbackable), to bugs in migrations' code, without forgetting
> unorthodoxes events which may have happend to a project's conf (eg.
> switching from a USER_MODEL to another, and thus having to fix stuffs the
> hard way, with "--fake" migrations if necessary), or the handling of apps
> that have no support for migrations.
>
> Sure, people encountering troubles might make like I did - start a brand
> new project and mysqldump it after whole migration - but if we can expose
> that information in a straightforward way, like it was before, it's all
> benefits for everyone, isn't it ?
>
> regards,
> Pascal
>
>
>
>
>
>
>
> 2015-03-30 5:59 GMT+02:00 Andrew Godwin :
>
>> I must also point out that the sqlmigrate command will still get you SQL
>> from migrations, though it sadly lacks the ability to take a range of
>> migrations, optimise them down, and output the SQL for _that_ - that's
>> probably the best thing for us to aim towards, and will almost entirely
>> recreate the functionality of sqlall without actually being wrong (sqlall
>> doesn't understand things like data migrations or custom SQL sections of
>> migrations, and so will sadly be wrong in if you have anything like a
>> moderately complex migration set).
>>
>> The plus side is that it would be possible to do this sort of thing as a
>> third-party command/backport if we wanted to have it usable on 1.8 and even
>> 1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it
>> in).
>>
>> Andrew
>>
>> On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee <
>> russ...@keith-magee.com> wrote:
>>
>>>
>>> On Mon, Mar 30, 2015 at 1:36 AM, Pkl  wrote:
>>>
>>>> Hello all,
>>>>
>>>> I've noticed that all sql* commands that were in django core management
>>>> have disappeared from the latest sources.
>>>>
>>>> However, even though Django now has builtin south-style migrations now,
>>>> these *sql** commands (especially sql-all) were priceless to debug
>>>> differences between the models and the actual DB schema.
>>>> Especially as long as some django apps haven't caught up with all these
>>>> recent changes - I myself ended up with missing tables after scrupulously
>>>> following upgrade instructions, with "manage.py migrate" saying "all is
>>>> fine".
>>>>
>>>> *What is the new way to dump the sql schema of currently installed
>>>> django appz ?* It'd maybe be worth that I provide a doc patch to
>>>> inform users about it.
>>>> *If there is none, is there an agreement to restore at least sqlall and
>>>> sqlclear ?*
>>>>
>>>
>>> No, there isn't :-)
>>>
>>> Also what was the reasoning behind, on the first, place, blocking access
>>>> to these utilities in django1.7 (they seemed to work, when I forced them),
>>>> and now removing them ? A simple warning about the new "migrations" stuffs
>>>> (or a "--force" argument) would most probably have sufficed to prevent
>>>> improper usage of these sql* commands, while letting django users getting
>>>> the info they need to move forward, wouldn't it ?
>>>>
>>>
>>> Right now, there are two (almost) completely independent implementations
>>> of SQL table generation logic:
>>>
>>>  * The legacy version, used by syncdb, sqlall, et al
>>>
>>>  * The new version, used by migrate.
>>>
>>> In the past, there was a good reason for sqlall to exist - you needed to
>>> be able to run syncdb and create tables. The only time you ever called
>>> "CREATE TABLE" was when a table was fresh, and after that, you never issued
>>> any command related to the table schema. It didn't hurt anyone to allow
>>> sqlall to generate "what would the CREATE TABLE be if we had clean
>>> databases", and it was at least partially useful as a migration assistant,
>>> so the sqlall functionality was available.
>&g

Re: Fate of sql* management commands

[Andrew Godwin]

That is correct - RunPython and some other operations (definitely
DeleteIndex) don't have single SQL statements that can be output. There's
not much that can be done here, really - if you want to use SQL to set up
databases, then you can't use RunPython to do it, you should just use
migrations directly (as they were intended).

Andrew

On Tue, Mar 31, 2015 at 12:20 AM, Joachim Jablon 
wrote:

> Not sure about this but wouldn't "RunPython" actually keep you from
> getting an proven accurate result ? Even if we could imagine running the
> migration in a transaction and capturing the SQL that would be run, it
> would be dependent on the current data in the DB among other thing (could
> even be dependant on the meteo, or random variables or anything)...
>
> And given that you can (as the doc says) use the scheme editor inside a
> RunPython migration, who knows what the proper SQL would be ?
>
> (again, I may be missing a point there)
>
> --
> Joachim Jablon
>
>
> Le lundi 30 mars 2015 06:00:10 UTC+2, Andrew Godwin a écrit :
>>
>> I must also point out that the sqlmigrate command will still get you SQL
>> from migrations, though it sadly lacks the ability to take a range of
>> migrations, optimise them down, and output the SQL for _that_ - that's
>> probably the best thing for us to aim towards, and will almost entirely
>> recreate the functionality of sqlall without actually being wrong (sqlall
>> doesn't understand things like data migrations or custom SQL sections of
>> migrations, and so will sadly be wrong in if you have anything like a
>> moderately complex migration set).
>>
>> The plus side is that it would be possible to do this sort of thing as a
>> third-party command/backport if we wanted to have it usable on 1.8 and even
>> 1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it
>> in).
>>
>> Andrew
>>
>> On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee <
>> rus...@keith-magee.com> wrote:
>>
>>>
>>> On Mon, Mar 30, 2015 at 1:36 AM, Pkl  wrote:
>>>
>>>> Hello all,
>>>>
>>>> I've noticed that all sql* commands that were in django core management
>>>> have disappeared from the latest sources.
>>>>
>>>> However, even though Django now has builtin south-style migrations now,
>>>> these *sql** commands (especially sql-all) were priceless to debug
>>>> differences between the models and the actual DB schema.
>>>> Especially as long as some django apps haven't caught up with all these
>>>> recent changes - I myself ended up with missing tables after scrupulously
>>>> following upgrade instructions, with "manage.py migrate" saying "all is
>>>> fine".
>>>>
>>>> *What is the new way to dump the sql schema of currently installed
>>>> django appz ?* It'd maybe be worth that I provide a doc patch to
>>>> inform users about it.
>>>> *If there is none, is there an agreement to restore at least sqlall and
>>>> sqlclear ?*
>>>>
>>>
>>> No, there isn't :-)
>>>
>>> Also what was the reasoning behind, on the first, place, blocking access
>>>> to these utilities in django1.7 (they seemed to work, when I forced them),
>>>> and now removing them ? A simple warning about the new "migrations" stuffs
>>>> (or a "--force" argument) would most probably have sufficed to prevent
>>>> improper usage of these sql* commands, while letting django users getting
>>>> the info they need to move forward, wouldn't it ?
>>>>
>>>
>>> Right now, there are two (almost) completely independent implementations
>>> of SQL table generation logic:
>>>
>>>  * The legacy version, used by syncdb, sqlall, et al
>>>
>>>  * The new version, used by migrate.
>>>
>>> In the past, there was a good reason for sqlall to exist - you needed to
>>> be able to run syncdb and create tables. The only time you ever called
>>> "CREATE TABLE" was when a table was fresh, and after that, you never issued
>>> any command related to the table schema. It didn't hurt anyone to allow
>>> sqlall to generate "what would the CREATE TABLE be if we had clean
>>> databases", and it was at least partially useful as a migration assistant,
>>> so the sqlall functionality was available.
>>>
>>> In the era of migrations, there's still a need t

Re: Fate of sql* management commands

[Andrew Godwin]

>
> >
> > An alternative would be to ignore migrations files, regenerate a fresh
> > set of migrations, and dump the corresponding SQL.
>
> I think this approach would be much preferable to using the totally
> separate legacy code path. Presented as a tool for debugging migrations
> issues, and with the appropriate documentation notes about RunSQL etc, I
> think it would be a useful addition.
>

This is in fact what I was suggesting when I said "tie the autodetector
into the SQL writer directly in-memory". It would do 99% of what people
want, and more than the old one did.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uphO_F9%3DnYh6K6BCOhAaiw45F1RYoARH%2BmdMZBmWK7SNQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting database default values in migrations (postgres)

[Andrew Godwin]

Hi Sam,

That was also my idea for some time during development; I'm open to the
idea, though it does mean bringing database defaults into Django as an
overall concept, with all that that entails (e.g. documenting which default
takes precedence; dealing with database-specific defaults; how to represent
SQL functions; etc.).

If you're interested in whipping up a rough proposal, though, I'd be happy
to look over it - people have been asking for this since migrations came
around (and long before, too), as while Django never needs DB defaults
anything else using the database does (plus some concurrency issues with
default functions can be solved by moving them to the DB layer).

Andrew

On Sun, Apr 26, 2015 at 1:09 PM, Sam Baron  wrote:

> Greetings, I know this is an old thread, but it's an issue I am coming up
> against and hopefully I can restart the conversation.  I do think it's
> important that we use as many native database features as possible for
> application portability, extensibility, and performance.
>
> I have a different solution to the database default problem.  Why not
> handle it like SQLAlchemy?  There are two field properties for default -
> one applied at model instance (current Django default behavior) and one
> sent to the database.  In SQLAlchemy, it's called 'server_default', but for
> Django, I think 'db_default' would be a more appropriate name.
>
> So for example:
> integer_column = models.IntegerField(default=0)
> OR
> integer_column = models.IntegerField(db_default="0")
>
> And I agree that callables would not be included in 'db_default'.  This
> new property would only be for pushing SQL.  With this solution, I think
> you avoid user errors by keeping the two defaults behavior separate.  Only
> the advanced database folks will end up using it.
>
> I have made the relatively minor code changes on my end.  I am in the
> process and starting a new project and will keep testing it out.  I am
> curious if this would be an acceptable solution.
>
> Thanks,
> Sam
>
>
> On Saturday, November 1, 2014 at 10:17:50 AM UTC-7, Shai Berger wrote:
>>
>> On Friday 31 October 2014 19:16:15 Jon Dufresne wrote:
>> > On Fri, Oct 31, 2014 at 9:46 AM, Andrew Godwin 
>> wrote:
>> > > So, bear in mind that you can easily set these defaults yourself in a
>> > > migration with RunSQL if you need them for legacy purposes; that way
>> > > they'll get applied
>> >
>> > Absolutely. I effectively have such a system in place at the moment.
>> >
>> > But, my point is I am also making an effort to match Django's expected
>> > schema while moving away from the legacy schema. I would prefer not to
>> > drift too far from Django's expectations as the goal is move entirely
>> > to Django. This is just one more thing to keep track of and handle
>> > semi-manually.
>> >
>> > All I'm saying is that if the described feature existed, it would
>> > benefit me and others that share my use case.
>>
>> So, we should be weighing the support-transition-from-legacy use-case
>> against
>> creating a situation where field defaults get a special treatment if they
>> are
>> primitive enough (callables are out, but I am really not sure about other
>> complex objects -- GIS would probably gain a new dimension of fun if it
>> were
>> to deal  with defaults, even when not callable).
>>
>> I think the correct way forward for migrations is to keep as it does
>> today --
>> requiring users to explicitly ask for db defaults (we could, as I said
>> earlier, give it a nicer API than RunSQL).
>>
>> As for "Django's expectations" -- while I don't think we should generate
>> db
>> defaults unless specifically asked to, I don't see where such defaults
>> could
>> get in our way. If you ever run into a situation where Django mishandles
>> some
>> table because it has defaults, that is almost for sure a bug.
>>
>> My 2 cents,
>> Shai.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/e7d99cf6-c6a5-4d32-994c-7dd4e07ad478%40g