Re: [Python-Dev] PEP 376 - Open questions

[Nick Coghlan]

Tarek Ziadé wrote:
> On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby wrote:
>> So I punned on that in order to imply that nobody who takes on the job of
>> packaging expects to be asked the kind of detailed, unanswerable questions
>> that come with the territory of packaging, or to be subjected to torture by
>> third parties merely for trying to do something useful.
>>
>> And, by further implication, I was thus expressing camaraderie with you on
>> your initiation into the wonderful world of trying to satisfy everyone in
>> Pythonland, and our shared experience in the area of "if I'd really known
>> what I was getting into, I probably wouldn't have".  ;-)
> 
> Hehe very nice, thanks for the explanation :)
> 
> (thanks to the others too)

I think is exchange actually a good reminder that the bar for acceptance
of PEP 376 shouldn't be "Addresses every possible packaging issue we can
come up with". That would be setting expectations far too high because
packaging cross-platform is so messy.

A more appropriate bar is probably:
  a) Better that what we have now; and
  b) Provides possible avenues for even more improvement in the future

For example, I quite like the concept behind the various ideas for
"location" or "prefix" definitions either in the RECORD file itself or
in a separate PREFIXES file, since such approaches feeds directly in to
part b) above. Once some kind prefix definition mechanism is in place in
the metadata, it becomes possible to leverage it to categorise files for
the benefit of system packagers while still allowing simple
internal-use-only packages to just bundle everything together.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Antoine Pitrou]

Nick Coghlan  gmail.com> writes:
> 
> I think is exchange actually a good reminder that the bar for acceptance
> of PEP 376 shouldn't be "Addresses every possible packaging issue we can
> come up with". That would be setting expectations far too high because
> packaging cross-platform is so messy.

+1

> For example, I quite like the concept behind the various ideas for
> "location" or "prefix" definitions either in the RECORD file itself or
> in a separate PREFIXES file, since such approaches feeds directly in to
> part b) above.

I don't understand the point. Since RECORD is specific to a particular machine
and installation (it is generated at install time), what would you gain by
adding a fake genericity to its entries?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Nick Coghlan]

Antoine Pitrou wrote:
> Nick Coghlan  gmail.com> writes:
>> For example, I quite like the concept behind the various ideas for
>> "location" or "prefix" definitions either in the RECORD file itself or
>> in a separate PREFIXES file, since such approaches feeds directly in to
>> part b) above.
> 
> I don't understand the point. Since RECORD is specific to a particular machine
> and installation (it is generated at install time), what would you gain by
> adding a fake genericity to its entries?

My understanding is that RECORD isn't generated at install time for the
bdist_* commands -it's generated at the time the binary distribution is
created. Hence the problems raised with the idea of having absolute
paths in RECORD - if they were written directly into RECORD they would
reflect where the files were when the distribution was created, not
where they end up when it is installed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 Nick Coghlan :
> Antoine Pitrou wrote:
>> Nick Coghlan  gmail.com> writes:
>>> For example, I quite like the concept behind the various ideas for
>>> "location" or "prefix" definitions either in the RECORD file itself or
>>> in a separate PREFIXES file, since such approaches feeds directly in to
>>> part b) above.
>>
>> I don't understand the point. Since RECORD is specific to a particular 
>> machine
>> and installation (it is generated at install time), what would you gain by
>> adding a fake genericity to its entries?
>
> My understanding is that RECORD isn't generated at install time for the
> bdist_* commands -it's generated at the time the binary distribution is
> created. Hence the problems raised with the idea of having absolute
> paths in RECORD - if they were written directly into RECORD they would
> reflect where the files were when the distribution was created, not
> where they end up when it is installed.

This is a key open question, IMHO. No-one has done any work on
updating the core bdist_xxx commands to the new structure, and the PEP
is silent on the matter, so it's still all speculation.

The current "best guess" assumption is that, given that bdist_xxx uses
a "setup.py install --root=xxx" incantation to generate the structure
to package, the RECORD can be picked up from this. There are a number
of issues with this assumption, though - it doesn't handle absolute
locations well (there are existing bugs), paths may be affected by the
environment at the time the installer is built, and it requires RECORD
to be relocatable.

An alternative solution would be for the bdist_xxx commands to ignore
the RECORD file generated in the temp area, and build its own on the
target machine when the installer is run. This is conceptuially far
cleaner, it's in line with the implicit assumptions in the PEP, and it
avoids a lot of the problems with path formats (the filenames in
RECORD could be absolute local pathnames, and that's the end of the
story). But - and it's a big but - this dumps RECORD file handling on
the bdist_xxx command, and they all need to implement the same code,
resulting in a lot of duplication and opportunity for inconsistency.

I have no answer here. But I think we need feedback from the people
who will ultimately be building bdist_xxx formats - Debian packagers,
people wrting alternate RPM builders, etc. There's also Phillip
commenting from the setuptools POV and MAL who has mentioned his build
process.

Bottom line - Is RECORD to be created on the target machine at install
time, or must it be relocatable?
Subsequent question - if it's to be relocatable, how will that work?

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Antoine Pitrou]

Paul Moore  gmail.com> writes:
> 
> I have no answer here. But I think we need feedback from the people
> who will ultimately be building bdist_xxx formats - Debian packagers,
> people wrting alternate RPM builders, etc.

I think any bdist_xxx command which targets externally handled package formats
(rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
dpkg, you will uninstall with the same tool, won't you?

> Bottom line - Is RECORD to be created on the target machine at install
> time, or must it be relocatable?

This is quite an uninformed opinion, but a relocatable RECORD looks like a can
of worms to me.

As for creating RECORD at install time, I don't understand the argument about
duplicate code. Distutils can provide useful primitives for generating RECORD,
so that interested subcommands don't have a lot of additional work to do.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 Antoine Pitrou :
> Paul Moore  gmail.com> writes:
>>
>> I have no answer here. But I think we need feedback from the people
>> who will ultimately be building bdist_xxx formats - Debian packagers,
>> people wrting alternate RPM builders, etc.
>
> I think any bdist_xxx command which targets externally handled package formats
> (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
> dpkg, you will uninstall with the same tool, won't you?

I thought the purpose of RECORD was (also) so that *other* installs
could check who "owned" an existing file, before potentially
overwriting it. That seems like it's the reasoning behind the uses()
function.

But the PEP is very light on practical applications, so it's hard to
tell what the methods are intended to be used for.

>> Bottom line - Is RECORD to be created on the target machine at install
>> time, or must it be relocatable?
>
> This is quite an uninformed opinion, but a relocatable RECORD looks like a can
> of worms to me.
>
> As for creating RECORD at install time, I don't understand the argument about
> duplicate code. Distutils can provide useful primitives for generating RECORD,
> so that interested subcommands don't have a lot of additional work to do.

The bdist_wininst code is in C, and the bdist_msi code is driven by
the MSI data format. I don't know what scripting options exist for
.deb and .rpm files. Different languages, so no code sharing seems
possible. Maybe I'm wrong, and something could be done along the lines
of a "pkgutil.generate_record_file()" function which could be run from
a post-install script written in Python, but I don't know how
plausible or robust that would be...

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

Antoine Pitrou wrote:

Paul Moore  gmail.com> writes:

I have no answer here. But I think we need feedback from the people
who will ultimately be building bdist_xxx formats - Debian packagers,
people wrting alternate RPM builders, etc.


I think any bdist_xxx command which targets externally handled package formats
(rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
dpkg, you will uninstall with the same tool, won't you?


I agree with this. For RPM's, there's a whole other database of what 
files were installed. Is it really the intent that a file will be 
managed by multiple different installers? That I can install with RPM 
but remove with some python-installer (or other) tool? That way lies 
madness. In fact, I see RECORD as an installer-specific detail that 
doesn't need to be standardized at all.


For bdist_rpm (which I'm contemplating taking over, but no promises), 
the only thing that needs to be done with the metadata is be able to 
build a .spec file. Once the .spec file is generated, rpmbuild (or 
equivalent) will produce the .rpm files. I image that .deb's are 
similar, but I'm no expert.


Which brings up the "static metadata" argument I was making at PyCon. I 
want there to be enough static metadata that the new bdist_rpm can read 
so that it produces a well-behaved .spec file. I need to know the intent 
of some of the files (which are documentation, which are config files, 
which are scripts, which are libraries, etc.) and not much else.


I envision the new bdist_rpm as a standalone tool, not a distutils 
extension.



Bottom line - Is RECORD to be created on the target machine at install
time, or must it be relocatable?


This is quite an uninformed opinion, but a relocatable RECORD looks like a can
of worms to me.


Agreed.


As for creating RECORD at install time, I don't understand the argument about
duplicate code. Distutils can provide useful primitives for generating RECORD,
so that interested subcommands don't have a lot of additional work to do.


I envision a "distlib" that contains new support code that's not part of 
the current distutils morass.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Define metatype and a type that uses it

[Erik Groeneveld]

L.S.

I am working on an extension that allows one to open Java .jar files
and use the objects defined in it (via CNI or JNI) in Python.  It
works like a charm, but I am stuck with the definition of a metatype
for the wrappers.  I have to do this in Python now, with a helper
class.

I did find examples of how to define metatypes, but I can't find
anything on how to declare one type using this metatype.

Could anyone point me to examples or documentation or help me fix my code below?

What I have is a type JObject that has a metatype JObjectMeta, but the
objects created from it seem not properly initialized.   If I call
PyString_Str(self) in JObjectMeta_init, it seg faults.  De Python code
that initiates this is:

class MyJObject(JObject):
pass

Below is the complete, contained code of the extension.

Thanks in advance,
Erik Groeneveld


#include 

/*** JObjectMeta ***/
typedef struct {
PyObject_HEAD
} JObjectMeta;

PyTypeObject JObjectMetaType = {
PyObject_HEAD_INIT(NULL)
};

int JObjectMeta_init(PyObject* self, PyObject* args, PyObject *kwargs) {
printf("before str\n");
PyObject_Str(self);
printf("after str\n");
return 0;
}

PyObject* JObjectMeta_getattro(PyObject *self, PyObject *name) {
return PyBaseObject_Type.tp_getattro(self, name);
}


/*** JObject ***/
typedef struct {
PyObject_HEAD
} JObject;

PyTypeObject JObjectType = {
PyObject_HEAD_INIT(&JObjectMetaType)
};

int JObject_init(PyObject* self, PyObject* args, PyObject *kwargs) {
return 0;
}

static PyMethodDef methods[] = {
{NULL, NULL, 0, NULL}
};

/*** INIT ***/

PyMODINIT_FUNC
initmetaclass(void) {
JObjectMetaType.tp_name = "metaclass.JObjectMeta";
JObjectMetaType.tp_basicsize= sizeof(JObjectMeta);
JObjectMetaType.tp_flags= Py_TPFLAGS_DEFAULT;
JObjectMetaType.tp_doc  = "JObject objects";
JObjectMetaType.tp_init = JObjectMeta_init;
JObjectMetaType.tp_base = &PyType_Type; // must ?
JObjectMetaType.tp_new  = PyType_GenericNew;
JObjectMetaType.tp_getattro = JObjectMeta_getattro;

JObjectType.tp_name = "metaclass.JObject";
JObjectType.tp_basicsize= sizeof(JObject);
JObjectType.tp_flags= Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
JObjectType.tp_doc  = "JObjectMeta objects";
JObjectType.tp_init = JObject_init;
JObjectType.tp_base = &JObjectMetaType; // must ?
JObjectType.tp_new  = PyType_GenericNew;

PyObject* module = Py_InitModule3("metaclass", methods, "Metaclass
try out.");
if (module == NULL) {
PyErr_Print();
exit(-1);
}
if (PyType_Ready(&JObjectMetaType) < 0) {
PyErr_Print();
exit(-1);
}
if (PyModule_AddObject(module, "JObjectMeta", (PyObject*)
&JObjectMetaType) < 0) {
PyErr_Print();
exit(-1);
}
if (PyType_Ready(&JObjectType) < 0) {
PyErr_Print();
exit(-1);
}
if (PyModule_AddObject(module, "JObject", (PyObject*) &JObjectType) < 0 ) {
PyErr_Print();
exit(-1);
}
}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Define metatype and a type that uses it

[Christian Heimes]

Erik Groeneveld wrote:
> L.S.
> 
> I am working on an extension that allows one to open Java .jar files
> and use the objects defined in it (via CNI or JNI) in Python.  It
> works like a charm, but I am stuck with the definition of a metatype
> for the wrappers.  I have to do this in Python now, with a helper
> class.

Did you know that Andi Vajda has already created a fantastic wrapper for
Java called JCC? http://pypi.python.org/pypi/JCC/2.3

Contrary to your project it doesn't create the wrappers in runtime. It
analyzes the jar file(s) with JNI, created C++ templates and wraps the
C++ code in Python wrappers. JCC even allows you to implement native
methods of Java classes in Python.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 Eric Smith :
> [Any reason you didn't respond to the list? I'd like to continue the
> discussion there. But I'm currently replying just to you.]

No, I just clicked the wrong button. Sorry. Redirecting back to the list.

> Paul Moore wrote:
>>
>> 2009/7/8 Eric Smith :
>>>
>>> I agree with this. For RPM's, there's a whole other database of what
>>> files
>>> were installed. Is it really the intent that a file will be managed by
>>> multiple different installers? That I can install with RPM but remove
>>> with
>>> some python-installer (or other) tool? That way lies madness. In fact, I
>>> see
>>> RECORD as an installer-specific detail that doesn't need to be
>>> standardized
>>> at all.
>>
>> I thought RECORD was to ensure that you can install with RPM, and
>> *not* have any other random uninstaller remove your files (because you
>> populate RECORD, and the other uninstaller uses
>> pkgutil.get_file_users(path) to find out that you care about the file,
>> and hence avoid deleting it).
>
> I really don't think this is a use case worth supporting. A single file
> managed by multiple installers is a mess waiting to happen. RPM won't let
> you do it. About the only use case I can think of is namespace packages, and
> that can be handled by factoring out the __init__.py file into it's own
> install package.

Personally, I think you're right. But I have essentially zero
experience with the non-Windows use cases, so I don't feel qualified
to judge.

It seems to me that a lot of the fancy detecting what other people do
stuff is motivated by setuptools. I'm not going to comment on that,
other than to say that it may be that setuptools is generating more
problems than it's solving...

>> Basically, other install methods can't read your (RPM) database, and
>> you can't read their database. RECORD is a simple common format,
>> providing just enough information for everyone to communicate the
>> message "I manage this file".
>
> The RPM uninstaller won't look at the RECORD file. I think the best we can
> say is that _if_ an installer wants to share state about what's installed
> with other installers, this is how it's done.

OK. I'd be stunned if bdist_wininst or bdist_msi ends up checking RECORD.

So that's 3 "not interested" votes.

Who's going to use the APIs based around the RECORD file? Anyone?

Seriously - who is actually going to USE any of the APIs being
proposed in PEP 376? If no-one's interested, we should stop right now.
Do any of the people from the summit read this list? If not, can
anyone contact them and get some feedback?

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:

An alternative solution would be for the bdist_xxx commands to ignore
the RECORD file generated in the temp area, and build its own on the
target machine when the installer is run. This is conceptuially far
cleaner, it's in line with the implicit assumptions in the PEP, and it
avoids a lot of the problems with path formats (the filenames in
RECORD could be absolute local pathnames, and that's the end of the
story). But - and it's a big but - this dumps RECORD file handling on
the bdist_xxx command, and they all need to implement the same code,
resulting in a lot of duplication and opportunity for inconsistency.


The PREFIXES approach I suggested doesn't have these problems; the 
bdist installer only needs to write a PREFIXES file, and doesn't need 
to touch RECORD (which would be generated by the install --root).  In 
fact, since PREFIXES is strictly advisory, it can be generated by the 
install --root and left alone at installation time.  And, if it's 
generated without the --root prefix included, RPMs would have a 
correct PREFIXES file without further manipulation.  Plus, the 
bdist_wininst .exe installer can easily write a PREFIXES file, since 
it has to know the prefixes in order to do an installation in the first place.


(We still need to hash out an initial list of prefixes, of course.)


At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
I agree with this. For RPM's, there's a whole other database of what 
files were installed. Is it really the intent that a file will be 
managed by multiple different installers? That I can install with 
RPM but remove with some python-installer (or other) tool? That way 
lies madness. In fact, I see RECORD as an installer-specific detail 
that doesn't need to be standardized at all.


This is a misunderstanding.  The purpose is to let an *installer* 
(like easy_install) know that these files belong to some other 
installer, and not allow overwriting them.  That's why there's also 
an INSTALLER file.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:

Who's going to use the APIs based around the RECORD file? Anyone?


The distutils uninstall facility, for starters.  easy_install and pip 
also will, eventually.


In all three cases, the use will be in order to avoid overwriting 
files belonging to a system package manager, and in order to 
uninstall/upgrade their own files.


(I can't personally commit that pip will need this; but it's 
currently the only installer besides the distutils that installs 
things in this format, and would therefore need this to prevent 
overwriting things.)


Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES 
appropriately for both development and actual installations.




Seriously - who is actually going to USE any of the APIs being
proposed in PEP 376? If no-one's interested, we should stop right now.
Do any of the people from the summit read this list? If not, can
anyone contact them and get some feedback?


Package management people aren't likely to be doing anything with 
this stuff directly; if they need a list of installed files they're 
probably using --root or --record at package build time, not needing 
to sniff stuff and installation time, and they already have 
installation facilities.  Their only real need in relation to this 
will be to set an option to generate an INSTALLER file that says you 
can't uninstall their packages manually, thereby preventing the 
package from being uninstalled by the distutils utility, or any other 
well-behaved PEP 376-based package management tool.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Define metatype and a type that uses it

[Amaury Forgeot d'Arc]

Hello,

2009/7/8 Erik Groeneveld :
> I am working on an extension that allows one to open Java .jar files
> and use the objects defined in it (via CNI or JNI) in Python.  It
> works like a charm, but I am stuck with the definition of a metatype
> for the wrappers.  I have to do this in Python now, with a helper
> class.
>
> I did find examples of how to define metatypes, but I can't find
> anything on how to declare one type using this metatype.
>
> Could anyone point me to examples or documentation or help me fix my code 
> below?
[...]

There are several changes to do in your code:
- Don't define a JObjectMeta struct, use JObjectType directly instead.
An instance of the metatype is the type itself!
- Don't set JObjectMetaType.tp_basicsize, let it inherit from the base
type (the correct value would be sizeof(PyHeapTypeObject), this is
important in order to create derived classes in python)
- Don't set JObjectMetaType.tp_new, let it inherit from the base type
(PyType_GenericNew just allocates memory; types are more complex...)
- JObjectType.tp_base = &JObjectMetaType" is wrong, it should be
"JObjectType.ob_type = &JObjectMetaType;", or better to be compatible
with python3: "Py_TYPE(JObjectType) = &JObjectMetaType;"

In summary, I made your example work with:
   JObjectMetaType.tp_name = "metaclass.JObjectMeta";
   JObjectMetaType.tp_flags= Py_TPFLAGS_DEFAULT;
   JObjectMetaType.tp_doc  = "JObjectMeta objects";
   JObjectMetaType.tp_init = JObjectMeta_init;
   JObjectMetaType.tp_base = &PyType_Type;
   JObjectMetaType.tp_getattro = JObjectMeta_getattro;

   Py_TYPE(&JObjectType)   = &JObjectMetaType;
   JObjectType.tp_name = "metaclass.JObject";
   JObjectType.tp_basicsize= sizeof(JObject);
   JObjectType.tp_flags= Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
   JObjectType.tp_doc  = "JObject objects";
   JObjectType.tp_init = JObject_init;


-- 
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Tarek Ziadé]

On Wed, Jul 8, 2009 at 5:11 PM, P.J. Eby wrote:
> At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:
>>
>> An alternative solution would be for the bdist_xxx commands to ignore
>> the RECORD file generated in the temp area, and build its own on the
>> target machine when the installer is run. This is conceptuially far
>> cleaner, it's in line with the implicit assumptions in the PEP, and it
>> avoids a lot of the problems with path formats (the filenames in
>> RECORD could be absolute local pathnames, and that's the end of the
>> story). But - and it's a big but - this dumps RECORD file handling on
>> the bdist_xxx command, and they all need to implement the same code,
>> resulting in a lot of duplication and opportunity for inconsistency.
>
> The PREFIXES approach I suggested doesn't have these problems; the bdist
> installer only needs to write a PREFIXES file, and doesn't need to touch
> RECORD (which would be generated by the install --root).  In fact, since
> PREFIXES is strictly advisory, it can be generated by the install --root and
> left alone at installation time.  And, if it's generated without the --root
> prefix included, RPMs would have a correct PREFIXES file without further
> manipulation.  Plus, the bdist_wininst .exe installer can easily write a
> PREFIXES file, since it has to know the prefixes in order to do an
> installation in the first place.
>
> (We still need to hash out an initial list of prefixes, of course.)

Instead of building this list of prefixes from scratch, let's use the
ones that are *actually* used
when the install command is called, have a look at the list I've suggested :

http://mail.python.org/pipermail/python-dev/2009-July/090514.html

And let's drop other absolute paths,
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 P.J. Eby :
> At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:
>>
>> Who's going to use the APIs based around the RECORD file? Anyone?
>
> The distutils uninstall facility, for starters.  easy_install and pip also
> will, eventually.

Is pip getting this via setuptools, or will it be independently
changing its code? (Who's the pip contact who could confirm this
directly?)

> In all three cases, the use will be in order to avoid overwriting files
> belonging to a system package manager, and in order to uninstall/upgrade
> their own files.

To uninstall/upgrade their own files counts as "an installer-specific
detail" in the sense that Eric stated it.

To avoid overwriting files belonging to a system package manager: That
sounds like we're looking at a use case where people use a mixture of
tools to install packages into the system Python. Personally, I never
do this (I use bdist_wininst exclusively). The cases I can imagine
are:

- no OS-specific package available, and the user can't build his own,
so he uses setup.py install.
- user using easy_install as well as the OS package

In case (1), I'd imagine the user is careful enough (given that he's
doing everything manually) to avoid installing something that's
already present.
Case (2) is setuptools-related (see below for setuptools-related comments).

> (I can't personally commit that pip will need this; but it's currently the
> only installer besides the distutils that installs things in this format,
> and would therefore need this to prevent overwriting things.)

Not the only one - MAL mentioned his (internal use?) packaging process.

> Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES
> appropriately for both development and actual installations.

If no-one else is using them, that's an internal setuptools detail
again. Or it's for interoperability between setuptools and distutils
(and other packagers don't benefit unless they are using PEP 376
APIs).

>> Seriously - who is actually going to USE any of the APIs being
>> proposed in PEP 376? If no-one's interested, we should stop right now.
>> Do any of the people from the summit read this list? If not, can
>> anyone contact them and get some feedback?
>
> Package management people aren't likely to be doing anything with this stuff
> directly; if they need a list of installed files they're probably using
> --root or --record at package build time, not needing to sniff stuff and
> installation time, and they already have installation facilities.  Their
> only real need in relation to this will be to set an option to generate an
> INSTALLER file that says you can't uninstall their packages manually,
> thereby preventing the package from being uninstalled by the distutils
> utility, or any other well-behaved PEP 376-based package management tool.

So if I follow this, there's a use for this in making sure the new
distutils uninstall option doesn't break something it shouldn't, but
as the uninstall uses RECORD, it won't be (accidentally) usable on
distributions that didn't follow PEP 376 and create a RECORD file.

There's also a use for this in terms of setuptools/easy_install
interoperability.

I'm now going to take of my "PEP supporter" hat, and put on my "End
user" hat. Please understand that the following is *not* unbiased,
whereas the rest of what I've said *is* intended to be reasonably
impartial.

If the only driver for this PEP is setuptools, then I'm -1 on it.

Unless someone working on a packaging tool *other* than setuptools (or
setuptools-derived projects) speaks up and says "I have code of my own
which uses distutils, and I would benefit as follows from PEP 376 and
I will be changing my code to conform to PEP 376" I think the PEP
should be rejected, and the energy put into some other PEP which will
benefit the wider (non-setuptools) community.

OK, end of the opinion bit. I'll go back to trying to keep things
impartial in future mails.

Paul.

PS Note - I like the idea of the PEP in theory. It's the practical
benefits that I'm trying to establish.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

P.J. Eby wrote:

At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
I agree with this. For RPM's, there's a whole other database of what 
files were installed. Is it really the intent that a file will be 
managed by multiple different installers? That I can install with RPM 
but remove with some python-installer (or other) tool? That way lies 
madness. In fact, I see RECORD as an installer-specific detail that 
doesn't need to be standardized at all.


This is a misunderstanding.  The purpose is to let an *installer* (like 
easy_install) know that these files belong to some other installer, and 
not allow overwriting them.  That's why there's also an INSTALLER file.


I really don't see this use case. Supporting multiple installers for the 
same file (or even just trying to prevent two installers from writing 
the same file)? Wouldn't you be better off just saying an installer 
can't overwrite any existing file?


Likewise, I don't see a use case for installing with one installer and 
uninstalling with another.


Put those two together, and the mechanism that an installer uses to 
record what files it installed is a private implementation detail.


Eric.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Current patch process question

[Scott David Daniels]

I am confused by the current hydra-like development.  I've recently
submitted a change (issue6422 -- a feature request) which is currently
moving through the process.  I am not particularly asking for special
attention on that issue, but I am confused about the process I should be
using (in the face of two trunks).

There seem to be two places to plausibly introduce a change: py3k and
trunk.  Should I submit changes for trunk, py3k, or both?  I tried
asking in the initial feature request, but nobody replied, and I finally
decided that I'd go for patching trunk in the interim.

For myself, I'd like to know if I should add a patch file for the py3k
branch, or will that simply make more work for someone else?  Certainly
it will be easier for me to do this sooner rather than later.

For the wider group, how should I have known the answer to these
questions?  Is there some page I overlooked?  Is there a better forum
in which to ask this question?

--Scott David Daniels
scott.dani...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Current patch process question

[Antoine Pitrou]

Hi,

> There seem to be two places to plausibly introduce a change: py3k and
> trunk.  Should I submit changes for trunk, py3k, or both?  I tried
> asking in the initial feature request, but nobody replied, and I finally
> decided that I'd go for patching trunk in the interim.

Patches are usually first checked into trunk, and then svnmerge'd to py3k.
So it makes more sense to submit your patch against trunk.

> For myself, I'd like to know if I should add a patch file for the py3k
> branch, or will that simply make more work for someone else?

If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take
care of it.
If, on the other hand, porting your patch requires many changes, or delicate
ones, providing a patch against py3k can make the committer's life easier.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 05:22 PM 7/8/2009 +0100, Paul Moore wrote:

2009/7/8 P.J. Eby :
> At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:
>>
>> Who's going to use the APIs based around the RECORD file? Anyone?
>
> The distutils uninstall facility, for starters.  easy_install and pip also
> will, eventually.

Is pip getting this via setuptools, or will it be independently
changing its code? (Who's the pip contact who could confirm this
directly?)


It would likely be changing its code directly, with respect to 
supporting overwrite-avoidance and uninstall-before-upgrade 
scenarios.  (Generation support would likely come from setuptools, 
dependent on who implemented first.)  Ian Bicking is pip's author.




In case (1), I'd imagine the user is careful enough (given that he's
doing everything manually) to avoid installing something that's
already present.


Obviously, you've never tried doing this yourself.  ;-)

My guess would actually be that in Unixy environments these days, 
people either use virtualenv or buildouts to manage such 
things.  Buildout handles uninstall-before-update (for eggs, anyway, 
I'm not sure about other things) and virtualenv makes it easy to 
create throwaway Python installs.




So if I follow this, there's a use for this in making sure the new
distutils uninstall option doesn't break something it shouldn't, but
as the uninstall uses RECORD, it won't be (accidentally) usable on
distributions that didn't follow PEP 376 and create a RECORD file.


That's an interesting point.  But for non-distutils tools, it'd be 
nice to be able to find out whether another package manager is 
administering some particular file.  For that matter, the distutils 
themselves should (ultimately) support uninstall-before-upgrade and 
avoid overwriting another package manager's files.




If the only driver for this PEP is setuptools, then I'm -1 on it.


If it were being driven by setuptools, I'd have just implemented it 
myself and presented it as a fait accompli.  I can't speak to Tarek's 
motives, but I assume that, as stated in the PEP, the primary driver 
is supporting the distutils being able to uninstall things, and 
secondarily to allow other tools to be built on top of the API.




a packaging tool *other* than setuptools (or setuptools-derived projects)


Is there really such a thing?  ;-)

AFAIK, every published tool for managing Python projects is either 
distutils-based or setuptools-based.  (Things like scons and pymake 
and various other project build tools don't seem to fall under 
"packaging tool" in this sense.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 01:58 PM 7/8/2009 -0400, Eric Smith wrote:
I really don't see this use case. Supporting multiple installers for 
the same file (or even just trying to prevent two installers from 
writing the same file)? Wouldn't you be better off just saying an 
installer can't overwrite any existing file?


Likewise, I don't see a use case for installing with one installer 
and uninstalling with another.


Put those two together, and the mechanism that an installer uses to 
record what files it installed is a private implementation detail.


You'll have to ask Tarek about that.  As I've said in another email, 
if I were doing this for setuptools I'd just go and do it; it's 
already on my backlog of things to do.  The point of the PEP is to 
provide an uninstall facility for the distutils, and to expose those 
records via an API so that other tools can be built on top of it.


Whether anything other than distutils-based tools will actually use 
the format directly is entirely irrelevant to the PEP's merits, since 
bdist_* tools and platform-specific distutils installers may need to 
read or write the format.  Hence, a PEP is appropriate to describe a 
format and API for the distutils family of tools to operate on.


The specific proposed format is based on an existing de facto 
standard for installed package metadata, so that the many existing 
packaging tools based on that other format and existing APIs can 
easily read or write this data, and so that the API can also be used 
as a way to detect the presence of already-installed distributions of 
software -- even ones installed other ways.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Martin v. Löwis]

>> Hehe very nice, thanks for the explanation :)
>>
>> (thanks to the others too)
> 
> I think is exchange actually a good reminder that the bar for acceptance
> of PEP 376 shouldn't be "Addresses every possible packaging issue we can
> come up with". That would be setting expectations far too high because
> packaging cross-platform is so messy.

OTOH, it *is* expected that the PEP explicitly lists aspects that it
didn't resolve, so that any reader can find out whether something was
discussed and rejected, or not discussed.

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Current patch process question

[Martin v. Löwis]

>> There seem to be two places to plausibly introduce a change: py3k and
>> trunk.  Should I submit changes for trunk, py3k, or both?  I tried
>> asking in the initial feature request, but nobody replied, and I finally
>> decided that I'd go for patching trunk in the interim.
> 
> Patches are usually first checked into trunk, and then svnmerge'd to py3k.
> So it makes more sense to submit your patch against trunk.
> 
>> For myself, I'd like to know if I should add a patch file for the py3k
>> branch, or will that simply make more work for someone else?
> 
> If porting the patch to py3k is trivial, you shouldn't care. svnmerge will 
> take
> care of it.
> If, on the other hand, porting your patch requires many changes, or delicate
> ones, providing a patch against py3k can make the committer's life easier.

I usually abstain from "me too" messages, but only this time:
as Antoine says.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Current patch process question

[Scott David Daniels]

Martin v. Löwis wrote:

There seem to be two places to plausibly introduce a change: py3k and
trunk.  Should I submit changes for trunk, py3k, or both?  

Patches are usually first checked into trunk, and then svnmerge'd to py3k.
So it makes more sense to submit your patch against trunk.


For myself, I'd like to know if I should add a patch file for the py3k
branch, or will that simply make more work for someone else?

If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take
care of it.
If, on the other hand, porting your patch requires many changes, or delicate
ones, providing a patch against py3k can make the committer's life easier.


I usually abstain from "me too" messages, but only this time:
as Antoine says.


Thanks to both of you for the quick response.  In this case, the port
should be dirt simple, so I'll consider myself done until comments
come back on the patch.

--Scott David Daniels
scott.dani...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 P.J. Eby :
> If it were being driven by setuptools, I'd have just implemented it myself
> and presented it as a fait accompli.  I can't speak to Tarek's motives, but
> I assume that, as stated in the PEP, the primary driver is supporting the
> distutils being able to uninstall things, and secondarily to allow other
> tools to be built on top of the API.

My understanding is that all of the various distutils PEPs were driven
by the "packaging summit" ay PyCon. The struggle here seems to be to
find *anyone* from that summit who will now comment on the discussion
:-(

>> a packaging tool *other* than setuptools (or setuptools-derived projects)
>
> Is there really such a thing?  ;-)

I wonder... :-)

> AFAIK, every published tool for managing Python projects is either
> distutils-based or setuptools-based.  (Things like scons and pymake and
> various other project build tools don't seem to fall under "packaging tool"
> in this sense.)

The big driver seems to be things that take distutils packages and
build Linux packages (debs, rpms, ...) from them. Unfortunately, I
know nothing about these tools, and no-one seems to be speaking up on
the subject.

Maybe this was all thrashed out on the distutils SIG, and consensus
reached there (I only skim that list). If so, maybe this list is the
wrong one to be discussing changes to the PEP on.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

Paul Moore wrote:

2009/7/8 P.J. Eby :

If it were being driven by setuptools, I'd have just implemented it myself
and presented it as a fait accompli.  I can't speak to Tarek's motives, but
I assume that, as stated in the PEP, the primary driver is supporting the
distutils being able to uninstall things, and secondarily to allow other
tools to be built on top of the API.


My understanding is that all of the various distutils PEPs were driven
by the "packaging summit" ay PyCon. The struggle here seems to be to
find *anyone* from that summit who will now comment on the discussion
:-(


I was there, and I've been commenting!

There might have been more discussion after the language summit and the 
one open space event I went to. But the focus as I recall was static 
metadata and version specification. When I originally brought up static 
metadata at the summit, I meant metadata describing the sources in the 
distribution, so that we can get rid of setup.py's. From that metadata, 
I want to be able to generate .debs, .rpms, .eggs, etc.


But I think we've veered into metadata that describes what has been 
installed. I don't think that's so useful. As I've said, this is private 
to the installers. If 2 installers want to communicate with each other 
about what they've installed, then they can agree on that data. I just 
don't find it generally useful for all installers, and therefore not 
useful for distutils.


I'd like to get back to the metadata that describes the source files. 
That's where the real value lies, in my opinion. I'll try and work on a 
post to distutils-sig explaining my thinking.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Paul Moore]

2009/7/8 Eric Smith :
> I was there, and I've been commenting!

Sorry, I hadn't realised that. Thanks for the correction.

> There might have been more discussion after the language summit and the one
> open space event I went to. But the focus as I recall was static metadata
> and version specification. When I originally brought up static metadata at
> the summit, I meant metadata describing the sources in the distribution, so
> that we can get rid of setup.py's. From that metadata, I want to be able to
> generate .debs, .rpms, .eggs, etc.
>
> But I think we've veered into metadata that describes what has been
> installed. I don't think that's so useful. As I've said, this is private to
> the installers. If 2 installers want to communicate with each other about
> what they've installed, then they can agree on that data. I just don't find
> it generally useful for all installers, and therefore not useful for
> distutils.
>
> I'd like to get back to the metadata that describes the source files. That's
> where the real value lies, in my opinion. I'll try and work on a post to
> distutils-sig explaining my thinking.

OK, that helps a lot. I see how your postings fit into things a little
better now.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[David Cournapeau]

On Thu, Jul 9, 2009 at 7:07 AM, Eric Smith wrote:
> Paul Moore wrote:
>>
>> 2009/7/8 P.J. Eby :
>>>
>>> If it were being driven by setuptools, I'd have just implemented it
>>> myself
>>> and presented it as a fait accompli.  I can't speak to Tarek's motives,
>>> but
>>> I assume that, as stated in the PEP, the primary driver is supporting the
>>> distutils being able to uninstall things, and secondarily to allow other
>>> tools to be built on top of the API.
>>
>> My understanding is that all of the various distutils PEPs were driven
>> by the "packaging summit" ay PyCon. The struggle here seems to be to
>> find *anyone* from that summit who will now comment on the discussion
>> :-(
>
> I was there, and I've been commenting!
>
> There might have been more discussion after the language summit and the one
> open space event I went to. But the focus as I recall was static metadata
> and version specification. When I originally brought up static metadata at
> the summit, I meant metadata describing the sources in the distribution, so
> that we can get rid of setup.py's. From that metadata, I want to be able to
> generate .debs, .rpms, .eggs, etc.

I agree wholeheartedly. Getting rid of setup.py for most packages
should be a goal IMHO. Most packages don't need anything fancy, and
static metadata are so much easier to use compared to
setup.py/distutils for 3rd party interop.

There was a discussion about how to describe/find the list of files to
form a distribution (for the different sdist/bdist_* commands), but no
agreement was reached. Some people strongly defend the setuptools
feature to get the list of files from the source control system, in
particular.

http://mail.python.org/pipermail/distutils-sig/2009-April/011226.html

David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Stephen J. Turnbull]

Eric Smith writes:

 > But I think we've veered into metadata that describes what has been 
 > installed. I don't think that's so useful. As I've said, this is private 
 > to the installers. If 2 installers want to communicate with each other 
 > about what they've installed, then they can agree on that data. I just 
 > don't find it generally useful for all installers, and therefore not 
 > useful for distutils.

ISTM that the problem that it solves is uninstall in the absence of
the original installer.  Am I to understand that you don't think that
use case is important?  Or that there's another way to do this?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Steven D'Aprano]

On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote:
> But I think we've veered into metadata that describes what has been
> installed. I don't think that's so useful. As I've said, this is
> private to the installers. If 2 installers want to communicate with
> each other about what they've installed, then they can agree on that
> data. I just don't find it generally useful for all installers, and
> therefore not useful for distutils.

But doesn't this metadata give any two installers a common language to 
use to communicate, instead of having every pair of installers create 
their own private communication method?

Personally, I like to be able to look at a package and say "What did 
that install?" Or contrary-wise, look at a file and say "What package 
installed that?" There are few things worse than discovering a bunch of 
mysterious executable files on your system that you don't remember 
installing, and then spending a few paranoid hours trying to determine 
whether you've stumbled across a root kit or virus or whether they have 
a legitimate reason to be there.



-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

Stephen J. Turnbull wrote:

Eric Smith writes:

 > But I think we've veered into metadata that describes what has been 
 > installed. I don't think that's so useful. As I've said, this is private 
 > to the installers. If 2 installers want to communicate with each other 
 > about what they've installed, then they can agree on that data. I just 
 > don't find it generally useful for all installers, and therefore not 
 > useful for distutils.


ISTM that the problem that it solves is uninstall in the absence of
the original installer.  Am I to understand that you don't think that
use case is important?  Or that there's another way to do this?


Yes, I think that's a use case that doesn't need to be supported by this 
PEP. It's an issue for a single installer, or a group of installers. 
It's not something that _all_ installers need to care about.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote:

Eric Smith writes:

 > But I think we've veered into metadata that describes what has been
 > installed. I don't think that's so useful. As I've said, this is private
 > to the installers. If 2 installers want to communicate with each other
 > about what they've installed, then they can agree on that data. I just
 > don't find it generally useful for all installers, and therefore not
 > useful for distutils.

ISTM that the problem that it solves is uninstall in the absence of
the original installer.


Or uninstall where the installer is "setup.py install", actually.



  Am I to understand that you don't think that
use case is important?  Or that there's another way to do this?


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - get_egginfo_files

[Sridhar Ratnakumar]

On Sun, 05 Jul 2009 11:46:58 -0700, Paul Moore  wrote:


2009/7/5 P.J. Eby :

At 05:26 PM 7/5/2009 +0100, Paul Moore wrote:


def get_distribution(name):
   for d in get_distributions():
   if d.name == name:
   return d
   return None


Btw, this is broken code anyway, because it's not handling
case-insensitivity or name canonicalization.  (I've mentioned these  
issue

previously on the distutils-sig.)

Fair point. (Although I don't recall your distutils-sig posting, so
I'm not sure what you mean by "name canonicalisation").


Correct me if I'm wrong - I believe he was referring to the `safe_name`  
function:


  http://mail.python.org/pipermail/distutils-sig/2005-September/005120.html

-srid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Sridhar Ratnakumar]

On Wed, 08 Jul 2009 09:22:52 -0700, Paul Moore  wrote:


If the only driver for this PEP is setuptools, then I'm -1 on it.
Unless someone working on a packaging tool *other* than setuptools (or
setuptools-derived projects) speaks up and says "I have code of my own
which uses distutils, and I would benefit as follows from PEP 376 and
I will be changing my code to conform to PEP 376" I think the PEP
should be rejected, and the energy put into some other PEP which will
benefit the wider (non-setuptools) community.


Other than easy_install/pip, there is also PyPM which is being developed  
at ActiveState. PyPM is the Python package manager much like what ppm is  
for ActivePerl.


We build packages using 'setup.py install --root', but when it comes  
installation on client .. there is a separate database of installed  
packages that has the following information for each package installed:


   - metadata (name, version, author, etc..)
   - installed_files (list of files installed on client)

So "pypm remove Foo" will simply remove all the 'installed_files' of Foo  
and remove Foo from the package database. The package database is  
currently an sqlite-based one.


PEP 376 definitely will benefit PyPM besides benefiting setuptools/pip. If  
all package managers rely on common API to 1) retrieve metadata of  
installed packages 2) write metadata into site-packages), then it is very  
possible that one package manager does not mess up with the installation  
of others. For example:


  $ pip install IPython
  $ pypm install ipython

Without PEP 376, pypm will overwrite pip's installation. But with PEP 376,  
as pip writes the RECORD file, pypm will reuse it (via the API) and  
co-operate with it.


Eventually, I'd like PEP 376 to support system packagers too. So for  
example, if you did "apt-get install python-pyqt4", then running "pip  
install python-pyqt4" should return without installing anything .. as  
RECORD will be part of the .deb previously installed. As for generating  
the RECORD file, I vote for generating it during install time (w/ absolute  
paths).


-srid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Absolute" paths in PEP 376 RECORD files

[Sridhar Ratnakumar]

Is there any reason why RECORD file can't be generated at runtime? Also,  
why should the RECORD file be generated at all by bdist* commands?


A .deb file contains "data.tar.gz" that is simply extracted over "/".  
There is no need for RECORD inside a .deb file because it is implicit in  
the contents on data.tar.gz.


-srid

On Tue, 07 Jul 2009 13:30:05 -0700, P.J. Eby  wrote:

Just an idea...  suppose that instead of using "real" absolute paths in  
the RECORD file for non-local files (scripts, data, etc) we changed the  
format to include a "prefix" field, containing something like LIBDIR,  
SHARE, SCRIPTS, etc., ala bdist_wininst internals?


Also, we could include a separate (optional) PREFIXES file defining what  
those locations translated to at install time.  Dumb bdists would still  
have their build paths there (or delete the file before packaging, or  
use real paths instead of the --root prefixed versions).


Upside: relocation can be detected and handled, RECORD remains  
cross-platform, and bdists are ok.


Downside: more complex API required to read/manipulate paths and delete  
files, since you need to be able to check that you have the right  
prefixes, and may have to ask the user for a prefix you don't recognize,  
if the default in PREFIXES doesn't match.


Thoughts?

(Also, as a special case, any file that's actually installed to LIBDIR  
or a subdirectory thereof (even if it's technically a "data" file or  
script), will probably need to be designated under LIBDIR prefix to  
prevent runtime breakage in the event any system package maintainers are  
tempted to use RECORD files as a way of forcing a HFS conformance.   
These are installation prefixes, *not* content types.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:  
http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Absolute" paths in PEP 376 RECORD files

[P.J. Eby]

At 06:51 PM 7/8/2009 -0700, Sridhar Ratnakumar wrote:

Is there any reason why RECORD file can't be generated at runtime?


Applications can be relocatable, yet require plugins and libraries to 
be upgraded, installed, uninstalled, etc.


(Also, RECORD is cross-platform to allow cross-platform 
installation/testing of cross-platform packages; that's why I propose 
keeping the prefix information separate.)




 Also,
why should the RECORD file be generated at all by bdist* commands?


bdist commands that use "install --root" will get it as a side 
effect, so they don't have to do anything special.  They needn't even 
generate the PREFIXES, if PREFIXES is also generated by "install 
--root", and omits the root from the prefixes written.  In other 
words, most bdist tools wouldn't change.  bdist_wininst and bdist_msi 
might need to regenerate PREFIXES at actual installation time, but 
AFAIK those are the only ones.  (And the spec would allow for the 
possibility that a PREFIXES file can be incorrect.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

Steven D'Aprano wrote:

On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote:

But I think we've veered into metadata that describes what has been
installed. I don't think that's so useful. As I've said, this is
private to the installers. If 2 installers want to communicate with
each other about what they've installed, then they can agree on that
data. I just don't find it generally useful for all installers, and
therefore not useful for distutils.


But doesn't this metadata give any two installers a common language to 
use to communicate, instead of having every pair of installers create 
their own private communication method?


I really don't get this use case of multiple installers trying to 
install the same package. There's just no way that running "yum install 
twisted" and "apt-get install twisted" and "pip install twisted" are 
going to coexist with each other. The best they can do is say "a file 
I'm trying to install already exists". Why try for anything else?


Personally, I like to be able to look at a package and say "What did 
that install?" Or contrary-wise, look at a file and say "What package 
installed that?"


I completely support that. All of the system installers I know of 
support this (at least on Linux; I'm familiar with the msi database 
layout (as exposed by Orca) on Windows, but I've never looked to see if 
they record the results of installing an msi). I just see this as an 
issue within each installer. Why would the installers need to 
communicate this with each other?


There are few things worse than discovering a bunch of 
mysterious executable files on your system that you don't remember 
installing, and then spending a few paranoid hours trying to determine 
whether you've stumbled across a root kit or virus or whether they have 
a legitimate reason to be there.


We can't solve the problem of "here's an arbitrary file on the system, 
where did it come from?". There are many, many installers outside of our 
control that might have installed something. Ruby has installers, Perl 
has installers, I'm sure there are hundreds of such installers. The user 
might have even used wget to install something.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

P.J. Eby wrote:

ISTM that the problem that it solves is uninstall in the absence of
the original installer.


Or uninstall where the installer is "setup.py install", actually.


I think we need to move away from "setup.py install". It's the 
antithesis of static metadata. setup.py needs to go away.


If we have a static metadata file called, say "setup.info", then I'd 
like to see:

"bdist_rpm setup.info" produce a .rpm
"bdist_msi setup.info" produce a .msi
"bdist_deb setup.info" produce a .deb
"bdist_egg setup.info" produce a .egg
and so on.

There might be a library (and I call dibs on the name "distlib" :) that 
provides support routines to parse setup.info, but there's no framework 
involved. And no need for a plugin system.


There might be support routines used to build setup.info. For those who 
think it's reasonable that the list of source files comes from your 
version control system, build a small command that produces setup.info 
from your vcs. If you need the version number to come from some other 
source, write a small command that reads that and sticks it into a 
setup.info. But the source distribution would be defined by a (possibly 
generated) setup.info.


The metadata in setup.info would be extensible. If setup.info contains 
information that's needed to build a .rpm that's not needed for a .deb, 
then that's fine. We can define the minimum information needed and an 
extension mechanism. As time goes by, some of the extensions might 
become standardized.


Then for each binary format, there's an installer (or more than one!), 
and it might handle dependency detection and automatic fetching, or it 
might not. Some of these exist (apt-get, yum, msi), some might need some 
tweaking (setuptools, pip) and others might need to be written from 
scratch. But the installer, its database of what it has installed, and 
how it handles uninstalls (if it does) is really not the issue.


It's the conflation of "build a package" with "install a package" that 
distutils and setuptools has given us (via setup.py) that leads to us 
never making progress. Let's just focus on "describe a package" and let 
the packagers handle "build a package" and let the installers handle 
"install and uninstall a package". Then the packagers (and their 
associated installers) can evolve independently of each other.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[Eric Smith]

Eventually, I'd like PEP 376 to support system packagers too. So for 
example, if you did "apt-get install python-pyqt4", then running "pip 
install python-pyqt4" should return without installing anything .. as 
RECORD will be part of the .deb previously installed. As for generating 
the RECORD file, I vote for generating it during install time (w/ 
absolute paths).


I think we should explicitly not support this. What if pip and apt-get 
(or rpm, or others) install the same package in different places because 
of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no 
way we're ever going to get this right, and it's not worth complicating 
our lives over it.


Seriously: Is there some real world use case I'm missing? Does any 
existing install system support this?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 11:20 PM 7/8/2009 -0400, Eric Smith wrote:

P.J. Eby wrote:

ISTM that the problem that it solves is uninstall in the absence of
the original installer.

Or uninstall where the installer is "setup.py install", actually.


I think we need to move away from "setup.py install". It's the 
antithesis of static metadata.


Please note that that's entirely out of scope for the PEP at hand.

That being said, the rest of your proposal is strikingly similar to a 
proposal I previously floated on the distutils-sig for a concept 
called BUILDS.  The main difference is that I suggested that the spec 
should include a standard interface for running build operations that 
would produce the manifest (equivalent to your setup.info), and that 
distutils and setuptools should provide setup.py commands to generate 
said manifest, to allow for a seamless transition.


There was very little comment on the proposal, perhaps because it 
involves a lot of work that most people are sane enough not to sign 
up for.  ;-)


(Or more to the point, it's the sort of thing that never gets off the 
ground for design by consensus.  The best way to make something like 
your proposal to happen is to go off and build it, and get it to take 
over.  If it can't win substantial market share on its own merits, it 
probably doesn't deserve to be blessed as a standard.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[P.J. Eby]

At 11:28 PM 7/8/2009 -0400, Eric Smith wrote:
Eventually, I'd like PEP 376 to support system packagers too. So 
for example, if you did "apt-get install python-pyqt4", then 
running "pip install python-pyqt4" should return without installing 
anything .. as RECORD will be part of the .deb previously 
installed. As for generating the RECORD file, I vote for generating 
it during install time (w/ absolute paths).


I think we should explicitly not support this. What if pip and 
apt-get (or rpm, or others) install the same package in different 
places because of system conventions (/usr vs. /usr/local vs. /opt, 
say)? There's no way we're ever going to get this right, and it's 
not worth complicating our lives over it.


Seriously: Is there some real world use case I'm missing? Does any 
existing install system support this?


The use case described above is supported right now by easy_install 
and pip; it doesn't require a RECORD file, though.  An .egg-info file 
or an .egg-info directory with a PKG-INFO is enough to prevent 
duplicate installation.


I don't know how pip handles file overwrites; easy_install OTOH never 
overwrites anything that's not in an .egg or a .pth.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[glyph]

On 02:55 am, e...@trueblade.com wrote:
I really don't get this use case of multiple installers trying to 
install the same package. There's just no way that running "yum install 
twisted" and "apt-get install twisted" and "pip install twisted" are 
going to coexist with each other. The best they can do is say "a file 
I'm trying to install already exists". Why try for anything else?


Here are some use-cases.

You do "apt-get install twisted", or, rather, Twisted comes preinstalled 
on your OS.  Later, you do "pip install 
py_awesome_network_thing_system", which depends on twisted.


You've configured pip to install files into ~/.local, though, and it's 
not sure if Twisted is installed or not.  So thanks to automatic 
dependency resolution, it downloads and installs another copy.  But 
that's not what you want: you want it to use the system-installed 
Twisted (and zope.interface, and pycrypto, and pytz, etc etc) and just 
install PANTS into your .local environment.  No files will conflict, but 
it would be helpful for the package installation tool to have an API to 
inspect both the system package installation and any other places where 
packages have been installed in order to make a decision about how it 
needs to satisfy its dependencies.


If you're not as interested as I am in per-user installation of 
packages, then consider the fact that Ubuntu stores Python packages in 
/usr/lib/pythonX.Y/dist-packages now.  Python package installers should 
be able to query that before putting things into site-packages. (which I 
*believe* is now reserved for non-OS-installed packages, but it's very 
hard to find documentation on the motivation behind debian's 
manipulations of distutils.  In any event, I digress.)


You're correct that worrying about 'yum install twisted' and 'apt-get 
install twisted' is way out of scope for any Python installation system; 
unfortunately, yum and apt are likely as not to have giant, opaque 
makefiles or shell scripts attached to these packages for obscure 
linux-y reasons that (hopefully) we don't need to care about, but would 
fight with each other if you tried to combine them, totally out of scope 
with distutils.


One hopes that regardless of whatever else they're doing, these packages 
*are* providing correct installation metadata.  (Unfortunately, I can 
bet you a nickel they will screw it up in the first few python packages 
that include it; see, for example, 
.  This 
spec should try to gracefully address what to do if system packagers 
have done something unusual or wrong.)


So, the filesystem and the OS package database and the Python package 
"database" (such as it is) are definitely three distinct things, but 
they have some overlap, and there should be APIs for querying each.  If 
for no other reason so you know when Python packages shouldn't stomp on 
OS packages, and vice versa.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[glyph]

On 03:28 am, e...@trueblade.com wrote:
Eventually, I'd like PEP 376 to support system packagers too. So for 
example, if you did "apt-get install python-pyqt4", then running "pip 
install python-pyqt4" should return without installing anything .. as 
RECORD will be part of the .deb previously installed. As for 
generating the RECORD file, I vote for generating it during install 
time (w/ absolute paths).


I think we should explicitly not support this. What if pip and apt-get 
(or rpm, or others) install the same package in different places 
because of system conventions (/usr vs. /usr/local vs. /opt, say)? 
There's no way we're ever going to get this right, and it's not worth 
complicating our lives over it.


As I understand it, the point of the PEP is just to *provide the 
information* that the package installers need to get it right, not to 
actually implement the interaction with system package managers.
Seriously: Is there some real world use case I'm missing? Does any 
existing install system support this?


System package managers already provide .egg-info metadata, so this is 
hardly unprecedented.  The antecedent of your "this" is somewhat 
ambiguous, so I'm not sure if any package manager provides the exact 
feature that you have in mind.  If you're talking about different 
package managers installing stuff on the same system, you can look a bit 
farther afield: 'alien' allows you to install RPM and debian packages on 
the same system, with some reconciliation of dependencies and other 
metadata.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com