Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
Am 14.02.2012 23:29, schrieb Barry Warsaw: > I think I will just state my reasoning one last time and then leave it to the > BDFL or BDFOP to make the final decision. I'd like to remind people what the original point of the PEP process was: to avoid going in cycles in discussions. To achieve this, the PEP author is supposed to record all objections in the PEP, even if he disagrees (and may state rebuttals for each objection that people brought up). So, Victor: please record all objections in a separate section of the PEP, rather than just rebutting in them in the PEP (as is currently the case). > My primary concern with the PEP is adding to users confusion when they have to > handle (at least) 5 different types[*] that represent time in Python. I agree with Barry here (despite having voiced support for using Decimal before): datetime.datetime *is* the right data type to represent time stamps. If it means that it needs to be improved before it can be used in practice, then so be it - improve it. I think improving datetime needs to go in two directions: a) arbitrary-precision second fractions. My motivation for proposing/supporting Decimal was that it can support arbitrary precision, unlike any of the alternatives (except for using numerator/denominator pairs). So just adding nanosecond resolution to datetime is not enough: it needs to support arbitrary decimal fractions (it doesn't need to support non-decimal fractions, IMO). b) distinction between universal time and local time. This distinction is currently blurred; there should be prominent API to determine whether a point-in-time is meant as universal time or local time. In terminology of the datetime documentation, there needs to be builtin support for "aware" (rather than "naive") UTC time, even if that's the only timezone that comes with Python. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 10:11, "Martin v. Löwis" wrote: >> My primary concern with the PEP is adding to users confusion when they have >> to >> handle (at least) 5 different types[*] that represent time in Python. > > I agree with Barry here (despite having voiced support for using Decimal > before): datetime.datetime *is* the right data type to represent time > stamps. If it means that it needs to be improved before it can be used > in practice, then so be it - improve it. > > I think improving datetime needs to go in two directions: > a) arbitrary-precision second fractions. My motivation for > proposing/supporting Decimal was that it can support arbitrary > precision, unlike any of the alternatives (except for using > numerator/denominator pairs). So just adding nanosecond resolution > to datetime is not enough: it needs to support arbitrary decimal > fractions (it doesn't need to support non-decimal fractions, IMO). > b) distinction between universal time and local time. This distinction > is currently blurred; there should be prominent API to determine > whether a point-in-time is meant as universal time or local time. > In terminology of the datetime documentation, there needs to be > builtin support for "aware" (rather than "naive") UTC time, even > if that's the only timezone that comes with Python. +1. And adding stuff to datetime to make it easier to get a unix timestamp out (as proposed by Victor before, IIRC) would also be a good thing in my book. I really want to be able to handle all my date+time needs without ever importing time or calendar. Cheers, Dirkjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 7:11 PM, "Martin v. Löwis" wrote: > I agree with Barry here (despite having voiced support for using Decimal > before): datetime.datetime *is* the right data type to represent time > stamps. If it means that it needs to be improved before it can be used > in practice, then so be it - improve it. By contrast, I think the only remotely viable choices for arbitrary precision low level timestamp APIs are decimal.Decimal and datetime.timedelta. The "unknown epoch" problem makes it impossible to consistently produce datetime.datetime objects, and an API that inconsistently returned either datetime.datetime or datetime.timedelta for operations that currently consistently return float objects would just be annoying. However, I still think that decimal.Decimal is the right choice. There's nothing wrong with layering APIs, and the core concept of a timestamp is simply a number representing a certain number of seconds. We already have a data type that lets us represent a numeric value to arbitrary precision: decimal.Decimal. Instead of trying to hoist all those APIs up to a higher semantic level, I'd prefer to just leave them as they are now: dealing with numbers (originally ints, then floats to support microseconds, now decimal.Decimal to support nanoseconds and any future increases in precision). If the higher level semantic API is incomplete, then we should *complete it* instead of trying to mash the two different layers together indiscriminately. > I think improving datetime needs to go in two directions: > a) arbitrary-precision second fractions. My motivation for > proposing/supporting Decimal was that it can support arbitrary > precision, unlike any of the alternatives (except for using > numerator/denominator pairs). So just adding nanosecond resolution > to datetime is not enough: it needs to support arbitrary decimal > fractions (it doesn't need to support non-decimal fractions, IMO). If our core timestamp representation is decimal.Decimal, this is trivial to implement for both datetime and timedelta - just store the seconds component as a decimal.Decimal instance. If not, we'd have to come up with some other way of obtaining arbitrary precision numeric storage (which seems rather wasteful). Even if we end up going down the datetime.timedelta path for the os module APIs, that's still the way I would want to go - arranging for timedelta.total_seconds() to return a Decimal value, rather than some other clumsy alternative like having a separate total_nanoseconds() function that returned a large integer. > b) distinction between universal time and local time. This distinction > is currently blurred; there should be prominent API to determine > whether a point-in-time is meant as universal time or local time. > In terminology of the datetime documentation, there needs to be > builtin support for "aware" (rather than "naive") UTC time, even > if that's the only timezone that comes with Python. As of 3.2, the datetime module already has full support for arbitrary fixed offsets from UTC, including datetime.timezone.utc (i.e. UTC+0), which allows timezone aware UTC. For 3.2+, you should only need a third party library like pytz if you want to support named timezones (including daylight savings changes). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
> I agree with Barry here (despite having voiced support for using Decimal > before): datetime.datetime *is* the right data type to represent time > stamps. If it means that it needs to be improved before it can be used > in practice, then so be it - improve it. Maybe I missed the answer, but how do you handle timestamp with an unspecified starting point like os.times() or time.clock()? Should we leave these function unchanged? My motivation for the PEP 410 is to provide nanosecond resolution for time.clock_gettime(time.CLOCK_MONOTONIC) and time.clock_gettime(time.CLOCK_REALTIME). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
2012/2/15 "Martin v. Löwis" : > I agree with Barry here (despite having voiced support for using Decimal > before): datetime.datetime *is* the right data type to represent time > stamps. If it means that it needs to be improved before it can be used > in practice, then so be it - improve it. Decimal and datetime.datetime are not necessary exclusive options. Using the API proposed in the PEP, we can add the Decimal type today, then improve datetime.datetime API, and finally add also datetime.datetime type. Such compromise would solve the unspecified starting date issue: an exception would be raised if the timestamp has an unspecified timestamp. In such case, you can still get the timestamp as a Decimal object with nanosecond resolution. Or we may add support of datetime and Decimal today, even if datetime only support microsecond, and improve datetime later to support nanosecond. It looks like there are use cases for Decimal and datetime, both are useful. At least, datetime has a nice object API related to time, whereas Decimal requires functions from other modules. I don't know yet if one type is enough to handle all use cases. I wrote a patch to demonstrate that my internal API can be extended (store more information for new types like datetime.datetime) to add new types later, without touching the public API (func(timestamp=type)). See timestamp_datetime.patch attached to the issue #13882 (the patch is now outside, I can update it if you would like to). For example: - time.time() would support float, Decimal and datetime - os.times() would support float and Decimal (but not datetime) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
> I'd like to remind people what the original point of the PEP process > was: to avoid going in cycles in discussions. To achieve this, the PEP > author is supposed to record all objections in the PEP, even if he > disagrees (and may state rebuttals for each objection that people > brought up). > > So, Victor: please record all objections in a separate section of the > PEP, rather than just rebutting in them in the PEP (as is currently the > case). Ok, I will try to list alternatives differently, e.g. by listing also advantages. I didn't know what a PEP is supposed to contain. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Feb 15, 2012, at 10:23 AM, Nick Coghlan wrote: >What should timedelta.total_seconds() return to avoid losing nanosecond >precision? How should this be requested when calling the API? See, I have no problem having this method return a Decimal for high precision values. This preserves the valuable abstraction of timedeltas, but also provides a useful method for interoperability. >The core "timestamp" abstraction is "just a number" that (in context) >represents a certain number of seconds. decimal.Decimal qualifies. >datetime.timedelta doesn't - it's a higher level construct that makes >the semantic context explicit (and currently refuses to interoperate >with other values that are just numbers). Right, but I think Python should promote the abstraction as the way to manipulate time-y data. Interoperability is an important principle to maintain, but IMO the right way to do that is to improve datetime and timedelta so that lower-level values can be extracted from, and added to, the higher-level abstract types. I think there are quite a few opportunities for improving the interoperability of datetime and timedelta, but that shouldn't be confused with bypassing them. Cheers, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Feb 15, 2012, at 10:11 AM, Martin v. Löwis wrote: >I think improving datetime needs to go in two directions: >a) arbitrary-precision second fractions. My motivation for > proposing/supporting Decimal was that it can support arbitrary > precision, unlike any of the alternatives (except for using > numerator/denominator pairs). So just adding nanosecond resolution > to datetime is not enough: it needs to support arbitrary decimal > fractions (it doesn't need to support non-decimal fractions, IMO). >b) distinction between universal time and local time. This distinction > is currently blurred; there should be prominent API to determine > whether a point-in-time is meant as universal time or local time. > In terminology of the datetime documentation, there needs to be > builtin support for "aware" (rather than "naive") UTC time, even > if that's the only timezone that comes with Python. +1 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
PEP author Victor asked (in http://mail.python.org/pipermail/python-dev/2012-February/116499.html): > Maybe I missed the answer, but how do you handle timestamp with an > unspecified starting point like os.times() or time.clock()? Should we > leave these function unchanged? If *all* you know is that it is monotonic, then you can't -- but then you don't really have resolution either, as the clock may well speed up or slow down. If you do have resolution, and the only problem is that you don't know what the epoch was, then you can figure that out well enough by (once per type per process) comparing it to something that does have an epoch, like time.gmtime(). -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
I just came to this thread. Having read the good arguments on both sides, I keep wondering why anybody would care about nanosecond precision in timestamps. Unless you're in charge of managing one of the few atomic reference clocks in the world, your clock is not going to tell time that accurate. (Hey, we don't even admit the existence of leap seconds in most places -- not that I mind. :-) What purpose is there to recording timestamps in nanoseconds? For clocks that start when the process starts running, float *is* (basically) good enough. For measuring e.g. file access times, there is no way that the actual time is know with anything like that precision (even if it is *recorded* as a number of milliseconds -- that's a different issue). Maybe it's okay to wait a few years on this, until either 128-bit floats are more common or cDecimal becomes the default floating point type? In the mean time for clock freaks we can have a few specialized APIs that return times in nanoseconds as a (long) integer. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, 15 Feb 2012 08:39:45 -0800 Guido van Rossum wrote: > > What purpose is there to recording timestamps in nanoseconds? For > clocks that start when the process starts running, float *is* > (basically) good enough. For measuring e.g. file access times, there > is no way that the actual time is know with anything like that > precision (even if it is *recorded* as a number of milliseconds -- > that's a different issue). The number one use case, as far as I understand, is to have bit-identical file modification timestamps where it can matter. I agree that the rest is anecdotical. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 8:47 AM, Antoine Pitrou wrote:
> On Wed, 15 Feb 2012 08:39:45 -0800
> Guido van Rossum wrote:
>>
>> What purpose is there to recording timestamps in nanoseconds? For
>> clocks that start when the process starts running, float *is*
>> (basically) good enough. For measuring e.g. file access times, there
>> is no way that the actual time is know with anything like that
>> precision (even if it is *recorded* as a number of milliseconds --
>> that's a different issue).
>
> The number one use case, as far as I understand, is to have
> bit-identical file modification timestamps where it can matter.
So that can be solved by adding extra fields st_{a,c,m}time_ns and an
extra os.utime_ns() call. Only the rare tool for making 100% faithful
backups of filesystems and the like would care.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
Does this need a pronouncement? Worrying about the speed of symlinks seems silly, and exactly how the links are created (hard or soft, chaining or direct) should be up to the distro; our own Makefile should create chaining symlinks just so the mechanism is clear. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
2012/2/15 Guido van Rossum : > I just came to this thread. Having read the good arguments on both > sides, I keep wondering why anybody would care about nanosecond > precision in timestamps. Python 3.3 exposes C functions that return timespec structure. This structure contains a timestamp with a resolution of 1 nanosecond, whereas the timeval structure has only a resolution of 1 microsecond. Examples of C functions -> Python functions: - timeval: gettimeofday() -> time.time() - timespec: clock_gettime() -> time.clock_gettime() - timespec: stat() -> os.stat() - etc. If we keep float, Python would have has worse precision than C just because it uses an inappropriate type (C uses two integers in timeval). Linux supports nanosecond timestamps since Linux 2.6, Windows supports 100 ns resolution since Windows 2000 or maybe before. It doesn't mean that Windows system clock is accurate: in practical, it's hard to get something better than 1 ms :-) But you may use QueryPerformanceCounter() is you need a bettre precision, it is used by time.clock() for example. > For measuring e.g. file access times, there > is no way that the actual time is know with anything like that > precision (even if it is *recorded* as a number of milliseconds -- > that's a different issue). If you need a real world example, here is an extract of http://en.wikipedia.org/wiki/Ext4: "Improved timestamps As computers become faster in general and as Linux becomes used more for mission-critical applications, the granularity of second-based timestamps becomes insufficient. To solve this, ext4 provides timestamps measured in nanoseconds. (...)" So nanosecond resolution is needed to check if a file is newer than another. Such test is common in build programs like make or scons. Filesystems resolution: - ext4: 1 ns - btrfs: 1 ns - NTFS: 100 ns - FAT32: 2 sec (yeah!) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
On Feb 15, 2012, at 09:20 AM, Guido van Rossum wrote: >Does this need a pronouncement? Worrying about the speed of symlinks >seems silly, and exactly how the links are created (hard or soft, >chaining or direct) should be up to the distro; our own Makefile >should create chaining symlinks just so the mechanism is clear. Works for me. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, 15 Feb 2012 18:23:55 +0100 Victor Stinner wrote: > > Linux supports nanosecond timestamps since Linux 2.6, Windows supports > 100 ns resolution since Windows 2000 or maybe before. It doesn't mean > that Windows system clock is accurate: in practical, it's hard to get > something better than 1 ms :-) Well, do you think the Linux system clock is nanosecond-accurate? A nanosecond is what it takes to execute a couple of CPU instructions. Even on a real-time operating system, your nanosecond-precise measurement is already obsolete when it starts being processed by the higher-level application. A single cache miss in the CPU will make the precision worthless. And in a higher-level language like Python, the execution times of individual instructions are not specified or stable, so the resolution brings you nothing. > "Improved timestamps > As computers become faster in general and as Linux becomes used > more for mission-critical applications, the granularity of > second-based timestamps becomes insufficient. To solve this, ext4 > provides timestamps measured in nanoseconds. (...)" This is a fallacy. Just because ext4 is able to *store* nanoseconds timestamps doesn't mean the timestamps are accurate up to that point. > Such test is common in build programs like make or scons. scons is written in Python and its authors have not complained, AFAIK, about timestamp precision. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 9:23 AM, Victor Stinner wrote: > 2012/2/15 Guido van Rossum : >> I just came to this thread. Having read the good arguments on both >> sides, I keep wondering why anybody would care about nanosecond >> precision in timestamps. > > Python 3.3 exposes C functions that return timespec structure. This > structure contains a timestamp with a resolution of 1 nanosecond, > whereas the timeval structure has only a resolution of 1 microsecond. > Examples of C functions -> Python functions: > > - timeval: gettimeofday() -> time.time() > - timespec: clock_gettime() -> time.clock_gettime() > - timespec: stat() -> os.stat() > - etc. > > If we keep float, Python would have has worse precision than C just > because it uses an inappropriate type (C uses two integers in > timeval). > > Linux supports nanosecond timestamps since Linux 2.6, Windows supports > 100 ns resolution since Windows 2000 or maybe before. It doesn't mean > that Windows system clock is accurate: in practical, it's hard to get > something better than 1 ms :-) But you may use > QueryPerformanceCounter() is you need a bettre precision, it is used > by time.clock() for example. > >> For measuring e.g. file access times, there >> is no way that the actual time is know with anything like that >> precision (even if it is *recorded* as a number of milliseconds -- >> that's a different issue). > > If you need a real world example, here is an extract of > http://en.wikipedia.org/wiki/Ext4: > > "Improved timestamps > As computers become faster in general and as Linux becomes used > more for mission-critical applications, the granularity of > second-based timestamps becomes insufficient. To solve this, ext4 > provides timestamps measured in nanoseconds. (...)" > > So nanosecond resolution is needed to check if a file is newer than > another. Such test is common in build programs like make or scons. > > Filesystems resolution: > - ext4: 1 ns > - btrfs: 1 ns > - NTFS: 100 ns > - FAT32: 2 sec (yeah!) This does not explain why microseconds aren't good enough. It seems none of the clocks involved can actually measure even relative time intervals more accurate than 100ns, and I expect that kernels don't actually keep their clock more accurate than milliseconds. (They may increment it by 1 microsecond approximately every microsecond, or even by 1 ns roughly every ns, but that doesn't fool me into believing all those digits of precision. I betcha that over say an hour even time deltas aren't more accurate than a microsecond, due to inevitable fluctuations in clock speed. It seems the argument goes simply "because Linux chose to go all the way to nanoseconds we must support nanoseconds" -- and Linux probably chose nanoseconds because that's what fits in 32 bits and there wasn't anything else to do with those bits. *Apart* from the specific use case of making an exact copy of a directory tree that can be verified by other tools that simply compare the nanosecond times for equality, I don't see any reason for complicating so many APIs to preserve the fake precision. As far as simply comparing whether one file is newer than another for tools like make/scons, I bet that it's in practice impossible to read a file and create another in less than a microsecond. (I actually doubt that you can do it faster than a millisecond, but for my argument I don't need that.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new dictionary implementation
Any opinions on my new dictionary implementation? I'm happy to take silence on the PEP as tacit approval, but the code definitely needs reviewing. Issue: http://bugs.python.org/issue13903 PEP: https://bitbucket.org/markshannon/cpython_new_dict/src/6c4d5d9dfc6d/pep-new-dict.txt Repository https://bitbucket.org/markshannon/cpython_new_dict Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
>> Linux supports nanosecond timestamps since Linux 2.6, Windows supports
>> 100 ns resolution since Windows 2000 or maybe before. It doesn't mean
>> that Windows system clock is accurate: in practical, it's hard to get
>> something better than 1 ms :-)
>
> Well, do you think the Linux system clock is nanosecond-accurate?
Test the following C program:
#include
#include
int main(int argc, char **argv, char **arge) {
struct timespec tps, tpe;
if ((clock_gettime(CLOCK_REALTIME, &tps) != 0)
|| (clock_gettime(CLOCK_REALTIME, &tpe) != 0)) {
perror("clock_gettime");
return -1;
}
printf("%lu s, %lu ns\n", tpe.tv_sec-tps.tv_sec,
tpe.tv_nsec-tps.tv_nsec);
return 0;
}
Compile it using gcc time.c -o time -lrt.
It gives me differences smaller than 1000 ns on Ubuntu 11.10 and a
Intel Core i5 @ 3.33GHz:
$ ./a.out
0 s, 781 ns
$ ./a.out
0 s, 785 ns
$ ./a.out
0 s, 798 ns
$ ./a.out
0 s, 818 ns
$ ./a.out
0 s, 270 ns
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
So using floats we can match 100ns precision, right?
On Wed, Feb 15, 2012 at 9:58 AM, Victor Stinner
wrote:
>>> Linux supports nanosecond timestamps since Linux 2.6, Windows supports
>>> 100 ns resolution since Windows 2000 or maybe before. It doesn't mean
>>> that Windows system clock is accurate: in practical, it's hard to get
>>> something better than 1 ms :-)
>>
>> Well, do you think the Linux system clock is nanosecond-accurate?
>
> Test the following C program:
>
> #include
> #include
>
> int main(int argc, char **argv, char **arge) {
> struct timespec tps, tpe;
> if ((clock_gettime(CLOCK_REALTIME, &tps) != 0)
> || (clock_gettime(CLOCK_REALTIME, &tpe) != 0)) {
> perror("clock_gettime");
> return -1;
> }
> printf("%lu s, %lu ns\n", tpe.tv_sec-tps.tv_sec,
> tpe.tv_nsec-tps.tv_nsec);
> return 0;
> }
>
> Compile it using gcc time.c -o time -lrt.
>
> It gives me differences smaller than 1000 ns on Ubuntu 11.10 and a
> Intel Core i5 @ 3.33GHz:
>
> $ ./a.out
> 0 s, 781 ns
> $ ./a.out
> 0 s, 785 ns
> $ ./a.out
> 0 s, 798 ns
> $ ./a.out
> 0 s, 818 ns
> $ ./a.out
> 0 s, 270 ns
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
Le mercredi 15 février 2012 à 18:58 +0100, Victor Stinner a écrit : > It gives me differences smaller than 1000 ns on Ubuntu 11.10 and a > Intel Core i5 @ 3.33GHz: > > $ ./a.out > 0 s, 781 ns > $ ./a.out > 0 s, 785 ns > $ ./a.out > 0 s, 798 ns > $ ./a.out > 0 s, 818 ns > $ ./a.out > 0 s, 270 ns What is it supposed to prove exactly? There is a difference between being able to *represent* nanoseconds and being able to *measure* them; let alone give a precise meaning to them. (and ironically, floating-point numbers are precise enough to represent these numbers unambiguously) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new dictionary implementation
On Mon, 13 Feb 2012 12:31:38 + Mark Shannon wrote: > Note that the json benchmark is unstable and should be ignored. Can you elaborate? If it's unstable it should be fixed, not ignored :) Also, there are two different mako results in your message, which one is the right one? Thanks Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
> *Apart* from the specific use case of making an exact copy of a > directory tree that can be verified by other tools that simply compare > the nanosecond times for equality, I don't see any reason for > complicating so many APIs to preserve the fake precision. As far as > simply comparing whether one file is newer than another for tools like > make/scons, I bet that it's in practice impossible to read a file and > create another in less than a microsecond. (I actually doubt that you > can do it faster than a millisecond, but for my argument I don't need > that.) But this leads to the issue with specialized APIs just for nanoseconds (as the one you just proposed): people will use them *just because they are there*. It's like the byte-oriented APIs to do file names: most applications won't need them, either because the file names convert into character strings just fine, or because the emulation that we (now) provide will fall back to some nearly-accurate representation. Still, just because we have the byte APIs, people use them, to then find out that they don't work on Windows, so they will write very complicated code to make their code 100% correct. The same will happen with specialized API for nanosecond time stamps: people will be told to use them because it might matter, and not knowing for sure that it won't matter to them, they will use them. Therefore, I feel that we must not introduced such specialized APIs. Not supporting ns timestamps is something I can readily agree to. However, contributors won't agree to that, and will insist that these be added (and keep writing patches to do so) until it does get added. Some of them are core contributors, so there is no easy way to stop them :-) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
Am 15.02.2012 19:10, schrieb Antoine Pitrou: > > Le mercredi 15 février 2012 à 18:58 +0100, Victor Stinner a écrit : >> It gives me differences smaller than 1000 ns on Ubuntu 11.10 and a >> Intel Core i5 @ 3.33GHz: >> >> $ ./a.out >> 0 s, 781 ns >> $ ./a.out >> 0 s, 785 ns >> $ ./a.out >> 0 s, 798 ns >> $ ./a.out >> 0 s, 818 ns >> $ ./a.out >> 0 s, 270 ns > > What is it supposed to prove exactly? There is a difference between > being able to *represent* nanoseconds and being able to *measure* them; > let alone give a precise meaning to them. Linux *actually* is able to measure time in nanosecond precision, even though it is not able to keep its clock synchronized to UTC with a nanosecond accuracy. The way Linux does that is to use the time-stamping counter of the processor (the rdtsc instructions), which (originally) counts one unit per CPU clock. I believe current processors use slightly different countings (e.g. through the APIC), but still: you get a resolution within the clock frequency of the CPU quartz. With the quartz in Victor's machine, a single clock takes 0.3ns, so three of them make a nanosecond. As the quartz may not be entirely accurate (and also as the CPU frequency may change) you have to measure the clock rate against an external time source, but Linux has implemented algorithms for that. On my system, dmesg shows [2.236894] Refined TSC clocksource calibration: 2793.000 MHz. [2.236900] Switching to clocksource tsc Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, 15 Feb 2012 20:56:26 +0100 "Martin v. Löwis" wrote: > > With the quartz in Victor's machine, a single clock takes 0.3ns, so > three of them make a nanosecond. As the quartz may not be entirely > accurate (and also as the CPU frequency may change) you have to measure > the clock rate against an external time source, but Linux has > implemented algorithms for that. On my system, dmesg shows > > [2.236894] Refined TSC clocksource calibration: 2793.000 MHz. > [2.236900] Switching to clocksource tsc But that's still not meaningful. By the time clock_gettime() returns, an unpredictable number of nanoseconds have elapsed, and even more when returning to the Python evaluation loop. So the nanosecond precision is just an illusion, and a float should really be enough to represent durations for any task where Python is suitable as a language. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
Antoine Pitrou wrote: On Wed, 15 Feb 2012 20:56:26 +0100 "Martin v. Löwis" wrote: With the quartz in Victor's machine, a single clock takes 0.3ns, so three of them make a nanosecond. As the quartz may not be entirely accurate (and also as the CPU frequency may change) you have to measure the clock rate against an external time source, but Linux has implemented algorithms for that. On my system, dmesg shows [2.236894] Refined TSC clocksource calibration: 2793.000 MHz. [2.236900] Switching to clocksource tsc But that's still not meaningful. By the time clock_gettime() returns, an unpredictable number of nanoseconds have elapsed, and even more when returning to the Python evaluation loop. So the nanosecond precision is just an illusion, and a float should really be enough to represent durations for any task where Python is suitable as a language. I reckon PyPy might be able to call clock_gettime() in a tight loop almost as frequently as the C program (although not with the overhead of converting to a decimal). Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
2012/2/15 Mark Shannon : > > I reckon PyPy might be able to call clock_gettime() in a tight loop > almost as frequently as the C program (although not with the overhead > of converting to a decimal). The nanosecond resolution is just as meaningless in C. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new dictionary implementation
Hello Mark,
First, I've back-ported your patch on python 3.2.2 (which was relatively
easy). Almost all tests pass, and those that don't are always failing on
my machine if I remember. The patch can be found here: http://goo.gl/nSzzY
Then, I compared memory footprint of one of our applications (300,000 LOC)
and saw it about 6% less than on vanilla python 3.2.2 (660 MB of reserved
process memory compared to 702 MB; Linux Gentoo 64bit) The application is
written in heavy OOP style (for instance, ~1000 classes are generated by our
ORM on the fly, and there are approximately the same amount of hand-written
ones) so I hoped for a much bigger saving.
As for the patch itself I found one use-case, where python with the patch
behaves differently::
class Foo:
def __init__(self, msg):
self.msg = msg
f = Foo('123')
class _str(str):
pass
print(f.msg)
print(getattr(f, _str('msg')))
The above snippet works perfectly on vanilla py3.2, but fails on the patched
one (even on 3.3 compiled from your 'cpython_new_dict' branch) I'm not sure
that it's a valid code, though. If not, then we need to fix some python
internals to add exact type check in 'getattr', in the 'operator.getattr',
etc.
And if it is - your patch needs to be fixed. In any case, I propose to add
the above code to the python test-suite, with either expecting a result or an
exception.
Cheers,
Yury
On 2012-02-15, at 12:58 PM, Mark Shannon wrote:
> Any opinions on my new dictionary implementation?
>
> I'm happy to take silence on the PEP as tacit approval,
> but the code definitely needs reviewing.
>
> Issue:
> http://bugs.python.org/issue13903
>
> PEP:
> https://bitbucket.org/markshannon/cpython_new_dict/src/6c4d5d9dfc6d/pep-new-dict.txt
>
> Repository
> https://bitbucket.org/markshannon/cpython_new_dict
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] best place for an atomic file API
Hi,
Issue #8604 aims at adding an atomic file API to make it easier to
create/update files atomically, using rename() on POSIX systems and
MoveFileEx() on Windows (which are now available through
os.replace()). It would also use fsync() on POSIX to make sure data is
committed to disk.
For example, it could be used by importlib to avoid races when
writting bytecode files (issues #13392, #13003, #13146), or more
generally by any application that wants to make sure to end up with a
consistent file even in face of crash (e.g. it seems that mercurial
implemented their own version).
Basically the usage would be, e.g.:
with AtomicFile('foo') as f:
pickle.dump(obj, f)
or
with AtomicFile('foo') as f:
chunk = heavyCrunch()
f.write(chunk)
chunk = CrunchSomeMore()
f.write(chunk)
What would be the best place for a such a class?
_pyio, tempfile, or a new atomicfile
Cheers,
cf
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 11:38 AM, "Martin v. Löwis" wrote:
>> *Apart* from the specific use case of making an exact copy of a
>> directory tree that can be verified by other tools that simply compare
>> the nanosecond times for equality, I don't see any reason for
>> complicating so many APIs to preserve the fake precision. As far as
>> simply comparing whether one file is newer than another for tools like
>> make/scons, I bet that it's in practice impossible to read a file and
>> create another in less than a microsecond. (I actually doubt that you
>> can do it faster than a millisecond, but for my argument I don't need
>> that.)
>
> But this leads to the issue with specialized APIs just for nanoseconds
> (as the one you just proposed): people will use them *just because they
> are there*.
>
> It's like the byte-oriented APIs to do file names: most applications
> won't need them, either because the file names convert into character
> strings just fine, or because the emulation that we (now) provide will
> fall back to some nearly-accurate representation. Still, just because we
> have the byte APIs, people use them, to then find out that they don't
> work on Windows, so they will write very complicated code to make their
> code 100% correct.
>
> The same will happen with specialized API for nanosecond time stamps:
> people will be told to use them because it might matter, and not knowing
> for sure that it won't matter to them, they will use them.
>
> Therefore, I feel that we must not introduced such specialized APIs.
You have a point, but applies just as much to the proposal in the PEP
-- floats and Decimal are often not quite compatible, but people will
pass type=Decimal to the clock and stat functions just because they
can. The problems with mixing floats and Decimal are probably just as
nasty as those with mixing byte and str. At least if people are mixing
nanoseconds (integers) and seconds (floats) they will quickly notice
results that are a billion times off.
> Not supporting ns timestamps is something I can readily agree to.
Me too.
> However, contributors won't agree to that, and will insist that these
> be added (and keep writing patches to do so) until it does get added.
> Some of them are core contributors, so there is no easy way to stop
> them :-)
Actually I think a rejected PEP would be an excellent way to stop this.
Maybe an alternative PEP could be written that supports the filesystem
copying use case only, using some specialized ns APIs? I really think
that all you need is st_{a,c,m}time_ns fields and os.utime_ns().
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Charles-François Natali wrote: Hi, Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. [...] What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile shutil perhaps? As a user, that's the third place I look for file utilities, after builtin functions and os module. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Thu, Feb 16, 2012 at 9:29 AM, Steven D'Aprano wrote: > Charles-François Natali wrote: >> What would be the best place for a such a class? >> _pyio, tempfile, or a new atomicfile > > > shutil perhaps? > > As a user, that's the third place I look for file utilities, after builtin > functions and os module. +1 for shutil from me. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): remove unused import
On 06/02/2012 17:57, Brett Cannon wrote: On Sun, Feb 5, 2012 at 19:53, Christian Heimes wrote: Am 06.02.2012 01:39, schrieb Brett Cannon: I'm going to assume pylint or pyflakes would throw too many warnings on the stdlib, but would it be worth someone's time to write a simple unused import checker to run over the stdlib on occasion? I bet even one that did nothing more than a regex search for matched import statements would be good enough. Zope 3 has an import checker that uses the compiler package and AST tree to check for unused imports. It seems like a better approach than a simple regex search. http://svn.zope.org/Zope3/trunk/utilities/importchecker.py?rev=25177&view=auto The importorder tool uses the tokenizer module to order import statements. http://svn.zope.org/Zope3/trunk/utilities/importorder.py?rev=25177&view=auto Both are written by Jim Fulton. Ah, but does it run against Python 3? If so then this is something to suggest on python-mentor for someone to get their feet wet for contributing. A possible alternative is the sfood-checker tool given here http://furius.ca/snakefood/ which I stumbled across whilst looking for something completely different. -- Cheers. Mark Lawrence. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Charles-François Natali writes: > Issue #8604 aims at adding an atomic file API to make it easier to > create/update files atomically, using rename() on POSIX systems and > MoveFileEx() on Windows (which are now available through > os.replace()). It would also use fsync() on POSIX to make sure data is > committed to disk. These make it quite OS-specific. […] > What would be the best place for a such a class? > _pyio, tempfile, or a new atomicfile I would expect to find it within ‘os’ or submodules of ‘os’. -- \“We should be less concerned about adding years to life, and | `\ more about adding life to years.” —Arthur C. Clarke, 2001 | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Wed, Feb 15, 2012 at 19:19, Ben Finney wrote: > Charles-François Natali writes: > >> Issue #8604 aims at adding an atomic file API to make it easier to >> create/update files atomically, using rename() on POSIX systems and >> MoveFileEx() on Windows (which are now available through >> os.replace()). It would also use fsync() on POSIX to make sure data is >> committed to disk. > > These make it quite OS-specific. That'll happen when solving problems on different OSes. Do you propose a more platform agnostic solution? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
Guido van Rossum wrote: > Does this need a pronouncement? Worrying about the speed of symlinks > seems silly I agree. I wonder if a hard-link was used for legacy reasons. Some very old versions of Unix didn't have symlinks. It looks like it was introduced in BSD 4.2, released in 1983. That seems a long time before the birth of Python but perhaps some SysV systems were around that didn't have it. Also, maybe speed was more of a concern at that time. In any case, those days are long, long gone. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Brian Curtin writes: > On Wed, Feb 15, 2012 at 19:19, Ben Finney wrote: > > Charles-François Natali writes: > > > >> […] using rename() on POSIX systems and MoveFileEx() on Windows > >> (which are now available through os.replace()). It would also use > >> fsync() on POSIX to make sure data is committed to disk. > > > > These make it quite OS-specific. > > That'll happen when solving problems on different OSes. Do you propose > a more platform agnostic solution? No, I have no objection to that implementation. I'm pointing that out only because the nature of the functionality implies I'd expect to find it within the ‘os’ module hierarchy. -- \ “The man who is denied the opportunity of taking decisions of | `\ importance begins to regard as important the decisions he is | _o__)allowed to take.” —C. Northcote Parkinson | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
+1 for using symlinks where possible. In deploying Python to different operating systems and filesystems I've often had to run a script to "fix" the hardlinking done by make install because the deployment mechanism or system couldn't be trusted to do the right thing with respect to minimising installation size. Symlinks are total win when disk use is a concern, and make intent clear. I'm not aware of any mainstream systems that don't support them. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On 16/02/12 06:43, Guido van Rossum wrote: This does not explain why microseconds aren't good enough. It seems none of the clocks involved can actually measure even relative time intervals more accurate than 100ns, and I expect that kernels don't actually keep their clock more accurate than milliseconds. I gather that modern x86 CPUs have a counter that keeps track of time down to a nanosecond or so by counting clock cycles. In principle it seems like a kernel should be able to make use of it in conjunction with other timekeeping hardware to produce nanosecond-resolution timestamps. Whether any existing kernel actually does that is another matter. It probably isn't worth the bother for things like file timestamps, where the time taken to execute the system call that modifies the file is likely to be several orders of magnitude larger. Until we have computers with terahertz clocks and gigahertz disk drives, it seems like a rather theoretical issue. And it doesn't look like Mr. Moore is going to give us anything like that any time soon. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
On Wed, Feb 15, 2012 at 5:26 PM, Neil Schemenauer wrote: > Guido van Rossum wrote: >> Does this need a pronouncement? Worrying about the speed of symlinks >> seems silly > > I agree. I wonder if a hard-link was used for legacy reasons. Some > very old versions of Unix didn't have symlinks. It looks like it > was introduced in BSD 4.2, released in 1983. That seems a long time > before the birth of Python but perhaps some SysV systems were around > that didn't have it. Also, maybe speed was more of a concern at > that time. In any case, those days are long, long gone. Actually I remember what was my motivation at the time (somewhere between 1995-1999 I think) that I decided to use a hard link. It was some trick whereby if you ran "make install" the target binary, e.g. "python1.3", was removed and then overwritten in such a way that code which was running it via "python" (a hard link to python1.3) would not be disturbed. Then a new hard link would be created atomically. But it was too clever, and it's long been replaced with a symlink. Anyway, I don't think anyone is objecting against the PEP allowing symlinks now. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 6:06 PM, Greg Ewing wrote: > On 16/02/12 06:43, Guido van Rossum wrote: >> >> This does not explain why microseconds aren't good enough. It seems >> none of the clocks involved can actually measure even relative time >> intervals more accurate than 100ns, and I expect that kernels don't >> actually keep their clock more accurate than milliseconds. > > > I gather that modern x86 CPUs have a counter that keeps track of > time down to a nanosecond or so by counting clock cycles. In > principle it seems like a kernel should be able to make use of > it in conjunction with other timekeeping hardware to produce > nanosecond-resolution timestamps. > > Whether any existing kernel actually does that is another > matter. It probably isn't worth the bother for things like > file timestamps, where the time taken to execute the system > call that modifies the file is likely to be several orders > of magnitude larger. Ironically, file timestamps are likely the only place where it matters. Read the rest of the thread. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Thu, Feb 16, 2012 at 11:49 AM, Ben Finney wrote: > No, I have no objection to that implementation. I'm pointing that out > only because the nature of the functionality implies I'd expect to find > it within the ‘os’ module hierarchy. The (very) rough rule of thumb is that the os module handles abstracting away cross-platform differences in implementation details, while the higher level shutil algorithms can be largely platform independent by using the shared abstractions in the os module layer. In this case, os.replace() is the cross platform abstraction, while the atomic file context manager is just a particular use case for that new feature. (MvL complained in the tracker issue about a lack of concrete use cases, but I think fixing race conditions when overwriting bytecode files in importlib and the existing distutils/packaging use cases cover that) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 394 request for pronouncement (python2 symlink in *nix systems)
On Thu, Feb 16, 2012 at 12:06 PM, Guido van Rossum wrote: > Anyway, I don't think anyone is objecting against the PEP allowing symlinks > now. Yeah, the onus is just back on me to do the final updates to the PEP and patch based on the discussion in this thread. Unless life unexpectedly intervenes, I expect that to happen on Saturday (my time). After that, the only further work is for Ned to supply whatever updates he needs to bring the 2.7 Mac OS X installers into line with the new naming scheme. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
Guido van Rossum writes: > On Wed, Feb 15, 2012 at 6:06 PM, Greg Ewing > wrote: > > It probably isn't worth the bother for things like file timestamps, > > where the time taken to execute the system call that modifies the > > file is likely to be several orders of magnitude larger. > > Ironically, file timestamps are likely the only place where it > matters. Read the rest of the thread. And log message timestamps. The *two* only places where it matters, file timestamps and log messages. And communication protocols. The *three* only places – I'll come in again. -- \ “Why should I care about posterity? What's posterity ever done | `\for me?” —Groucho Marx | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On 02/15/2012 09:43 AM, Guido van Rossum wrote:
*Apart* from the specific use case of making an exact copy of a
directory tree that can be verified by other tools that simply compare
the nanosecond times for equality,
A data point on this specific use case. The following code throws its
assert ~90% of the time in Python 3.2.2 on a modern Linux machine
(assuming "foo" exists and "bar" does not):
import shutil
import os
shutil.copy2("foo", "bar")
assert os.stat("foo").st_mtime == os.stat("bar").st_mtime
The problem is with os.utime. IIUC stat() on Linux added nanosecond
atime/mtime support back in 2.5. But the corresponding utime()
functions to write nanosecond atime/mtime didn't appear until relatively
recently--and Python 3.2 doesn't use them. With stat_float_times turned
on, os.stat effectively reads with ~100-nanosecond precision, but
os.utime still only writes with microsecond precision. I fixed this in
trunk last September (issue 12904); os.utime now preserves all the
precision that Python currently conveys.
One way of looking at it: in Python 3.2 it's already pretty bad and
almost nobody is complaining. (There's me, I guess, but I scratched my
itch.)
/arry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] folding cElementTree behind ElementTree in 3.3
On Feb 13, 2012, at 8:44 PM, Nick Coghlan wrote:
> On Tue, Feb 14, 2012 at 2:25 PM, Eli Bendersky wrote:
>> With the deprecation warning being silent, is there much to lose, though?
>
> Yes, it creates problems for anyone that deliberately converts all
> warnings to errors when running their test suites. This forces them to
> spend time switching over to a Python version dependent import of
> either cElementTree or ElementTree that could have been spent doing
> something actually productive instead of mere busywork.
>
> And, of course, even people that *don't* convert warnings to errors
> when running tests will have to make the same switch when the module
> is eventually removed.
What about a PendingDeprecationWarning? I think you're usually only going to
convert DeprecationWarnings to errors (with python -W error::DeprecationWarning
or warnings.simplefilter('error', DeprecationWarning))
--
Philip Jenvey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] folding cElementTree behind ElementTree in 3.3
On Thu, Feb 16, 2012 at 1:40 PM, Philip Jenvey wrote:
> What about a PendingDeprecationWarning? I think you're usually only going to
> convert DeprecationWarnings to errors (with python -W
> error::DeprecationWarning or warnings.simplefilter('error',
> DeprecationWarning))
Think "-Wall" for strict testing regimes :)
If you trawl back in the archives a few years, you'll find I've
historically been on the *other* side of this kind of argument. I've
since come to recognise that programmatic deprecation really is a big
hammer that hits the wider Python community - it needs to be treated
with appropriate respect.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 410 (Decimal timestamp): the implementation is ready for a review
On Wed, Feb 15, 2012 at 7:28 PM, Larry Hastings wrote:
>
> On 02/15/2012 09:43 AM, Guido van Rossum wrote:
>>
>> *Apart* from the specific use case of making an exact copy of a
>> directory tree that can be verified by other tools that simply compare
>> the nanosecond times for equality,
>
>
> A data point on this specific use case. The following code throws its
> assert ~90% of the time in Python 3.2.2 on a modern Linux machine (assuming
> "foo" exists and "bar" does not):
>
> import shutil
> import os
> shutil.copy2("foo", "bar")
> assert os.stat("foo").st_mtime == os.stat("bar").st_mtime
>
> The problem is with os.utime. IIUC stat() on Linux added nanosecond
> atime/mtime support back in 2.5. But the corresponding utime() functions to
> write nanosecond atime/mtime didn't appear until relatively recently--and
> Python 3.2 doesn't use them. With stat_float_times turned on, os.stat
> effectively reads with ~100-nanosecond precision, but os.utime still only
> writes with microsecond precision. I fixed this in trunk last September
> (issue 12904); os.utime now preserves all the precision that Python
> currently conveys.
>
> One way of looking at it: in Python 3.2 it's already pretty bad and almost
> nobody is complaining. (There's me, I guess, but I scratched my itch.)
So, essentially you fixed this particular issue without having to do
anything as drastic as the proposed PEP...
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
