On Fri, Aug 19, 2011 at 11:07 AM, Charles R Harris < charlesr.har...@gmail.com> wrote:
> > > On Fri, Aug 19, 2011 at 11:55 AM, Bruce Southey <bsout...@gmail.com>wrote: > >> On Fri, Aug 19, 2011 at 10:48 AM, Mark Wiebe <mwwi...@gmail.com> wrote: >> > On Fri, Aug 19, 2011 at 7:15 AM, Bruce Southey <bsout...@gmail.com> >> wrote: >> >> >> >> On 08/18/2011 04:43 PM, Mark Wiebe wrote: >> >> >> >> It's taken a lot of changes to get the NA mask support to its current >> >> point, but the code ready for some testing now. You can read the >> >> work-in-progress release notes here: >> >> >> >> >> https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes.rst >> >> To try it out, check out the missingdata branch from my github account, >> >> here, and build in the standard way: >> >> https://github.com/m-paradox/numpy >> >> The things most important to test are: >> >> * Confirm that existing code still works correctly. I've tested against >> >> SciPy and matplotlib. >> >> * Confirm that the performance of code not using NA masks is the same >> or >> >> better. >> >> * Try to do computations with the NA values, find places they don't >> work >> >> yet, and nominate unimplemented functionality important to you to be >> next on >> >> the development list. The release notes have a preliminary list of >> >> implemented/unimplemented functions. >> >> * Report any crashes, build problems, or unexpected behaviors. >> >> In addition to adding the NA mask, I've also added features and done a >> few >> >> performance changes here and there, like letting reductions like sum >> take >> >> lists of axes instead of being a single axis or all of them. These >> changes >> >> affect various bugs >> >> like http://projects.scipy.org/numpy/ticket/1143 and >> http://projects.scipy.org/numpy/ticket/533. >> >> Thanks! >> >> Mark >> >> Here's a small example run using NAs: >> >> >>> import numpy as np >> >> >>> np.__version__ >> >> '2.0.0.dev-8a5e2a1' >> >> >>> a = np.random.rand(3,3,3) >> >> >>> a.flags.maskna = True >> >> >>> a[np.random.rand(3,3,3) < 0.5] = np.NA >> >> >>> a >> >> array([[[NA, NA, 0.11511708], >> >> [ 0.46661454, 0.47565512, NA], >> >> [NA, NA, NA]], >> >> [[NA, 0.57860351, NA], >> >> [NA, NA, 0.72012669], >> >> [ 0.36582123, NA, 0.76289794]], >> >> [[ 0.65322748, 0.92794386, NA], >> >> [ 0.53745165, 0.97520989, 0.17515083], >> >> [ 0.71219688, 0.5184328 , 0.75802805]]]) >> >> >>> np.mean(a, axis=-1) >> >> array([[NA, NA, NA], >> >> [NA, NA, NA], >> >> [NA, 0.56260412, 0.66288591]]) >> >> >>> np.std(a, axis=-1) >> >> array([[NA, NA, NA], >> >> [NA, NA, NA], >> >> [NA, 0.32710662, 0.10384331]]) >> >> >>> np.mean(a, axis=-1, skipna=True) >> >> >> >> >> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2474: >> >> RuntimeWarning: invalid value encountered in true_divide >> >> um.true_divide(ret, rcount, out=ret, casting='unsafe') >> >> array([[ 0.11511708, 0.47113483, nan], >> >> [ 0.57860351, 0.72012669, 0.56435958], >> >> [ 0.79058567, 0.56260412, 0.66288591]]) >> >> >>> np.std(a, axis=-1, skipna=True) >> >> >> >> >> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2707: >> >> RuntimeWarning: invalid value encountered in true_divide >> >> um.true_divide(arrmean, rcount, out=arrmean, casting='unsafe') >> >> >> >> >> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2730: >> >> RuntimeWarning: invalid value encountered in true_divide >> >> um.true_divide(ret, rcount, out=ret, casting='unsafe') >> >> array([[ 0. , 0.00452029, nan], >> >> [ 0. , 0. , 0.19853835], >> >> [ 0.13735819, 0.32710662, 0.10384331]]) >> >> >>> np.std(a, axis=(1,2), skipna=True) >> >> array([ 0.16786895, 0.15498008, 0.23811937]) >> >> >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion@scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> Hi, >> >> That is great news! >> >> (Python2.x will be another email.) >> >> >> >> Python3.1 and Python3.2 failed with building >> 'multiarraymodule_onefile.o' >> >> but I could not see any obvious reason. >> > >> > I've pushed a change to fix the Python 3 build, it was a use >> > of Py_TPFLAGS_CHECKTYPES, which is no longer in Python3 but is always >> > default now. Tested with 3.2. >> > Thanks! >> > Mark >> > >> >> >> >> I had removed my build directory and then 'python3 setup.py build' but >> I >> >> saw this message: >> >> Running from numpy source directory. >> >> numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch >> detected, >> >> the C API version numbers have to be updated. Current C api version is >> 6, >> >> with checksum ef5688af03ffa23dd8e11734f5b69313, but recorded checksum >> for C >> >> API version 6 in codegen_dir/cversions.txt is >> >> e61d5dc51fa1c6459328266e215d6987. If functions were added in the C API, >> you >> >> have to update C_API_VERSION in numpy/core/setup_common.py. >> >> MismatchCAPIWarning) >> >> >> >> Upstream of the build log is below. >> >> >> >> Bruce >> >> >> >> In file included from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0: >> >> numpy/core/src/multiarray/na_singleton.c: At top level: >> >> numpy/core/src/multiarray/na_singleton.c:708:25: error: >> >> ‘Py_TPFLAGS_CHECKTYPES’ undeclared here (not in a function) >> >> numpy/core/src/multiarray/common.c:48:1: warning: ‘_use_default_type’ >> >> defined but not used >> >> numpy/core/src/multiarray/ctors.h:93:1: warning: ‘_arrays_overlap’ >> >> declared ‘static’ but never defined >> >> numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning: >> >> ‘gentype_getsegcount’ defined but not used >> >> numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning: >> >> ‘gentype_getcharbuf’ defined but not used >> >> numpy/core/src/multiarray/mapping.c:110:1: warning: ‘_array_ass_item’ >> >> defined but not used >> >> numpy/core/src/multiarray/number.c:266:1: warning: ‘array_divide’ >> defined >> >> but not used >> >> numpy/core/src/multiarray/number.c:464:1: warning: >> ‘array_inplace_divide’ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:25:1: warning: ‘array_getsegcount’ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:58:1: warning: ‘array_getwritebuf’ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:71:1: warning: ‘array_getcharbuf’ >> >> defined but not used >> >> numpy/core/src/multiarray/na_mask.c:681:1: warning: >> >> ‘PyArray_GetMaskInversionFunction’ defined but not used >> >> In file included from numpy/core/src/multiarray/scalartypes.c.src:25:0, >> >> from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:10: >> >> numpy/core/src/multiarray/_datetime.h:9:1: warning: function >> declaration >> >> isnât a prototype >> >> In file included from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:13:0: >> >> numpy/core/src/multiarray/datetime.c:33:1: warning: function >> declaration >> >> isnât a prototype >> >> In file included from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:17:0: >> >> numpy/core/src/multiarray/arraytypes.c.src: In function âVOID_getitemâ: >> >> numpy/core/src/multiarray/arraytypes.c.src:643:9: warning: passing >> >> argument 2 of âPyArray_SetBaseObjectâ from incompatible pointer type >> >> >> >> >> build/src.linux-x86_64-3.2/numpy/core/include/numpy/__multiarray_api.h:763:12: >> >> note: expected âstruct PyObject *â but argument is of type âstruct >> >> PyArrayObject *â >> >> In file included from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:44:0: >> >> numpy/core/src/multiarray/nditer_pywrap.c: In function >> >> ânpyiter_subscriptâ: >> >> numpy/core/src/multiarray/nditer_pywrap.c:2395:29: warning: passing >> >> argument 1 of âPySlice_GetIndicesâ from incompatible pointer type >> >> /usr/local/include/python3.2m/sliceobject.h:38:5: note: expected >> âstruct >> >> PyObject *â but argument is of type âstruct PySliceObject *â >> >> numpy/core/src/multiarray/nditer_pywrap.c: In function >> >> ânpyiter_ass_subscriptâ: >> >> numpy/core/src/multiarray/nditer_pywrap.c:2440:29: warning: passing >> >> argument 1 of âPySlice_GetIndicesâ from incompatible pointer type >> >> /usr/local/include/python3.2m/sliceobject.h:38:5: note: expected >> âstruct >> >> PyObject *â but argument is of type âstruct PySliceObject *â >> >> In file included from >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0: >> >> numpy/core/src/multiarray/na_singleton.c: At top level: >> >> numpy/core/src/multiarray/na_singleton.c:708:25: error: >> >> âPy_TPFLAGS_CHECKTYPESâ undeclared here (not in a function) >> >> numpy/core/src/multiarray/common.c:48:1: warning: â_use_default_typeâ >> >> defined but not used >> >> numpy/core/src/multiarray/ctors.h:93:1: warning: â_arrays_overlapâ >> >> declared âstaticâ but never defined >> >> numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning: >> >> âgentype_getsegcountâ defined but not used >> >> numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning: >> >> âgentype_getcharbufâ defined but not used >> >> numpy/core/src/multiarray/mapping.c:110:1: warning: â_array_ass_itemâ >> >> defined but not used >> >> numpy/core/src/multiarray/number.c:266:1: warning: âarray_divideâ >> defined >> >> but not used >> >> numpy/core/src/multiarray/number.c:464:1: warning: >> âarray_inplace_divideâ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:25:1: warning: âarray_getsegcountâ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:58:1: warning: âarray_getwritebufâ >> >> defined but not used >> >> numpy/core/src/multiarray/buffer.c:71:1: warning: âarray_getcharbufâ >> >> defined but not used >> >> numpy/core/src/multiarray/na_mask.c:681:1: warning: >> >> âPyArray_GetMaskInversionFunctionâ defined but not used >> >> error: Command "gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall >> >> -Wstrict-prototypes -fPIC -Inumpy/core/include >> >> -Ibuild/src.linux-x86_64-3.2/numpy/core/include/numpy >> >> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >> >> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray >> -Inumpy/core/src/umath >> >> -Inumpy/core/src/npysort -Inumpy/core/include >> >> -I/usr/local/include/python3.2m >> >> -Ibuild/src.linux-x86_64-3.2/numpy/core/src/multiarray >> >> -Ibuild/src.linux-x86_64-3.2/numpy/core/src/umath -c >> >> numpy/core/src/multiarray/multiarraymodule_onefile.c -o >> >> >> build/temp.linux-x86_64-3.2/numpy/core/src/multiarray/multiarraymodule_onefile.o" >> >> failed with exit status 1 >> >> >> >> >> >> >> >> >> Thanks for the prompt responses. >> >> That fixes the build problem for both Python3.1 and Python3.2. >> >> I got some test errors below but I guess you are working on those. >> >> >> Bruce >> >> >> >> $ python3 -c "import numpy; numpy.test()" >> Running unit tests for numpy >> NumPy version 2.0.0.dev-965a5c6 >> NumPy is installed in /usr/lib64/python3.2/site-packages/numpy >> Python version 3.2 (r32:88445, Feb 21 2011, 21:11:06) [GCC 4.6.0 >> 20110212 (Red Hat 4.6.0-0.7)] >> nose version 1.0.0 >> >> ..............S.......EFF.....E............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...................................................................................................................................................................................................K..................................................................................................K......................K..........................................................................................................S......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./usr/lib64/python3.2/site-packages/numpy/lib/format.py:575: >> ResourceWarning: unclosed file <_io.BufferedReader >> name='/tmp/tmpfmmo7x'> >> mode=mode, offset=offset) >> >> ......................................................................................................................................................................................................................../usr/lib64/python3.2/subprocess.py:460: >> ResourceWarning: unclosed file <_io.BufferedReader name=3> >> return Popen(*popenargs, **kwargs).wait() >> /usr/lib64/python3.2/subprocess.py:460: ResourceWarning: unclosed file >> <_io.BufferedReader name=8> >> return Popen(*popenargs, **kwargs).wait() >> >> .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... >> ====================================================================== >> ERROR: test_datetime_array_str (test_datetime.TestDateTime) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py", >> line 510, in test_datetime_array_str >> assert_equal(str(a), "['2011-03-16' '1920-01-01' '2013-05-19']") >> File "/usr/lib64/python3.2/site-packages/numpy/core/numeric.py", >> line 1400, in array_str >> return array2string(a, max_line_width, precision, suppress_small, >> ' ', "", str) >> File "/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py", >> line 459, in array2string >> separator, prefix, formatter=formatter) >> File "/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py", >> line 331, in _array2string >> _summaryEdgeItems, summary_insert)[:-1] >> File "/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py", >> line 502, in _formatArray >> word = format_function(a[-i]) + separator >> File "/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py", >> line 770, in __call__ >> casting=self.casting) >> TypeError: Cannot create a local timezone-based date string from a >> NumPy datetime without forcing 'unsafe' casting >> >> ====================================================================== >> ERROR: test_datetime_divide (test_datetime.TestDateTime) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py", >> line 926, in test_datetime_divide >> assert_equal(tda / tdb, 6.0 / 9.0) >> TypeError: internal error: could not find appropriate datetime inner >> loop in true_divide ufunc >> >> ====================================================================== >> FAIL: test_datetime_as_string (test_datetime.TestDateTime) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py", >> line 1166, in test_datetime_as_string >> '1959') >> File "/usr/lib64/python3.2/site-packages/numpy/testing/utils.py", >> line 313, in assert_equal >> raise AssertionError(msg) >> AssertionError: >> Items are not equal: >> ACTUAL: b'1959' >> DESIRED: '1959' >> >> ====================================================================== >> FAIL: test_datetime_as_string_timezone (test_datetime.TestDateTime) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py", >> line 1277, in test_datetime_as_string_timezone >> '2010-03-15T06:30Z') >> File "/usr/lib64/python3.2/site-packages/numpy/testing/utils.py", >> line 313, in assert_equal >> raise AssertionError(msg) >> AssertionError: >> Items are not equal: >> ACTUAL: b'2010-03-15T06:30Z' >> DESIRED: '2010-03-15T06:30Z' >> >> ---------------------------------------------------------------------- >> Ran 3063 tests in 37.701s >> >> FAILED (KNOWNFAIL=4, SKIP=2, errors=2, failures=2) >> > > The 3.2 test errors aren't new. I'd fix the tests except I'm not sure if > Mark wants to modify the datetime stuff instead. > I left them largely untouched because I found it weird that the 'S' data type doesn't return strings in Python 3... I guess maybe the datetime_as_string function should convert to 'U' data type on Python 3 after building the 'S' array to work around this design choice. I'll look at it after the NA stuff is wrapped up. -Mark > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
