Dear GDAL Developers
I’m finding that when
saving a relatively small array (shape =
128,4000,128 ie 128 band x 4000 lines x
128 columns of float data = 256Mb total) to a ENVI file format using the call:
gdalnumeric.SaveArray(img,strf_out,'ENVI')
performance is extremely slow (~260 seconds). When I save the same array to
Geotif format using;
gdalnumeric.SaveArray(img,strf_out,'GTIFF')
the operation
takes ~24 seconds which is much more
acceptable.
I’m currently
using gdal version 1.11.1 .
My question to
the forum is then -why is there such a
large discrepancy between the performance of the two operations and is there a
way
to improve the performance when writing an array to an ENVI file format?
I have tried changing
the GDAL Cache options but performance doesn’t seem to alter. When profiling
the code which saves the array
to GeoTIFF format I get the output contained in the attached file
timing_savearray_GTIFF.txt. When profiling the code which saves the array
to ENVI format I get the output contained in the attached file
timing_savearray_ENVI.txt
I would like to
save the data to ENVI file format. Your
thoughts on this matter would be appreciated.
Thanks
Regards
mike
Finished save
3790 function calls in 24.405 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 24.405 24.405 <string>:1(<module>)
3 0.038 0.013 0.601 0.200 _methods.py:115(_std)
6 0.000 0.000 0.000 0.000
_methods.py:39(_count_reduce_items)
3 0.000 0.000 0.116 0.039 _methods.py:49(_mean)
3 0.368 0.123 0.563 0.188 _methods.py:71(_var)
128 0.000 0.000 0.001 0.000 fromnumeric.py:122(reshape)
3 0.000 0.000 0.000 0.000 fromnumeric.py:38(_wrapit)
3 0.000 0.000 0.000 0.000 fromnumeric.py:492(transpose)
1 0.000 0.000 0.000 0.000 gdal.py:332(GetDescription)
1 0.000 0.000 0.000 0.000 gdal.py:340(GetMetadataDomainList)
3 0.000 0.000 0.000 0.000 gdal.py:344(GetMetadata_Dict)
3 0.000 0.000 0.000 0.000 gdal.py:367(GetMetadata)
3 0.000 0.000 6.777 2.259 gdal.py:390(CreateCopy)
8 0.000 0.000 0.000 0.000 gdal.py:655(<lambda>)
1 0.000 0.000 0.000 0.000 gdal.py:656(GetDriver)
384 0.000 0.000 0.001 0.000 gdal.py:660(GetRasterBand)
1 0.000 0.000 0.000 0.000 gdal.py:664(GetProjection)
1 0.000 0.000 0.000 0.000 gdal.py:676(GetGeoTransform)
1 0.000 0.000 0.927 0.927 gdal.py:707(FlushCache)
128 0.000 0.000 0.000 0.000 gdal.py:962(GetNoDataValue)
3 0.000 0.000 0.000 0.000
gdal_array.py:104(GetArrayFilename)
128 0.000 0.000 15.647 0.122
gdal_array.py:108(BandRasterIONumPy)
3 0.000 0.000 0.001 0.000 gdal_array.py:152(OpenArray)
385 0.001 0.000 0.001 0.000 gdal_array.py:165(flip_code)
128 0.000 0.000 0.001 0.000
gdal_array.py:184(NumericTypeCodeToGDALTypeCode)
257 0.000 0.000 0.001 0.000
gdal_array.py:189(GDALTypeCodeToNumericTypeCode)
1 0.018 0.018 15.788 15.788 gdal_array.py:192(LoadFile)
3 0.013 0.004 6.791 2.264 gdal_array.py:199(SaveArray)
1 0.002 0.002 15.764 15.764
gdal_array.py:206(DatasetReadAsArray)
128 0.001 0.000 15.651 0.122 gdal_array.py:240(BandReadAsArray)
3 0.000 0.000 0.000 0.000 numeric.py:392(asarray)
9 0.000 0.000 0.000 0.000 numeric.py:462(asanyarray)
1 0.000 0.000 0.000 0.000 pyPlugger.py:104(pyGetMetadata)
1 0.000 0.000 0.000 0.000 pyPlugger.py:113(pyStringtoList)
1 0.000 0.000 0.000 0.000
pyPlugger.py:130(pyExtractENVImetadata)
1 0.000 0.000 0.009 0.009 pyPlugger.py:160(pyOpenImage)
1 0.000 0.000 0.010 0.010 pyPlugger.py:168(pyQueryImage)
1 0.000 0.000 0.716 0.716
pyPlugger.py:231(pyPluggerCalStats)
1 0.000 0.000 0.000 0.000
pyPlugger.py:246(pyPluggerGenerateMask)
1 0.001 0.001 0.001 0.001
pyPlugger.py:288(pyPluggerMaskAnalyse)
1 0.001 0.001 0.002 0.002
pyPlugger.py:341(pyPluggerApplyCorrection)
1 0.170 0.170 24.405 24.405 pyPlugger.py:378(pyPlugger)
128 0.000 0.000 0.000 0.000 {_gdal.Band_GetNoDataValue}
1 0.927 0.927 0.927 0.927 {_gdal.Dataset_FlushCache}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetDriver}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetGeoTransform}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetProjection}
384 0.001 0.000 0.001 0.000 {_gdal.Dataset_GetRasterBand}
3 6.777 2.259 6.777 2.259 {_gdal.Driver_CreateCopy}
3 0.000 0.000 0.000 0.000 {_gdal.GetDriverByName}
1 0.000 0.000 0.000 0.000 {_gdal.MajorObject_GetDescription}
1 0.000 0.000 0.000 0.000
{_gdal.MajorObject_GetMetadataDomainList}
3 0.000 0.000 0.000 0.000
{_gdal.MajorObject_GetMetadata_Dict}
5 0.017 0.003 0.017 0.003 {_gdal.Open}
128 15.646 0.122 15.646 0.122 {_gdal_array.BandRasterIONumPy}
3 0.000 0.000 0.000 0.000 {_gdal_array.GetArrayFilename}
3 0.000 0.000 0.000 0.000 {getattr}
531 0.000 0.000 0.000 0.000 {isinstance}
3 0.000 0.000 0.000 0.000 {issubclass}
3 0.000 0.000 0.000 0.000 {max}
384 0.000 0.000 0.000 0.000 {method 'append' of 'list'
objects}
24 0.001 0.000 0.001 0.000 {method 'astype' of
'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profiler' objects}
128 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'lstrip' of 'str' objects}
3 0.000 0.000 0.116 0.039 {method 'mean' of 'numpy.ndarray'
objects}
4 0.001 0.000 0.001 0.000 {method 'nonzero' of
'numpy.ndarray' objects}
9 0.310 0.034 0.310 0.034 {method 'reduce' of 'numpy.ufunc'
objects}
132 0.001 0.000 0.001 0.000 {method 'reshape' of
'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
3 0.000 0.000 0.601 0.200 {method 'std' of 'numpy.ndarray'
objects}
3 0.000 0.000 0.000 0.000 {method 'transpose' of
'numpy.ndarray' objects}
12 0.000 0.000 0.000 0.000 {numpy.core.multiarray.array}
1 0.109 0.109 0.109 0.109
{numpy.core.multiarray.concatenate}
128 0.002 0.000 0.002 0.000 {numpy.core.multiarray.empty}
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
3 0.000 0.000 0.000 0.000 {range}Finished correction
Finished save
3790 function calls in 266.837 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 266.837 266.837 <string>:1(<module>)
3 0.041 0.014 0.661 0.220 _methods.py:115(_std)
6 0.000 0.000 0.000 0.000
_methods.py:39(_count_reduce_items)
3 0.000 0.000 0.117 0.039 _methods.py:49(_mean)
3 0.419 0.140 0.621 0.207 _methods.py:71(_var)
128 0.000 0.000 0.001 0.000 fromnumeric.py:122(reshape)
3 0.000 0.000 0.001 0.000 fromnumeric.py:38(_wrapit)
3 0.000 0.000 0.001 0.000 fromnumeric.py:492(transpose)
1 0.000 0.000 0.000 0.000 gdal.py:332(GetDescription)
1 0.000 0.000 0.000 0.000 gdal.py:340(GetMetadataDomainList)
3 0.000 0.000 0.000 0.000 gdal.py:344(GetMetadata_Dict)
3 0.000 0.000 0.000 0.000 gdal.py:367(GetMetadata)
3 0.000 0.000 227.863 75.954 gdal.py:390(CreateCopy)
8 0.000 0.000 0.000 0.000 gdal.py:655(<lambda>)
1 0.000 0.000 0.000 0.000 gdal.py:656(GetDriver)
384 0.000 0.000 0.001 0.000 gdal.py:660(GetRasterBand)
1 0.000 0.000 0.000 0.000 gdal.py:664(GetProjection)
1 0.000 0.000 0.000 0.000 gdal.py:676(GetGeoTransform)
1 0.000 0.000 20.384 20.384 gdal.py:707(FlushCache)
128 0.000 0.000 0.000 0.000 gdal.py:962(GetNoDataValue)
3 0.000 0.000 0.000 0.000
gdal_array.py:104(GetArrayFilename)
128 0.000 0.000 17.308 0.135
gdal_array.py:108(BandRasterIONumPy)
3 0.000 0.000 0.001 0.000 gdal_array.py:152(OpenArray)
385 0.001 0.000 0.002 0.000 gdal_array.py:165(flip_code)
128 0.000 0.000 0.001 0.000
gdal_array.py:184(NumericTypeCodeToGDALTypeCode)
257 0.000 0.000 0.001 0.000
gdal_array.py:189(GDALTypeCodeToNumericTypeCode)
1 0.016 0.016 17.481 17.481 gdal_array.py:192(LoadFile)
3 0.018 0.006 227.883 75.961 gdal_array.py:199(SaveArray)
1 0.002 0.002 17.458 17.458
gdal_array.py:206(DatasetReadAsArray)
128 0.001 0.000 17.312 0.135 gdal_array.py:240(BandReadAsArray)
3 0.000 0.000 0.001 0.000 numeric.py:392(asarray)
9 0.000 0.000 0.000 0.000 numeric.py:462(asanyarray)
1 0.000 0.000 0.000 0.000 pyPlugger.py:104(pyGetMetadata)
1 0.000 0.000 0.000 0.000 pyPlugger.py:113(pyStringtoList)
1 0.000 0.000 0.000 0.000
pyPlugger.py:130(pyExtractENVImetadata)
1 0.000 0.000 0.012 0.012 pyPlugger.py:160(pyOpenImage)
1 0.000 0.000 0.013 0.013 pyPlugger.py:168(pyQueryImage)
1 0.000 0.000 0.779 0.779
pyPlugger.py:231(pyPluggerCalStats)
1 0.098 0.098 0.124 0.124
pyPlugger.py:246(pyPluggerGenerateMask)
1 0.013 0.013 0.014 0.014
pyPlugger.py:288(pyPluggerMaskAnalyse)
1 0.002 0.002 0.003 0.003
pyPlugger.py:341(pyPluggerApplyCorrection)
1 0.135 0.135 266.837 266.837 pyPlugger.py:378(pyPlugger)
128 0.000 0.000 0.000 0.000 {_gdal.Band_GetNoDataValue}
1 20.384 20.384 20.384 20.384 {_gdal.Dataset_FlushCache}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetDriver}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetGeoTransform}
1 0.000 0.000 0.000 0.000 {_gdal.Dataset_GetProjection}
384 0.001 0.000 0.001 0.000 {_gdal.Dataset_GetRasterBand}
3 227.863 75.954 227.863 75.954 {_gdal.Driver_CreateCopy}
3 0.000 0.000 0.000 0.000 {_gdal.GetDriverByName}
1 0.000 0.000 0.000 0.000 {_gdal.MajorObject_GetDescription}
1 0.000 0.000 0.000 0.000
{_gdal.MajorObject_GetMetadataDomainList}
3 0.000 0.000 0.000 0.000
{_gdal.MajorObject_GetMetadata_Dict}
5 0.020 0.004 0.020 0.004 {_gdal.Open}
128 17.307 0.135 17.307 0.135 {_gdal_array.BandRasterIONumPy}
3 0.000 0.000 0.000 0.000 {_gdal_array.GetArrayFilename}
3 0.000 0.000 0.000 0.000 {getattr}
531 0.001 0.000 0.001 0.000 {isinstance}
3 0.000 0.000 0.000 0.000 {issubclass}
3 0.000 0.000 0.000 0.000 {max}
384 0.000 0.000 0.000 0.000 {method 'append' of 'list'
objects}
24 0.027 0.001 0.027 0.001 {method 'astype' of
'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profiler' objects}
128 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'lstrip' of 'str' objects}
3 0.000 0.000 0.117 0.039 {method 'mean' of 'numpy.ndarray'
objects}
4 0.021 0.005 0.021 0.005 {method 'nonzero' of
'numpy.ndarray' objects}
9 0.318 0.035 0.318 0.035 {method 'reduce' of 'numpy.ufunc'
objects}
132 0.001 0.000 0.001 0.000 {method 'reshape' of
'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects}
3 0.000 0.000 0.661 0.220 {method 'std' of 'numpy.ndarray'
objects}
3 0.000 0.000 0.000 0.000 {method 'transpose' of
'numpy.ndarray' objects}
12 0.001 0.000 0.001 0.000 {numpy.core.multiarray.array}
1 0.141 0.141 0.141 0.141
{numpy.core.multiarray.concatenate}
128 0.002 0.000 0.002 0.000 {numpy.core.multiarray.empty}
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
3 0.000 0.000 0.000 0.000 {range}
Finished_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
