any thoughts on a ogr netcdf driver? I have a simple one for madis data that needs work in my sandbox
Brian On Sun, 2011-08-28 at 14:56 -0700, Etienne Tourigny wrote: > I have created a new wiki page at > http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements , following Even's > suggestion (Thanks Even). > > The following topics have been included: > > > Topics: > > 1 - UNIDATA NetCDF Conventions > > 2 - Metadata > 3 - Datum issues > 4 - Dimension issues > 5 - Compatibility with other software > 6 - Test files > 7 - Contribution > Discussion > > I encourage interested parties in participating in this discussion in the > wiki and in the mailing list. > > > In particular, it would be useful to collect information on contributors and > obtain test files with issues and the expected outcome. > > regards, Etienne > > > ----- Original Message ----- > From: Even Rouault <even.roua...@mines-paris.org> > To: gdal-dev@lists.osgeo.org; Etienne Tourigny <etienne...@yahoo.com> > Cc: > Sent: Friday, August 26, 2011 4:20:06 PM > Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver > and CF-1 convention > > Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a écrit : > > My thoughts, just from a process point of view, not being myself deeply > involved in the NetCDF format : > > * A wiki page (I'd suggest either a new page - > http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements - or at least a clearly > identified section in the current page, to avoid confusion for outsiders > about > what is currently implemented and what is not) can be a convenient way of > structuring and editing the content by anyone with a osgeo id. But > discussions > on the mailing list are also OK, at least to notify new content has been > added > (provided that it doesn't cause flooding on the mailing list ;-)) > > * A http://download.osgeo.org/gdal/data/netcdf directory would be the right > place to store NetCDF sample files, provided they are free under the terms of > http://download.osgeo.org/gdal/data/COPYING . As write access to it is > restricted to people with appropriate rights, having those rights, I can help > uploading files over there. Having small (typically < or ~ 10 KB ) sample > files > is also good so they can be directly included in SVN under the > autotest/gdrivers/data directory and thus easily used by regression tests. > > * Tickets can be used to track the implementation of specific items and > bugfixes. But I've found they are not generally suitable for brainstorming > when > the discussion is lasting, as you cannot edit already posted content. Wiki is > clearly better for this. > > * Looking at past history, I see we haven't yet used the RFC formalism for > work specific to one driver, while it doesn't require changes or extensions > to > the GDAL API. But it wouldn't hurt to create one if the need arises. Anyway, > a > point that might require care and some consensus among the interested parties > is about how backward compatibility issues, in particular with respect to > files > produced by older GDAL versions, are adressed. > http://trac.osgeo.org/gdal/wiki/rfc1_pmc details when a vote is required. > > Best regards, > > Even > > > Hi all, > > > > Since a few people have shown an interest in helping out, I would suggest > > to set up space where we can post ideas, goals and test files. This could > > take the form of one of the following: > > > > - a new wiki entry in trac, or the existing one at > > http://trac.osgeo.org/gdal/wiki/NetCDF - a file repository at > > http://download.osgeo.org/gdal/data/netcdf as suggested some time ago by > > warmerdam > > > > and/or > > > > - a new ticket in trac (which allows to post files) > > > > What are you thoughts about this? Would it make sense to eventually > > prepare an RFC? > > > > > > > > I would suggest that each group identify the problems with the current > > driver in terms of importing and exporting, post the files that they would > > like to be supported, which other software they use with the netcdf files, > > and their proposed contribution (code, test, ideas, etc.). > > > > Those that cannot participate in code development can help out in testing > > and discussion. Personally, I can write and test code, given example files > > and expected outcome. > > > > > > Now to your specific comments, Knut-Frode: > > > > 1) Yes CF-1.5 should be the goal, although I'm not sure we need to support > > upgrading/downgrading. > > > > 5) The WGS84 issue needs further research. Weather and ocean data are in > > lat/lon, but what is the sphere radius assumed by the various data > > providers and the models which output/import netcdf files? I agree that a > > small error does not make a difference with low-resolution dataset, but > > I'd like to be sure in any case. We also need a means to translate the > > CF-1.x datum information into EPSG/Proj.4 values. > > > > > > 7) There is some support of a z-axis right now, in fact they driver assumes > > that a third dimension is vertical if I understand correctly. However, I > > have not tested nor looked into that. This issue definitely needs come > > planning and an RFC, IMHO. > > > > > > Kyle, thanks for you help and interest! > > > > > > cheers, > > Etienne > > > > > > ________________________________ > > From: Knut-Frode Dagestad <knutfrodesop...@hotmail.com> > > To: gdal-dev@lists.osgeo.org > > Sent: Friday, August 26, 2011 4:41:02 AM > > Subject: [gdal-dev] Re: discussion on improvements to the NetCDF driver and > > CF-1 convention > > > > Etienne, > > > > My group is also very much in support of the suggested improvements, as > > NetCDF/CF is becoming the most important standard for data > > storage/exchange in our community (climate, meteorology, oceanography). My > > group is not much into GDAL driver development (at least not yet), but we > > are now building our future very much around GDAL, so we are interested to > > join the discussion around improved support for NetCDF/CF reading and > > writing. > > > > About some of the mentioned issues: > > > > 1) The latest version of CF is 1.5, so it would be better to target 1.5 (or > > 1.x) than 1.0. Ideally the driver could be made generic for smooth upgrade > > to newer (or downgrade to older) versions of the CF conventions. > > > > 5) It sounds meaningful to assume WGS84 datum if not specified. I believe > > at least 99% of data typically stored in NetCDF/CF will have this datum, > > and even when that is not the case, positioning errors less than 100 m > > should normally have little impact for the types of data. Precision on > > datum level is probably more relevant for more typical GIS-data with > > roads, borders etc. A warning could however be issued each time the driver > > has to assume WGS84. > > > > 7) We agree in particular that proper handling of time axis is of high > > value. This is one of the issues which is highly relevant for climate-data > > (often 4-dimensional), but not well supported by typical GIS-software or > > standards designed with 2-dimensional data in mind. A general > > representation of a time-axis (or even vertical axis) in GDAL would be > > ideal, but that would probably be too much an effort for anyone to > > develop. > > > > > > Knut-Frode > > Nansen Environmental and Remote Sensing Center > > > > On 24/08/2011 22:37, Etienne Tourigny wrote: > > > Hi all, > > > > > > I would like to start a discussion with those interested about fixing > > > various issues in the NetCDF driver. > > > > > > As it stands, it is not fully CF-1.0 compliant, and produces > > > geographical grids that are not valid for other software. > > > Also, the infamous "up-side down" problem has been addressed for > > > importing netcdfs, but needs to be addressed for exporting. > > > > > > Related ticket: http://trac.osgeo.org/gdal/ticket/2129 > > > I have been submitting patches for some time to fix the related issues. > > > > > > I have identified a number of issues, of which the following have been > > > fixed partially in my proposed patch. > > > > > > 1- conform to the cf-1.0 standard for geographical grids (creating lat > > > and lon dimensions and variables) > > > 2- make netcdf grids "bottom-up" by default, as all software out there > > > uses that convention > > > 3- fix problems related to floating-point / string conversions (also > > > reported in issue #4200 ) > > > 4- fix metadata handling and duplication (as reported in issue #4204), > > > and keep band (variable) metadata outside of the global metadata > > > > > > Pending issues: > > > > > > 5- how to deal with netcdfs which have no datum? Can we assume WGS84? > > > These files are extremely common, and CF-1.0 has provisions for > > > identifying the key variables, but no Proj/Wkt codes. > > > 6- fix proper export of projected CRS (this is not fully necessary > > > though) with lat/lon values respecting CF-1.0 > > > 7- handle time axis properly (at least when creating a netcdf) - that > > > would be great for import/export of netcdf > > > > > > Regards, > > > Etienne > > > > > > > > > _______________________________________________ > > > gdal-dev mailing list > > > gdal-dev@lists.osgeo.org > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
