Hi all, I've been successfully reading OSM PBF files using GDAL Python bindings, even files larger than 5Gb. I did have to use OGR_INTERLEAVED_READING=YES and OSM_USE_CUSTOM_INDEXING=NO, but eventually OSM PBF is read and I can extract data.
If I use osmconvert [0] and osmfilter [1] to preprocess data, Python script complains that OSM_USE_CUSTOM_INDEXING is not used. In the debug output, it will report that OSM_USE_CUSTOM_INDEXING is used... """ Tue Apr 1 19:04:56 2014: Shape: DBF Codepage = UTF-8 for /tmp/out/world1.shp Tue Apr 1 19:04:56 2014: Shape: Treating as encoding 'UTF-8'. Tue Apr 1 19:04:57 2014: OSM: Using SQLite indexing for points Tue Apr 1 19:04:57 2014: OGR: OGROpen(/go_video/admin-levels.pbf/0x1d4a570) succeeded as OSM. Tue Apr 1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available field. ERROR 1: 'admin_level' not recognised as an available field. Tue Apr 1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available field. ERROR 1: 'admin_level' not recognised as an available field. Tue Apr 1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available field. ERROR 1: 'admin_level' not recognised as an available field. Tue Apr 1 19:04:57 2014: OSM: OGR_INTERLEAVED_READING = 1 Tue Apr 1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available field. ERROR 1: 'admin_level' not recognised as an available field. Tue Apr 1 19:04:57 2014: OSM: Got bounds : minx=-180.000000, miny=-90.000000, maxx=180.000000, maxy=90.000000 Tue Apr 1 19:04:57 2014: OSM: For layer points, new max size is 128 Tue Apr 1 19:04:57 2014: OSM: For layer points, new max size is 320 Tue Apr 1 19:04:57 2014: OSM: For layer points, new max size is 608 Tue Apr 1 19:04:57 2014: OSM: For layer points, new max size is 1040 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 1688 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 2660 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 4118 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 6305 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 9585 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 14505 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 21885 Tue Apr 1 19:04:58 2014: OSM: For layer points, new max size is 32955 Tue Apr 1 19:05:01 2014: OSM: all_tags/other_tags field truncated for feature 249399339 Tue Apr 1 19:05:01 2014: OSM: For layer points, new max size is 49560 Tue Apr 1 19:05:03 2014: OSM: For layer points, new max size is 74468 Tue Apr 1 19:05:06 2014: OSM: /vsimem/osm_importer/osm_temp_0x1d4a570.sqlite too big for RAM. Transfering it onto disk in /tmp/osm_tmp1067730688_0 Tue Apr 1 19:05:20 2014: OSM: For layer points, new max size is 111830 Tue Apr 1 19:05:26 2014: CPLError: Too many features have accumulated in points layer. Use OGR_INTERLEAVED_READING=YES mode ERROR 1: Too many features have accumulated in points layer. Use OGR_INTERLEAVED_READING=YES mode Tue Apr 1 19:05:26 2014: OGR: ReleaseDataSource(/go_video/admin-levels.pbf/0x1d4a570) on unshared datasource! Deleting directly. Tue Apr 1 19:05:26 2014: OSM: Number of bytes read in file : 143840051 Tue Apr 1 19:05:26 2014: OGR: ReleaseDataSource(/tmp/out/world1.shp/0x19a93d0) on unshared datasource! Deleting directly. Tue Apr 1 19:05:26 2014: GDAL: In GDALDestroy - unloading GDAL shared library. """ However, using ogr2ogr utility on the command line works like a charm. If I convert PBF->OSM, script will work, but exported data is missing some of the features. I can provide test data and the script if required. Thank you, Dražen [0] https://wiki.openstreetmap.org/wiki/Osmconvert [1] https://wiki.openstreetmap.org/wiki/Osmfilter _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
