[gdal-dev] Spatial references and geometry operations

2023-09-28 Thread adamgutonski via gdal-dev
Hello, I am using geometry object functions like Buffer and Value. I am wondering if and when it is necessary to assign a spatial reference to an ogr.Geometry before calling these functions? For example if I want to get a point along the line using Value is the output affected by whether or not

Re: [gdal-dev] GRIB2 file extent not matchi GRIB2 metadata (grib_dump)

2023-09-28 Thread Even Rouault via gdal-dev
Hi Louis-Philippe, yes this is a matter of conventions. GRIB indeed uses a center-of-pixel registration whereas GDAL uses a top-left corner of pixel ones. So GDAL adds a half-pixel shift to expose GRIB georeferencing using its convention, which is a feature. What is annoying is that it leads t

[gdal-dev] GRIB2 file extent not matchi GRIB2 metadata (grib_dump)

2023-09-28 Thread Rousseau Lambert, Louis-Philippe (ECCC) via gdal-dev
Hi, I am facing an issue with GRIB2 data, and the extent reported by a gdal_info command. Here is a sample data to test: https://drive.google.com/file/d/1URvQs2qHXgRkq3YfC7ZR8VGYK3SKXB59/view?usp=drive_link Here is what I am testing with: * Ubuntu 20.04.6 LTS * GDAL 3.5.2, released 202

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Scott via gdal-dev
Increasing swap was my last resort as I'd prefer not to do this across different systems. However, that's exactly what I'll do! Thanks for the help people! On 9/28/23 11:36, Cainã K. Campos via gdal-dev wrote: I believe that you could try to increase your swap RAM, for linux it is pretty strai

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
Le 28/09/2023 à 20:17, Scott a écrit : Thanks for digging into that Even! Can I create my new .fgb in sections? If I limit the number of source rows with -sql, doing that multiple times with -update, will it still build the entire R-tree when writing to the destination? That would actually b

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Cainã K . Campos via gdal-dev
I believe that you could try to increase your swap RAM, for linux it is pretty straightforward, and having a SSD or NVME it will perform good. Free disk space is a must have to this to work, as you are going to need about 10 - 20 Gb disk space as swap, according to Even calcs + 8Gb that you have. N

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Scott via gdal-dev
Thanks for digging into that Even! Can I create my new .fgb in sections? If I limit the number of source rows with -sql, doing that multiple times with -update, will it still build the entire R-tree when writing to the destination? I'm looking for a way to get the desired results. On 9/28/2

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
ok, that now makes sense. Writing a .fgb files comes into those exceptions where RAM consumption might be important, as it involves building a packed Hilbert R-Tree in memory. With the current implementation, you need at least the number of features times some constant amount of RAM, at least t

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Scott via gdal-dev
I get the same error on OS AWS Linux 2. Also, on either OS with source .parquet instead of .fgb. On 9/28/23 10:17, Scott via gdal-dev wrote: USA.fgb is 36 GB. I've renamed it from its original source which can be found here: https://beta.source.coop/vida/google-microsoft-open-buildings ogr2o

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Scott via gdal-dev
USA.fgb is 36 GB. I've renamed it from its original source which can be found here: https://beta.source.coop/vida/google-microsoft-open-buildings ogr2ogr -sql "select area_in_meters from bfp_USA" -nln footprints footprints.fgb ~/Downloads/USA.fgb GDAL 3.7.1 OS Debian Buster Output from ogrin

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit : I should have been more specific. One particular machine has 8GB of memory. When I try to do the most simple ogr2ogr command on large files, the host runs out of memory (vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more.

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Scott via gdal-dev
I should have been more specific. One particular machine has 8GB of memory. When I try to do the most simple ogr2ogr command on large files, the host runs out of memory (vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more. The data formats I have experienced this with are

Re: [gdal-dev] GDAL version in pip requirements file

2023-09-28 Thread Kurt Schwehr via gdal-dev
I am not sure if there is anything in this, but take a look at https://packaging.python.org/en/latest/guides/ Also, I might suggest avoiding command line solutions. You can get the version from the python module On Mon, Sep 25, 2023, 11:55 PM Luca Delucchi via gdal-dev < gdal-dev@lists.osgeo.org>

Re: [gdal-dev] GDAL version in pip requirements file

2023-09-28 Thread Luca Delucchi via gdal-dev
Il mar 26 set 2023, 09:07 Stefan Blumentrath ha scritto: > Hi Luca, > Hi Stefan, > > You could parse the gdal config version output into an environment > variable and then specify the version in requirements.txt like this: > GDAL==${GDAL_VERSION} > Thanks a lot it looks a good solution > > Ch

Re: [gdal-dev] GDAL version in pip requirements file

2023-09-28 Thread Luca Delucchi via gdal-dev
Hi Peter, Il mar 26 set 2023, 16:33 Peter Schmitt ha scritto: > I know you mention requirements.txt, but I write my setup.py with a > function that uses gdal-config to get Python bindings for the version of > GDAL I have installed. It looks something like this: > Thanks for your help but I'm l

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

2023-09-28 Thread Jonathan Moules via gdal-dev
Well, it seems that PROJ_DATA isn't set in their environment. But it's not set in mine either (`print(os.environ['PROJ_DATA']`)! So no idea why mine works just fine without it (Windows 11 thing?). Creating a PROJ_DATA env var didn't fix anything. Even adding it explicitly in Python. Their lo

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

2023-09-28 Thread Rahkonen Jukka via gdal-dev
Hi, Then they should add that environment if they do not know that they do not belong to "most users" https://proj.org/en/9.3/usage/environmentvars.html -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: gdal-dev Puolesta Jonathan Moules via gdal-dev Lähetetty: torstai 28. syyskuuta 20

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

2023-09-28 Thread Jonathan Moules via gdal-dev
Hi Even, The colleague doesn't have either a PROJ_LIB or a PROJ_DATA environment variable. I asked another colleague to try it; they're on Ubuntu 20.04, and it worked for them. I believe using the same setup instructions. Cheers, Jonathan On 2023-09-24 22:37, Jonathan Moules via gdal-dev

Re: [gdal-dev] About installing EXPAT

2023-09-28 Thread Uhrig, Stefan via gdal-dev
Hi Abel, I can recommend vcpkg (https://vcpkg.io/) if you’d like to build GDAL on Windows with Visual Studio or Visual Studio Community Edition. vcpkg builds packages from sources and can build the mandatory dependencies of GDAL (and a lot of optional ones). You get a working starting set via

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Rahkonen Jukka via gdal-dev
Hi, Do you mean that your conversions fail because ogr2ogr does not get enough memory, or do you want to tell ogr2ogr to use max xxx MB of memory even there is much more available? -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: gdal-dev Puolesta Scott via gdal-dev Lähetetty: keski

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Laurențiu Nicola via gdal-dev
Hello, You might want to say which drivers you're using. I can imagine PostGIS with PG_USE_COPY using more memory with a larger batch size (ogr2ogr -gt), for example. But most likely you're not going to be able to do anything about it. So how much memory are we talking about, and which formats

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Abel Pau via gdal-dev
Hi Scott, As a programmer, I can see that when translating from one format to another, there is a minimal amount of memory to work with. The primary objective for a programmer is to be cautious about this and strive to stay within this limit. The easy solution would be to request as much memory