This is an interesting idea and approach, thanks Piero for the suggestion and the proposed solution.
I agree with Piero and Adam, it would be a great addition to the raster field. Especially for large volumes of data (not only for individual large files, but also for many small files). Raster data can eat up a lot of storage quickly, and delegating that to an off-db storage is useful. I have been looking at the out-of-db feature for PostGIS rasters, but I am not sure if it can be used in the Django context. The problem I see is the storage location. If I understand it right, then the raster files need to be on the same system as the PostGIS instance, i.e. they need to be available on the filesystem of the postgis server. Like that, PostGIS can access the files from within its raster operations (such as intersections, getting pixel values etc), and it will work as if the data was stored in the db directly. In that case however, the out-of-db field would only work if Django is running on the same server as the PostGIS instance. So I think it would fail for more distributed systems, where you have multiple application instances, and a remote PostGIS, which might have replicas etc. In your experience Piero, is that correct? How are the out-of-db raster files stored by PostGIS? I have never used the out-of-db raster option so I am not sure if I understood the storage mechanism correclty. Coincidently, I recently had a similar problem, and I have been testing a version of a raster field which is a subclass of the regular Django FileField. In this approach, the storage can be any Django compatible storage (including remote object storages such as S3). The downside is that PostGIS will no longer recognize the data as rasters, so lookups and PostGIS internal functions will not be available. This is still quite experimental (it uses a patched Django with an extension of the GDALRaster) and is slightly hacky, but does convert the remote rasters into GDALRaster instances when opened. https://github.com/geodesign/django-raster/compare/raster_file_field An ideal solution would be a mixture of the two, but I am not sure if PostGIS can handle remote storages. On Tuesday, December 20, 2016 at 4:33:21 PM UTC, Tim Graham wrote: > > Daniel Wiesmann did all the work for RasterField. I'm not sure if he > follows this list but you can find his email address in the Django commit > longs and mail him to ask for his input. > > https://github.com/yellowcap > > On Tuesday, December 20, 2016 at 11:27:25 AM UTC-5, Piero Toffanin wrote: >> >> It's stored on the file system. This is to improve performance when >> storing large geospatial datasets. >> >> This would only work on PostGIS. >> >> On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam Johnson wrote: >>> >>> I can't say I'm that familiar with GeoDjango, but that does sound like a >>> useful feature. Where does the data get stored if not in the DB? And does >>> this feature exist on any of the other database backends that GeoDjango >>> supports? >>> >>> On 14 December 2016 at 18:40, Piero Toffanin <pier...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> Not sure this is the right place to post this, if not, could somebody >>>> point me to the right place? >>>> >>>> I recently had the need to use GeoDjango to define a model that uses a >>>> RasterField to store a GeoTIFF raster. >>>> https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield >>>> >>>> Upon testing, one of the users of the application found that loading a >>>> large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the >>>> report are available here: >>>> https://github.com/OpenDroneMap/WebODM/issues/55 >>>> >>>> I went around the problem by using a relatively new feature of PostGIS >>>> that allows raster files to be stored off db. I noticed that RasterField >>>> does not support such feature, so I wrote the code to enable support for >>>> it >>>> via a new OffdbRasterField ( >>>> https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). >>>> The from_pgraster and to_pgraster functions are modified versions of the >>>> same functions found here: >>>> https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py >>>> >>>> Just wanted to see if there was an interest in adding off db raster >>>> support into GeoDjango core. Perhaps by modifying RasterField to have an >>>> additional parameter "offdb=True|False" and implement the necessary logic? >>>> >>>> Thanks, >>>> >>>> -Piero >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Django developers (Contributions to Django itself)" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to django-develop...@googlegroups.com. >>>> To post to this group, send email to django-d...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/django-developers. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Adam >>> >> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2e2125c7-8275-4189-982d-ab77d6d686f3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
