Are rasters on the GeoDjango roadmap?
Hi All I am wondering if integration of Raster data is on the roadmap of GeoDjango. I am asking because I have started to work on a package to include Rasters into Django (see link below). This is still very early stage, but I already use the package in a Django project I am working on. For that project I will continue extending the package and would like to work towards integration with GeoDjango (although I am aware that this is a long way to go). https://pypi.python.org/pypi/django-raster/ However, before spending more time on this I wanted to make sure I am not duplicating something that is already underway. Does anyone know about the state of rasters in GeoDjango and/or if its on the roadmap for future development? Thanks, Daniel -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7bf8a60a-a138-4c87-91d6-61070e30d09b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Spatial lookups for RasterField
Hello Everyone I have been working on adding spatial lookups for the RasterField in the GIS module. https://github.com/django/django/pull/5460#issuecomment-152606283 I would like to get feedback regarding API design for the lookups. The challenge is that one needs to specify two parameters for the lookup: the raster itself and a band index. I think everything should default to the first band, as many rasters only have one band. For the special cases, an index has to be specified. On the right hand side (rhs) the proposal is to use a syntax similar to the existing distance lookups, providing a tuple: (raster, band_index). That follows previous art and should be ok. But on the left hand side (lhs) its less clear to me what to do. For instance, to compare band 2 from a raster field "elevation" with a geometry, I can see two options A specify the index with a double underscore: elevation__2__intersects=geom B pass the band index on the rhs: elevation__intersects=(geom, 2) The current implementation in the PR above uses the second option. But I have doubts because that mixes the lhs with the rhs. Here are my questions: - Are there other options for the api that shoudl be considered? - If not, which of the two approaches would be better? Thanks, Daniel -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a67874c5-f1d4-44f1-a0c6-97765544125a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: GeoDjango OffdbRasterField
This is an interesting idea and approach, thanks Piero for the suggestion and the proposed solution. I agree with Piero and Adam, it would be a great addition to the raster field. Especially for large volumes of data (not only for individual large files, but also for many small files). Raster data can eat up a lot of storage quickly, and delegating that to an off-db storage is useful. I have been looking at the out-of-db feature for PostGIS rasters, but I am not sure if it can be used in the Django context. The problem I see is the storage location. If I understand it right, then the raster files need to be on the same system as the PostGIS instance, i.e. they need to be available on the filesystem of the postgis server. Like that, PostGIS can access the files from within its raster operations (such as intersections, getting pixel values etc), and it will work as if the data was stored in the db directly. In that case however, the out-of-db field would only work if Django is running on the same server as the PostGIS instance. So I think it would fail for more distributed systems, where you have multiple application instances, and a remote PostGIS, which might have replicas etc. In your experience Piero, is that correct? How are the out-of-db raster files stored by PostGIS? I have never used the out-of-db raster option so I am not sure if I understood the storage mechanism correclty. Coincidently, I recently had a similar problem, and I have been testing a version of a raster field which is a subclass of the regular Django FileField. In this approach, the storage can be any Django compatible storage (including remote object storages such as S3). The downside is that PostGIS will no longer recognize the data as rasters, so lookups and PostGIS internal functions will not be available. This is still quite experimental (it uses a patched Django with an extension of the GDALRaster) and is slightly hacky, but does convert the remote rasters into GDALRaster instances when opened. https://github.com/geodesign/django-raster/compare/raster_file_field An ideal solution would be a mixture of the two, but I am not sure if PostGIS can handle remote storages. On Tuesday, December 20, 2016 at 4:33:21 PM UTC, Tim Graham wrote: > > Daniel Wiesmann did all the work for RasterField. I'm not sure if he > follows this list but you can find his email address in the Django commit > longs and mail him to ask for his input. > > https://github.com/yellowcap > > On Tuesday, December 20, 2016 at 11:27:25 AM UTC-5, Piero Toffanin wrote: >> >> It's stored on the file system. This is to improve performance when >> storing large geospatial datasets. >> >> This would only work on PostGIS. >> >> On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam Johnson wrote: >>> >>> I can't say I'm that familiar with GeoDjango, but that does sound like a >>> useful feature. Where does the data get stored if not in the DB? And does >>> this feature exist on any of the other database backends that GeoDjango >>> supports? >>> >>> On 14 December 2016 at 18:40, Piero Toffanin wrote: >>> >>>> Hello, >>>> >>>> Not sure this is the right place to post this, if not, could somebody >>>> point me to the right place? >>>> >>>> I recently had the need to use GeoDjango to define a model that uses a >>>> RasterField to store a GeoTIFF raster. >>>> https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield >>>> >>>> Upon testing, one of the users of the application found that loading a >>>> large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the >>>> report are available here: >>>> https://github.com/OpenDroneMap/WebODM/issues/55 >>>> >>>> I went around the problem by using a relatively new feature of PostGIS >>>> that allows raster files to be stored off db. I noticed that RasterField >>>> does not support such feature, so I wrote the code to enable support for >>>> it >>>> via a new OffdbRasterField ( >>>> https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). >>>> The from_pgraster and to_pgraster functions are modified versions of the >>>> same functions found here: >>>> https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py >>>> >>>> Just wanted to see if there was an interest in adding off db raster >>>> support into GeoDjango core. Perhaps by modifying RasterField to have an >>>> additional parameter "offdb=True|False" and
Re: GeoDjango OffdbRasterField
I am not sure what the policy is for components with such strong restrictions. So I guess this is a question back to the group: What are requirements for components within Django with regard to system architecture? Are components required to work on distributed systems? If there is no general answer to this question, please advise on this specific case. On 29-12-2016 20:07, Piero Toffanin wrote: > Your analysis is spot on, PostGIS references the path to off-db rasters > using an absolute path, which needs to be resolved on the PostGIS server. > > I guess a developer could map a folder (via samba for example) on the > PostGIS server to match that of the Django server so that both servers > share the same path structure. There would probably be some problems > with this approach as well, especially when mixing different operating > systems, or synchronization issues depending on the type of network > mapping used. > > If this addition is too problematic to be added in core, I could just > isolate the code and release it as a plugin, with some documentation > about the limits of the approach. > > On Thursday, December 29, 2016 at 2:35:59 PM UTC-5, Daniel Wiesmann wrote: > > This is an interesting idea and approach, thanks Piero for the > suggestion and the proposed solution. > > I agree with Piero and Adam, it would be a great addition to the > raster field. Especially for large volumes of data (not only for > individual large files, but also for many small files). Raster data > can eat up a lot of storage quickly, and delegating that to an > off-db storage is useful. > > I have been looking at the out-of-db feature for PostGIS rasters, > but I am not sure if it can be used in the Django context. The > problem I see is the storage location. > > If I understand it right, then the raster files need to be on the > same system as the PostGIS instance, i.e. they need to be available > on the filesystem of the postgis server. Like that, PostGIS can > access the files from within its raster operations (such as > intersections, getting pixel values etc), and it will work as if the > data was stored in the db directly. > > In that case however, the out-of-db field would only work if Django > is running on the same server as the PostGIS instance. So I think it > would fail for more distributed systems, where you have multiple > application instances, and a remote PostGIS, which might have > replicas etc. > > In your experience Piero, is that correct? How are the out-of-db > raster files stored by PostGIS? > > I have never used the out-of-db raster option so I am not sure if I > understood the storage mechanism correclty. > > Coincidently, I recently had a similar problem, and I have been > testing a version of a raster field which is a subclass of the > regular Django FileField. In this approach, the storage can be any > Django compatible storage (including remote object storages such as > S3). The downside is that PostGIS will no longer recognize the data > as rasters, so lookups and PostGIS internal functions will not be > available. > > > This is still quite experimental (it uses a patched Django with an > extension of the GDALRaster) and is slightly hacky, but does convert > the remote rasters into GDALRaster instances when opened. > > > https://github.com/geodesign/django-raster/compare/raster_file_field > <https://github.com/geodesign/django-raster/compare/raster_file_field> > > > An ideal solution would be a mixture of the two, but I am not sure > if PostGIS can handle remote storages. > > > On Tuesday, December 20, 2016 at 4:33:21 PM UTC, Tim Graham wrote: > > Daniel Wiesmann did all the work for RasterField. I'm not sure > if he follows this list but you can find his email address in > the Django commit longs and mail him to ask for his input. > > https://github.com/yellowcap > > On Tuesday, December 20, 2016 at 11:27:25 AM UTC-5, Piero > Toffanin wrote: > > It's stored on the file system. This is to improve > performance when storing large geospatial datasets. > > This would only work on PostGIS. > > On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam > Johnson wrote: > > I can't say I'm that familiar with GeoDjango, but that > does sound like a useful feature. Where does the data > get stored if not in the DB? And does this feature exist > on any
