I am not sure what the policy is for components with such strong restrictions. So I guess this is a question back to the group:
What are requirements for components within Django with regard to system architecture? Are components required to work on distributed systems? If there is no general answer to this question, please advise on this specific case. On 29-12-2016 20:07, Piero Toffanin wrote: > Your analysis is spot on, PostGIS references the path to off-db rasters > using an absolute path, which needs to be resolved on the PostGIS server. > > I guess a developer could map a folder (via samba for example) on the > PostGIS server to match that of the Django server so that both servers > share the same path structure. There would probably be some problems > with this approach as well, especially when mixing different operating > systems, or synchronization issues depending on the type of network > mapping used. > > If this addition is too problematic to be added in core, I could just > isolate the code and release it as a plugin, with some documentation > about the limits of the approach. > > On Thursday, December 29, 2016 at 2:35:59 PM UTC-5, Daniel Wiesmann wrote: > > This is an interesting idea and approach, thanks Piero for the > suggestion and the proposed solution. > > I agree with Piero and Adam, it would be a great addition to the > raster field. Especially for large volumes of data (not only for > individual large files, but also for many small files). Raster data > can eat up a lot of storage quickly, and delegating that to an > off-db storage is useful. > > I have been looking at the out-of-db feature for PostGIS rasters, > but I am not sure if it can be used in the Django context. The > problem I see is the storage location. > > If I understand it right, then the raster files need to be on the > same system as the PostGIS instance, i.e. they need to be available > on the filesystem of the postgis server. Like that, PostGIS can > access the files from within its raster operations (such as > intersections, getting pixel values etc), and it will work as if the > data was stored in the db directly. > > In that case however, the out-of-db field would only work if Django > is running on the same server as the PostGIS instance. So I think it > would fail for more distributed systems, where you have multiple > application instances, and a remote PostGIS, which might have > replicas etc. > > In your experience Piero, is that correct? How are the out-of-db > raster files stored by PostGIS? > > I have never used the out-of-db raster option so I am not sure if I > understood the storage mechanism correclty. > > Coincidently, I recently had a similar problem, and I have been > testing a version of a raster field which is a subclass of the > regular Django FileField. In this approach, the storage can be any > Django compatible storage (including remote object storages such as > S3). The downside is that PostGIS will no longer recognize the data > as rasters, so lookups and PostGIS internal functions will not be > available. > > > This is still quite experimental (it uses a patched Django with an > extension of the GDALRaster) and is slightly hacky, but does convert > the remote rasters into GDALRaster instances when opened. > > > https://github.com/geodesign/django-raster/compare/raster_file_field > <https://github.com/geodesign/django-raster/compare/raster_file_field> > > > An ideal solution would be a mixture of the two, but I am not sure > if PostGIS can handle remote storages. > > > On Tuesday, December 20, 2016 at 4:33:21 PM UTC, Tim Graham wrote: > > Daniel Wiesmann did all the work for RasterField. I'm not sure > if he follows this list but you can find his email address in > the Django commit longs and mail him to ask for his input. > > https://github.com/yellowcap > > On Tuesday, December 20, 2016 at 11:27:25 AM UTC-5, Piero > Toffanin wrote: > > It's stored on the file system. This is to improve > performance when storing large geospatial datasets. > > This would only work on PostGIS. > > On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam > Johnson wrote: > > I can't say I'm that familiar with GeoDjango, but that > does sound like a useful feature. Where does the data > get stored if not in the DB? And does this feature exist > on any of the other database backends that GeoDjango > supports? > > On 14 December 2016 at 18:40, Piero Toffanin > <pier...@gmail.com> wrote: > > Hello, > > Not sure this is the right place to post this, if > not, could somebody point me to the right place? > > I recently had the need to use GeoDjango to define a > model that uses a RasterField to store a GeoTIFF > raster. > https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield > > <https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield> > > Upon testing, one of the users of the application > found that loading a large GeoTIFF (100Mb+) caused > PostgreSQL to fail. More details about the report > are available > here: https://github.com/OpenDroneMap/WebODM/issues/55 > <https://github.com/OpenDroneMap/WebODM/issues/55> > > I went around the problem by using a relatively new > feature of PostGIS that allows raster files to be > stored off db. I noticed that RasterField does not > support such feature, so I wrote the code to enable > support for it via a new OffdbRasterField > > (https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py > > <https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py>). > The from_pgraster and to_pgraster functions are > modified versions of the same functions found here: > > https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py > > <https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py> > > Just wanted to see if there was an interest in > adding off db raster support into GeoDjango core. > Perhaps by modifying RasterField to have an > additional parameter "offdb=True|False" and > implement the necessary logic? > > Thanks, > > -Piero > > -- > You received this message because you are subscribed > to the Google Groups "Django developers > (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving > emails from it, send an email to > django-develop...@googlegroups.com. > To post to this group, send email to > django-d...@googlegroups.com. > Visit this group at > https://groups.google.com/group/django-developers > <https://groups.google.com/group/django-developers>. > To view this discussion on the web visit > > https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com > > <https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit > https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > > > -- > Adam > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/Drb2qYuuuKo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com > <mailto:django-developers+unsubscr...@googlegroups.com>. > To post to this group, send email to django-developers@googlegroups.com > <mailto:django-developers@googlegroups.com>. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/28150c14-089d-4025-b52b-f616fb0a600e%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/28150c14-089d-4025-b52b-f616fb0a600e%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/30e98148-9502-312a-9aff-96d7e120806a%40gmail.com. For more options, visit https://groups.google.com/d/optout.
