[openstreetmap/openstreetmap-website] [notes API] Prevent duplication of the same note with equal coordinates and text (Issue #6100)
biodranik created an issue (openstreetmap/openstreetmap-website#6100) ### URL https://osm.org/note/4801754 ### How to reproduce the issue? In some cases, e.g. when a client didn't receive a success response from the notes API (bad network, app killed by the system, etc.), it may try to recreate the same note again and again. That leads to unnecessary duplication on the server and an unnecessary burden for the community to clean up such duplicate notes. Example: https://api.openstreetmap.org/api/0.6/notes?closed=-1&bbox=116.53912,40.41724,116.53914,40.41726 Is there a reason why the OSM notes API implementation doesn't check if exactly the same note (text) already exists at exactly the same location? The proposal is to add a server-side check in the [Create Notes API call](https://wiki.openstreetmap.org/wiki/API_v0.6#Create_a_new_note:_POST_/api/0.6/notes), that compares the coordinates and text of a new note with already existing one. And if they are equal, the server doesn't create a new note, but: Option 1: - Returns HTTP 200 as if the note was created, with its `id`. Option 2: - Return HTTP 409 to indicate that the duplicate note wasn't created. Context: Organic Maps tries to upload collected map edits in the background. Looks like the system often kills background service/communication right when HTTP requests are going on, which [leads to duplicate OSM notes](https://github.com/organicmaps/organicmaps/issues/2071). ### Screenshot(s) or anything else? _No response_ -- Reply to this email directly or view it on GitHub: https://github.com/openstreetmap/openstreetmap-website/issues/6100 You are receiving this because you are subscribed to this thread. Message ID: ___ rails-dev mailing list rails-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/rails-dev
Re: [openstreetmap/openstreetmap-website] [notes API] Prevent duplication of the same note with equal coordinates and text (Issue #6100)
biodranik left a comment (openstreetmap/openstreetmap-website#6100) Thanks for the reference to a similar #2201 issue. I read comments there, and do not understand why this (and that) issue was closed: 1. #2201 was closed just because nobody commented there. The problem of duplication in the database has not been solved yet, right? Why close both issues? Before creating this one, I was looking for something related and didn't find anything. 2. How many duplicated data (excluding notes) are now in the OSM database? Is it still an issue? With notes, it is an annoying issue because they are resolved manually by people. Here are some stats https://dupnotes.webmapping.cyou/ with a [subset of Organic Maps duplicated notes](https://dupnotes.webmapping.cyou/organic-maps/om-dupe-notes.json). So it is still an issue. 3. How do you propose to solve it on a client without changes on the server? Query the data first before creating a note? That will double the HTTP server load. Clients are losing network connection frequently, as was mentioned already many times. This is the reality. 4. #2201 discusses different _complex_ approaches to solve the problem on a server or server+client side, and I agree with @tomhughes that proposed solutions there are _too complex_ to implement. 5. I did not check the current OSM notes DB schema and don't know the current typical server load (if it is a bottleneck already). @tomhughes is it expensive to compare coordinates+text from a note create request with content in the DB before actually creating a record, without storing/modifying the database with any hashes? Or use coords+initial note text as a unique index in the DB? -- Reply to this email directly or view it on GitHub: https://github.com/openstreetmap/openstreetmap-website/issues/6100#issuecomment-2972847642 You are receiving this because you are subscribed to this thread. Message ID: ___ rails-dev mailing list rails-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/rails-dev
Re: [openstreetmap/openstreetmap-website] [notes API] Prevent duplication of the same note with equal coordinates and text (Issue #6100)
biodranik left a comment (openstreetmap/openstreetmap-website#6100) 1. It would be great to reopen the issue and continue the discussion, to highlight it. 2. > One thing I find interesting is that StreetComplete, for some reason, manages > to create almost 120 times fewer duplicate notes compared to Organic Maps StreetComplete uploads edits while users are in the app. OM and MM are uploading edits when the app has moved to the background for a number of reasons. It was a good working solution when we introduced it in 2015, but modern OSes are actively killing/limiting background apps, e.g. https://dontkillmyapp.com/ 3. We will continue working on fixing/workarounding the client side in any case. 4. Moving to an idempotent OSM API (at least in some cases) would benefit everyone in any case, and may also speed up the database, right? -- Reply to this email directly or view it on GitHub: https://github.com/openstreetmap/openstreetmap-website/issues/6100#issuecomment-2973945666 You are receiving this because you are subscribed to this thread. Message ID: ___ rails-dev mailing list rails-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/rails-dev
