On 13/04/21 at 11:49 -0700, Felix Lechner wrote: > Hi, > > On Tue, Apr 13, 2021 at 11:27 AM Lucas Nussbaum <lu...@debian.org> wrote: > > > > From the UDD point of view, I would very much prefer to get a full dump > > something I can import every few hours, than having to deal with a > > stream of updates or with querying a per-package API. > > Since few users ever need *all* data, would it make sense to > re-conceive UDD as a "query broker" to help people get the data they > actually need?
Well if you adopt it, feel free to reimplement it the way you want :) > > Currently the full import (that runs twice a day) takes about 10 minutes > > The power of COPY. The Lintian website currently takes 12 hours to > import a single run across the archive in 42 bulk UPSERTS via JSON > (but will eventually cease to generate data that way). No, it's just PREPARE/EXECUTE (inside a single big transaction -- I think) Lucas
