Bug#1034667: ITP: pg-failover-slots -- High-availability support for PostgreSQL logical replication
Package: wnpp Severity: wishlist Owner: Marco Nenciarini X-Debbugs-Cc: debian-devel@lists.debian.org, team+postgre...@tracker.debian.org * Package name: pg-failover-slots Version : 1.0.0 Upstream Contact: EnterpriseDB * URL : https://github.com/EnterpriseDB/pg_failover_slots * License : Postgres License Programming Lang: C Description : High-availability support for PostgreSQL logical replication This extension is for anyone with Logical Replication Slots on Postgres databases that are also part of a Physical Streaming Replication architecture. Since logical replication slots are only maintained on the primary node, downstream subscribers don't receive any new changes from a newly promoted primary until the slot is created, which is unsafe because the information that includes which data a subscriber has confirmed receiving and which log data still needs to be retained for the subscriber will have been lost, resulting in an unknown gap in data changes. PG Failover Slots makes logical replication slots usable across a physical failover using the following features: - Copies any missing replication slots from the primary to the standby - Removes any slots from the standby that aren't found on the primary - Periodically synchronizes the position of slots on the standby based on the primary - Ensures that selected standbys receive data before any of the logical slot walsenders can send data to consumers PostgreSQL 11 on higher is required.
Re: DEP 17: Improve support for directory aliasing in dpkg
Hello, On Mon, 03 Apr 2023, Helmut Grohne wrote: > Please consider it to be a piece of best > intentions at reconciling feedback wherever I could. At the time of this > writing it certainly is not consensus, but consensus is what I seek > here. Without further ado, the full DEP text follows after my name > while it also is available at > https://salsa.debian.org/dep-team/deps/-/merge_requests/5 I'd like to express some disappointment that nobody replied publicly sofar. Last year's developer survey concluded that "Debian should complete the merged-/usr transition" was the most important project for Debian [1] (among those proposed in the survey). That's what we are trying to do here and it would be nice to build some sort of consensus on what it means in terms of changes for dpkg. I know that Guillem (dpkg's maintainer) is generally opposed to the approach that Debian has followed to implement merged-/usr but I have yet to read his concerns on the changes proposed here (besides the fact that they ought to not be needed because we should redo the transition in another way, but that's a ship that has sailed a long time ago...). The rough project consensus seems to be that we should modify dpkg to avoid the cases where some files can disappear upon upgrades. Most people don't really care how we modify dpkg for this, and I can't blame them, but given that dpkg's maintainer seems unwilling to work on this problem, someone else has to come up with a design, implement it and get it applied on Debian's version of dpkg. We are committed to work on the design and implementation but we want to make sure the design is sound and agreed upon by the persons who are technically knowledgeable on this issue and who have thought a lot on this issue. There aren't that many persons in that set but it is also not empty. So please read the DEP and share your feedback, even if it's just "I have read it and it sounds fine", it will definitely help. Thank you! [1] page 28-32 of https://debian.pages.debian.net/dd-surveys/dd-survey-analysis-2022.pdf -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: DEP 17: Improve support for directory aliasing in dpkg
Hello, I'd like to offer some food for thoughts on this issue. >From what I have understood, Guillem would rather avoid committing to a new public interface for this specific use-case, i.e. the fact that the DEP is suggesting "dpkg --add-alias" is problematic because that feature will be useless when we will have moved to .deb shipping files in /usr only. However the problem of file loss through aliased directories is a broader problem, it is not specific to this transition. It's quite possible that a package is shipping a symlink pointing to a directory and to have other packages installing files through that symlink (and then move those files between binary packages and between their two possible locations). Let's try to tackle that problem in a generic way without requiring any external information... it ought to be doable. You did consider it partly already: On Mon, 03 Apr 2023, Helmut Grohne wrote: > Naive solution > == > > In theory, `dpkg` could resolve this automatically. For every file it > touches, it could canonicalize the location using the actual filesystem > and check whether any other installed file has the same canonicalized > location. Unfortunately, `dpkg` cannot know which filenames can > collide, so it would check every filename in its database. For > canonicalization, it would `stat()` every component of every filename. > This easily amounts to a million or more `stat()` calls on larger > installations. Caching could reduce the impact somewhat, but since > Debian introduces aliases during maintainer scripts, it would have to > invalidate the cache after maintainer scripts have been run. The > resulting performance would be unacceptable. Here you are considering all files, but for the purpose of our issue, we can restrict ourselves to the directories known by dpkg. We really only care about directories that have been turned into symlinks (or packaged symlinks that are pointing to directories). That's a a much lower number of paths that we would have to check. You are speaking of having some sort of cache and I certainly agree that it would make sense to have such a cache. We could decide that /var/lib/dpkg/aliases is that cache, it would be the result of a scan of all directories known by dpkg (i.e. all paths known by dpkg where files are installed through that path) and it would list the target directory in case that path is a symlink. The absence of a directory in that file would mean that, according to dpkg, the directory ought to be a real directory. Thus this time-consuming operation would be done once, the first time that the updated dpkg starts and when /var/lib/dpkg/aliases does not yet exist. That cache file would be kept up-to-date by the various dpkg invocations: - when you install a new .deb containing a symlink pointing to a directory, that new "aliased path" is added to this file - when dpkg removes a symlink that is listed in the aliases file, we drop it too We don't add any new public interface to dpkg, but we also have the possibility to remove to /var/lib/dpkg/aliases to force an new scan (some sort of "dpkg --refresh-aliases" without an official name). It might still be cleaner to have that "dpkg --refresh-aliases" command so that we can invoke it for example in "dpkg-maintscript-helper symlink_to_dir/dir_to_symlink" when we are voluntarily turning a directory into a symlink (or vice-versa). In any case, now that you have a database of aliases, you can do the other modifications to detect conflicting files and avoid file losses. How does that sound? > Implement aliasing after metadata tracking > -- > > The [metadata > tracking](https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking) > feature enhances `dpkg` with knowledge about filesystem metadata for > installed files. This includes knowledge of symbolic links, which would > help with tracking aliasing. Unfortunately, progress on this is fairly > slow and we think that aliasing support is more urgent. The proposal I made above is not a real database in the sense that we don't record what was shipped by the .deb when we installed the files... it's rather the opposite, it analyzes the system to detect possible conflicts with dpkg's view of the system. It can be seen as complimentary to it. In any case, I don't see how implementing metadata tracking would help to solve the problem that we have today. dpkg would know that all .deb have /bin as a directory and not as a symlink, and it would be able to conclude that the directory has been replaced by a symlink by something external, but that's it. It should still accept that replacement and do its best to work with it. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: DEP 17: Improve support for directory aliasing in dpkg
On Fri, 21 Apr 2023 at 13:16, Raphael Hertzog wrote: > > Hello, > > On Mon, 03 Apr 2023, Helmut Grohne wrote: > > Please consider it to be a piece of best > > intentions at reconciling feedback wherever I could. At the time of this > > writing it certainly is not consensus, but consensus is what I seek > > here. Without further ado, the full DEP text follows after my name > > while it also is available at > > https://salsa.debian.org/dep-team/deps/-/merge_requests/5 > > I'd like to express some disappointment that nobody replied publicly > sofar. Last year's developer survey concluded that "Debian should complete > the merged-/usr transition" was the most important project for Debian [1] > (among those proposed in the survey). That's what we are trying to do > here and it would be nice to build some sort of consensus on what it means > in terms of changes for dpkg. > > I know that Guillem (dpkg's maintainer) is generally opposed to the > approach that Debian has followed to implement merged-/usr but I have > yet to read his concerns on the changes proposed here There was an answer from the maintainer, 2 weeks ago: https://lists.debian.org/debian-dpkg/2023/04/msg1.html Essentially, the answer was "no", so... After Bookworm ships I plan to propose a policy change to the CTTE and policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild, plus MBF for the small % that does not use dh and a piuparts test to stop migration for anything that is uploaded and doesn't comply. That should bring the matter to an end, without needing to modify dpkg. Kind regards, Luca Boccassi
Bug#1034682: ITP: tuba -- Browse the Fediverse
Package: wnpp Severity: wishlist Owner: Danial Behzadi X-Debbugs-Cc: debian-devel@lists.debian.org, dani.be...@ubuntu.com * Package name: tuba Version : 0.2.0 Upstream Contact: GeopJr * URL : https://tuba.geopjr.dev/ * License : GPL Programming Lang: Vala Description : Browse the Fediverse A fediverse app suporting multiple ActivityPub servers incl. Mastodon, GoToSocial, Akkoma, Takahē, etc.
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Fri, Apr 21, 2023 at 02:15:54PM +0200, Raphael Hertzog wrote: I'd like to express some disappointment that nobody replied publicly sofar. There were a few replies on the dpkg mailing list. Last year's developer survey concluded that "Debian should complete the merged-/usr transition" was the most important project for Debian [1] (among those proposed in the survey). That's what we are trying to do here and it would be nice to build some sort of consensus on what it means in terms of changes for dpkg. The first thing we need consensus on, IMO, is the definition of "complete". The maintainers of the usrmerge package consider the status quo an acceptable technical solution, so their definition of "complete" is to roll out the change to the remaining users. Other people view the transition as complete when the transition handling code can be removed, which, with the current implementation, is bookworm+2 because that is the first time we could ever have ld.so in /usr/lib. The handling code for bookworm can never be removed from the various bootstrapping tools, because users will rightfully expect to be able to install older versions. I believe cross debootstrap is also currently broken. The problem for dpkg is that there is already a considerable amount of cases it handles. Paths are owned by specific packages, ownership can be transferred, they can be marked as conffiles and diverted. That alone gives us a massive testsuite already, including such elaborate cases as a package providing two files that are subsequently replaced by files in two other packages so no files remain, which automatically makes the first package disappear. Extending these cases to include aliases is a monumental task, and limiting the allowed combinations does not reduce that, it only inverts the expected return code from that test case. We need defined behaviour for what happens if someone tries to register an alias for /etc (not allowed because there are conffiles inside), and we need error handling paths for cases like /lib -> /usr/lib, where creation of the /lib symlink fails and the original /lib needs to be restored before exiting with an error. There are several possible designs here, including a variant that is similar to dpkg-divert, which is called from a maintainer script, safely performs the filesystem transformation and updates dpkg's database, which is then reloaded after the maintainer script finishes. This variant has the drawback that even in future releases we can never move ld.so to /usr in the packages, because this would be safe only if we could guarantee that the alias is registered. We can approximate that by giving ownership of the alias to a package, which allows declaring a dependency, but then the maintainer scripts need to be able to handle transfer of ownership at some point or we're again creating technical debt. Another variant would be a declarative approach, which would, conveniently, work for initial installation for foreign architectures, and permit moving files to /usr eventually (one or two releases later). The drawback there is that we need a defined point when that declaration is read, and at this point the main dpkg binary (which for cross installation is the host binary) is responsible for moving the files over. This also gives strong ownership of the alias to a package, but now it is dpkg that is responsible for ownership transfer and removal of the alias when the last package providing it vanishes. That solution again spans up a massive space of possible edge cases. Either we handle the alias through the normal file handling code, which gives us ownership transfer for almost free, but introduces lots of special cases into that code, or we add separate handling, which introduces the special case where all files in a package providing an alias are gone and the package should be deinstalled -- do we deinstall when the alias is gone as well, or when all regular files are? This is the problem with doing it right: there are so many corner cases that need to be handled, even if the handling consists of an error message about an unsupported configuration and a rollback to the last valid state. The alternative would be a consensus that dpkg is simply not expected to always leave the system in a useful state if it encounters certain invalid situations, and hoping that we will also be able to point to a few million installations where that has not exploded and call it a success, but that would need to be communicated. I know that Guillem (dpkg's maintainer) is generally opposed to the approach that Debian has followed to implement merged-/usr but I have yet to read his concerns on the changes proposed here (besides the fact that they ought to not be needed because we should redo the transition in another way, but that's a ship that has sailed a long time ago...). A sensible solution needs to be able to perform the tr
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi,
On 21.04.23 15:03, Raphael Hertzog wrote:
Here you are considering all files, but for the purpose of our issue,
we can restrict ourselves to the directories known by dpkg. We really
only care about directories that have been turned into symlinks (or
packaged symlinks that are pointing to directories). That's a a much lower
number of paths that we would have to check.
Having all paths in the database is cheaper, because doubling the number
of paths multiplies the (average) cost by log_{262144} 2 only, and we do
significantly more lookups than inserts.
The other problem is that we do not know all of these paths, because the
file system has been modified externally without informing dpkg. The
closest thing we can do is scan everything that is supposed to be a
directory.
As an additional complication, dpkg silently resolves
symlink-vs-directory conflicts in favour of the directory (which happens
seldom, but third-party tools sometimes generate broken packages like
that, so it is useful to keep it that way).
Thus this time-consuming operation would be done once, the first
time that the updated dpkg starts and when /var/lib/dpkg/aliases
does not yet exist.
That is still a public interface. :/
In any case, now that you have a database of aliases, you can do the other
modifications to detect conflicting files and avoid file losses.
How does that sound?
Alas, that is the easy part. My branch already implements most of that,
including the logic to trigger a reload after a maintainer script if the
stat information changed (like for diversions).
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files...
it's rather the opposite, it analyzes the system to detect possible
conflicts with dpkg's view of the system.
That is going to be slow, and it changes dpkg's public interface to a
more complex one where our tight loop that handles unpacking files gains
additional error states.
It can be seen as complimentary to it. In any case, I don't see how
implementing metadata tracking would help to solve the problem that we
have today. dpkg would know that all .deb have /bin as a directory and
not as a symlink, and it would be able to conclude that the directory
has been replaced by a symlink by something external, but that's it.
It should still accept that replacement and do its best to work with it.
That means there are two sources of truth: packages and the file system.
We then need a (lowercase) policy how to resolve conflicts between
these, which becomes our public interface, and thus part of (uppercase)
Policy.
I'd also single out the usrmerge transition here. This package operates
in a grey area of Policy where technically a grave bug is warranted
because it manipulates files belonging to other packages without going
through one of the approved interfaces, but since we accidentally
shipped that, we need to deal with it now. That does not mean this is
acceptable, it just wasn't enforced.
To me it would also be acceptable to just hardcode "if usrmerge or
usr-is-merged is installed, take over the known aliases and silently
discard that package", then salt the earth in dak that no package of
this name can ever be shipped again until bookworm+3.
That would be significantly easier than finding a generic solution that
covers all existing use cases.
Simon
Bug#1034703: ITP: node-react-hot-loader -- Node.js module to tweak ReactJS components in real time
Package: wnpp Severity: wishlist Owner: Yadd X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: node-react-hot-loader Version : 4.13.1 Upstream Contact: https://github.com/gaearon/react-hot-loader/issues * URL : https://github.com/gaearon/react-hot-loader * License : Expat Programming Lang: JavaScript Description : Node.js module to tweak ReactJS components in real time React-Hot-Loader provides a hot reloading system for ReactJS. It's a dependency of Jupyterlab and will be maintained under JS Team umbrella.
