On Mon, Feb 6, 2023 at 2:26 AM Christian MAUDERER <christian.maude...@embedded-brains.de> wrote: > > On 2023-02-06 10:02, Christian MAUDERER wrote: > > On 2023-02-05 11:38, Christian Mauderer wrote: > >> > >> > >> Am 04.02.23 um 01:25 schrieb Gedare Bloom: > >>> On Fri, Feb 3, 2023 at 3:29 PM Christian Mauderer <o...@c-mauderer.de> > >>> wrote: > >>>> > >>>> > >>>> > >>>> Am 3. Februar 2023 22:52:48 MEZ schrieb Gedare Bloom > >>>> <ged...@rtems.org>: > >>>>> On Fri, Feb 3, 2023 at 2:39 PM Christian Mauderer > >>>>> <o...@c-mauderer.de> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> Am 3. Februar 2023 22:12:06 MEZ schrieb Gedare Bloom > >>>>>> <ged...@rtems.org>: > >>>>>>> On Fri, Feb 3, 2023 at 12:52 PM <o...@c-mauderer.de> wrote: > >>>>>>>> > >>>>>>>> Hello Gedare, > >>>>>>>> > >>>>>>>> Am 03.02.23 um 19:51 schrieb Gedare Bloom: > >>>>>>>>> On Thu, Feb 2, 2023 at 11:24 PM Christian MAUDERER > >>>>>>>>> <christian.maude...@embedded-brains.de> wrote: > >>>>>>>>>> > >>>>>>>>>> Hello Karel, > >>>>>>>>>> > >>>>>>>>>> On 2023-02-02 12:43, Karel Gardas wrote: > >>>>>>>>>>> > >>>>>>>>>>> Guys, > >>>>>>>>>>> > >>>>>>>>>>> recently I needed to work with RTEMS/NFS. As this is provided > >>>>>>>>>>> by libbsd > >>>>>>>>>>> I took this and following two sentences below from master branch > >>>>>>>>>>> description provided in README I took as granted that master > >>>>>>>>>>> does have > >>>>>>>>>>> all the features which are currently available and provided > >>>>>>>>>>> by the project: > >>>>>>>>>>> > >>>>>>>>>>> "This branch must be used for libbsd development. Back ports > >>>>>>>>>>> to the > >>>>>>>>>>> 6-freebsd-12 are allowed." > >>>>>>>>>>> > >>>>>>>>>>> I was surprised to be proven wrong then by Fabrizio here: > >>>>>>>>>>> https://devel.rtems.org/ticket/4723 > >>>>>>>>>>> > >>>>>>>>>>> and by later investigation which shows that 6-freebsd-12 branch > >>>>>>>>>>> accumulated NFS work by Chris done in 2021 which is not > >>>>>>>>>>> presented on > >>>>>>>>>>> master. I've investigated just NFS as this was my focus here. > >>>>>>>>>>> > >>>>>>>>>>> So if 6-freebsd-12 became development branch of some sort, > >>>>>>>>>>> then it would > >>>>>>>>>>> be great to have that clarified in the project README file to > >>>>>>>>>>> prevent > >>>>>>>>>>> users confusion? Or if the policy is still the same, then > >>>>>>>>>>> perhaps some > >>>>>>>>>>> branch sync is needed here? > >>>>>>>>>> > >>>>>>>>>> That currently is an open issue. Basically there is a pending > >>>>>>>>>> patch set > >>>>>>>>>> that should fix that since several months. But there is a > >>>>>>>>>> disagreement > >>>>>>>>>> about some of the changes in that patch set (and about the > >>>>>>>>>> patches > >>>>>>>>>> checked in to 6-freebsd-12). Therefore, it still hasn't been > >>>>>>>>>> merged. > >>>>>>>>>> > >>>>>>>>>> If you want to know some more about the problematic points, I > >>>>>>>>>> recommend > >>>>>>>>>> reading this (long) thread: > >>>>>>>>>> > >>>>>>>>>> https://lists.rtems.org/pipermail/devel/2023-January/074164.html > >>>>>>>>>> > >>>>>>>>>> The statement that development has to happen on the master > >>>>>>>>>> branch is > >>>>>>>>>> still true. The master is intended to track the FreeBSD upstream > >>>>>>>>>> development. Only changes on that branch are guaranteed to > >>>>>>>>>> live through > >>>>>>>>>> an upgrade to a newer base version of FreeBSD. It's very > >>>>>>>>>> unfortunate, > >>>>>>>>>> that there are some patches on the 6-freebsd-12 branch only. > >>>>>>>>>> On the long > >>>>>>>>>> term, that issue has to be resolved. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> I have been investigating this problem in the background, and I > >>>>>>>>> have > >>>>>>>>> some findings and some questions. First, I have found that > >>>>>>>>> there is a > >>>>>>>>> most-common ancestor between master and 6-freebsd-12 at commit > >>>>>>>>> https://git.rtems.org/rtems-libbsd/commit/?h=6-freebsd-12&id=2ce13cf6dc73855f28bc7edbbc64dc4b482a4976 > >>>>>>>>> This is at least promising that the discrepancy between the > >>>>>>>>> branches > >>>>>>>>> can be resolved. > >>>>>>>>> > >>>>>>>>> The proposed pending patch set to "fix" the NFS issue does not > >>>>>>>>> fix the > >>>>>>>>> underlying problem. Instead, it introduces further divergence > >>>>>>>>> between > >>>>>>>>> the branches. I would instead suggest that we should resolve to > >>>>>>>>> fix > >>>>>>>>> the underlying problem. I can see two paths forward. > >>>>>>>>> > >>>>>>>>> 1. Abandon 6-freebsd-12 after releasing 6. This is probably not > >>>>>>>>> ideal > >>>>>>>>> since what I understand is some users have projects based on > >>>>>>>>> 6-freebsd-12 and would like an upgrade path. (I guess there is > >>>>>>>>> also > >>>>>>>>> the option to abandon master, which also makes little sense.) > >>>>>>>> > >>>>>>>> A variant for this would be to introduce a 6-freebsd-13 that is > >>>>>>>> based on > >>>>>>>> the master branch as soon as we have one. That would allow a longer > >>>>>>>> maintenance because FreeBSD 12 reaches it's EoL December 2023. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> 2. Pull commits from 6-freebsd-12 into master to make sure > >>>>>>>>> master is > >>>>>>>>> the development head. in the future, reject patches that only go > >>>>>>>>> toward release branches. This has its own problems too. It can > >>>>>>>>> realistically only be done in three ways: > >>>>>>>> > >>>>>>>> Please note that Sebastian mentioned that the file descriptors > >>>>>>>> broke the > >>>>>>>> NTP support (at least I think it was NTP; possible that it was > >>>>>>>> another > >>>>>>>> submodule). So picking the current version of the patches into the > >>>>>>>> master without adding fixes makes the master unusable for some > >>>>>>>> cases. > >>>>>>>> > >>>>>>> Fixing the problems after making the branches the same will be > >>>>>>> better > >>>>>>> for the long-term, if we can find a path to do it. > >>>>>>> > >>>>>>>>> > >>>>>>>>> 2a: Rebase master and cherry-pick commits from 6-freebsd-12 and > >>>>>>>>> master > >>>>>>>>> back into master. This rewrites the history of master, and > >>>>>>>>> unfortunately will cause the head of 5-freebsd-12 and the tags for > >>>>>>>>> rtems-5 to no longer exist on the master branch. They will > >>>>>>>>> still exist > >>>>>>>>> in the '5' branch. The advantage is in the end there will be a > >>>>>>>>> linear > >>>>>>>>> history of development on master that reflects the timeline of > >>>>>>>>> actual > >>>>>>>>> development that spanned both branches. Theoretically, this should > >>>>>>>>> make it easier to git-bisect. > >>>>>>>>> > >>>>>>>>> 2b: Cherry-pick commits from 6-freebsd-12 to master and fix > >>>>>>>>> conflicts. > >>>>>>>>> This puts all the missing commits from 6-freebsd-12 on to the > >>>>>>>>> current > >>>>>>>>> head of master. I don't know how messy this would be. It ends up > >>>>>>>>> making the history of master convoluted to understand, with > >>>>>>>>> fairly old > >>>>>>>>> commits from 2018 being placed on top of newer commits from 2020s. > >>>>>>>>> > >>>>>>>>> 2c: Merge 6-freebsd-12 into master and fixup conflicts in the > >>>>>>>>> merge > >>>>>>>>> commit. This is pretty similar to 2a but ends up with a non-linear > >>>>>>>>> history and a merge commit. It may be a fairly complex merge > >>>>>>>>> commit. > >>>>>>>> > >>>>>>>> For all of the 2x solutions: The commits from 6-freebsd-12 can't > >>>>>>>> just be > >>>>>>>> cherry-picked. You have to re-import the NFS files from the FreeBSD > >>>>>>>> master version that is used as base for the current libbsd master. > >>>>>>>> Otherwise we mix different FreeBSD source versions. We had that > >>>>>>>> some > >>>>>>>> time back in libbsd and Sebastian needed a lot of time cleaning > >>>>>>>> that up. > >>>>>>>> > >>>>>>> Understood. > >>>>>>> > >>>>>>>>> > >>>>>>>>> To get a sense of the difference between the two branches, I > >>>>>>>>> have done > >>>>>>>>> the following command: > >>>>>>>>> $ git log --pretty=oneline master...6-freebsd-12 > ../log.txt > >>>>>>>>> This uses the ... (three-dot) Symmetric Difference Notation. The > >>>>>>>>> result of that is a 750 line file, so 750 commits are different > >>>>>>>>> between the two branches. Some of those commits are actually > >>>>>>>>> the same > >>>>>>>>> content, but they have different parents so different hashes. In a > >>>>>>>>> rebase or merge situation, those commits should end up the > >>>>>>>>> same. There > >>>>>>>>> may be other git-fu to find just the patches that are unique in > >>>>>>>>> the > >>>>>>>>> two branches. > >>>>>>>> > >>>>>>>> 750 commits are a bit too much. The order of patches on master and > >>>>>>>> 6-freebsd-12 isn't always the same. I wrote a small python > >>>>>>>> script to > >>>>>>>> find the differences somewhere in 2021 when I needed the > >>>>>>>> differences in > >>>>>>>> a discussion with (I think) Joel. It compares based on author and > >>>>>>>> subject which gives a quite good estimate for libbsd. You can > >>>>>>>> find the > >>>>>>>> script here: > >>>>>>>> > >>>>>>>> https://gist.github.com/a82e21eb250cb96c3a36f107b92dab09 > >>>>>>>> > >>>>>>>> That shows 138 different commits. 86 are only on 6-freebsd-12. > >>>>>>>> Complete > >>>>>>>> output is here: > >>>>>>>> > >>>>>>>> https://gist.github.com/93a2b19a5bf4cd8a6263ae29aa359ac2 > >>>>>>>> > >>>>>>>> From these you can ignore the "Update to FreeBSD..." commits > >>>>>>>> and some > >>>>>>>> of the cleanup patches. I assume that quite some of them can be > >>>>>>>> cherry-picked with only small or no changes. For example patches > >>>>>>>> that > >>>>>>>> only add or update drivers in rtemsbsd should work without > >>>>>>>> problems. So > >>>>>>>> I think a more realistic number of problematic patches should be > >>>>>>>> 30 to 50. > >>>>>>>> > >>>>>>> Oh, thanks, this is nice. It may be a reasonable place to start by > >>>>>>> looking at what is "Only on master" and to consider back-porting to > >>>>>>> 6-freebsd-12. That may reduce the delta somewhat in a way that makes > >>>>>>> sense. > >>>>>> > >>>>>> We really should do the other direction. Not all features oft > >>>>>> master have to be on the stable branch. But all features of the > >>>>>> stable one should be on master. > >>>>>> > >>>>> I agree, but since there are many fewer changes only on master, if we > >>>>> can merge those into 6-freebsd-12, then it becomes simpler to just > >>>>> call 6-freebsd-12 as the new master. We can replace the current master > >>>>> with this branch, and then update freebsd versions. I think this can > >>>>> be a lower burden process to follow, except it requires acceptance of > >>>>> the commits that are on 6-freebsd-12, which is a different matter. > >>>> > >>>> You can't just make 6-freebsd-12 a new master. It tracks the FreeBSD > >>>> 12 branch. > >>>> > >>>> Every block of updates to a newer FreeBSD base revision was several > >>>> days of work for Sebastian. There are a few on master. > >>>> > >>>> An upgrade from a branch tracking FreeBSD 12 to one tracking FreeBSD > >>>> 13 is most likely more something in the several week range of work. > >>>> > >>>>> > >>>>>> master tracks the FreeBSD master. As soon as it reaches the point > >>>>>> where FreeBSD branched off version 13, we can create a > >>>>>> 6-freebsd-13 or 7-freebsd-13 from master (depending on which RTEMS > >>>>>> version we have then). All features not on master would be lost > >>>>>> when we do that. > >>>>>> > >>>>>> The current 6-freebsd-12 on the other hand should be considered > >>>>>> stable. We won't upgrade it to a 7-freensd-12 or similar. It will > >>>>>> be a maintenance branch as soon as we reach the release. We can > >>>>>> backport features that have been tested on master if there are > >>>>>> users for that. But we don't have to. Just like we don't backport > >>>>>> every new feature from RTEMS 6 to 5. > >>>>>> > >>>>> I view this as equivalent to the proposals to revert commits that are > >>>>> on 6-freebsd-12. Doing this means that we are discarding quite a lot > >>>>> of software engineering effort that was done on 6-freebsd-12. I would > >>>>> rather that we determine a way that can retain that effort. At this > >>>>> point, there is probably little way to make everyone happy. > >>>> > >>>> I think you misunderstood my intention. We definitively should port > >>>> the patches from 6-freebsd-12 to master so that they are on a future > >>>> libbsd branch. Like said above: I just don't think it's realistic to > >>>> base a new developed branch on the 12 one. > >>>> > >>> I see. This is good to consider. > >>> > >>>> If I remember correctly, Sebastian's branch that contains his > >>>> suggested fixes already ported Chris NFS work to master first and > >>>> then added Sebastian's patches that revert the parts where he > >>>> disagrees. > >>>> > >>> I would leave it to them to determine if the NFSv4 is fully supported. > >>> > >>>> That leaves some driver patches on the 12 branch that are most > >>>> likely a lot simpler to port to master. > >>>> > >>>> My guess would be: Porting the stuff from 6-freebsd-12 to master > >>>> should be a few days maximum to a compile clean state. Upgrading > >>>> 6-freebsd-12 to something that would follow FreeBSD master again > >>>> should be something in the range of weeks. > >>>> > >>> OK, then we should likely follow the path of porting from 6-freebsd-12 > >>> to master. I would leave the issue of what to do with FDs as a > >>> separate concern that can be addressed once we solve this underlying > >>> problem. > >>> > >>> What would you think is the best way to handle the process though? It > >>> would seem to still require some kind of cherry-picking + patching > >>> FreeBSD and passing some tests? Perhaps it is best to do it that way, > >>> on top of the current master branch, attempting to pick up patches > >>> from the shared ancestor. ("Only in 6-freebsd-12"). > >> > >> I would usually just cherry-pick commits that don't import anything > >> from FreeBSD. All commits that do import files need a bit more work: > >> For these, the files from FreeBSD master have to be re-importet. > >> > >> It's basically the same process that I use to back-port something from > >> master to 6-freebsd-12. And it's why CONTRIBUTING.md tells that you > >> should have one commit that only adds the unmodified imported files. > >> With that it's easy to re-import the files from a different version > >> and re-apply the patches. > >> > >> Best regards > >> > >> Christian > > > > I tried to cherry-pick some of the commits yesterday (hobby time on > > Sunday!). The first few patches should work quite well. There are some > > minor problems that still have to be cleaned up to make it compile clean. > > > > One of the early ones that makes problems is the path-mappings feature > > from Jan Sommer because on master something changed the same code parts. > > We need someone who understands both changes to port that. > > > > About starting with Chris kernel symbol namespace tool patch > > (59f652fe88) the changes start to get bigger and the patches are harder > > to apply. I think the kernel symbol namespace is one of the core patches > > that would be important to port, but I don't know the code surrounding > > it well enough that I can do it fast. > > > > How should we approach that work? One patch at a time? Maybe together > > with pinging the original authors to ask them whether they can port > > their patches to master or at least test them there? > > > > For the VFS and NFS parts, it depends a bit on the solution of the > > discussion. Sebastian already worked on a re-implementation of NFS on > > master (not yet entirely finished): > > > > https://github.com/sebhub/rtems-libbsd/commits/master-update > > > > Of course that is based on his solution without the file descriptors and > > without the central system calls. So it depends on the results of the > > discussion between Sebastian and Chris whether that patch can be used to > > have the NFS functionality in master. > > > > When re-reading the text I noted that mentioning that branch here can > lead into a wrong direction. To make my intention clear: I think we > should concentrate on all unrelated patches at the moment. Parallel we > should try to push the NFS/VFS discussion and help Chris and Sebastian > to agree on one solution that is acceptable for both. Depending on that > solution, it could be simpler to just use that branch and maybe port > only small parts to have a functionally equivalent solution on both > branches with a slightly different history. > Yes, thanks. This looks like a good analysis. I would definitely prefer to get master and 6-freebsd-12 into harmony. Then it is a little simpler to discuss the other problems in libbsd.
Vijay had similar kind of success (and problems) as you did with cherry-picking off of 6-freebsd-12. I think he made it a little further. I guess one possible route forward would be to begin working on this effort and share results, pushing to master when some relatively stable milestones are reached. Gedare > Best regards > > Christian > > > Note that Sebastian also mentioned, that FreeBSD changed a lot in the > > NFS/VFS in the last few months. So it's possible that both solutions > > need quite some changes after we reach that version of FreeBSD. > > > > Best regards > > > > Christian > > > >> > >>> > >>>> Best regards > >>>> > >>>> Christian > >>>> > >>>>> > >>>>> it will be much easier to debate the merits of proposed patches to > >>>>> address optimization etc., if we have a shared basis of features and > >>>>> development. The way this has been heading is going to lead us > >>>>> directly into a non-viable mode of trying to support several open > >>>>> development branches of libbsd. I can easily see someone wanting to > >>>>> carry forward 6-freebsd-12 commits to rtems7. If we don't have a way > >>>>> to do that, we will definitely end up with something like > >>>>> 7-freebsd-12. > >>>>> > >>>>> We have to deal with the fact that there are legitimate developments > >>>>> on 6-freebsd-12 that users may expect in future RTEMS releases. I > >>>>> cannot agree with having a release of 6-freebsd-12 with rtems6 that is > >>>>> followed by a release of 7-freebsd-1x that drops some features such as > >>>>> support for NFSv4. It is not pragmatic. But we also don't have the > >>>>> resources to be maintaining multiple branches of libbsd. > >>>>> > >>>>>> For example, I don't know of a user for the BBB framebuffer on > >>>>>> RTEMS 6. It's nice that it is there on master. It will become a > >>>>>> stable feature on the next major branch (one of 6 or 7-freebsd-13 > >>>>>> or 14). But there is no necessity to backport it to the mostly > >>>>>> stable 6-freebsd-12. > >>>>>> > >>>>> I have asked Vijay to look at backporting it anyway, to reduce the > >>>>> difference between the branches. I want to get us out of this backward > >>>>> way of development / change management that has evolved over time in > >>>>> libbsd. How best to get us to a state where new development only goes > >>>>> to master is for me still the main question. Once we have this fixed, > >>>>> it is my intention to thoroughly reject any development that is only > >>>>> submitted to a "stable" branch in the future. > >>>>> > >>>>> Gedare > >>>>> > >>>>>> Best regards > >>>>>> > >>>>>> Christian > >>>>>> > >>>>>>> > >>>>>>>> Best regards > >>>>>>>> > >>>>>>>> Christian > >>>>>>>> > >>>>>>>>> > >>>>>>>>> In any case, doing this in a way that ensures the commits build > >>>>>>>>> and > >>>>>>>>> tests run is challenging due to the interactions with the > >>>>>>>>> rtems.git, > >>>>>>>>> toolchain, and the submodules. > > >>>>>>>>> After the 6-freebsd-12 and master are made consistent, then it > >>>>>>>>> becomes > >>>>>>>>> possible to update freebsd and also to consider what ways can be > >>>>>>>>> chosen to fix problems in 6-freebsd-12. > >>>>>>>>> > >>>>>>>>> -Gedare > >>>>>>>>> > >>>>>>>>>> Best regards > >>>>>>>>>> > >>>>>>>>>> Christian > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I'm fine with either way, as a user I just need clear not > >>>>>>>>>>> confusing > >>>>>>>>>>> project message... > >>>>>>>>>>> > >>>>>>>>>>> Thanks! > >>>>>>>>>>> Karel > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> devel mailing list > >>>>>>>>>>> devel@rtems.org > >>>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> -------------------------------------------- > >>>>>>>>>> embedded brains GmbH > >>>>>>>>>> Herr Christian MAUDERER > >>>>>>>>>> Dornierstr. 4 > >>>>>>>>>> 82178 Puchheim > >>>>>>>>>> Germany > >>>>>>>>>> email: christian.maude...@embedded-brains.de > >>>>>>>>>> phone: +49-89-18 94 741 - 18 > >>>>>>>>>> mobile: +49-176-152 206 08 > >>>>>>>>>> > >>>>>>>>>> Registergericht: Amtsgericht München > >>>>>>>>>> Registernummer: HRB 157899 > >>>>>>>>>> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, > >>>>>>>>>> Thomas Dörfler > >>>>>>>>>> Unsere Datenschutzerklärung finden Sie hier: > >>>>>>>>>> https://embedded-brains.de/datenschutzerklaerung/ > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> devel mailing list > >>>>>>>>>> devel@rtems.org > >>>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel > >>>>>>>>> _______________________________________________ > >>>>>>>>> devel mailing list > >>>>>>>>> devel@rtems.org > >>>>>>>>> http://lists.rtems.org/mailman/listinfo/devel > >>>>>>> > >>>>> > >>> > > > > -- > -------------------------------------------- > embedded brains GmbH > Herr Christian MAUDERER > Dornierstr. 4 > 82178 Puchheim > Germany > email: christian.maude...@embedded-brains.de > phone: +49-89-18 94 741 - 18 > mobile: +49-176-152 206 08 > > Registergericht: Amtsgericht München > Registernummer: HRB 157899 > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > Unsere Datenschutzerklärung finden Sie hier: > https://embedded-brains.de/datenschutzerklaerung/ _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
