This sounds like an awesome idea! The article being 9 years old makes me wonder though... is it something that'll be done in the foreseeable future?
thx, e. On Tue, Oct 20, 2015 at 9:57 PM, David Boyce <david.s.bo...@gmail.com> wrote: > My view is that this is really a variant of the longstanding > discussion about getting make to not rely on timestamps at all and use > MD5 or similar digital signatures instead. There are a number of > reasons why timestamps can be insufficient beside this race condition, > so I think it might be better to put effort into getting that working > than doing more work on the timestamp implementation. > > Note that there are published mechanisms for using checksums with > existing make. For instance > http://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes > . > > David Boyce > > On Tue, Oct 20, 2015 at 11:36 AM, Egmont Koblinger <egm...@gmail.com> > wrote: > > Hi, > > > > Let me tell 2 real-life examples: > > > > 1. The one that made me find this issue: > > > > My hobby project takes 45 seconds to compile, out of which the biggest > .cc > > file that I modify most of the time is 15 seconds. In order to > understand > > what's going on (it was not written by me, I'm just fixing bugs and > adding > > new features) I often insert statements like printf("A\n"); so that I'll > see > > the code is there. I copy-paste it to multiple places, replacing the > letter > > with B, C etc. > > > > Once I forgot to replace the letter. Saved, launched make, and > immediately > > realized this mistake. So went back, fixed the file, saved, and > re-launched > > make (probably just pressed the Up arrow and Enter, not even caring > whether > > the previous make had already finished or not). I got hit by this race, > and > > seeing only A's in the program's output mislead me into incorrectly > thinking > > that B is not reached in the code, wasting a lot of time. > > > > I prefer uninterrupted runs of development with full focus for maybe an > hour > > or two at a time, re-compiling the app every minute or two in certain > phases > > of development. Standing up and grabbing a coffee is not a viable > option. > > (And yes, I do know about gdb, I just prefer printf-debugging whenever I > > can.) > > > > 2. Another example that also happens to me relatively frequently: > > > > I download a mid-size piece of free software (let's say it compiles in 10 > > minutes) and apply a couple of patches to it. Let's assume applying the > > patches takes 5 minutes (I have to figure out the order, whether they're > -p0 > > or -p1, some of them slightly conflict and I have to manually merge). > If I > > perform these two steps sequentially, it's a total of 15 minutes. So, > > instead, what I do is: Start a make (or "make -k" preferably), and > > immediately afterwards start applying the patches. This way we use the > CPU > > time and my time in parallel. Finally run "make" once again to catch up. > > This can reduce the total time in this example from 15 minutes to hardly > > more than 10 minutes. Alas, chances that I'll be bitten by this race > > condition is not negligible. > > > > --- > > > >> I don't think make should be worried about this potential race; it's an > >> obvious case of user error with an easy discipline for the user to apply > >> to avoid any problems. > > > > Is it really obvious? Is it documented somewhere? Do you really think > > pretty much all users of make think about it? I, for one, first used > make > > (not just typed the command, but understood its basics, wrote a tiny > > Makefile for a tiny project) about 18-20 years ago, yet this race > condition > > never occurred to me until recently. > > > > Many years ago I was wondering what "make" would do if the source and > > destination files had the exact timestamp. I had better things to do > than > > actually try it out or look it up, I just assumed that developers had put > > proper thought to it and found the solution that Just Works™. Nowadays > with > > nanosecond timestamps I just don't care anymore. > > > > Had I thought about it, I probably would have just guessed that I'm not > the > > first one with this race condition problem, would have assumed that > there's > > already a clever solution for this in place which, again, Just Works™. > > Seeing that there isn't any, I'm kindly asking you to think about it and > > come up with one. > > > >> It's a rough-and-ready tool for developers, with > >> no pretenses of being intended for use by those who don't think about > >> how their tools work. > > > > Do you really claim I should deeply think about all corner cases of the > > tools before using them? Given that "make" is just a tiny little > fraction > > of all the tools I'm using, it's practically impossible. In fact, I > think > > it's quite the opposite! The creators of such tools should do their > best to > > make sure that the tools work reliably as expected. > > > > I didn't think about the internals of make -- why should have I? I knew > the > > goal it was serving: to create an up-to-date build in as little overall > time > > as possible. With my particular development pattern it fails to achieve > > this goal. I don't think it's me who should've thought of this, nor that > > it's me who should change my workflow (touch the file after make > completes, > > or not save while make is running, or start drinking coffee, or accept > that > > I can not parallelize my time with the CPU's time, effectively taking > away > > long minutes of my life many times for no good reason). > > > > In my firm opinion, it should be make's job to do whatever it can to > produce > > an up-to-date build under any circumstance. Hey, that's what make is all > > about! > > > >> Of course, if a file-system had a separate "up to date at" file-stamp, > >> that make can set (to the newest dependency's time-stamp) and later > >> read, or if make were to store this same datum in a separate database, > >> it would solve your problem. It would also make it possible to have > >> make rules that tentatively build the new file, compare to existing, > >> replace if (meaningfully) different but only revise "up to date at" > >> otherwise. > > > > Such a tiny "database" could indeed solve this problem without race > > condition. I would be more than happy to see this implemented! It > wouldn't > > have to be a "database", just a simple ephemeral text file that's > removed on > > clean exit of make, listing all the currently running rules that'd have > to > > be rerun if this file exists on startup (meaning that the previous run > was > > interrupted or crashed). > > > >> As you note, every command that might be used in a make rule would need > >> to do this "right", which is a prohibitive requirement. > >> > >> Requiring every command that could be used as a make rule to abide by > >> this is, again, prohibitive. [...] > > > > Some of these ideas I've thrown in might not work in practice, I agree. > > > > Please note that I had a simple propsal: re-check the timestamps of input > > files when a rule completes and re-run (moving away or dating back the > > previous output file first) if any of them changed. This would not > elimite > > the race condition completely, but would make the window magnitudes > smaller > > (e.g. fraction of a millisecond instead of 15 seconds), and as such, > would > > already be a great improvement! > > > > > > cheers, > > egmont > > > > > > _______________________________________________ > > Bug-make mailing list > > Bug-make@gnu.org > > https://lists.gnu.org/mailman/listinfo/bug-make > > >
_______________________________________________ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
