This is mixing up two things. The article discusses ways of doing this at the makefile level, i.e. without any mods needed to make per se. There’s also a separate longstanding discussion with multiple proposals for how to modify make to add a native checksum capability.
David On Tue, Oct 20, 2015 at 1:03 PM, Egmont Koblinger <[email protected]> wrote: > This sounds like an awesome idea! > > The article being 9 years old makes me wonder though... is it something > that'll be done in the foreseeable future? > > thx, > e. > > On Tue, Oct 20, 2015 at 9:57 PM, David Boyce <[email protected]> > wrote: >> >> My view is that this is really a variant of the longstanding >> discussion about getting make to not rely on timestamps at all and use >> MD5 or similar digital signatures instead. There are a number of >> reasons why timestamps can be insufficient beside this race condition, >> so I think it might be better to put effort into getting that working >> than doing more work on the timestamp implementation. >> >> Note that there are published mechanisms for using checksums with >> existing make. For instance >> >> http://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes. >> >> David Boyce >> >> On Tue, Oct 20, 2015 at 11:36 AM, Egmont Koblinger <[email protected]> >> wrote: >> > Hi, >> > >> > Let me tell 2 real-life examples: >> > >> > 1. The one that made me find this issue: >> > >> > My hobby project takes 45 seconds to compile, out of which the biggest >> > .cc >> > file that I modify most of the time is 15 seconds. In order to >> > understand >> > what's going on (it was not written by me, I'm just fixing bugs and >> > adding >> > new features) I often insert statements like printf("A\n"); so that I'll >> > see >> > the code is there. I copy-paste it to multiple places, replacing the >> > letter >> > with B, C etc. >> > >> > Once I forgot to replace the letter. Saved, launched make, and >> > immediately >> > realized this mistake. So went back, fixed the file, saved, and >> > re-launched >> > make (probably just pressed the Up arrow and Enter, not even caring >> > whether >> > the previous make had already finished or not). I got hit by this race, >> > and >> > seeing only A's in the program's output mislead me into incorrectly >> > thinking >> > that B is not reached in the code, wasting a lot of time. >> > >> > I prefer uninterrupted runs of development with full focus for maybe an >> > hour >> > or two at a time, re-compiling the app every minute or two in certain >> > phases >> > of development. Standing up and grabbing a coffee is not a viable >> > option. >> > (And yes, I do know about gdb, I just prefer printf-debugging whenever I >> > can.) >> > >> > 2. Another example that also happens to me relatively frequently: >> > >> > I download a mid-size piece of free software (let's say it compiles in >> > 10 >> > minutes) and apply a couple of patches to it. Let's assume applying the >> > patches takes 5 minutes (I have to figure out the order, whether they're >> > -p0 >> > or -p1, some of them slightly conflict and I have to manually merge). >> > If I >> > perform these two steps sequentially, it's a total of 15 minutes. So, >> > instead, what I do is: Start a make (or "make -k" preferably), and >> > immediately afterwards start applying the patches. This way we use the >> > CPU >> > time and my time in parallel. Finally run "make" once again to catch >> > up. >> > This can reduce the total time in this example from 15 minutes to hardly >> > more than 10 minutes. Alas, chances that I'll be bitten by this race >> > condition is not negligible. >> > >> > --- >> > >> >> I don't think make should be worried about this potential race; it's an >> >> obvious case of user error with an easy discipline for the user to >> >> apply >> >> to avoid any problems. >> > >> > Is it really obvious? Is it documented somewhere? Do you really think >> > pretty much all users of make think about it? I, for one, first used >> > make >> > (not just typed the command, but understood its basics, wrote a tiny >> > Makefile for a tiny project) about 18-20 years ago, yet this race >> > condition >> > never occurred to me until recently. >> > >> > Many years ago I was wondering what "make" would do if the source and >> > destination files had the exact timestamp. I had better things to do >> > than >> > actually try it out or look it up, I just assumed that developers had >> > put >> > proper thought to it and found the solution that Just Works™. Nowadays >> > with >> > nanosecond timestamps I just don't care anymore. >> > >> > Had I thought about it, I probably would have just guessed that I'm not >> > the >> > first one with this race condition problem, would have assumed that >> > there's >> > already a clever solution for this in place which, again, Just Works™. >> > Seeing that there isn't any, I'm kindly asking you to think about it and >> > come up with one. >> > >> >> It's a rough-and-ready tool for developers, with >> >> no pretenses of being intended for use by those who don't think about >> >> how their tools work. >> > >> > Do you really claim I should deeply think about all corner cases of the >> > tools before using them? Given that "make" is just a tiny little >> > fraction >> > of all the tools I'm using, it's practically impossible. In fact, I >> > think >> > it's quite the opposite! The creators of such tools should do their >> > best to >> > make sure that the tools work reliably as expected. >> > >> > I didn't think about the internals of make -- why should have I? I knew >> > the >> > goal it was serving: to create an up-to-date build in as little overall >> > time >> > as possible. With my particular development pattern it fails to achieve >> > this goal. I don't think it's me who should've thought of this, nor >> > that >> > it's me who should change my workflow (touch the file after make >> > completes, >> > or not save while make is running, or start drinking coffee, or accept >> > that >> > I can not parallelize my time with the CPU's time, effectively taking >> > away >> > long minutes of my life many times for no good reason). >> > >> > In my firm opinion, it should be make's job to do whatever it can to >> > produce >> > an up-to-date build under any circumstance. Hey, that's what make is >> > all >> > about! >> > >> >> Of course, if a file-system had a separate "up to date at" file-stamp, >> >> that make can set (to the newest dependency's time-stamp) and later >> >> read, or if make were to store this same datum in a separate database, >> >> it would solve your problem. It would also make it possible to have >> >> make rules that tentatively build the new file, compare to existing, >> >> replace if (meaningfully) different but only revise "up to date at" >> >> otherwise. >> > >> > Such a tiny "database" could indeed solve this problem without race >> > condition. I would be more than happy to see this implemented! It >> > wouldn't >> > have to be a "database", just a simple ephemeral text file that's >> > removed on >> > clean exit of make, listing all the currently running rules that'd have >> > to >> > be rerun if this file exists on startup (meaning that the previous run >> > was >> > interrupted or crashed). >> > >> >> As you note, every command that might be used in a make rule would need >> >> to do this "right", which is a prohibitive requirement. >> >> >> >> Requiring every command that could be used as a make rule to abide by >> >> this is, again, prohibitive. [...] >> > >> > Some of these ideas I've thrown in might not work in practice, I agree. >> > >> > Please note that I had a simple propsal: re-check the timestamps of >> > input >> > files when a rule completes and re-run (moving away or dating back the >> > previous output file first) if any of them changed. This would not >> > elimite >> > the race condition completely, but would make the window magnitudes >> > smaller >> > (e.g. fraction of a millisecond instead of 15 seconds), and as such, >> > would >> > already be a great improvement! >> > >> > >> > cheers, >> > egmont >> > >> > >> > _______________________________________________ >> > Bug-make mailing list >> > [email protected] >> > https://lists.gnu.org/mailman/listinfo/bug-make >> > > > _______________________________________________ Bug-make mailing list [email protected] https://lists.gnu.org/mailman/listinfo/bug-make
