I've tried setting Stack's package build parallelism to j20 or j30 instead of the j10 it infers fro my core count but it just leads to more sleepy threads.
On Fri, Jul 8, 2016 at 2:34 PM, Michael Sloan <[email protected]> wrote: > Oh, and it's also worth mentioning that stack also takes "-j" to specify > building more packages concurrently. It defaults to your processor count. > It can be beneficial to use a higher value. > > To pass the -j to the cabal builds, use > > --ghc-options "-j2" > > On Fri, Jul 8, 2016 at 12:31 PM, Michael Sloan <[email protected]> wrote: >> >> On Fri, Jul 8, 2016 at 11:53 AM, Erik Rantapaa <[email protected]> >> wrote: >>> >>> Hi all, >>> >>> I've been building a large number of packages with a single `stack >>> install` command, e.g. something like: >>> >>> stack install --resolver ... --keep-going $(cat all-packages) >>> >>> On a 8-core / 16 GB box I've noticed that the worker threads are not >>> always kept busy. The overall %-idle time is at least 20%. >> >> >>> >>> >>> So I'm wondering... >>> >>> - Is there anything I can do on the command line to improve CPU >>> utilization? >> >> >> >> You can definitely increase CPU utilization by specifying "-jN" options >> for GHC (e.g. "-j2"). Unfortunately, in my experience it usually doesn't >> help as much as I'd hope with overall build time. One issue is that it's >> possible for it to interact poorly with package-level parallelism. For >> example, if stack is building 5 packages with "-j5", suddenly there's >> potentially 25 processes each trying to use 100% of a core. >> >>> >>> >>> - Where can I find the code that does the job scheduling? I would like to >>> modify how package builds are prioritized to improve throughput. >> >> >> Scheduling is done here: >> https://github.com/commercialhaskell/stack/blob/master/src/Control/Concurrent/Execute.hs >> >> Converting the build tasks to actions for scheduling is here: >> https://github.com/commercialhaskell/stack/blob/4304711d91f41a137931502ac4b1d00b3a4bc842/src/Stack/Build/Execute.hs#L573 >> >> >>> Thanks, >>> Erik >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "haskell-stack" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/haskell-stack/329a3117-b2ff-411c-954d-156001b622d6%40googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "haskell-stack" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/haskell-stack/CAEYHaY6jQkuKUpa5E0pqNGgLOv47%2Bpqtb49wPat18%2Br1rBE9Jw%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- Chris Allen Currently working on http://haskellbook.com -- You received this message because you are subscribed to the Google Groups "haskell-stack" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CADnndOr-e8tL85zbjy-5wesOH1g%3DxDezJG_aP%3Dzv0i_M%2BD1iOw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
