> From: Alan W. Irwin, Sent: 25 November 2018 01:48 Many thanks Alan, answers and questions below
> On 2018-11-25 00:23-0000 Kris Thielemans wrote: > > > I'm trying to add doxygen-generated comments to my Python module build > > via SWIG using https://github.com/m7thon/doxy2swig. This means I need > > to run doxygen first, then run doxy2swig, and only then run swig. > > However, I'm getting reports that parallel builds are failing (sometimes). > > > > > > > > My strategy is based on CMake advice and some blogs (e.g. at > > https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta rgets-and-files-and-custom-commands/ ) and relies on creating > > custom commands to generate the files and custom targets to be able to > > specify dependencies on those generated files. Somewhat simplified > > code is below (original is at https://github.com/UCL/STIR/pull/280) > > > > > > > > # command that runs doxygen > > > > add_custom_command( > > > > OUTPUT doxygen.stamp > > > > DEPENDS ${doxyfile} > > > > COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile} > > > > COMMAND cmake -E touch doxygen.stamp > > > > .) > > > > # corresponding target > > > > add_custom_target( doc ALL DEPENDS doxygen.stamp) > > > > > > > > # command that runs doxy2swig > > > > add_custom_command( > > > > OUTPUT STIR_DOXY2SWIG.i > > > > DEPENDS doc > > > > COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml > > STIR_DOXY2SWIG.i > > > > ) > > > > # corresponding target > > > > add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i) > > > > > > > > # now add that target to the SIWG module > > > > add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig) > > > > > > > > Any suggestions on what I'm doing wrong? > > > > > > > > Many thanks! > > > > Kris > > > > > > > > PS: I note that FindDoxygen's doxygen_add_docs merges the > > add_custom_command and target into one add_custom_target, see > > > https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb153c876a1 a/Modules/FindDoxygen.cmake#L1104 > > > > I thought that couldn't be done but I guess I was wrong. > > > > PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp > > file to prevent re-runs though, is that not needed then? > > Hi Kris: > > To answer your last set of questions first, you can execute all desired > commands with add_custom_target, but the issue is that COMMAND always > reruns. So your way of doing it above (with paired > add_custom_command/add_custom_target) is the recommended procedure > which should only re-run if the OUTPUT file is non-existent or older than a > dependent file. > Ok. That makes sense to me. thanks! > Anyhow, I think what you have outlined above is generally correct. > For example, your "DEPENDS doc" argument to add_custom_command where > doc is a custom target argument is correct according to the documentation at > <https://cmake.org/cmake/help/latest/command/add_custom_command.html > >. > However, your simplified code above has several deviations from the pattern I > always use. I know my pattern generally works, and I think at least your first > two deviations from it are important. > > 1. I always like to specify the full pathname for all files (e.g., > ${CMAKE_CURRENT_BINARY_DIR}/doxygen.stamp and > ${CMAKE_CURRENT_BINARY_DIR}/STIR_DOXY2SWIG.i. This is true not only for > OUTPUT files and the corresponding DEPENDS but also for input (which > typically start with ${CMAKE_CURRENT_SOURCE_DIR} and output files for > COMMANDs. > > 2. At the same time I make sure I run all commands in > ${CMAKE_CURRENT_BINARY_DIR} by specifying that as the working directory. > So that all unmentioned files on the command line are generated in the build > tree to keep the source tree as clean as possible. > > 3. I don't think it matters above but I always use the VERBATIM attribute for > COMMANDS. > Thanks for these suggestions. RE 2 and 3, yes, I did the same. For the sake of brevity I cut some of the relevant bits. Sorry! (full code at https://github.com/UCL/STIR/pull/280) I'm indeed not using the full pathnames. However, given the WORKING_DIRECTORY option, all files are "local" as far as CMake concerns, it seems to me this is not relevant, although I should give it try. > I have found one of my most difficult CMake tasks is to avoid build race > conditions that can occur for parallel builds. So your real code is most > important in that regard rather than a simplified example of your CMake logic. > Perhaps others here would like to comment on the most reliable way to find > such race conditions. But what I do is process "make -j<jobs> <target_name>" > output results to remove the progress percentage marks, sort those results, > and look for anything that is repeated that shouldn't be. Sounds painful :-; The person who reported the problem can currently not provide me with a log file. He's on OSX but I don't know yet what CMake/make versions he's using. So instead I did some more digging in the generated makefiles. From these, it seems all is ok (see end of email). I'm now puzzled of course. > Finally, although I have found parallel builds to be reliable on Linux, Arjen > Markus, one of the PLplot developers, has discovered that parallel builds are > currently unreliable for all Unix-like Windows platforms, e.g., classical > MinGW/MSYS, its modern replacement MinGW-w64/MSYS2, and Cygwin. So if > some of your users are reporting parallel build troubles on any of those > platforms it is likely an issue with the make command on those platforms > rather than your build system. > That is pretty amazing. Who knows, maybe "make" on OSX should be added to the list of problematic "make" implementations as well. If I can get dig out anything else on reproducing the problem or debugging, I'll let you know. Thanks again for your help! Kris PS: some checks on build files == Visual Studio 2015 generated files (with CMake 3.11.4): My swig module _stir has a reference "doxy2swig", which has a reference "doc", so that seems fine. (I'm assuming that a reference means it'll check dependencies). == Unix Makefiles generated by CMake 3.7 and 3.13.0: I first got confused by the fact that cross-directory dependencies in the makefiles in each sub-dir are non-existent, but I realised that "make" is called for each of these from the "master" makefile. Checking CMakeFiles/Makefile2 in the root of my build, everything seems fine as this contains dependencies: src/swig/CMakeFiles/doxy2swig.dir/all: src/CMakeFiles/doc.dir/all src/swig/CMakeFiles/_stir.dir/all: src/swig/CMakeFiles/doxy2swig.dir/all (I've now learned that using "make" from in one of the subdirs is a dangerous thing to do as it doesn't check dependencies properly, but that makes some sense.) -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
