Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark, PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore. If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS. Barry $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc (main=) arch-main $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc (main=) arch-main $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o > On May 15, 2021, at 8:44 PM, Junchao Zhang <junchao.zh...@gmail.com> wrote: > > Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper > parses the options > > PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname > > <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> > /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` > NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx > /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper > --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler > -fPIC -O3 -gencode arch=compute_80,code=sm_80 > -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include > -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include > -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include > -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include > -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include > /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o > arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show > > --Junchao Zhang > > > On Sat, May 15, 2021 at 7:14 PM Mark Adams <mfad...@lbl.gov > <mailto:mfad...@lbl.gov>> wrote: > Thanks, > Now I get an error in make all. THis was working a few weeks ago. > Make.log was empty but here is the output. > .... > CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o > PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname > > <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> > /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` > NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx > /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper > --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler > -fPIC -O3 -gencode arch=compute_80,code=sm_80 > -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include > -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include > -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include > -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include > -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include > /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o > arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o > g++: error: arch=compute_80: No such file or directory > g++: error: code=sm_80: No such file or directory > gmake[3]: *** [gmakefile:188: > arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 > gmake[3]: *** Waiting for unfinished jobs.... > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o > CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o > FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o > CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o > CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o > gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] > Error 2 > **************************ERROR************************************* > Error during compile, check > arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log > Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to > petsc-ma...@mcs.anl.gov <mailto:petsc-ma...@mcs.anl.gov> > ******************************************************************** > gmake[1]: *** [makefile:40: all] Error 1 > make: *** [GNUmakefile:9: all] Error 2 > > On Sat, May 15, 2021 at 5:30 PM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > Bitbucket is off-line. Do you need fortran stubs at this moment? If not use > --with-fortran-interface=0 > > >> On May 15, 2021, at 2:51 PM, Mark Adams <mfad...@lbl.gov >> <mailto:mfad...@lbl.gov>> wrote: >> >> I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with >> Kokkos. >> Any suggestions? >> Thanks, >> Mark >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> >> Trying to >> download git://https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING >> >> >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> >> Trying to >> download git://https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING >> >> >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> >> Trying to >> download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz >> <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING >> >> >> >> ============================================================================================= >> >> >> >> >> >> >> >> ******************************************************************************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> ------------------------------------------------------------------------------- >> Error during download/extract/detection of SOWING: >> Unable to clone sowing >> Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> >> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": >> Cloning into >> '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... >> fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ >> <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned >> error: 503 >> Unable to download package SOWING from: >> git://https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> >> * If URL specified manually - perhaps there is a typo? >> * If your network is disconnected - please reconnect and rerun ./configure >> * Or perhaps you have a firewall blocking the download >> * You can run with --with-packages-download-dir=/adirectory and ./configure >> will instruct you what packages to download manually >> * or you can download the above URL manually, to /yourselectedlocation >> and use the configure option: >> --download-sowing=/yourselectedlocation >> Unable to clone sowing >> Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> >> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": >> Cloning into >> '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... >> fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ >> <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned >> error: 503 >> Unable to download package SOWING from: >> git://https://bitbucket.org/petsc/pkg-sowing.git >> <https://bitbucket.org/petsc/pkg-sowing.git> >> * If URL specified manually - perhaps there is a typo? >> * If your network is disconnected - please reconnect and rerun ./configure >> * Or perhaps you have a firewall blocking the download >> * You can run with --with-packages-download-dir=/adirectory and ./configure >> will instruct you what packages to download manually >> * or you can download the above URL manually, to /yourselectedlocation >> and use the configure option: >> --download-sowing=/yourselectedlocation >> file could not be opened successfully >> Downloaded package SOWING from: >> https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz >> <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a >> tarball. >> [or installed python cannot process compressed files] >> * If you are behind a firewall - please fix your proxy and rerun ./configure >> For example at LANL you may need to set the environmental variable >> http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov >> <http://proxyout.lanl.gov/> >> * You can run with --with-packages-download-dir=/adirectory and ./configure >> will instruct you what packages to download manually >> * or you can download the above URL manually, to >> /yourselectedlocation/v1.1.26-p1.tar.gz >> and use the configure option: >> --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz >> ******************************************************************************* >> <configure.log> >