Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. 
But handles either of them separately correctly.

  Mark,

    PETSc configure now handles  the -arch and --with-kokkos-cuda-arch=  
business automatically so you do not, and generally shouldn't pass -arch sm_70 
in  --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.

    If you need to set the -arch to something earlier than the system supports 
(rare) you can now use -with-cuda-gencodearch to set that instead of using 
CUDAFLAGS.

  Barry


$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx 
/home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c 
-Xcompiler -fPIC -g  -gencode arch=compute_70,code=sm_70  
-I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets   
-I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include 
-I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include
      /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o 
arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 
-Wno-deprecated-gpu-targets -ccbin mpicxx -g 
-I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include 
-I/home/bsmith/petsc/arch-main/include 
-I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include
 -Xcompiler -fPIC -x cu 
/home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o 
arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
~/petsc (main=) arch-main

$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx 
/home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c 
-arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 
-DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3  -gencode 
arch=compute_80,code=sm_80  
-I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include 
-Wno-deprecated-gpu-targets 
nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin 
mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC 
-DLANDAU_MAX_Q=4 -O3 
-I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler 
-rdynamic,-fPIC,arch=compute_80,code=sm_80 -c
~/petsc (main=) arch-main

$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx 
/home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c 
-Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70  
-I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets   
-I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include 
-I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include
      /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o 
arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin 
mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include 
-I/home/bsmith/petsc/arch-main/include 
-I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include
 -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu 
/home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o 
arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o


> On May 15, 2021, at 8:44 PM, Junchao Zhang <junchao.zh...@gmail.com> wrote:
> 
> Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper 
> parses the options
> 
> PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname
>  
> <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname>
>  /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` 
> NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx
>  /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper 
> --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 
> -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler 
> -fPIC -O3  -gencode arch=compute_80,code=sm_80  
> -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include 
> -Wno-deprecated-gpu-targets   -I/global/homes/m/madams/petsc/include 
> -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include 
> -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include 
> -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include      
> /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o 
> arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
> 
> --Junchao Zhang
> 
> 
> On Sat, May 15, 2021 at 7:14 PM Mark Adams <mfad...@lbl.gov 
> <mailto:mfad...@lbl.gov>> wrote:
> Thanks,
> Now I get an error in make all. THis was working a few weeks ago.
> Make.log was empty but here is the output.
>  ....
>        CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
> PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname
>  
> <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname>
>  /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` 
> NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx
>  /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper 
> --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 
> -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler 
> -fPIC -O3  -gencode arch=compute_80,code=sm_80  
> -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include 
> -Wno-deprecated-gpu-targets   -I/global/homes/m/madams/petsc/include 
> -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include 
> -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include 
> -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include      
> /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o 
> arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o
> g++: error: arch=compute_80: No such file or directory
> g++: error: code=sm_80: No such file or directory
> gmake[3]: *** [gmakefile:188: 
> arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1
> gmake[3]: *** Waiting for unfinished jobs....
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o
>           CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o
>           FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o
>        CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o
>    CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o
> gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] 
> Error 2
> **************************ERROR*************************************
>   Error during compile, check 
> arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log
>   Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to 
> petsc-ma...@mcs.anl.gov <mailto:petsc-ma...@mcs.anl.gov>
> ********************************************************************
> gmake[1]: *** [makefile:40: all] Error 1
> make: *** [GNUmakefile:9: all] Error 2
> 
> On Sat, May 15, 2021 at 5:30 PM Barry Smith <bsm...@petsc.dev 
> <mailto:bsm...@petsc.dev>> wrote:
> 
>   Bitbucket is off-line. Do you need fortran stubs at this moment? If not use 
>  --with-fortran-interface=0 
> 
> 
>> On May 15, 2021, at 2:51 PM, Mark Adams <mfad...@lbl.gov 
>> <mailto:mfad...@lbl.gov>> wrote:
>> 
>> I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with 
>> Kokkos.
>> Any suggestions?
>> Thanks,
>> Mark
>> 
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                          
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                                Trying to 
>> download git://https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING                      
>>                                                                              
>>                                                                              
>>                                             
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                          
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                                Trying to 
>> download git://https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING                      
>>                                                                              
>>                                                                              
>>                                             
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                          
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                                Trying to 
>> download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz 
>> <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING    
>>                                                                              
>>                                                                              
>>                                                   
>> =============================================================================================
>>                                                                              
>>                                                                              
>>                                                                              
>>                                                                              
>>                                                                              
>>                                                                              
>>                                                      
>> *******************************************************************************
>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for 
>> details):
>> -------------------------------------------------------------------------------
>> Error during download/extract/detection of SOWING:
>> Unable to clone sowing
>> Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git> 
>> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']":
>> Cloning into 
>> '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'...
>> fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ 
>> <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned 
>> error: 503
>> Unable to download package SOWING from: 
>> git://https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git>
>> * If URL specified manually - perhaps there is a typo?
>> * If your network is disconnected - please reconnect and rerun ./configure
>> * Or perhaps you have a firewall blocking the download
>> * You can run with --with-packages-download-dir=/adirectory and ./configure 
>> will instruct you what packages to download manually
>> * or you can download the above URL manually, to /yourselectedlocation
>>   and use the configure option:
>>   --download-sowing=/yourselectedlocation
>>     Unable to clone sowing
>> Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git> 
>> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']":
>> Cloning into 
>> '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'...
>> fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ 
>> <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned 
>> error: 503
>> Unable to download package SOWING from: 
>> git://https://bitbucket.org/petsc/pkg-sowing.git 
>> <https://bitbucket.org/petsc/pkg-sowing.git>
>> * If URL specified manually - perhaps there is a typo?
>> * If your network is disconnected - please reconnect and rerun ./configure
>> * Or perhaps you have a firewall blocking the download
>> * You can run with --with-packages-download-dir=/adirectory and ./configure 
>> will instruct you what packages to download manually
>> * or you can download the above URL manually, to /yourselectedlocation
>>   and use the configure option:
>>   --download-sowing=/yourselectedlocation
>>     file could not be opened successfully
>> Downloaded package SOWING from: 
>> https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz 
>> <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a 
>> tarball.
>> [or installed python cannot process compressed files]
>> * If you are behind a firewall - please fix your proxy and rerun ./configure
>>   For example at LANL you may need to set the environmental variable 
>> http_proxy (or HTTP_PROXY?) to  http://proxyout.lanl.gov 
>> <http://proxyout.lanl.gov/>
>> * You can run with --with-packages-download-dir=/adirectory and ./configure 
>> will instruct you what packages to download manually
>> * or you can download the above URL manually, to 
>> /yourselectedlocation/v1.1.26-p1.tar.gz
>>   and use the configure option:
>>   --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
>> *******************************************************************************
>> <configure.log>
> 

Reply via email to