On 2020/05/18 12:51, Klemens Nanni wrote:
> Some ports have huge distfiles of which we only need a tiny fraction,
> see the coreboot-utils thread.
>
> My idea was to pass a list of files to the extract phase and it works
> just fine as seen in the diff below, based on this EXTRACT_FILES has no
> effect for .shar or .gz distfiles, but I don't see this as an issue.
>
> Technically, EXTRACT_FILES is a list of glob patterns read by tar(1).
>
> Diff below has the bsd.port.mk(5) bits as well as two examples:
> coreboot-utils dropping WRKDIR size from 205M to 7M and iwm-firmware
> reduced from 408M to 93M - no PLIST changes.
>
> Passing EXTRACT_FILES as is has the advantage of using sh(1)'s brace
> expansion as shown in the diff, using make's :QL would have the
> advantage of not preventing porters from passing bogus values, e.g.
> passing unquoted or unescaped asterisks which would mistakenly be
> interpreted by sh(1) instead of tar(1).
>
> I pass it as is because brace expansion makes it easy to use and
> EXTRACT_FILES wouldn't be the first variable to pass unsanitized shell
> code into parts of bsd.port.mk where arbitrary code is not meant to be.
>
> Feedback?
I like it, can you work up something for the bsd.port.mk(5) manual please?
I agree the brace expansion is helpful for how this is used.
> Index: infrastructure/mk/bsd.port.mk
> ===================================================================
> RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
> retrieving revision 1.1534
> diff -u -p -U0 -r1.1534 bsd.port.mk
> --- infrastructure/mk/bsd.port.mk 20 Apr 2020 11:49:57 -0000 1.1534
> +++ infrastructure/mk/bsd.port.mk 18 May 2020 10:41:40 -0000
> @@ -1344,0 +1345 @@ EXTRACT_CASES ?=
> +EXTRACT_FILES ?=
> @@ -1353 +1354 @@ EXTRACT_CASES += *.zip) \
> - ${UNZIP} -oq ${FULLDISTDIR}/$$archive -d ${WRKDIR};;
> + ${UNZIP} -oq ${FULLDISTDIR}/$$archive -d ${WRKDIR} ${EXTRACT_FILES};;
> @@ -1360 +1361 @@ EXTRACT_CASES += *.tar.xz) \
> - xzdec <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + xzdec <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
> @@ -1362 +1363 @@ EXTRACT_CASES += *.tar.lzma) \
> - lzmadec <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + lzmadec <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
> @@ -1368 +1369 @@ EXTRACT_CASES += *.tar.lz) \
> - lunzip <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + lunzip <${FULLDISTDIR}/$$archive | ${TAR} -xf - -- ${EXTRACT_FILES};;
> @@ -1378 +1379 @@ EXTRACT_CASES += *.tar.bz2|*.tbz2|*.tbz)
> - ${BZIP2} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + ${BZIP2} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
> ${EXTRACT_FILES};;
> @@ -1387 +1388 @@ EXTRACT_CASES += *.tar) \
> - ${TAR} xf ${FULLDISTDIR}/$$archive;;
> + ${TAR} -xf ${FULLDISTDIR}/$$archive -- ${EXTRACT_FILES};;
> @@ -1393 +1394 @@ EXTRACT_CASES += *.tar.gz|*.tgz) \
> - ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
> ${EXTRACT_FILES};;
> @@ -1397 +1398 @@ EXTRACT_CASES += *) \
> - ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} xf -;;
> + ${GZIP_CMD} -d <${FULLDISTDIR}/$$archive | ${TAR} -xf - --
> ${EXTRACT_FILES};;
> Index: sysutils/firmware/iwm/Makefile
> ===================================================================
> RCS file: /cvs/ports/sysutils/firmware/iwm/Makefile,v
> retrieving revision 1.13
> diff -u -p -r1.13 Makefile
> --- sysutils/firmware/iwm/Makefile 18 Nov 2019 11:36:56 -0000 1.13
> +++ sysutils/firmware/iwm/Makefile 18 May 2020 10:41:46 -0000
> @@ -5,6 +5,7 @@ FW_VER= 20191022
> REVISION= 0
> DISTNAME= linux-firmware-${FW_VER}
> EXTRACT_SUFX= .tar.xz
> +EXTRACT_FILES=
> ${DISTNAME}/{LICENCE.iwlwifi_firmware,iwlwifi-\*-\*.ucode}
>
> HOMEPAGE= https://wireless.wiki.kernel.org/en/users/Drivers/iwlwifi
> MASTER_SITES= https://cdn.kernel.org/pub/linux/kernel/firmware/
> Index: sysutils/coreboot-utils/Makefile
> ===================================================================
> RCS file: /cvs/ports/sysutils/coreboot-utils/Makefile,v
> retrieving revision 1.3
> diff -u -p -r1.3 Makefile
> --- sysutils/coreboot-utils/Makefile 14 May 2020 19:20:27 -0000 1.3
> +++ sysutils/coreboot-utils/Makefile 18 May 2020 10:26:38 -0000
> @@ -24,6 +24,8 @@ WANTLIB = c
> MASTER_SITES = https://coreboot.org/releases/
>
> EXTRACT_SUFX = .tar.xz
> +# extract only what's needed (7M) instead of everything (205M)
> +EXTRACT_FILES = ${DISTNAME}/{src/commonlib,util}
>
> USE_GMAKE = Yes
>
>
