Thank you Gabriel,
Just for future readers. Below is a base R way to address this common
problem, as instructed by you (+stopifnot to suppress print).
Rscript -e 'stopifnot(file.copy("DESCRIPTION",
file.path(tdir<-tempdir(), "PACKAGES")));
db<-available.packages(paste0("file://", tdir));
install.packages(setdiff(tools::package_dependencies(read.dcf("DESCRIPTION",
fields="Package")[[1L]], db, which="most")[[1L]],
installed.packages(priority="high")[,"Package"]))'
3 liner, 310 chars long command, far from ideal, but does work.
Best,
Jan
On Fri, Oct 28, 2022 at 10:42 PM Gabriel Becker <[email protected]> wrote:
>
> Hi Jan,
>
>
> On Fri, Oct 28, 2022 at 1:57 PM Jan Gorecki <[email protected]> wrote:
>>
>> Gabriel,
>>
>> It is the most basic CI use case. One wants to install only
>> dependencies only of the package, and run R CMD check on the package.
>
>
> Really what you're looking for though, is to install all the dependencies
> which aren't present right? Excluding base packages is just a particular way
> to do that under certain assumptions about the CI environment.
>
> So
>
>
> needed_pkgs <- setdiff(package_dependencies(...),
> installed.packages()[,"Package"])
> install.packages(needed_pkgs, repos = fancyrepos)
>
>
> will do what you want without installing the package itself, if that is
> important. This will filter out base and recommended packages (which will be
> already installed in your CI container, since R is).
>
>
> Now this does not take into account versioned dependencies, so it's not
> actually fully correct (whereas installing the package is), but it gets you
> where you're trying to go. And in a clean CI container without cached package
> installation for the deps, its equivalent.
>
>
> Also, as an aside, if you need to get the base packages, you can do
>
> installed.packages(priority="base")[,"Package"]
>
> base compiler datasets graphics grDevices grid
>
> "base" "compiler" "datasets" "graphics" "grDevices" "grid"
>
> methods parallel splines stats stats4 tcltk
>
> "methods" "parallel" "splines" "stats" "stats4" "tcltk"
>
> tools utils
>
> "tools" "utils"
>
>
> (to get base and recommended packages use 'high' instead of 'base')
>
> No need to be reaching down into unexported functions. So if you *really*
> only want to exclude base functions (which likely will give you some
> protection from versioned dep issues), you can change the code above to
>
> needed_pkgs <- setdiff(package_dependencies(...), installed.packages(priority
> = "high")[,"Package"])
> install.packages(needed_pkgs, repos = fancyrepos)
>
> Best,
> ~G
>
>>
>> On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <[email protected]> wrote:
>> >
>> > Hi Jan,
>> >
>> > The reason, I suspect without speaking for R-core, is that by design you
>> > should not be specifying package dependencies as additional packages to
>> > install. install.packages already does this for you, as it did in the
>> > construct of a repository code that I provided previously in the thread.
>> > You should be *only* doing
>> >
>> > install.packages(<pkg in question>, repos = *)
>> >
>> > Then everything happens automatically via extremely well tested very
>> > mature code.
>> >
>> > I (still) don't understand why you'd need to pass install.packages the
>> > vector of dependencies yourself, as that is counter to install.packages'
>> > core design.
>> >
>> > Does that make sense?
>> >
>> > Best,
>> > ~G
>> >
>> > On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <[email protected]> wrote:
>> >>
>> >> Gabriel,
>> >>
>> >> I am trying to design generic solution that could be applied to
>> >> arbitrary package. Therefore I went with the latter solution you
>> >> proposed.
>> >> If we wouldn't have to exclude base packages, then its a 3 liner
>> >>
>> >> file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
>> >> db<-available.packages(paste0("file://", tdir));
>> >> utils::install.packages(tools::package_dependencies("pkgname", db,
>> >> which="most")[[1L]])
>> >>
>> >> As you noticed, we still have to filter out base packages. Otherwise
>> >> it won't be a robust utility that can be used in CI. Therefore we have
>> >> to add a call to tools:::.get_standard_package_names() which is an
>> >> internal function (as of now). Not only complicating the call but also
>> >> putting the functionality outside of safe use.
>> >>
>> >> Considering above, don't you agree that the following one liner could
>> >> nicely address the problem? The problem that hundreds/thousands of
>> >> packages are now addressing in their CI scripts by using a third party
>> >> packages.
>> >>
>> >> utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
>> >>
>> >> It is hard to me to understand why R members don't consider this basic
>> >> functionality to be part of base R. Possibly they just don't need it
>> >> themselves. Yet isn't this sufficient that hundreds/thousands of
>> >> packages does need this functionality?
>> >>
>> >> Best regards,
>> >> Jan
>> >>
>> >> On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <[email protected]> wrote:
>> >> >
>> >> > Gabriel and Simon
>> >> >
>> >> > I completely agree with what you are saying.
>> >> > The thing is that obtaining recursive deps, all/most whatever, is
>> >> > already well supported in core R. What is missing is just this single
>> >> > functionality I am requesting.
>> >> >
>> >> > If you will look into the branch you can see there is mirror.packages
>> >> > function meant to mirror a slice of CRAN. It is doing exactly what you
>> >> > described: package_dependencies; to obtain recursive deps, then
>> >> > download all, etc.
>> >> > I would love to have this function provided by core R as well, but we
>> >> > need to start somewhere.
>> >> >
>> >> > There are other use cases as well.
>> >> > For example CI, where one wants to install all/most dependencies and
>> >> > then run R CMD check. Then we don't worry about recursive deps are they
>> >> > will be resolved automatically.
>> >> > I don't think it's reasonable to force users to use 3rd party packages
>> >> > to handle such a common and simple use case. Otherwise one has to hard
>> >> > code deps in CI script. Not robust at all.
>> >> >
>> >> > packages.dcf and repos.dcf makes all that way easier, and are solid
>> >> > base for building customized orchestration like mirroring slice of CRAN.
>> >> >
>> >> > Best regards
>> >> > Jan
>> >> >
>> >> > On Sun, Oct 16, 2022, 01:31 Simon Urbanek <[email protected]>
>> >> > wrote:
>> >> >>
>> >> >> Jan,
>> >> >>
>> >> >> I think using a single DCF as input is not very practical and would
>> >> >> not be useful in the context you describe (creating self contained
>> >> >> repos) since they typically concern a list of packages, but
>> >> >> essentially splitting out the part of install.packages() which
>> >> >> determines which files will be pulled from where would be very useful
>> >> >> as it would be trivial to use it to create repository (what we always
>> >> >> do in corporate environments) instead of installing the packages. I
>> >> >> suspect that install packages is already too complex so instead of
>> >> >> adding a flag to install.packages one could move that functionality
>> >> >> into a separate function - we all do that constantly for the sites we
>> >> >> manage, so it would be certainly something worthwhile.
>> >> >>
>> >> >> Cheers,
>> >> >> Simon
>> >> >>
>> >> >>
>> >> >> > On Oct 15, 2022, at 7:14 PM, Jan Gorecki <[email protected]>
>> >> >> > wrote:
>> >> >> >
>> >> >> > Hi Gabriel,
>> >> >> >
>> >> >> > It's very nice usage you provided here. Maybe instead of adding new
>> >> >> > function we could extend packages_depenedncies then? To accept file
>> >> >> > path to
>> >> >> > dsc file.
>> >> >> >
>> >> >> > What about repos.dcf? Maybe additional repositories could be an
>> >> >> > attribute
>> >> >> > attached to returned character vector.
>> >> >> >
>> >> >> > The use case is to, for a given package sources, obtain its
>> >> >> > dependencies,
>> >> >> > so one can use that for installing them/mirroring CRAN subset, or
>> >> >> > whatever.
>> >> >> > The later is especially important for a production environment where
>> >> >> > one
>> >> >> > wants to have fixed version of packages, and mirroring relevant
>> >> >> > subset of
>> >> >> > CRAN is the most simple, and IMO reliable, way to manage such
>> >> >> > environment.
>> >> >> >
>> >> >> > Regards
>> >> >> > Jan
>> >> >> >
>> >> >> > On Fri, Oct 14, 2022, 23:34 Gabriel Becker <[email protected]>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi Jan and Jan,
>> >> >> >>
>> >> >> >> Can you explain a little more what exactly you want the
>> >> >> >> non-recursive,
>> >> >> >> non-version aware dependencies from an individual package for?
>> >> >> >>
>> >> >> >> Either way package_dependencies will do this for you* with a little
>> >> >> >> "aggressive convincing". It wants output from available.packages,
>> >> >> >> but who
>> >> >> >> really cares what it wants? It's a function and we are people :)
>> >> >> >>
>> >> >> >>> library(tools)
>> >> >> >>> db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
>> >> >> >>> package_dependencies("rtables", db, which = intersect(c("Depends",
>> >> >> >> "Suggests", "Imports", "LinkingTo"), colnames(db)))
>> >> >> >> $rtables
>> >> >> >> [1] "methods" "magrittr" "formatters" "dplyr" "tibble"
>> >> >> >> [6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
>> >> >> >> [11] "flextable" "officer" "stats" "htmltools" "grid"
>> >> >> >>
>> >> >> >>
>> >> >> >> The only gotcha that I see immediately is that "LinkingTo" isn't
>> >> >> >> always
>> >> >> >> there (whereas it is with real output from available.packages). If
>> >> >> >> you
>> >> >> >> know your package doesn't have that (or that it does) at call time
>> >> >> >> , this
>> >> >> >> becomes a one-liner:
>> >> >> >>
>> >> >> >> package_dependencies("rtables", db =
>> >> >> >> read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
>> >> >> >> c("Depends", "Suggests", "Imports"))
>> >> >> >> $rtables
>> >> >> >> [1] "methods" "magrittr" "formatters" "dplyr" "tibble"
>> >> >> >> [6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
>> >> >> >> [11] "flextable" "officer" "stats" "htmltools" "grid"
>> >> >> >>
>> >> >> >> You can also trick it a slightly different way by giving it what it
>> >> >> >> actually wants
>> >> >> >>
>> >> >> >>> tdir <- tempdir()
>> >> >> >>> file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
>> >> >> >>> file.path(tdir,
>> >> >> >> "PACKAGES"))
>> >> >> >> [1] TRUE
>> >> >> >>> avl <- available.packages(paste0("file://", tdir))
>> >> >> >>> library(tools)
>> >> >> >>> package_dependencies("rtables", avl)
>> >> >> >> $rtables
>> >> >> >> [1] "methods" "magrittr" "formatters" "stats" "htmltools"
>> >> >> >> [6] "grid"
>> >> >> >>
>> >> >> >>> package_dependencies("rtables", avl, which = "all")
>> >> >> >> $rtables
>> >> >> >> [1] "methods" "magrittr" "formatters" "stats" "htmltools"
>> >> >> >> [6] "grid" "dplyr" "tibble" "tidyr" "testthat"
>> >> >> >> [11] "xml2" "knitr" "rmarkdown" "flextable" "officer"
>> >> >> >>
>> >> >> >> So the only real benefits I see that we'd be picking up here is
>> >> >> >> automatic
>> >> >> >> filtering by priority, and automatic extraction of the package name
>> >> >> >> from
>> >> >> >> the DESCRIPTION file. I'm not sure either of those warrant a new
>> >> >> >> exported
>> >> >> >> function that R-core has to maintain forever.
>> >> >> >>
>> >> >> >> Best,
>> >> >> >> ~G
>> >> >> >>
>> >> >> >> * I haven't tested this across all OSes, but I dont' know of any
>> >> >> >> reason it
>> >> >> >> wouldn't work generally.
>> >> >> >>
>> >> >> >> On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <[email protected]>
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hello Jan,
>> >> >> >>>
>> >> >> >>> Thanks for confirming about many packages reinventing this missing
>> >> >> >>> functionality.
>> >> >> >>> packages.dcf was not meant handle versions. It just extracts names
>> >> >> >>> of
>> >> >> >>> dependencies... Yes, such a simple thing, yet missing in base R.
>> >> >> >>>
>> >> >> >>> Versions of packages can be controlled when setting up R pkgs
>> >> >> >>> repo. This
>> >> >> >>> is
>> >> >> >>> how I used to handle it. Making a CRAN subset mirror of fixed
>> >> >> >>> version
>> >> >> >>> pkgs.
>> >> >> >>> BTW. function for that is also included in mentioned branch. I am
>> >> >> >>> just not
>> >> >> >>> proposing it, to increase the chance of having at least this
>> >> >> >>> simple,
>> >> >> >>> missing, functionality merged.
>> >> >> >>>
>> >> >> >>> Best
>> >> >> >>> Jan
>> >> >> >>>
>> >> >> >>> On Fri, Oct 14, 2022, 15:14 Jan Netík <[email protected]> wrote:
>> >> >> >>>
>> >> >> >>>> Hello Jan,
>> >> >> >>>>
>> >> >> >>>> I have seen many packages that implemented dependencies
>> >> >> >>>> "extraction" on
>> >> >> >>>> their own for internal purposes and today I was doing exactly
>> >> >> >>>> that for
>> >> >> >>>> mine. It's not a big deal using read.dcf on DESCRIPTION. It was
>> >> >> >>> sufficient
>> >> >> >>>> for me, but I had to take care of some \n chars (the overall
>> >> >> >>>> returned
>> >> >> >>> value
>> >> >> >>>> has some rough edges, in my opinion). However, the function from
>> >> >> >>>> the
>> >> >> >>> branch
>> >> >> >>>> seems to not care about version requirements, which are crucial
>> >> >> >>>> for me.
>> >> >> >>>> Maybe that is something to reconsider before merging.
>> >> >> >>>>
>> >> >> >>>> Best,
>> >> >> >>>> Jan
>> >> >> >>>>
>> >> >> >>>> pá 14. 10. 2022 v 2:27 odesílatel Jan Gorecki
>> >> >> >>>> <[email protected]>
>> >> >> >>>> napsal:
>> >> >> >>>>
>> >> >> >>>>> Dear R devs,
>> >> >> >>>>>
>> >> >> >>>>> I would like to raise a request for a simple helper function.
>> >> >> >>>>> Utility function to extract package dependencies from
>> >> >> >>>>> DESCRIPTION file.
>> >> >> >>>>>
>> >> >> >>>>> I do think that tools package is better place, for such a
>> >> >> >>>>> fundamental
>> >> >> >>>>> functionality, than community packages.
>> >> >> >>>>>
>> >> >> >>>>> tools pkg seems perfect fit (having already great function
>> >> >> >>>>> write_PACKAGES).
>> >> >> >>>>>
>> >> >> >>>>> Functionality I am asking for is already in R svn repository
>> >> >> >>>>> since
>> >> >> >>> 2016,
>> >> >> >>>>> in
>> >> >> >>>>> a branch tools4pkgs. Function is called 'packages.dcf'.
>> >> >> >>>>> Another one 'repos.dcf' would be a good functional complementary
>> >> >> >>>>> to it.
>> >> >> >>>>>
>> >> >> >>>>> Those two simple helper functions really makes it easier for
>> >> >> >>> organizations
>> >> >> >>>>> to glue together usage of their own R packages repos and CRAN
>> >> >> >>>>> repo in a
>> >> >> >>>>> smooth way. That could possibly help to offload CRAN from new
>> >> >> >>> submissions.
>> >> >> >>>>>
>> >> >> >>>>> gh mirror link for easy preview:
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>> https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419
>> >> >> >>>>>
>> >> >> >>>>> Regards
>> >> >> >>>>> Jan Gorecki
>> >> >> >>>>>
>> >> >> >>>>> [[alternative HTML version deleted]]
>> >> >> >>>>>
>> >> >> >>>>> ______________________________________________
>> >> >> >>>>> [email protected] mailing list
>> >> >> >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>>
>> >> >> >>> [[alternative HTML version deleted]]
>> >> >> >>>
>> >> >> >>> ______________________________________________
>> >> >> >>> [email protected] mailing list
>> >> >> >>> https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> > [[alternative HTML version deleted]]
>> >> >> >
>> >> >> > ______________________________________________
>> >> >> > [email protected] mailing list
>> >> >> > https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >
>> >> >>
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
