Thanks for pushing on this, Ankit, Thomas, and Paul. As Doug mentioned in
another thread, Apple is heading into our holiday shutdown until the new year,
so I think we should schedule the actual evolution review for the first week of
January. Let's get this hashed out in the meantime and ready for that review
now, though.
Here are my comments:
– I know Cargo uses these terms already, but --lock and "lockfile" are very
generic terms. A "package manager lockfile" could easily refer to a file
multiple package manager processes use to avoid corrupting a shared database. I
think calling this a "deplock" file (for "dependency lock") is much more
specific without being much more verbose. And I'd suggest calling the option
something like --lock-deps.
– Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps
instead. (If we have this flag at all... see below).
– I'm concerned about the lockfile being a 2nd source of truth (vs
Package.swift) that could easily get out of sync. For example, if you update
Package.swift to require a new minimum version, but forget to update the
lockfile (or forget to commit the updated lockfile), users will wind up
silently using a different version than that allowed by the Package.swift's
specified version.
– I'm also uneasy with the lockfile being toml while Package.swift is swift.
That seems inconsistent and requires users to work in two different
configuration file syntaxes (even if toml is very simple).
– To address both of the above points, maybe the dep-lock info should be stored
in Package.swift itself. In this case, any tool which automatically modifies
your Package.swift for you would autoclear the dependency lock when it updates
a package version, and since that data is in the same file, those two changes
would get checked in together. When you're hand-editing the file, it'd be a lot
easier to remember to clear (or update) the locked dependency. And you'd ensure
that the Package.swift data is always in sync with your dependency lock across
revisions and branches, since both data is in the same file (at least for
direct dependencies).
Since dependency locks apply to non-direct dependencies as well, we would need
to add a new package property for modeling the dependency lock for an
otherwise-unspecified dependency. And we might require that the locks fully
specify the properties of the dependency they apply to, so if the required
version of an indirect dependency changes, we can tell that the dependency lock
is out of date. For example, say your "FooApp" package depends on "Lib1" and
"Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on
LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify
LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency
on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)),
and that's what we'd record for the dependencyLock:
let package =
Package(
name: "FooApp",
dependencies: [
.Package(url: "ssh://[email protected]/Lib1.git"),
.Package(url: "ssh://[email protected]/Lib2.git"),
],
dependencyLocks: [
.DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485,
.Package(url: "ssh://[email protected]/Lib1.git"),
.DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2,
.Package(url: "ssh://[email protected]/Lib2.git"),
.DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc,
.Package(url: "ssh://[email protected]/LibBar.git", versions:
Version(1,0,1)..<Version(1,5,0)),
],
)
If Lib2 then updated its version specification to
Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out
of date and prompt you to create a new lock.
Note that we wouldn't expect you to have to hand-author the .DependencyLock
(and manually repeat the version range for packages you depend on); normally
this would be autogenerated by the package manager.
– It would be nice if we could warn, when building with dependency locks, if
your dependencyLock revision does not match a dependency's version specifier.
For example, if your dependency specifies (2,0,1)..<Version(2,1,0), and your
dependencyLock is db2e873d530c72023cf00ce7fe9a37211b8d2fbc, we would check and
make sure that some revision 2.0.1 or greater contains
db2e873d530c72023cf00ce7fe9a37211b8d2fbc, but that it's not reachable from
2.1.0 or greater. That said, `git tag --contains` is probably not fast enough
for nontrivial repositories to run on each build for the purposes of issuing
this warning, so this might be a non-starter for performance reasons.
– Should we always automatically use the deplock info? Ankit's proposal said
no, while Thomas and Paul said yes. I think that it makes sense for most use
cases to use a stable version of your dependencies and only update to a newer
version explicitly, instead of having that happen implicitly when you build if
there happens to be a new version. That favors always using the deplock info.
Thus I think I like Paul's proposal to "always generate .lock if absent, always
use the locked version if present, and use a separate command to update the
locked version." So I'd suggest:
– `swift build --update-deps` updates all dependencies to the latest
allowed version and sets/updates dependency locks for each dependency.
– `swift build --update-dep=<package name>` updates the named
dependency to the latest allowed version and sets/updates dependency lock for
that dependency.
– `swift build --lock-deps` looks at the HEAD of all cloned
dependencies and sets the dependency lock in the top-level Package.swift to
that HEAD commit. This is useful when you want to explicitly lock to something
other than the latest allowed version of a package.
– `swift build --lock-dep=<package name> looks at HEAD of the cloned
named dependency and sets the dependency lock in the the top-level
Package.swift for that one dependency.
– `swift build` clones any dependencies that aren't already cloned,
checks out the locked commit for all dependencies (whether they were already
cloned before or not), and adds dependency locks to Package.swift for any
packages that don't have a lock already. This also warns if the dependency
lock's package specifier doesn't match the actual package specifier where that
dependency is defined, which indicates that your locks are out of date with
respect to your dependency specifications.
One downside to this behavior is it makes it easy to mess up when modifying
your dependencies locally. If you make an edit to a cloned dependency and
commit it, and then `swift build` the top-level package, swiftpm will
automatically revert HEAD of that dependency to the locked commit, so it won't
actually build your change. In order to avoid this, you need to run `swift
build --lock-deps` after committing a local change to a cloned dependency, and
it's easy to forget to do so. That said, the alternative behavior – where
`swift build` preserves the state of your dependencies by default – means that
if you've built a package in the past, and you pull and get a new Package.swift
with new dependency locks, you won't automatically get those dependencies
updated when you build, since you already have cloned dependencies whose HEADs
are stale and don't match the new dependency locks. The latter problem seems
worse than the former. I'm open to ideas to how to solve both problems nicely;
the ways I've thought of so far make this proposal even more complex.
– This proposal still doesn't fully address how you can use a branch for your
dependency instead of a version tag, which was one of the reasons this topic
came up in the first place. You could do so by checking out the branch in your
cloned dependency and then using `swift build --lock-deps` to lock to that
commit, but if you do a `swift build --update-deps` we'll blow that away. To
solve this, perhaps a dependency lock could have an additional optional
"overridingRef" property which, if specified, overrides the version specifier
for the package. That means that `swift build --update-deps` will now update
the package to what that ref points to. The --lock-dep option could allow a
follow-on option --lock-overriding-ref which takes the overriding ref to set.
– Likewise, this mechanism could be used to allow you to override the source of
a dependency for your indirect dependencies. For example, if you depend on
"Lib1", which depends on [email protected]:Somewhere/LibFoo.git, but you actually
want to use your own fork of LibFoo – [email protected]:YourName/LibFoo.git – the
dependency lock would allow that override. This would be done with an
"overridingURL" property on the dependency lock.
– I am concerned about the complexity and additional learning curve this
behavior brings to the package manager. That said, this seems like important
functionality.
Thoughts?
- Rick
> On Dec 20, 2015, at 1:22 PM, Paul Cantrell via swift-evolution
> <[email protected]> wrote:
>
> +1 for Ankit’s general idea. Details of the proposal aside, I’ll say from
> experience with bundler that it’s immensely useful — a lifesaver! — to know
> the exact version of the dependencies another author was using. This has
> saved my neck more than once.
>
> IMO it’s useful to have a lock file checked in even for libraries — just not
> pushed forward to client projects. You still want to know what versions the
> library’s tests last passed against, both for CI and for diagnosing
> downstream breakage.
>
> -1 to this:
>
>> [The] lock file will only be re-modified by $ swift build if Package.swift
>> is modified by the user.
>> $ swift build always ignores the lock file and uses local state of Packages
>> dir / Package.swift
>> …
>> To lock the current state of Packages user can run $ swift build --lock
>
>
> A couple of problems with that:
>
> (1) Package.swift can specify a version range. You may want to update to the
> latest patch release without actually modifying Package.swift. I agree with
> Thomas: there should be a command to update dependencies to the latest
> matching version. This command should also be able update a single dependency:
>
> swift build --update SomePackage
>
> (2) I don’t like the idea of the build system running in two separate modes,
> where sometimes the lock file is ignored and sometimes takes precedence. (If
> there’s a desire to run in an “unlocked” mode, how about it just doesn't
> generate the .lock if not already present, and always uses it if it is
> present?) In practice, though, I’ve found the bundler model works quite well:
> always generate .lock if absent, always use the locked version if present,
> and use a separate command to update the locked version.
>
> Cheers,
>
> Paul
>
>> On Dec 20, 2015, at 9:51 AM, Thomas Guthrie via swift-evolution
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>> Personally, I’d be more in favour of having something similar to Cargo
>> (Rust’s package/crate manager):
>>
>> 1. `swift build`
>>
>> Almost the same as it is now, expect if there’s no Package.lock it creates
>> one and subsequent builds use the same dependencies.
>>
>> 2. `swift build --update` or maybe eventually `swift update`
>>
>> Updates the dependencies in Package.lock to their newest versions and writes
>> them to disk. It should probably build the project as well but possibly
>> makes less sense if its `swift update`.
>>
>> Similar to Bundler and Cargo you’d check in your Package.lock for app
>> projects and ignore it for library projects.
>>
>> I’m not really sure what their motivation was for having a lock file always
>> created, it definitely favours “app” projects heavily, but I’ve been messing
>> around with Rust recently and it works pretty well honestly. Maybe there’s a
>> way of making the experience better when the package is solely a library?
>> Personally, if you’re developing a library and `swift build` updates a
>> dependency that breaks everything it’s probably better to know then, whereas
>> with an app you probably want to be working to a lock file and checking what
>> happens when you update dependencies individually.
>>
>> As for the format of Package.lock, I think it might have to be more
>> complicated than shown to be able to handle the possibility of multiple
>> versions of a dependency etc? Haven’t had a chance to mess around with
>> swiftpm enough yet to say though.
>>
>> (/end ramble of first thoughts)
>>
>> — Thomas
>>
>>> On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution
>>> <[email protected] <mailto:[email protected]>> wrote:
>>>
>>> Lock File for Swift Package Manager
>>> Introduction
>>> A Package.lock file containing list of resolved dependencies generated by
>>> swiftpm.
>>>
>>> Motivation
>>> Package.lock file can be helpful in situations like :
>>>
>>> Reproduce exact versions of dependencies on different machine
>>>
>>> * Multiple developers working on a package would want to use the exact
>>> versions (including minor and patch) of the dependencies declared in the
>>> manifest file
>>> * Also helpful when a build is being performed on a remote machine eg CI
>>> Pointing a dependency to an untagged commit
>>>
>>> Sometimes it might be helpful to lock a dependency to a particular commit
>>> ref for which a tagged version is unavailable in cases such as :
>>>
>>> * Forking a 3rd party library and making it swiftpm compatible for
>>> temporary use until officially supported by the author
>>> * Package is in active development and not ready for a release tag
>>> Proposed Solution
>>> swiftpm generates a simple Package.lock file after resolving the dependency
>>> graph for that package in some simple format.
>>>
>>> Detailed Design
>>> 1. Initial$ swift build resolves the dependency graph and generates a
>>> Package.lock file similar to :
>>> ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
>>> http://github.com/foo/baz <http://github.com/foo/baz> "v1.0.0"
>>> ../local/git/repo "v3.4.4"
>>> lock file will only be re-modified by $ swift build if Package.swift is
>>> modified by the user.
>>> $ swift build always ignores the lock file and uses local state of Packages
>>> dir / Package.swift
>>>
>>> 2. User modifies the cloned packages in Packages dir and when satisfied
>>> with the current code of the dependency, commits and pushes it.
>>> To lock the current state of Packages user can run $ swift build --lock
>>> which might result something similar to
>>> ssh://github.com/foo/bar <http://github.com/foo/bar>
>>> "248441ff375a19c4365d00d6b0706e11173074f6"
>>> http://github.com/foo/baz <http://github.com/foo/baz> "v1.0.0"
>>> ../local/git/repo "v3.4.4"
>>> the lock file is committed into the package repo for others to use.
>>>
>>> 3. A command like $ swift build --bootstrap will always use the lock file
>>> to fetch and checkout the dependencies.
>>> This is useful because running $ swift build might give a higher patch or
>>> minor version of the dependency.
>>>
>>> 4. If some dependency depends on commit hash (ie non-tagged commit) the
>>> author mentions that in their readme and the end user and maybe other
>>> parallel dependencies will have to use only that commit hash in order to
>>> avoid dependency hell.
>>>
>>> 5. Allow declaring a dependency without versions in the manifest file for
>>> user wanting to use a untagged dependency. This should probably only be
>>> allowed in debug configuration.
>>> Impact on existing code
>>> None as this would be additional functionality to swift package manager
>>> Alternatives Considered
>>> One alternative is to allow mentioning refs in manifest file while
>>> declaring a dependency but as discussed in this
>>> <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html>
>>> thread it might not be the best idea.
>>>
>>> --
>>> Ankit
>>> _______________________________________________
>>> swift-evolution mailing list
>>> [email protected] <mailto:[email protected]>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>> <https://lists.swift.org/mailman/listinfo/swift-evolution>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> [email protected] <mailto:[email protected]>
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>> <https://lists.swift.org/mailman/listinfo/swift-evolution>
>
> _______________________________________________
> swift-evolution mailing list
> [email protected] <mailto:[email protected]>
> https://lists.swift.org/mailman/listinfo/swift-evolution
> <https://lists.swift.org/mailman/listinfo/swift-evolution>
_______________________________________________
swift-evolution mailing list
[email protected]
https://lists.swift.org/mailman/listinfo/swift-evolution
