mainly this addresses the following use case:
1. Someone creates a template with build.zig.zon, id field included
(note that zig init does not create this problem since it generates
fresh id every time it runs).
2. User A uses the template, changing package name to "example" but not
id field.
3. User B uses the same template, changing package name also to
"example", also not changing the id field.
Here, both packages have unintentional conflicting logical ids.
By making the field a combination of name checksum + random id, this
accident is avoided. "nonce" is an OK name for this.
Also relaxes errors on remote packages when using `zig fetch`.
Introduces the `id` field to `build.zig.zon`.
Together with name, this represents a globally unique package
identifier. This field should be initialized with a 16-bit random number
when the package is first created, and then *never change*. This allows
Zig to unambiguously detect when one package is an updated version of
another.
When forking a Zig project, this id should be regenerated with a new
random number if the upstream project is still maintained. Otherwise,
the fork is *hostile*, attempting to take control over the original
project's identity.
`0x0000` is invalid because it obviously means a random number wasn't
used.
`0xffff` is reserved to represent "naked" packages.
Tracking issue #14288
Additionally:
* Fix bad path in error messages regarding build.zig.zon file.
* Manifest validates that `name` and `version` field of build.zig.zon
are maximum 32 bytes.
* Introduce error for root package to not switch to enum literal for
name.
* Introduce error for root package to omit `id`.
* Update init template to generate `id`
* Update init template to populate `minimum_zig_version`.
* New package hash format changes:
- name and version limited to 32 bytes via error rather than truncation
- truncate sha256 to 192 bits rather than 40 bits
- include the package id
This means that, given only the package hashes for a complete dependency
tree, it is possible to perform version selection and know the final
size on disk, without doing any fetching whatsoever. This prevents
wasted bandwidth since package versions not selected do not need to be
fetched.
This branch regressed from master by switching to binary rather than hex
digest, allowing null bytes to end up in identifiers in the zig file.
This commit fixes it by changing the "hash" to be literally equal to the
sub_path (with a prefix '/' to indicate "global") if it can fit. If it
is too long then it is actually hashed, and that value used instead.
This prevents bogus "error: file exists in multiple modules" errors due
to file paths looking like:
```
note: root of module foo/freetype/
note: root of module foo/fontconfig/../freetype/
```
It also enables checking for dependency paths outside the root package.
* add Module instances for each package's build.zig and attach it to the
dependencies.zig module with the hash digest hex string as the name.
* fix incorrectly skipping the wrong packages for creating
dependencies.zig
* a couple more renaming of "package" to "module"
Finish the work started in 4c4fb839972f66f55aa44fc0aca5f80b0608c731.
Now the compiler compiles again.
Wire up dependency tree fetching code in the CLI for `zig build`.
Everything is hooked up except for `createDependenciesModule` is not yet
implemented.
* start renaming "package" to "module" (see #14307)
- build system gains `main_mod_path` and `main_pkg_path` is still
there but it is deprecated.
* eliminate the object-oriented memory management style of what was
previously `*Package`. Now it is `*Package.Module` and all pointers
point to externally managed memory.
* fixes to get the new Fetch.zig code working. The previous commit was
work-in-progress. There are still two commented out code paths, the
one that leads to `Compilation.create` and the one for `zig build`
that fetches the entire dependency tree and creates the required
modules for the build runner.
Organize everything around a Fetch task which does a bunch of stuff in a
worker thread without touching any shared state, and then queues up
Fetch tasks for its dependencies.
This isn't the theoretical optimal package fetching performance because
CPU cores don't necessarily map 1:1 with I/O tasks, and each fetch task
contains a mixture of computations and I/O. However, it is expected for
this to significantly outperform master branch, which fetches everything
recursively with only one thread.
The logic is now a lot more linear and easy to follow. Everything that
is embarassingly parallel is done on the thread pool, and then after
everything is fetched, the worker threads are joined and the main thread
does the finishing touches of stitching together the dependencies.zig
import files. There is only one tiny little critical section and it does
not even have any error handling in it.
This also lays the groundwork for #14281 because in system mode, all
this fetching logic will be skipped, but the "finishing touches"
mentioned above still need to be done. With this branch, that logic is
separated out and no longer recursively tangled with fetching stuff.
Additionally, this branch:
* Implements inclusion directives in `build.zig.zon` for deciding which
files belong the package (#14311).
* Adds basic documentation for `build.zig.zon` files.
* Adds support for fetching dependencies with the `file://` protocol
scheme (#17364).
* Adds a workaround for a Linux/btrfs file system bug (#17282).
This commit is a work-in-progress. Still todo:
1. Hook up the CLI to the new system.
2. Restore the module table creation logic after all the fetching is
done.
3. Fix compilation errors, get the tests passing, and regression test
against real world projects.
zig fetch [options] <url>
zig fetch [options] <path>
Fetches a package which is found at <url> or <path> into the global
cache directory, printing the package hash to stdout.
Closes#16972
Related to #14280
Additionally, this commit:
* Adds uncompressed .tar support to package fetching
* Introduces symlink support to package fetching
Closes#14298
This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data.
Git URLs can be specified in `build.zig.zon` as follows:
```zig
.xml = .{
.url = "git+https://github.com/ianprime0509/zig-xml#7380d59d50f1cd8460fd748b5f6f179306679e2f",
.hash = "122085c1e4045fa9cb69632ff771c56acdb6760f34ca5177e80f70b0b92cd80da3e9",
},
```
The fragment part of the URL may specify a commit ID (SHA1 hash), branch
name, or tag. It is an error to omit the fragment: if this happens, the
compiler will prompt the user to add it, using the commit ID of the HEAD
commit of the repository (that is, the latest commit of the default
branch):
```
Fetch Packages... xml... /var/home/ian/src/zig-gobject/build.zig.zon:6:20: error: url field is missing an explicit ref
.url = "git+https://github.com/ianprime0509/zig-xml",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: try .url = "git+https://github.com/ianprime0509/zig-xml#dfdc044f3271641c7d428dc8ec8cd46423d8b8b6",
```
This implementation currently supports only version 2 of Git's wire
protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was first
introduced in Git 2.19 (2018) and made the default in 2.26 (2020).
The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. #14295).
This commit makes the following changes:
* Disallow file:/// URIs
* Allow only relative paths in the .path field of build.zig.zon
* Remote now-unneeded shlwapi dependency
Addresses #17015 by introducing a new startWithOptions. The only option is currently is a flag
to use the provided URI as is, without modification when passed to the server. Normally, this
is not needed nor desired. However, some REST APIs may have requirements that cannot be satisfied
with the default handling.
The new `@depedencies` module contains generated code like the
following (where strings like "abc123" represent hashes):
```zig
pub const root_deps = [_]struct { []const u8, []const u8 }{
.{ "foo", "abc123" },
};
pub const packages = struct {
pub const abc123 = struct {
pub const build_root = "/home/mlugg/.cache/zig/blah/abc123";
pub const build_zig = @import("abc123");
pub const deps = [_]struct { []const u8, []const u8 }{
.{ "bar", "abc123" },
.{ "name", "ghi789" },
};
};
};
```
Each package contains a build root string, the build.zig import, and a
mapping from dependency names to package hashes. There is also such a
mapping for the root package dependencies.
In theory, we could now remove the `dep_prefix` field from `std.Build`,
since its main purpose is now handled differently. I believe this is a
desirable goal, as it doesn't really make sense to assign a single FQN
to any package (because it may appear in many different places in the
package hierarchy). This commit does not remove that field, as it's used
non-trivially in a few places in the build runner and compiler tests:
this will be a future enhancement.
Resolves: #16354Resolves: #17135
`TailQueue` was implemented as a doubly-linked list, but named after an
abstract data type. This was inconsistent with `SinglyLinkedList`, which
can be used to implement an abstract data type, but is still named after
the implementation. Renaming `TailQueue` to `DoublyLinkedList` improves
consistency between the two type names, and should help discoverability.
`TailQueue` is now a deprecated alias of `DoublyLinkedList`.
Related to issues #1629 and #8233.
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
Allows the package manager to download gitlab tarballs from urls such as
https://gitlab.com/<namespace>/<project>/-/archive/<sha>/<project>-<sha>.tar.gz
Such http requests have headers Content-Type=application/octet-stream
and Content-Disposition='attachment; filename="<project>-<sha>.tar.gz"'.
The package manager doesn't yet support these headers. This patch
doesn't attempt to properly parse the content-disposition header.
Instead it checks that it starts with 'attachment;' and ends with
'.tar.gz"'.
