At the expense of a slight special case in the build runner, we can make
the handling of dependencies between modules a little shorter and much
easier to follow.
When module and step graphs are being constructed during the "configure"
phase, we do not set up step dependencies triggered by modules. Instead,
after the configure phase, the build runner traverses the whole
step/module graph, starting from the root top-level steps, and
configures all step dependencies implied by modules. The "make" phase
then proceeds as normal. Also, the old `Module.dependencyIterator` logic
is replaced by two separate iterables. `Module.getGraph` takes the root
module of a compilation, and returns all modules in its graph; while
`Step.Compile.getCompileDependencies` takes a `*Step.Compile` and
returns all `*Step.Compile` it depends on, recursively, possibly
excluding dynamic libraries. The old `Module.dependencyIterator`
combined these two functions into one unintuitive iterator; they are now
separated, which in particular helps readability at the usage sites
which only need one or the other.
This commit changes the `root_module` field of `std.Build.Step.Compile`
to be a `*Module` rather than a `Module`. This is a breaking change, but
an incredibly minor one (the full potential extent of the breakage can
be seen in the modified standalone test).
This change will be necessary for an upcoming improvement, so it was
convenient to make it here.
We have deduced that it seems the sporadic BrokenPipe failures happening
on the CI runners (e.g.
https://github.com/ziglang/zig/actions/runs/12035916948/job/33555963190)
are likely caused by the test runner's stdin pipe abnormally closing,
likely due to the process crashing. Here, we introduce error handling
for this case, so that if these writes fail, the step is marked as
failed correctly, and we still collect the child's stderr to report.
This won't fix the CI issues, but it should promote them to proper error
messages including child stderr, which -- at least in theory -- should
allow us to ultimately track down where the errors come from.
Note that this change is desirable regardless of bugs in the test runner
or similar, since the child process could terminate abnormally for any
number of reasons (e.g. a crashing test), and such cases should be
correctly reported by the build runner.
When using Build.Step.Run.captureStdOut with a program that prints more
than 10 megabytes of output, the build process hangs.
This is because evalGeneric returns an error without reading child's
stdin till the end, so we subsequently get stuck in `try child.wait()`.
To fix this, make sure to kill the child in case of an error!
Output before this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
[3/8] steps
└─ run gh
^C
λ # an hour of debugging
Output after this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
install
└─ install generated to ../tigerbeetle
└─ run build_mutliversion (tigerbeetle)
└─ run unzip
└─ run gh failure
error: unable to spawn gh: StdoutStreamTooLong
Build Summary: 3/8 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install generated to ../tigerbeetle transitive failure
└─ run build_mutliversion (tigerbeetle) transitive failure
└─ run unzip transitive failure
└─ run gh failure
error: the following build command failed with exit code 1:
/home/matklad/p/tb/work/.zig-cache/o/c0e3f5e66ff441cd16f9a1a7e1401494/build /home/matklad/p/tb/work/zig/zig /home/matklad/p/tb/work /home/matklad/p/tb/work/.zig-cache /home/matklad/.cache/zig --seed 0xc1d4efc8 -Zaecc61299ff08765 -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.
This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
A compilation build step for which the binary is not required could not
be compiled previously. There were 2 issues that caused this:
- The compiler communicated only the results of the emitted binary and
did not properly communicate the result if the binary was not emitted.
This is fixed by communicating the final hash of the artifact path (the
hash of the corresponding /o/<hash> directory) and communicating this
instead of the entire path. This changes the zig build --listen protocol
to communicate hashes instead of paths, and emit_bin_path is accordingly
renamed to emit_digest.
- There was an error related to the default llvm object path when
CacheUse.Whole was selected. I'm not really sure why this didn't manifest
when the binary is also emitted.
This was fixed by improving the path handling related to flush() and
emitLlvmObject().
In general, this commit also improves some of the path handling throughout
the compiler and standard library.
* libfuzzer: close file after mmap
* fuzzer/main.js: connect with EventSource and debug dump the messages.
currently this prints how many fuzzer runs have been attempted to
console.log.
* extract some `std.debug.Info` logic into `std.debug.Coverage`.
Prepares for consolidation across multiple different executables which
share source files, and makes it possible to send all the
PC/SourceLocation mapping data with 4 memcpy'd arrays.
* std.Build.Fuzz:
- spawn a thread to watch the message queue and signal event
subscribers.
- track coverage map data
- respond to /events URL with EventSource messages on a timer
* new .zig-cache subdirectory: 'v'
- stores coverage information with filename of hash of PCs that want
coverage. This hash is a hex encoding of the 64-bit coverage ID.
* build runner
* fixed bug in file system inputs when a compile step has an
overridden zig_lib_dir field set.
* set some std lib options optimized for the build runner
- no side channel mitigations
- no Transport Layer Security
- no crypto fork safety
* add a --port CLI arg for choosing the port the fuzzing web interface
listens on. it defaults to choosing a random open port.
* introduce a web server, and serve a basic single page application
- shares wasm code with autodocs
- assets are created live on request, for convenient development
experience. main.wasm is properly cached if nothing changes.
- sources.tar comes from file system inputs (introduced with the
`--watch` feature)
* receives coverage ID from test runner and sends it on a thread-safe
queue to the WebServer.
* test runner
- takes a zig cache directory argument now, for where to put coverage
information.
- sends coverage ID to parent process
* fuzzer
- puts its logs (in debug mode) in .zig-cache/tmp/libfuzzer.log
- computes coverage_id and makes it available with
`fuzzer_coverage_id` exported function.
- the memory-mapped coverage file is now namespaced by the coverage id
in hex encoding, in `.zig-cache/v`
* tokenizer
- add a fuzz test to check that several properties are upheld
The flag makes compiler_rt and libfuzzer be in debug mode.
Also:
* fuzzer: override debug logs and disable debug logs for frequently
called functions
* std.Build.Fuzz: fix bug of rerunning the old unit test binary
* report errors from rebuilding the unit tests better
* link.Elf: additionally add tsan lib and fuzzer lib to the hash
This flag makes the build runner rebuild unit tests after the pipeline
finishes, if it finds any unit tests.
I did not make this integrate with file system watching yet.
The test runner is updated to detect which tests are fuzz tests.
Run step is updated to track which test indexes are fuzz tests.
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.
Also adds global zig cache to the compiler cache prefixes.
Closes#20600
The docs for setting stdio to "inherit" say:
It also means that this step will obtain a global lock to prevent other
steps from running in the meantime.
The implementation of this lock was missing but is now provided by this
commit.
closes#20119
Introduces `disable_zig_progress` which prevents the build runner from
assigning the child process a progress node.
This is needed for the empty_env test which requires the environment to
be completely empty.
This allows running commands that take an output directory argument. The
main thing that was needed for this feature was generated file subpaths,
to allow access to the files in a generated directory. Additionally, a
minor change was required to so that the correct directory is created
for output directory args.
* `doc/langref` formatting
* upgrade `.{ .path = "..." }` to `b.path("...")`
* avoid using arguments named `self`
* make `Build.Step.Id` usage more consistent
* add `Build.pathResolve`
* use `pathJoin` and `pathResolve` everywhere
* make sure `Build.LazyPath.getPath2` returns an absolute path
This replaces `extra_file_dependencies` with support for lazy paths.
Also assert output file basenames are not empty, avoid improper use of
field default values, ensure stored strings are duplicated, and
prefer `ArrayListUnmanaged` to discourage misuse of direct field access
which wouldn't add step dependencies.
This also makes a long-overdue change of extracting common state from
Build into a shared Graph object.
Getting the semantics right for these flags turned out to be quite
tricky. In the end it works like this:
* The override only happens when the target is fully native, with no
additional query parameters, such as versions or CPU features added.
* The override affects the resolved Target but leaves the original Query
unmodified.
* The "is native?" detection logic operates on the original, unmodified
query. This makes it possible to provide invalid host target
information, causing confusing errors to occur. Don't do that.
There are some minor breaking changes to std.Build API such as the fact
that `b.zig_exe` is now moved to `b.graph.zig_exe`, as well as a handful
of other similar flags.
* Specifically recognize stderr as a different concept than an error
message in Step results.
* Display it differently when only stderr occurs but the build proceeds
successfully.
closes#18473
This change is seemingly insignificant but I actually agonized over this
for three days. Some other things I considered:
* (status quo in master branch) make Compile step creation functions
accept a Target.Query and delete the ResolvedTarget struct.
- downside: redundantly resolve target queries many times
* same as before but additionally add a hash map to cache target query
resolutions.
- downside: now there is a hash map that doesn't actually need to
exist, just to make the API more ergonomic.
* add is_native_os and is_native_abi fields to std.Target and use it
directly as the result of resolving a target query.
- downside: they really don't belong there. They would be available
as comptime booleans via `@import("builtin")` but they should not
be exposed that way.
With this change the downsides are:
* the option name of addExecutable and friends is `target` instead of
`resolved_target` matching the type name.
- upside: this does not break compatibility with existing build
scripts
* you likely end up seeing `target.result.cpu.arch` rather than
`target.cpu.arch`.
- upside: this is an improvement over `target.target.cpu.arch` which
it was before this commit.
- downside: `b.host.target` is now `b.host.result`.
