An embedded manifest file is really just XML data embedded as a RT_MANIFEST resource (ID = 24). Typically, the Windows-only 'Manifest Tool' (`mt.exe`) is used to embed manifest files, and `mt.exe` also seems to perform some transformation of the manifest data before embedding, but in testing it doesn't seem like the transformations are necessary to get the intended result.
So, to handle embedding manifest files, Zig now takes the following approach:
- Generate a .rc file with the contents `1 24 "path-to-manifest.manifest"`
- Compile that generated .rc file into a .res file
- Link the .res file into the final binary
This effectively achieves the same thing as `mt.exe` minus the validation/transformations of the XML data that it performs.
How this is used:
On the command line:
```
zig build-exe main.zig main.manifest
```
(on the command line, specifying a .manifest file when the target object format is not COFF is an error)
or in build.zig:
```
const exe = b.addExecutable(.{
.name = "manifest-test",
.root_source_file = .{ .path = "main.zig" },
.target = target,
.optimize = optimize,
.win32_manifest = .{ .path = "main.manifest" },
});
```
(in build.zig, the manifest file is ignored if the target object format is not COFF)
Note: Currently, only one manifest file can be specified per compilation. This is because the ID of the manifest resource is currently always 1. Specifying multiple manifests could be supported if a way for the user to specify an ID for each manifest is added (manifest IDs must be a u16).
Closes#17406
options
* start renaming "package" to "module" (see #14307)
- build system gains `main_mod_path` and `main_pkg_path` is still
there but it is deprecated.
* eliminate the object-oriented memory management style of what was
previously `*Package`. Now it is `*Package.Module` and all pointers
point to externally managed memory.
* fixes to get the new Fetch.zig code working. The previous commit was
work-in-progress. There are still two commented out code paths, the
one that leads to `Compilation.create` and the one for `zig build`
that fetches the entire dependency tree and creates the required
modules for the build runner.
The new `@depedencies` module contains generated code like the
following (where strings like "abc123" represent hashes):
```zig
pub const root_deps = [_]struct { []const u8, []const u8 }{
.{ "foo", "abc123" },
};
pub const packages = struct {
pub const abc123 = struct {
pub const build_root = "/home/mlugg/.cache/zig/blah/abc123";
pub const build_zig = @import("abc123");
pub const deps = [_]struct { []const u8, []const u8 }{
.{ "bar", "abc123" },
.{ "name", "ghi789" },
};
};
};
```
Each package contains a build root string, the build.zig import, and a
mapping from dependency names to package hashes. There is also such a
mapping for the root package dependencies.
In theory, we could now remove the `dep_prefix` field from `std.Build`,
since its main purpose is now handled differently. I believe this is a
desirable goal, as it doesn't really make sense to assign a single FQN
to any package (because it may appear in many different places in the
package hierarchy). This commit does not remove that field, as it's used
non-trivially in a few places in the build runner and compiler tests:
this will be a future enhancement.
Resolves: #16354Resolves: #17135
When using `std.Build.dependency` with target options, dependencies
would sometimes get targets which are equivalent but have distinct
names, e.g. `native` vs `native-native`. This is a somewhat broad issue,
and it's unclear how to fix it more generally - perhaps we should
special-case CrossTarget in options passing, or maybe targets should
have a canonical name which we guarantee to use everywhere aside from
raw user input.
However, this commit fixes the most egregious issue, which was an active
blocker to using the package manager for some users. This was caused by
the CPU changing from `native` to a specific descriptor (e.g.
`skylake+sgx`), which then changed the behavior of `zigTriple`.
Resolves: #16856
* introduce LazyPath.cwd_relative variant and use it for --zig-lib-dir. closes#12685
* move overrideZigLibDir and setMainPkgPath to options fields set once
and then never mutated.
* avoid introducing Build/util.zig
* use doc comments for deprecation notices so that they show up in
generated documentation.
* introduce InstallArtifact.Options, accept it as a parameter to
addInstallArtifact, and move override_dest_dir into it. Instead of
configuring the installation via Compile step, configure the
installation via the InstallArtifact step. In retrospect this is
obvious.
* remove calls to pushInstalledFile in InstallArtifact. See #14943
* rewrite InstallArtifact to not incorrectly observe whether a Compile
step has any generated outputs. InstallArtifact is meant to trigger
output generation.
* fix child process evaluation code handling of `-fno-emit-bin`.
* don't store out_h_filename, out_ll_filename, etc., pointlessly. these
are all just simple extensions appended to the root name.
* make emit_directory optional. It's possible to have nothing outputted,
for example, if you're just type-checking.
* avoid passing -femit-foo/-fno-emit-foo when it is the default
* rename ConfigHeader.getTemplate to getOutput
* deprecate addOptionArtifact
* update the random number seed of Options step caching.
* avoid using `inline for` pointlessly
* avoid using `override_Dest_dir` pointlessly
* avoid emitting an executable pointlessly in test cases
Removes forceBuild and forceEmit. Let's consider these additions separately.
Nearly all of the usage sites were suspicious.
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
Also get rid of the TTY wrapper struct, which was exlusively used as a
namespace - this is done by the tty.zig root struct now.
detectTTYConfig has been renamed to just detectConfig, which is enough
given the new namespace. Additionally, a doc comment had been added.
* build.zig: the result of b.option() can be assigned directly in many
cases thanks to the return type being an optional
* std.Build: make the build system aware of the
std.Build.Step.Compile.BuildId type when used as an option.
- remove extraneous newlines in error logs
* simplify caching logic
* simplify hexstring parsing tests and use a doc test
* simplify hashing logic. don't use an optional when the `none` tag
already provides this meaning.
* CLI: fix incorrect linker arg parsing
The CI now runs C backend tests in addition to compiling them. It uses
-std=c99 -pedantic -Werror in order to catch non-conformant C code.
This necessitated disabling a test case that caused a C compile error,
in addition to disabling a handful of warnings that are already being
triggered by Zig's C backend output for the behavior tests.
The upshot is that I was able to, very cleanly, integrate the C backend
tests into the build system, so that it communicates via the test runner
protocol along with all the other behavior tests.
* remove setName, setFilter, and setTestRunner. Please set these
options directly when creating the CompileStep.
* removed unused field
* remove computeOutFileNames and inline the logic, making clear the
goal of avoiding state mutations after the build step is created.
* use the same hash function as the rest of the steps
* fix race condition due to a macOS oddity.
* fix race condition due to file truncation (rename into place instead)
* integrate with marking Step.result_cached. check if the file already
exists with fs.access before doing anything else.
* use a directory so that the file basename can be "options.zig"
instead of a hash digest.
* better error reporting in case of file system failures.
This is useful for tests that want to `execve` zig directly. The string
is already null-terminated, so this will just expose it as such,
removing an extra allocation from the test.
Will be used in #14462
std.Build.addTest creates a CompileStep as before, however, this kind of
step no longer actually runs the unit tests. Instead it only compiles
it, and one must additionally create a RunStep from the CompileStep in
order to actually run the tests.
RunStep gains integration with the default test runner, which now
supports the standard --listen=- argument in order to communicate over
stdin and stdout. It also reports test statistics; how many passed,
failed, and leaked, as well as directly associating the relevant stderr
with the particular test name that failed.
This separation of CompileStep and RunStep means that
`CompileStep.Kind.test_exe` is no longer needed, and therefore has been
removed in this commit.
* build runner: show unit test statistics in build summary
* added Step.writeManifest since many steps want to treat it as a
warning and emit the same message if it fails.
* RunStep: fixed error message that prints the failed command printing
the original argv and not the adjusted argv in case an interpreter
was used.
* RunStep: fixed not passing the command line arguments to the
interpreter.
* move src/Server.zig to std.zig.Server so that the default test runner
can use it.
* the simpler test runner function which is used by work-in-progress
backends now no longer prints to stderr, which is necessary in order
for the build runner to not print the stderr as a warning message.
Previously this code used SipHash(1, 3) directly; now that we have the
cache system available in the build system, borrow the same
implementation as is being used everywhere else.
This is for bypassing the package manager and directly depending on
another package via the build system. For this to work the anonymous
package must be found on the file system relative to the current
package's build.zig.
CLI tests are now ported over to the new std.Build API and thus work
properly with concurrency.
* add `std.Build.addCheckFile` for creating a
`std.Build.CheckFileStep`.
* add `std.Build.makeTempPath`. This function is intended to be called
in the `configure` phase only. It returns an absolute directory path,
which is potentially going to be a source of API breakage in the
future, so keep that in mind when using this function.
* add `std.Build.CheckFileStep.setName`.
* `std.Build.CheckFileStep`: better error message when reading the
input file fails.
* `std.Build.RunStep`: add a `has_side_effects` flag for when you need
to override the autodetection.
* `std.Build.RunStep`: add the ability to obtain a FileSource for the
directory that contains the written files.
* `std.Build.WriteFileStep`: add a way to write bytes to an arbitrary
path - absolute or relative to the package root. Be careful with this
because it updates source files. This should not be used as part of
the normal build process, but as a utility occasionally run by a
developer with intent to modify source files and then commit those
changes to version control. A file added this way is not available
with `getFileSource`.
The problem is that one may execute too many subprocesses concurrently
that, together, exceed an RSS value that causes the OOM killer to kill
something problematic such as the window manager. Or worse, nothing, and
the system freezes.
This is a real world problem. For example when building LLVM a simple
`ninja install` will bring your system to its knees if you don't know
that you should add `-DLLVM_PARALLEL_LINK_JOBS=1`.
In particular: compiling the zig std lib tests takes about 2G each,
which at 16x at once (8 cores + hyperthreading) is using all 32GB of my
RAM, causing the OOM killer to kill my window manager
The idea here is that you can annotate steps that might use a high
amount of system resources with an upper bound. So for example I could
mark the std lib tests as having an upper bound peak RSS of 3 GiB.
Then the build system will do 2 things:
1. ulimit the child process, so that it will fail if it would exceed
that memory limit.
2. Notice how much system RAM is available and avoid running too many
concurrent jobs at once that would total more than that.
This implements (1) not with an operating system enforced limit, but by
checking the maxrss after a child process exits.
However it does implement (2) correctly.
The available memory used by the build system defaults to the total
system memory, regardless of whether it is used by other processes at
the time of spawning the build runner. This value can be overridden with
the new --maxrss flag to `zig build`. This mechanism will ensure that
the sum total of upper bound RSS memory of concurrent tasks will not
exceed this value.
This system makes it so that project maintainers can annotate
problematic subprocesses, avoiding bug reports from users, who can
blissfully execute `zig build` without worrying about the project's
internals.
Nobody's computer crashes, and the build system uses as much parallelism
as possible without risking OOM. Users do not need to unnecessarily
resort to -j1 when the build system can figure this out for them.
* remove std.Build.updateFile. I noticed some people use it from
build.zig (declare phase) when it is intended only for use in the
make phase.
- This also was incorrectly reporting errors with std.log.
* std.Build.InstallArtifactStep
- report better errors on failure
- report whether the step was cached or not
* std.Build.InstallDirStep: report better error on failure
* std.Build.InstallFileStep: report better error on failure
Rework std.Build.Step to have an `owner: *Build` field. This
simplified the implementation of installation steps, as well as provided
some much-needed common API for the new parallelized build system.
--verbose is now defined very concretely: it prints to stderr just
before spawning a child process.
Child process execution is updated to conform to the new
parallel-friendly make() function semantics.
DRY up the failWithCacheError handling code. It now integrates properly
with the step graph instead of incorrectly dumping to stderr and calling
process exit.
In the main CLI, fix `zig fmt` crash when there are no errors and stdin
is used.
Deleted steps:
* EmulatableRunStep - this entire thing can be removed in favor of a
flag added to std.Build.RunStep called `skip_foreign_checks`.
* LogStep - this doesn't really fit with a multi-threaded build runner
and is effectively superseded by the new build summary output.
build runner:
* add -fsummary and -fno-summary to override the default behavior,
which is to print a summary if any of the build steps fail.
* print the dep prefix when emitting error messages for steps.
std.Build.FmtStep:
* This step now supports exclude paths as well as a check flag.
* The check flag decides between two modes, modify mode, and check
mode. These can be used to update source files in place, or to fail
the build, respectively.
Zig's own build.zig:
* The `test-fmt` step will do all the `zig fmt` checking that we expect
to be done. Since the `test` step depends on this one, we can simply
remove the explicit call to `zig fmt` in the CI.
* The new `fmt` step will actually perform `zig fmt` and update source
files in place.
std.Build.RunStep:
* expose max_stdio_size is a field (previously an unchangeable
hard-coded value).
* rework the API. Instead of configuring each stream independently,
there is a `stdio` field where you can choose between
`infer_from_args`, `inherit`, or `check`. These determine whether the
RunStep is considered to have side-effects or not. The previous
field, `condition` is gone.
* when stdio mode is set to `check` there is a slice of any number of
checks to make, which include things like exit code, stderr matching,
or stdout matching.
* remove the ill-defined `print` field.
* when adding an output arg, it takes the opportunity to give itself a
better name.
* The flag `skip_foreign_checks` is added. If this is true, a RunStep
which is configured to check the output of the executed binary will
not fail the build if the binary cannot be executed due to being for
a foreign binary to the host system which is running the build graph.
Command-line arguments such as -fqemu and -fwasmtime may affect
whether a binary is detected as foreign, as well as system
configuration such as Rosetta (macOS) and binfmt_misc (Linux).
- This makes EmulatableRunStep no longer needed.
* Fix the child process handling to properly integrate with the new
bulid API and to avoid deadlocks in stdout/stderr streams by polling
if necessary.
std.Build.RemoveDirStep now uses the open build_root directory handle
instead of an absolute path.
