Instead of always using std.testing.allocator, the test harness now follows
the same logic as self-hosted for choosing an allocator - that is - it
uses C allocator when linking libc, std.testing.allocator otherwise, and
respects `-Dforce-gpa` to override the decision. I did this because
I found GeneralPurposeAllocator to be prohibitively slow when doing
multi-threading, even in the context of a debug build.
There is now a second thread pool which is used to spawn each
test case. The stage2 tests are passed the first thread pool. If it were
only multi-threading the stage1 tests then we could use the same thread
pool for everything. However, the problem with this strategy with stage2
is that stage2 wants to spawn tasks and then call wait() on the main
thread. If we use the same thread pool for everything, we get a deadlock
because all the threads end up all hanging at wait() and nothing is
getting done. So we use our second thread pool to simulate a "process pool"
of sorts.
I spent most of the time working on this commit scratching my head trying
to figure out why I was getting ETXTBSY when spawning the test cases.
Turns out it's a fundamental Unix design flaw, already a known, unsolved
issue by Go and Java maintainers:
https://github.com/golang/go/issues/22315https://bugs.openjdk.org/browse/JDK-8068370
With this change, the following command, executed on my laptop, went from
6m24s to 1m44s:
```
stage1/bin/zig build test-cases -fqemu -fwasmtime -Denable-llvm
```
closes#11818
* migrate runtime safety tests to the new test harness
- this required adding compare output / execution support for stage1
to the test harness.
* rename `zig build test-stage2` to `zig build test-cases` since it now
does quite a bit of stage1 testing actually. I named it this way
since the main directory in the source tree associated with these
tests is "test/cases/".
* add some documentation for the test manifest format.
stage2: change logic for detecting whether the main package is inside
the std package. Previously it relied on realpath() which is not portable.
This uses resolve() which is how imports already work.
* stage2: fix cleanup bug when creating Module
* flatten lib/std/special/* to lib/*
- this was motivated by making main_pkg_is_inside_std false for
compiler_rt & friends.
* rename "mini libc" to "universal libc"
We can't yet run the behavior tests with stage3, but at least we can run
them with stage2, and we can use the proper test matrix.
This commit also adds use_llvm and ofmt to the zig build system.
* `-Dskip-compile-errors` is removed; `-Dskip-stage1` is added.
* Use `std.testing.allocator` instead of a new instance of GPA.
- Fix the memory leaks this revealed.
* Show the file name when it is not parsed correctly such as when the
manifest is missing.
- Better error messages when test files are not parsed correctly.
* Ignore unknown files such as swap files.
* Move logic from declarative file to the test harness implementation.
* Move stage1 tests to stage2 tests where appropriate.
using the provided -L directories before checking if we need to integrate
with system library paths. This prevents false positive of invoking
system cc to find native paths when in fact all dependencies are
satisfied by -L (or --search-prefix to zig build).
* build.zig: remove detour through zig0. I'll add it back as an option
later. This means you can build stage1 with a freshly built zig
without detouring through zig0 again.
* remove unused file windows_script.bat
* drone.yml: remove unnecessary steps
* ninja doesn't need a jobs parameter
* build the release version of zig with zig build instead of cmake.
* build stage2 with -Dstatic-llvm -Duse-zig-libcxx
Instead of assuming that the zig version used with `zig build` is
appropriate for building the self-hosted compiler component, we follow
the same path as the cmake build script and build zig0, using that to
produce zig1.o, re-linking to produce stage1.
This allows an arbitrarily old Zig version and corresponding build.zig
script to build all future versions of Zig from source via the bootstrap
compiler. In other words, it allows us to use `zig build` as an
alternative to cmake when bootstrapping.
In accordance with the requesting issue (#10750):
- `zig test` skips any tests that it cannot spawn, returning success
- `zig run` and `zig build` exit with failure, reporting the command the cannot be run
- `zig clang`, `zig ar`, etc. already punt directly to the appropriate clang/lld main(), even before this change
- Native `libc` Detection is not supported
Additionally, `exec()` and related Builder functions error at run-time, reporting the command that cannot be run
Instead of defaulting to false, just keep the option as optional to
communicate default to the build system.
Fixes one problem with building the compiler for single-threaded
targets.
An attempt to normalize some of the function names in build.zig. Normalize add*Dir to add*Path. Also use "Library" instead of the "Lib" abbreviation.
The PR does not remove the old names, only adds the new normalized ones to faciliate a transition period.
Replaces the inflate API from `inflateStream(reader: anytype, window_slice: []u8)` to
`decompressor(allocator: mem.Allocator, reader: anytype, dictionary: ?[]const u8)` and
`compressor(allocator: mem.Allocator, writer: anytype, options: CompressorOptions)`
from zig-specific options to generally recognized zig build options that
any project can take advantage of. See the updated usage text for more
details.
Each element of the output JSON has the VM address of the generated
binary nondecreasing (some elements might occupy the same VM address
for example the atom and the relocation might coincide in the address
space).
The generated JSON can be inspected manually or via a preview tool
`zig-snapshots` that I am currently working on and will allow the user
to inspect interactively the state of the linker together with the
positioning of sections, symbols, atoms and relocations within each
snapshot state, and in the future, between snapshots too. This should
allow for quicker debugging of the linker which is nontrivial when
run in the incremental mode.
Note that the state will only be dumped if the compiler is built with
`-Dlink-snapshot` flag on, and then the compiler is passed `--debug-link-snapshot`
flag upon compiling a source/project.
This supports the case when it is known that LLVM, Clang, LLD were built
with Clang (or `zig c++`). This commit updates the Linux CI script to
pass this since we build using a zig tarball.
There was some new code in master branch enumerating all the targets and
a new target was added so we needed to add the glue code.
This commit also introduces some build options to support experimental
LLVM targets.
* rework `build_options` to integrate with the FileSource abstraction
* support mapping as an arbitrarily named package
* support mapping to multiple different artifacts
* use hash of contents for options filename
This way, we can explicitly signal if a test requires the presence
of macOS SDK to build. For instance, when testing our in-house
MachO linker for correctly linking Objective-C, we require the
presence of the SDK on the host system, and we can enforce this
with `-Denable-macos-sdk` flag to `zig build test-standalone`.
The motivation for this commit is that there exists source files which
produce ast-check errors, but crash stage1 or otherwise trigger stage1
bugs. Previously to this commit, Zig would run AstGen, collect the
compile errors, run stage1, report stage1 compile errors and exit if
any, and then report AstGen compile errors.
The main change in this commit is to report AstGen errors prior to
invoking stage1, and in fact if any AstGen errors occur, do not invoke
stage1 at all.
This caused most of the compile error tests to fail due to things such
as unused local variables and mismatched stage1/stage2 error messages.
It was taking a long time to update the test cases one-by-one, so I
took this opportunity to unify the stage1 and stage2 testing harness,
specifically with regards to compile errors. In this way we can start
keeping track of which tests pass for 1, 2, or both.
`zig build test-compile-errors` no longer works; it is now integrated
into `zig build test-stage2`.
This is one step closer to executing compile error tests in parallel; in
fact the ThreadPool object is already in scope.
There are some cases where the stage1 compile errors were actually
better; those are left failing in this commit, to be addressed in a
follow-up commit.
Other changes in this commit:
* build.zig: improve support for -Dstage1 used with the test step.
* AstGen: minor cosmetic changes to error messages.
* stage2: add -fstage1 and -fno-stage1 flags. This now allows one to
download a binary of the zig compiler and use the llvm backend of
self-hosted. This was also needed for hooking up the test harness.
However, I realized that stage1 calls exit() and also has memory
leaks, so had to complicate the test harness by not using this flag
after all and instead invoking as a child process.
- These CLI flags will disappear once we start shipping the
self-hosted compiler as the main compiler. Until then, they can be
used to try out the work-in-progress stage2.
* stage2: select the LLVM backend by default for release modes, as long
as the target architecture is supported by LLVM.
* test harness: support setting the optimize mode
Before this change, when one or more of name or value are not known at
comptime, build.zig files must allocate and do the concatanation, which can be
cumbersome, and also adds a redundant allocation when name and value are
slices. The new version only does a single allocation directly in the builder's
allocator to concatonate name and value.
The origional behavior is available in defineCMacroRaw, for use in situations
such as parseing c compiler arguments.
Additionally, several places have been updated to use the new funtions.
