Commit graph

65 commits

Author SHA1 Message Date
mlugg
0fe3fd01dd
std: update std.builtin.Type fields to follow naming conventions
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.

This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
2024-08-28 08:39:59 +01:00
Robin Voetter
43f73af359
fix various issues related to Path handling in the compiler and std
A compilation build step for which the binary is not required could not
be compiled previously. There were 2 issues that caused this:

- The compiler communicated only the results of the emitted binary and
  did not properly communicate the result if the binary was not emitted.

  This is fixed by communicating the final hash of the artifact path (the
  hash of the corresponding /o/<hash> directory) and communicating this
  instead of the entire path. This changes the zig build --listen protocol
  to communicate hashes instead of paths, and emit_bin_path is accordingly
  renamed to emit_digest.

- There was an error related to the default llvm object path when
  CacheUse.Whole was selected. I'm not really sure why this didn't manifest
  when the binary is also emitted.

  This was fixed by improving the path handling related to flush() and
  emitLlvmObject().

In general, this commit also improves some of the path handling throughout
the compiler and standard library.
2024-08-19 19:09:11 +02:00
Andrew Kelley
e64a00950e fuzzer web ui: introduce entry points
so you can have somewhere to start browsing
2024-08-07 00:48:32 -07:00
Andrew Kelley
517cfb0dd1 fuzzing: progress towards web UI
* libfuzzer: close file after mmap
* fuzzer/main.js: connect with EventSource and debug dump the messages.
  currently this prints how many fuzzer runs have been attempted to
  console.log.
* extract some `std.debug.Info` logic into `std.debug.Coverage`.
  Prepares for consolidation across multiple different executables which
  share source files, and makes it possible to send all the
  PC/SourceLocation mapping data with 4 memcpy'd arrays.
* std.Build.Fuzz:
  - spawn a thread to watch the message queue and signal event
    subscribers.
  - track coverage map data
  - respond to /events URL with EventSource messages on a timer
2024-08-07 00:48:32 -07:00
Andrew Kelley
e0ffac4e3c introduce a web interface for fuzzing
* new .zig-cache subdirectory: 'v'
  - stores coverage information with filename of hash of PCs that want
    coverage. This hash is a hex encoding of the 64-bit coverage ID.
* build runner
  * fixed bug in file system inputs when a compile step has an
    overridden zig_lib_dir field set.
  * set some std lib options optimized for the build runner
    - no side channel mitigations
    - no Transport Layer Security
    - no crypto fork safety
  * add a --port CLI arg for choosing the port the fuzzing web interface
    listens on. it defaults to choosing a random open port.
  * introduce a web server, and serve a basic single page application
    - shares wasm code with autodocs
    - assets are created live on request, for convenient development
      experience. main.wasm is properly cached if nothing changes.
    - sources.tar comes from file system inputs (introduced with the
      `--watch` feature)
  * receives coverage ID from test runner and sends it on a thread-safe
    queue to the WebServer.
* test runner
  - takes a zig cache directory argument now, for where to put coverage
    information.
  - sends coverage ID to parent process
* fuzzer
  - puts its logs (in debug mode) in .zig-cache/tmp/libfuzzer.log
  - computes coverage_id and makes it available with
    `fuzzer_coverage_id` exported function.
  - the memory-mapped coverage file is now namespaced by the coverage id
    in hex encoding, in `.zig-cache/v`
* tokenizer
  - add a fuzz test to check that several properties are upheld
2024-08-07 00:48:32 -07:00
Andrew Kelley
a3c74aca99 add --debug-rt CLI arg to the compiler + bonus edits
The flag makes compiler_rt and libfuzzer be in debug mode.

Also:
* fuzzer: override debug logs and disable debug logs for frequently
  called functions
* std.Build.Fuzz: fix bug of rerunning the old unit test binary
* report errors from rebuilding the unit tests better
* link.Elf: additionally add tsan lib and fuzzer lib to the hash
2024-07-25 18:52:21 -07:00
Andrew Kelley
bce3b1efb0 build runner sends a start_fuzzing message to test runner 2024-07-25 18:52:21 -07:00
Andrew Kelley
711ed56ce3 build runner: extract logic to std.Build.Fuzz 2024-07-25 18:52:21 -07:00
Andrew Kelley
047640383e add --fuzz CLI argument to zig build
This flag makes the build runner rebuild unit tests after the pipeline
finishes, if it finds any unit tests.

I did not make this integrate with file system watching yet.

The test runner is updated to detect which tests are fuzz tests.

Run step is updated to track which test indexes are fuzz tests.
2024-07-25 18:52:20 -07:00
Alex Rønne Petersen
be1e1fa180 std.Build.Step.Run: Fix invocation syntax for Wasmtime 14+.
https://github.com/bytecodealliance/wasmtime/issues/7384
2024-07-23 12:39:40 -07:00
Andrew Kelley
33c7984183 add std.testing.random_seed
closes #17609
2024-07-23 11:43:12 -07:00
Andrew Kelley
eb4028bf30 add std.fmt.hex
converts an unsigned integer into an array
2024-07-20 01:06:29 -07:00
Andrew Kelley
abf8955951 make zig compiler processes live across rebuilds
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.

Also adds global zig cache to the compiler cache prefixes.

Closes #20600
2024-07-14 19:51:16 -07:00
Andrew Kelley
1d20ff11d7
Merge pull request #20580 from ziglang/watch
introduce file system watching features to the zig build system
2024-07-12 16:56:17 -07:00
xtex
0d79aa0176
std.Build.Step.Run: support prefixed artifact args
Just like how addPrefixedFileArg and addPrefixedDirectoryArg works, we can make it for artifacts.

Signed-off-by: Bingwu Zhang <xtexchooser@duck.com>
2024-07-12 11:06:06 +02:00
Andrew Kelley
2e42969786 std.Build.Step.Run: integrate with --watch 2024-07-12 00:14:08 -07:00
Krzysztof Wolicki
6327a3994a
std.Build: remove deprecated b.host in favor of b.graph.host (#20237) 2024-06-08 12:22:38 -07:00
Andrew Kelley
4918c2ce2d std.Build.Step.Run: global lock when child inherits stderr
The docs for setting stdio to "inherit" say:

It also means that this step will obtain a global lock to prevent other
steps from running in the meantime.

The implementation of this lock was missing but is now provided by this
commit.

closes #20119
2024-05-31 18:59:53 -04:00
Andrew Kelley
a5e4fe5487 std.Build.Step.Run: account for new environment variable
Introduces `disable_zig_progress` which prevents the build runner from
assigning the child process a progress node.

This is needed for the empty_env test which requires the environment to
be completely empty.
2024-05-27 20:56:49 -07:00
Andrew Kelley
9331da8fe3 std.Build.Step.Run: don't create empty progress node 2024-05-27 20:56:48 -07:00
Andrew Kelley
f97c2f28fd update the codebase for the new std.Progress API 2024-05-27 20:56:48 -07:00
Jacob Young
ad72172293 Build.Step.Run: fix cache management when there are side effects
Closes #19947
2024-05-20 07:45:31 -04:00
Andrew Kelley
661a9be66b std.Build.Step.Run: add missing call to writeManifest
fixes #19938
2024-05-11 06:28:45 -07:00
Jacob Young
dee9f82f69 Run: add output directory arguments
This allows running commands that take an output directory argument. The
main thing that was needed for this feature was generated file subpaths,
to allow access to the files in a generated directory. Additionally, a
minor change was required to so that the correct directory is created
for output directory args.
2024-05-05 15:58:08 -04:00
Jacob Young
e3424332d3 Build: cleanup
* `doc/langref` formatting
 * upgrade `.{ .path = "..." }` to `b.path("...")`
 * avoid using arguments named `self`
 * make `Build.Step.Id` usage more consistent
 * add `Build.pathResolve`
 * use `pathJoin` and `pathResolve` everywhere
 * make sure `Build.LazyPath.getPath2` returns an absolute path
2024-05-05 09:42:51 -04:00
Jacob Young
d582575aba Run: add lazy path file inputs
This replaces `extra_file_dependencies` with support for lazy paths.

Also assert output file basenames are not empty, avoid improper use of
field default values, ensure stored strings are duplicated, and
prefer `ArrayListUnmanaged` to discourage misuse of direct field access
which wouldn't add step dependencies.
2024-05-05 09:42:51 -04:00
Ryan Liptak
b86c4bde64 Rename Dir.writeFile2 -> Dir.writeFile and update all callsites
writeFile was deprecated in favor of writeFile2 in f645022d16. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-03 13:29:22 -07:00
Jacob Young
eb723a4070 Update uses of @fieldParentPtr to use RLS 2024-03-30 20:50:48 -04:00
Jacob Young
e409afb79b Update uses of @fieldParentPtr to pass a pointer type 2024-03-30 20:50:48 -04:00
Andrew Kelley
c52a2c338d std.Build.Step.Run: documentation for addFileArg and friends
Someone on IRC got these functions mixed up, and it sure would have been
helpful to have these docs in the first place, eh?
2024-03-18 23:52:52 -07:00
Nguyễn Gia Phong
f13401ac69 Fix stuttering stderr in Zig test log
Before this fix, the stderr FIFO was advanced by the length
of the trimmed message, thus the next error log contained
the tail of that message.
2024-03-15 16:45:58 +02:00
Andrew Kelley
54bbc73f85
Merge pull request #18712 from Vexu/std.options
std: make options a struct instance instead of a namespace
2024-02-09 13:38:42 -08:00
Andrew Kelley
370438943e std.Build: revert moving some fields to Graph
On second thought, let's keep a bunch of these flags how they already
were.

Partial revert of the previous commit.
2024-02-02 20:43:01 -07:00
Andrew Kelley
105db13536 std.Build: implement --host-target, --host-cpu, --host-dynamic-linker
This also makes a long-overdue change of extracting common state from
Build into a shared Graph object.

Getting the semantics right for these flags turned out to be quite
tricky. In the end it works like this:
* The override only happens when the target is fully native, with no
  additional query parameters, such as versions or CPU features added.
* The override affects the resolved Target but leaves the original Query
  unmodified.
* The "is native?" detection logic operates on the original, unmodified
  query. This makes it possible to provide invalid host target
  information, causing confusing errors to occur. Don't do that.

There are some minor breaking changes to std.Build API such as the fact
that `b.zig_exe` is now moved to `b.graph.zig_exe`, as well as a handful
of other similar flags.
2024-02-02 20:43:01 -07:00
Veikka Tuominen
a4f27e8987 remove std.io.Mode 2024-02-01 15:22:36 +02:00
Andrew Kelley
45ec851733 zig build: handle stderr more elegantly
* Specifically recognize stderr as a different concept than an error
  message in Step results.
* Display it differently when only stderr occurs but the build proceeds
  successfully.

closes #18473
2024-01-10 17:11:26 -08:00
Andrew Kelley
f64205b445
Merge pull request #18262 from ziglang/fix-18259
std.Build.Step.Run: fix depfile support
2024-01-03 13:23:40 -08:00
Loris Cro
65878c16ee std.Build.Step.Run: fix depfile support 2024-01-02 11:54:32 +01:00
Andrew Kelley
b92e30ff0b std.Build.ResolvedTarget: rename target field to result
This change is seemingly insignificant but I actually agonized over this
for three days. Some other things I considered:

* (status quo in master branch) make Compile step creation functions
  accept a Target.Query and delete the ResolvedTarget struct.
  - downside: redundantly resolve target queries many times
* same as before but additionally add a hash map to cache target query
  resolutions.
  - downside: now there is a hash map that doesn't actually need to
    exist, just to make the API more ergonomic.
* add is_native_os and is_native_abi fields to std.Target and use it
  directly as the result of resolving a target query.
  - downside: they really don't belong there. They would be available
    as comptime booleans via `@import("builtin")` but they should not
    be exposed that way.

With this change the downsides are:
* the option name of addExecutable and friends is `target` instead of
  `resolved_target` matching the type name.
  - upside: this does not break compatibility with existing build
    scripts
* you likely end up seeing `target.result.cpu.arch` rather than
  `target.cpu.arch`.
  - upside: this is an improvement over `target.target.cpu.arch` which
    it was before this commit.
  - downside: `b.host.target` is now `b.host.result`.
2024-01-01 17:51:18 -07:00
Andrew Kelley
dbdb87502d std.Target: add DynamicLinker 2024-01-01 17:51:18 -07:00
Andrew Kelley
767eb5da77 fix infinite loop in Run step
Previously this relied on recursion; now it relies on the recursive
iteration which is better, but means the recursive call needs to be
deleted.
2024-01-01 17:51:18 -07:00
Andrew Kelley
04480f72d8 fix linker test regressions
Caused by problems with transitive dependencies
2024-01-01 17:51:18 -07:00
Andrew Kelley
142471fcc4 zig build system: change target, compilation, and module APIs
Introduce the concept of "target query" and "resolved target". A target
query is what the user specifies, with some things left to default. A
resolved target has the default things discovered and populated.
In the future, std.zig.CrossTarget will be rename to std.Target.Query.
Introduces `std.Build.resolveTargetQuery` to get from one to the other.

The concept of `main_mod_path` is gone, no longer supported. You have to
put the root source file at the module root now.

* remove deprecated API
* update build.zig for the breaking API changes in this branch
* move std.Build.Step.Compile.BuildId to std.zig.BuildId
* add more options to std.Build.ExecutableOptions, std.Build.ObjectOptions,
  std.Build.SharedLibraryOptions, std.Build.StaticLibraryOptions, and
  std.Build.TestOptions.
* remove `std.Build.constructCMacro`. There is no use for this API.
* deprecate `std.Build.Step.Compile.defineCMacro`. Instead,
  `std.Build.Module.addCMacro` is provided.
  - remove `std.Build.Step.Compile.defineCMacroRaw`.
* deprecate `std.Build.Step.Compile.linkFrameworkNeeded`
  - use `std.Build.Module.linkFramework`
* deprecate `std.Build.Step.Compile.linkFrameworkWeak`
  - use `std.Build.Module.linkFramework`
* move more logic into `std.Build.Module`
* allow `target` and `optimize` to be `null` when creating a Module.
  Along with other fields, those unspecified options will be inherited
  from parent `Module` when inserted into an import table.
* the `target` field of `addExecutable` is now required. pass `b.host`
  to get the host target.
2024-01-01 17:51:18 -07:00
Andrew Kelley
c20ad51c62 introduce std.Build.Module and extract some logic into it
This moves many settings from `std.Build.Step.Compile` and into
`std.Build.Module`, and then makes them transitive.

In other words, it adds support for exposing Zig modules in packages,
which are configured in various ways, such as depending on other link
objects, include paths, or even a different optimization mode.

Now, transitive dependencies will be included in the compilation, so you
can, for example, make a Zig module depend on some C source code, and
expose that Zig module in a package.

Currently, the compiler frontend autogenerates only one
`@import("builtin")` module for the entire compilation, however, a
future enhancement will be to make it honor the differences in modules,
so that modules can be compiled with different optimization modes, code
model, valgrind integration, or even target CPU feature set.

closes #14719
2024-01-01 17:51:18 -07:00
mlugg
51595d6b75
lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
Jan Philipp Hafer
fd2239bde9 child_process + Build: rename exec to run + all related code
Justification: exec, execv etc are unix concepts and portable version
should be called differently.

Do no touch non-Zig code. Adjust error names as well, if associated.
Closes #5853.
2023-10-22 14:47:20 -04:00
Krzysztof Wolicki
7edba14d7c
Step.Run: change cwd to ?Build.LazyPath (#17418)
closes #17409
2023-10-10 18:39:44 -04:00
Jakub Konka
df9462690f std: fix memory bug in getExternalExecutor
Until now, we would pass `candidate: NativeTargetInfo` which creates
a copy of the `NativeTargetInfo.DynamicLinker` buffer. We would then
return this buffer in `bad_dl: []const u8` which would goes out-of-scope
the moment we leave this function frame yielding garbage. To fix this,
we just need to remember to pass by const-pointer
`candidate: *const NativeTargetInfo`.
2023-10-06 12:43:00 +02:00
Krzysztof Wolicki
e919fbea9f
Step.Run: fix assert of the wrong value (#17303)
closes #16866
2023-09-29 14:14:42 -04:00
Jacob Young
750998eef6 Build: fail tests that log errors, like zig test does 2023-08-25 15:36:25 -07:00