Commit graph

256 commits

Author SHA1 Message Date
Andrew Kelley
e0ffac4e3c introduce a web interface for fuzzing
* new .zig-cache subdirectory: 'v'
  - stores coverage information with filename of hash of PCs that want
    coverage. This hash is a hex encoding of the 64-bit coverage ID.
* build runner
  * fixed bug in file system inputs when a compile step has an
    overridden zig_lib_dir field set.
  * set some std lib options optimized for the build runner
    - no side channel mitigations
    - no Transport Layer Security
    - no crypto fork safety
  * add a --port CLI arg for choosing the port the fuzzing web interface
    listens on. it defaults to choosing a random open port.
  * introduce a web server, and serve a basic single page application
    - shares wasm code with autodocs
    - assets are created live on request, for convenient development
      experience. main.wasm is properly cached if nothing changes.
    - sources.tar comes from file system inputs (introduced with the
      `--watch` feature)
  * receives coverage ID from test runner and sends it on a thread-safe
    queue to the WebServer.
* test runner
  - takes a zig cache directory argument now, for where to put coverage
    information.
  - sends coverage ID to parent process
* fuzzer
  - puts its logs (in debug mode) in .zig-cache/tmp/libfuzzer.log
  - computes coverage_id and makes it available with
    `fuzzer_coverage_id` exported function.
  - the memory-mapped coverage file is now namespaced by the coverage id
    in hex encoding, in `.zig-cache/v`
* tokenizer
  - add a fuzz test to check that several properties are upheld
2024-08-07 00:48:32 -07:00
Carter Snook
219acaa1d6 std.fs.Dir: Refactor atomicSymLink from std.fs
Deprecates std.fs.atomicSymLink and removes the allocator requirement
from the new std.fs.Dir.atomicSymLink. Replaces the two usages of this
within std.

I did not include the TODOs from the original code that were based
off of `switch (err) { ..., else => return err }` not having correct
inference that cases handled in `...` are impossible in the error
union return type because these are not specified in many places but
I can add them back if wanted.

Thank you @squeek502 for help with fixing buffer overflows!
2024-07-28 20:08:52 -07:00
Andrew Kelley
a3c74aca99 add --debug-rt CLI arg to the compiler + bonus edits
The flag makes compiler_rt and libfuzzer be in debug mode.

Also:
* fuzzer: override debug logs and disable debug logs for frequently
  called functions
* std.Build.Fuzz: fix bug of rerunning the old unit test binary
* report errors from rebuilding the unit tests better
* link.Elf: additionally add tsan lib and fuzzer lib to the hash
2024-07-25 18:52:21 -07:00
Andrew Kelley
bce3b1efb0 build runner sends a start_fuzzing message to test runner 2024-07-25 18:52:21 -07:00
Andrew Kelley
711ed56ce3 build runner: extract logic to std.Build.Fuzz 2024-07-25 18:52:21 -07:00
Andrew Kelley
047640383e add --fuzz CLI argument to zig build
This flag makes the build runner rebuild unit tests after the pipeline
finishes, if it finds any unit tests.

I did not make this integrate with file system watching yet.

The test runner is updated to detect which tests are fuzz tests.

Run step is updated to track which test indexes are fuzz tests.
2024-07-25 18:52:20 -07:00
Andrew Kelley
26d2a7960e default "trace pc guard" coverage off
* Add -f(no-)sanitize-coverage-trace-pc-guard CLI flag which defaults to
  off. This value lowers to TracePCGuard = true (LLVM backend) and -Xclang
  -fsanitize-coverage-trace-pc-guard. These settings are not
  automatically included with -ffuzz.
* Add `Build.Step.Compile` flag for sanitize_coverage_trace_pc_guard
  with appropriate documentation.
* Add `zig cc` integration for the respective flags.
* Avoid crashing in ELF linker code when -ffuzz -femit-llvm-ir used
  together.
2024-07-23 17:30:53 -07:00
Alex Rønne Petersen
be1e1fa180 std.Build.Step.Run: Fix invocation syntax for Wasmtime 14+.
https://github.com/bytecodealliance/wasmtime/issues/7384
2024-07-23 12:39:40 -07:00
Andrew Kelley
33c7984183 add std.testing.random_seed
closes #17609
2024-07-23 11:43:12 -07:00
Eric Joldasov
d263f1ec0e zig build: respect PKG_CONFIG environment variable
`PKG_CONFIG` environment variable is used to override path to
pkg-config executable, for example when it's name is prepended by
target triple for cross-compilation purposes:

```
PKG_CONFIG=/usr/bin/aarch64-unknown-linux-gnu-pkgconf zig build
```

Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
2024-07-21 02:32:33 -07:00
Andrew Kelley
eb4028bf30 add std.fmt.hex
converts an unsigned integer into an array
2024-07-20 01:06:29 -07:00
Andrew Kelley
716b128a24 frontend: add -fincremental, -fno-incremental flag
Remove --debug-incremental

This flag is also added to the build system. Importantly, this tells
Compile step whether or not to keep the compiler running between
rebuilds. It defaults off because it is currently crashing
zirUpdateRefs.
2024-07-14 21:18:09 -07:00
Andrew Kelley
abf8955951 make zig compiler processes live across rebuilds
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.

Also adds global zig cache to the compiler cache prefixes.

Closes #20600
2024-07-14 19:51:16 -07:00
kcbanner
373e53d7c5 Compile: Pass the default --zig-lib-dir along to child processes
main: print the self_exe_path when `findZigLibDirFromSelfExe` fails in all cases
2024-07-13 19:04:55 -04:00
Andrew Kelley
1d20ff11d7
Merge pull request #20580 from ziglang/watch
introduce file system watching features to the zig build system
2024-07-12 16:56:17 -07:00
Andrew Kelley
4f9a8b6843 update build system unit test
need to add another field to initialize now
2024-07-12 11:00:52 -07:00
xtex
0d79aa0176
std.Build.Step.Run: support prefixed artifact args
Just like how addPrefixedFileArg and addPrefixedDirectoryArg works, we can make it for artifacts.

Signed-off-by: Bingwu Zhang <xtexchooser@duck.com>
2024-07-12 11:06:06 +02:00
Andrew Kelley
818f9cb5a0 std.Build.Step.ObjCopy: remove random bytes from cache hash
The cache hash already has the zig version in there, so it's not really
needed.
2024-07-12 00:14:08 -07:00
Andrew Kelley
2e42969786 std.Build.Step.Run: integrate with --watch 2024-07-12 00:14:08 -07:00
Andrew Kelley
6fcb1897d2 std.Build.Step.WriteFile: remove random bytes from cache hash
The cache hash already has the zig version in there, so it's not really
needed.
2024-07-12 00:14:08 -07:00
Andrew Kelley
a966eee090 std.Build.Step.WriteFile: fix handling of directories
and add file system watching integration.

`addDirectoryWatchInput` now returns a `bool` which helps remind the
caller to
1. call addDirectoryWatchInputFromPath on any derived paths
2. but only if the dependency is not already captured by a step
   dependency edge.

The make function now recursively walks all directories and adds the
found files to the cache hash rather than incorrectly only adding the
directory name to the cache hash.

closes #20571
2024-07-12 00:14:08 -07:00
Andrew Kelley
5c3fae3a32 td.Build.Step.InstallDir: leave hint for wrong cached status
Since I spent a couple minutes debugging this, hopefully this saves
someone some future trouble doing the same.
2024-07-12 00:14:08 -07:00
Andrew Kelley
7bccef3e4e std.Build.Watch: introduce special file "." to watch entire dir
And use it to implement InstallDir Step watch integration.

I'm not seeing any events triggered when I run `mkdir` in the watched
directory, however, and I have not yet figured out why.
2024-07-12 00:14:08 -07:00
Andrew Kelley
d1c14f2f52 std.Build.Step.WriteFile: extract UpdateSourceFiles
This has been planned for quite some time; this commit finally does it.

Also implements file system watching integration in the make()
implementation for UpdateSourceFiles and fixes the reporting of step
caching for both.

WriteFile does not yet have file system watching integration.
2024-07-12 00:14:08 -07:00
Andrew Kelley
0cc492a272 make more build steps integrate with the watch system 2024-07-12 00:14:08 -07:00
Andrew Kelley
bbd90a562e build runner: implement --watch (work-in-progress)
I'm still learning how the fanotify API works but I think after playing
with it in this commit, I finally know how to implement it, at least on
Linux. This commit does not accomplish the goal but I want to take the
code in a different direction and still be able to reference this point
in time by viewing a source control diff.

I think the move is going to be saving the file_handle for the parent
directory, which combined with the dirent names is how we can correlate
the events back to the Step instances that have registered file system
inputs. I predict this to be similar to implementations on other
operating systems.
2024-07-12 00:14:07 -07:00
Andrew Kelley
26d506c0f8 std.Build: remove the "push installed file" mechanism
Tracked by #14943
2024-07-12 00:14:07 -07:00
Andrew Kelley
6e025fc2e2 build system: add --watch flag and report source file in InstallFile
This direction is not quite right because it mutates shared state in a
threaded context, so the next commit will need to fix this.
2024-07-12 00:14:07 -07:00
Krzysztof Wolicki
7205756a69 Step.TranslateC: fix defineCMacro 2024-07-07 12:28:28 +02:00
Jonathan Marler
d9f1a952b8 build: fix WriteFile and addCSourceFiles not adding LazyPath deps
Adds a missing call to addLazyPathDependenciesOnly in
std.Build.Module.addCSourceFiles.  Also fixes an issue in
std.Build.Step.WriteFile where it wasn't updating all the GeneratedFile
instances for every directory.  To fix the second issue, I removed
all the GeneratedFile instances and now all files/directories reference
the steps main GeneratedFile via sub paths.
2024-07-04 13:34:17 -04:00
Sashko
604a332e21 Extract getZigArgs function in std.Build.Step.Compile 2024-07-01 16:36:13 -07:00
Michael Bradshaw
642093e04b Rename *[UI]LEB128 functions to *[UI]leb128 2024-06-23 04:30:12 +01:00
Andrew Kelley
fda2458f6a
Merge pull request #20321 from ziglang/build-system-fmt
enhance `std.Build.Step.Fmt` and use it more
2024-06-17 01:04:57 -04:00
Alex Rønne Petersen
6220c754cf std.Build: Add Step.Fail and addFail() function.
Closes #15373.
2024-06-16 23:13:12 -04:00
Andrew Kelley
eb8201afde std.Build.Step.Fmt: display non-conforming files
When in --check mode, and files are found to not conform, emit them
explicitly as step errors. Previously this stdout data was being
ignored.
2024-06-16 19:30:14 -07:00
Krzysztof Wolicki
6327a3994a
std.Build: remove deprecated b.host in favor of b.graph.host (#20237) 2024-06-08 12:22:38 -07:00
MrDmitry
84d1580873 Report error on missing values for addConfigHeader 2024-06-08 15:18:56 -04:00
wooster0
aef66eebe3 objcopy build step: don't accept multiple sections
The actual `zig objcopy` does not accept keeping multiple sections.
If you pass multiple `-j .section` arguments to `zig objcopy`, it will
only respect the last one passed.

Originally I changed `zig objcopy` to accept multiple sections
and then concatenate them instead of returning after outputting the
first section (see emitElf) but I realized concatenating probably doesn't make sense.
2024-06-05 19:39:43 -04:00
Andrew Kelley
8f27fdb84e
Merge pull request #20120 from vahur/move-consts-to-rdata
mark anondecls as constants in llvm ir
2024-06-05 14:24:00 -04:00
Veikka Tuominen
c01d8c8b20 pass -fno-builtin when testing lib/c.zig, lib/compiler_rt.zig 2024-06-02 10:10:35 +03:00
Andrew Kelley
4918c2ce2d std.Build.Step.Run: global lock when child inherits stderr
The docs for setting stdio to "inherit" say:

It also means that this step will obtain a global lock to prevent other
steps from running in the meantime.

The implementation of this lock was missing but is now provided by this
commit.

closes #20119
2024-05-31 18:59:53 -04:00
Linus Groh
b5e99b5295 std.Build.step.Compile: Fix lib{c,cpp} mixup in dependsOnSystemLibrary() 2024-05-28 20:39:42 -04:00
Andrew Kelley
a5e4fe5487 std.Build.Step.Run: account for new environment variable
Introduces `disable_zig_progress` which prevents the build runner from
assigning the child process a progress node.

This is needed for the empty_env test which requires the environment to
be completely empty.
2024-05-27 20:56:49 -07:00
Andrew Kelley
9331da8fe3 std.Build.Step.Run: don't create empty progress node 2024-05-27 20:56:48 -07:00
Andrew Kelley
f97c2f28fd update the codebase for the new std.Progress API 2024-05-27 20:56:48 -07:00
Jakub Konka
8e52d54c25 Step/CheckObject: handle stab entries when dumping MachO symtab 2024-05-24 22:32:14 +02:00
Jacob Young
ad72172293 Build.Step.Run: fix cache management when there are side effects
Closes #19947
2024-05-20 07:45:31 -04:00
Jakub Konka
6a65561e3e
Merge pull request #19958 from ziglang/macho-check-object
Step.CheckObject: add support for dumping raw section contents for MachO
2024-05-14 07:11:41 +02:00
Jakub Konka
2d537da6bd Step.CheckObject: support dumping raw section contents for MachO 2024-05-13 09:14:35 +02:00
Jakub Konka
5d619da2de Step.CheckObject: put MachO object parsing logic in ObjectContext wrapper 2024-05-13 08:56:15 +02:00