Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
There are many different types of Windows paths, and there are a few different possible namespaces on top of that. Before this commit, NT namespaced paths were somewhat supported, and for Win32 paths (those without a namespace prefix), only relative and drive absolute paths were supported. After this commit, all of the following are supported:
- Device namespaced paths (`\\.\`)
- Verbatim paths (`\\?\`)
- NT-namespaced paths (`\??\`)
- Relative paths (`foo`)
- Drive-absolute paths (`C:\foo`)
- Drive-relative paths (`C:foo`)
- Rooted paths (`\foo`)
- UNC absolute paths (`\\server\share\foo`)
- Root local device paths (`\\.` or `\\?` exactly)
Plus:
- Any of the path types and namespace types can be mixed and matched together as appropriate.
- All of the `std.os.windows.*ToPrefixedFileW` functions will accept any path type, prefixed or not, and do the appropriate thing to convert them to an NT-prefixed path if necessary.
This is achieved by making the `std.os.windows.*ToPrefixedFileW` functions behave like `ntdll.RtlDosPathNameToNtPathName_U`, but with a few differences:
- Does not allocate on the heap (this is why we can't use `ntdll.RtlDosPathNameToNtPathName_U` directly, it does internal heap allocation).
- Relative paths are kept as relative unless they contain too many .. components, in which case they are treated as 'drive relative' and resolved against the CWD (this is how it behaved before this commit as well).
- Special case device names like COM1, NUL, etc are not handled specially (TODO)
- `.` and space are not stripped from the end of relative paths (potential TODO)
Most of the non-trivial conversion of non-relative paths is done via `ntdll.RtlGetFullPathName_U`, which AFAIK is used internally by `ntdll.RtlDosPathNameToNtPathName_U`.
Some relevant reading on Windows paths:
- https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html
- https://chrisdenton.github.io/omnipath/Overview.htmlCloses#8205
Might close (untested) #12729
Note:
- This removes checking for illegal characters in `std.os.windows.sliceToPrefixedFileW`, since the previous solution (iterate the whole string and error if any illegal characters were found) was naive and won't work for all path types. This is further complicated by things like file streams (where `:` is used as a delimiter, e.g. `file.ext:stream_name:$DATA`) and things in the device namespace (where a path like `\\.\GLOBALROOT\??\UNC\localhost\C$\foo` is valid despite the `?`s in the path and is effectively equivalent to `C:\foo`). Truly validating paths is complicated and would need to be tailored to each path type. The illegal character checking being removed may open up users to more instances of hitting `OBJECT_NAME_INVALID => unreachable` when using `fs` APIs.
+ This is related to https://github.com/ziglang/zig/issues/15607
The majority of these are in comments, some in doc comments which might
affect the generated documentation, and a few in parameter names -
nothing that should be breaking, however.
Notably the Darwin (XNU) kernel the maxrss field is number of bytes
and not kilobytes (kibibytes) like other platforms (e.g. Linux, BSD).
watchOS and tvOS are not supported because they do not have the ability
to spawn a child process. iOS is enabled but due to OS sandboxing it
should fail with a permission error.
`GetProcessMemoryInfo` is implemented using `NtQueryInformationProcess`
with `ProcessVmCounters` to obtain `VM_COUNTERS`. The structs, enum
definitions are found in `winternl.h` or `ntddk.h` in the latest WDK.
This should give the same results as using `K32GetProcessMemoryInfo`
In Windows, the equivalent to maxrss is PeakWorkingSetSize which is
found in PROCESS_MEMORY_COUNTERS in bytes.
Currently, this is done by calling `GetProcessMemoryInfo` in kernel32.
* Eliminate all uses of `std.debug.print` in make() functions, instead
properly using the step failure reporting mechanism.
* Introduce the concept of skipped build steps. These do not cause the
build to fail, and they do allow their dependants to run.
* RunStep gains a new flag, `skip_foreign_checks` which causes the
RunStep to be skipped if stdio mode is `check` and the binary cannot
be executed due to it being a foreign executable.
- RunStep is improved to automatically use known interpreters to
execute binaries if possible (integrating with flags such as
-fqemu and -fwasmtime). It only does this after attempting a native
execution and receiving a "exec file format" error.
- Update RunStep to use an ArrayList for the checks rather than this
ad-hoc reallocation/copying mechanism.
- `expectStdOutEqual` now also implicitly adds an exit_code==0 check
if there is not already an expected termination. This matches
previously expected behavior from older API and can be overridden by
directly setting the checks array.
* Add `dest_sub_path` to `InstallArtifactStep` which allows choosing an
arbitrary subdirectory relative to the prefix, as well as overriding
the basename.
- Delete the custom InstallWithRename step that I found deep in the
test/ directory.
* WriteFileStep will now update its step display name after the first
file is added.
* Add missing stdout checks to various standalone test case build
scripts.
Today I found out that posix_spawn is trash. It's actually implemented
on top of fork/exec inside of libc (or libSystem in the case of macOS).
So, anything posix_spawn can do, we can do better. In particular, what
we can do better is handle spawning of child processes that are
potentially foreign binaries. If you try to spawn a wasm binary, for
example, posix spawn does the following:
* Goes ahead and creates a child process.
* The child process writes "foo.wasm: foo.wasm: cannot execute binary file"
to stderr (yes, it prints the filename twice).
* The child process then exits with code 126.
This behavior is indistinguishable from the binary being successfully
spawned, and then printing to stderr, and exiting with a failure -
something that is an extremely common occurrence.
Meanwhile, using the lower level fork/exec will simply return ENOEXEC
code from the execve syscall (which is mapped to zig error.InvalidExe).
The posix_spawn behavior means the zig build runner can't tell the
difference between a failure to run a foreign binary, and a binary that
did run, but failed in some other fashion. This is unacceptable, because
attempting to excecve is the proper way to support things like Rosetta.
Previously, this API had pid, to be used on POSIX systems, and handle,
to be used on Windows.
This commit unifies the API, defining an Id type that is either the pid
or the HANDLE depending on the target OS.
This commit also prepares for the future by allowing one to import via
`std.process.Child` which is the fully qualified namespace that I intend
to migrate to in the future.
Previously, the implementation would essentially check `startsWith` instead of `eql` (e.g. it would return true for `.exec` because it erroneously 'matched' `.exe`).
Follow up to #13993
Fixes a regression caused by https://github.com/ziglang/zig/pull/13983
From the added comment:
We still search the path if the cwd is absolute because of the
"cwd set in ChildProcess is in effect when choosing the executable path
to match posix semantics" behavior--we don't want to skip searching
the PATH just because we were trying to set the cwd of the child process.
The name of the game here is to avoid CreateProcessW calls at all costs,
and only ever try calling it when we have a real candidate for execution.
Secondarily, we want to minimize the number of syscalls used when checking
for each PATHEXT-appended version of the app name.
An overview of the technique used:
- Open the search directory for iteration (either cwd or a path from PATH)
- Use NtQueryDirectoryFile with a wildcard filename of `<app name>*` to
check if anything that could possibly match either the unappended version
of the app name or any of the versions with a PATHEXT value appended exists.
- If the wildcard NtQueryDirectoryFile call found nothing, we can exit early
without needing to use PATHEXT at all.
This allows us to use a <open dir, NtQueryDirectoryFile, close dir> sequence
for any directory that doesn't contain any possible matches, instead of having
to use a separate look up for each individual filename combination (unappended +
each PATHEXT appended). For directories where the wildcard *does* match something,
we only need to do a maximum of <number of supported PATHEXT extensions> more
NtQueryDirectoryFile calls.
---
In addition, we now only evaluate the extensions in PATHEXT that we know we can handle (.COM, .EXE, .BAT, .CMD) and ignore the rest.
---
This commit also makes two edge cases match Windows behavior:
- If an app name has the extension .exe and it is attempted to be executed, that is now treated as unrecoverable and InvalidExe is immediately returned no matter where the .exe is (cwd or in the PATH). This matches the behavior of the Windows cmd.exe.
- If the app name contains more than just a filename (e.g. it has path separators), then it is excluded from PATH searching and only does a cwd search. This matches the behavior of Windows cmd.exe.
This matches `cmd.exe` behavior. For example, if there is only a file named `mycommand` in the cwd but it is a Linux executable, then running the command `mycommand` will result in:
'mycommand' is not recognized as an internal or external command, operable program or batch file.
However, if there is *both* a `mycommand` (that is a Linux executable) and a `mycommand.exe` that is a valid Windows exe, then running the command `mycommand` will successfully run `mycommand.exe`.
Avoid a lot of unnecessary utf8 -> utf16 conversion and use a single ArrayList buffer for all the joined paths instead of a separate allocation for each join
For example, if the command is specified as `something.exe`, the retry will now try:
```
C:\some\path\something.exe
C:\some\path\something.exe.COM
C:\some\path\something.exe.EXE
C:\some\path\something.exe.BAT
... etc ...
```
whereas before it would only try the versions with an added extension from `PATHEXT`, which would cause the retry to fail on things that it should find.
When linking libc and compiling natively, Zig tries to integrate with
the system C compiler. However, this caused Zig to fail when no system C
compiler is installed, despite the fact that Zig is perfectly capable of
compiling & linking libc without one.
This commit makes Zig fall back to using its own ability to provide libc
in the case that no C compiler is installed. For glibc, it means
sometimes getting the warning "zig cannot build new glibc version abc,
providing instead xyz".
Ideally, Zig would do some more validation about the system libraries
being linked against, and report an error in case it could not provide
the exact correct libc version of the system libraries (or that the
system libraries themselves conflict with each other), however, I think
it is fair to call that a separate enhancement.
This tight coupling causes problems for various targets, requires
hacky "get args" functionality, and bungles relative file system paths,
making invalid assumptions about the zig-cache directory.
In short, these are not unit tests; these should be standalone tests
instead.
Reverts e5d4a694ea
Reverts d976456ef6
Reverts dbbda0f41aCloses#11542
Instead, just return ChildProcess directly. This structure does not
require a stable address, so we can put it on the stack just fine. If
someone wants it on the heap they should do.
const proc = try allocator.create(ChildProcess);
proc.* = ChildProcess.init(args, allocator);
This reverts commit 7f13f5cd5f.
I'd like to review this one before it goes in. This is an awfully
specific API that I don't think belongs in std.testing. Also I don't
want any code snippets in doc strings. We have doctests for that.
* remove need for manual string concatenation for building binaries in test blocks
* include small program snippet to show how to get binary path with subslicing
