* define std.crypto.sha2.Sha512224
* rename blunder
* add sha512-224 and sha512-256 tests
* fix Sha2x64 for variations that aren't a multiple of 64 bits
Also removes the LOCK namespace from std.c.wasi because wasi libc does
not have flock.
closes#19336
related to #19352
Co-authored-by: Ryan Liptak <squeek502@hotmail.com>
This is a partial revert of 105db13536.
As we learned from Void Linux packaging, these options are not actually
helpful since the distribution package manager may very well want to
cross-compile the packages that it is building.
So, let's not overcomplicate things. There are already the standard
options: -Dtarget, -Dcpu, and -Ddynamic-linker.
These options are generally provided when the project generates machine
code artifacts, however, there may be a project that does no such thing,
in which case it makes sense for these options to be missing. The Zig
Build System is a general-purpose build system, after all.
When building zig natively from source on an RK3588 SoC board, the
generated stage3 compiler will use unsupported 'sha256h.4s' instructions
due to mis-identified CPU features. This happens when trying to
compile a new project generated with "zig init"
$ ~/devel/zig/build/stage3/bin/zig build
thread 919 panic: Illegal instruction at address 0x1fdc0c4
/home/dliviu/devel/zig/lib/std/crypto/sha2.zig:223:29: 0x1fdc0c4 in round (zig)
asm volatile (
^
/home/dliviu/devel/zig/lib/std/crypto/sha2.zig:168:20: 0x1fdca87 in final (zig)
d.round(&d.buf);
^
/home/dliviu/devel/zig/lib/std/crypto/sha2.zig:180:20: 0x1c8bb33 in finalResult (zig)
d.final(&result);
^
/mnt/home/dliviu/devel/zig/src/Package/Fetch.zig:754:49: 0x1a3a8eb in relativePathDigest (zig)
return Manifest.hexDigest(hasher.finalResult());
^
/mnt/home/dliviu/devel/zig/src/main.zig:5128:53: 0x1a37413 in cmdBuild (zig)
Package.Fetch.relativePathDigest(build_mod.root, global_cache_directory),
^
???:?:?: 0xff09ffffffffffff in ??? (???)
Unwind information for `???:0xff09ffffffffffff` was not available, trace may be incomplete
???:?:?: 0x3f7f137 in ??? (???)
Aborted (core dumped)
system/linux.zig parses "/proc/cpuinfo" to determine the CPU features, but it
seems to generate the wrong set of features from:
$ cat /proc/cpuinfo
processor : 0
BogoMIPS : 48.00
Features : fp asimd evtstrm crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x2
CPU part : 0xd05
CPU revision : 0
.....
processor : 4
BogoMIPS : 48.00
Features : fp asimd evtstrm crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x4
CPU part : 0xd0b
CPU revision : 0
.....
To fix this, use the Linux kernel way of reading the feature registers as documented
here: https://www.kernel.org/doc/html/latest/arch/arm64/cpu-feature-registers.html
arm.zig already has the code to parse the feature register values, we just need to
collect them in an array and pass them for identification.
Signed-off-by: Liviu Dudau <liviu@dudau.co.uk>
This commit reverts the handling of partially-undefined values in
bitcasting to transform these bits into an arbitrary numeric value,
like happens on `master` today.
As @andrewrk rightly points out, #19634 has unfortunate consequences
for the standard library, and likely requires more thought. To avoid
a major breaking change, it has been decided to revert this design
decision for now, and make a more informed decision further down the
line.
We've got a big one here! This commit reworks how we represent pointers
in the InternPool, and rewrites the logic for loading and storing from
them at comptime.
Firstly, the pointer representation. Previously, pointers were
represented in a highly structured manner: pointers to fields, array
elements, etc, were explicitly represented. This works well for simple
cases, but is quite difficult to handle in the cases of unusual
reinterpretations, pointer casts, offsets, etc. Therefore, pointers are
now represented in a more "flat" manner. For types without well-defined
layouts -- such as comptime-only types, automatic-layout aggregates, and
so on -- we still use this "hierarchical" structure. However, for types
with well-defined layouts, we use a byte offset associated with the
pointer. This allows the comptime pointer access logic to deal with
reinterpreted pointers far more gracefully, because the "base address"
of a pointer -- for instance a `field` -- is a single value which
pointer accesses cannot exceed since the parent has undefined layout.
This strategy is also more useful to most backends -- see the updated
logic in `codegen.zig` and `codegen/llvm.zig`. For backends which do
prefer a chain of field and elements accesses for lowering pointer
values, such as SPIR-V, there is a helpful function in `Value` which
creates a strategy to derive a pointer value using ideally only field
and element accesses. This is actually more correct than the previous
logic, since it correctly handles pointer casts which, after the dust
has settled, end up referring exactly to an aggregate field or array
element.
In terms of the pointer access code, it has been rewritten from the
ground up. The old logic had become rather a mess of special cases being
added whenever bugs were hit, and was still riddled with bugs. The new
logic was written to handle the "difficult" cases correctly, the most
notable of which is restructuring of a comptime-only array (for
instance, converting a `[3][2]comptime_int` to a `[2][3]comptime_int`.
Currently, the logic for loading and storing work somewhat differently,
but a future change will likely improve the loading logic to bring it
more in line with the store strategy. As far as I can tell, the rewrite
has fixed all bugs exposed by #19414.
As a part of this, the comptime bitcast logic has also been rewritten.
Previously, bitcasts simply worked by serializing the entire value into
an in-memory buffer, then deserializing it. This strategy has two key
weaknesses: pointers, and undefined values. Representations of these
values at comptime cannot be easily serialized/deserialized whilst
preserving data, which means many bitcasts would become runtime-known if
pointers were involved, or would turn `undefined` values into `0xAA`.
The new logic works by "flattening" the datastructure to be cast into a
sequence of bit-packed atomic values, and then "unflattening" it; using
serialization when necessary, but with special handling for `undefined`
values and for pointers which align in virtual memory. The resulting
code is definitely slower -- more on this later -- but it is correct.
The pointer access and bitcast logic required some helper functions and
types which are not generally useful elsewhere, so I opted to split them
into separate files `Sema/comptime_ptr_access.zig` and
`Sema/bitcast.zig`, with simple re-exports in `Sema.zig` for their small
public APIs.
Whilst working on this branch, I caught various unrelated bugs with
transitive Sema errors, and with the handling of `undefined` values.
These bugs have been fixed, and corresponding behavior test added.
In terms of performance, I do anticipate that this commit will regress
performance somewhat, because the new pointer access and bitcast logic
is necessarily more complex. I have not yet taken performance
measurements, but will do shortly, and post the results in this PR. If
the performance regression is severe, I will do work to to optimize the
new logic before merge.
Resolves: #19452Resolves: #19460
* fix UB when Thread.spawn fails, and we try to join uninitialized
threads (through new `thread_count` variable).
* make sure that the tests pins down synchronization guarantees: in the
main thread, we can observe `1` due to synchronization from
`Thread.join()`. To make sure that once uses Acq/Rel, and not just
relaxed, we should also additionally check that each thread observes
1, regardless of whether it was the one to call once.
On Windows, the command line arguments of a program are a single WTF-16 encoded string and it's up to the program to split it into an array of strings. In C/C++, the entry point of the C runtime takes care of splitting the command line and passing argc/argv to the main function.
https://github.com/ziglang/zig/pull/18309 updated ArgIteratorWindows to match the behavior of CommandLineToArgvW, but it turns out that CommandLineToArgvW's behavior does not match the behavior of the C runtime post-2008. In 2008, the C runtime argv splitting changed how it handles consecutive double quotes within a quoted argument (it's now considered an escaped quote, e.g. `"foo""bar"` post-2008 would get parsed into `foo"bar`), and the rules around argv[0] were also changed.
This commit makes ArgIteratorWindows match the behavior of the post-2008 C runtime, and adds a standalone test that verifies the behavior matches both the MSVC and MinGW argv splitting exactly in all cases (it checks that randomly generated command line strings get split the same way).
The motivation here is roughly the same as when the same change was made in Rust (https://github.com/rust-lang/rust/pull/87580), that is (paraphrased):
- Consistent behavior between Zig and modern C/C++ programs
- Allows users to escape double quotes in a way that can be more straightforward
Additionally, the suggested mitigation for BatBadBut (https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/) relies on the post-2008 argv splitting behavior for roundtripping of the arguments given to `cmd.exe`. Note: it's not necessary for the suggested mitigation to work, but it is necessary for the suggested escaping to be parsed back into the intended argv by ArgIteratorWindows after being run through a `.bat` file.
signature/s:
Algorithm Before After
---------------+---------+-------
ecdsa-p256 3707 4396
ecdsa-p384 1067 1332
ecdsa-secp256k1 4490 5147
Add ECDSA to the benchmark by the way.
This makes the host http header have the port if and only if it differs
from the defaults based on the protocol.
This is an alternate implementation that closes#19624.
This flips things around such that std/hash/crc.zig is generated
by the catalog-based generation tool, and the real code that used
to be in that file is moved out to std/hash/crc/impl.zig. The
generated tests are moved to std/hash/crc/test.zig. By going this
route, we eliminate the need for usingnamespace without changing
anything for callers of these interfaces. The Crc32 tests are
simply added to the fixed part of the generated output and
compactified a bit.
This was the second-to-last usage of usingnamespace left in std.
Don't know why UEFI wasn't excluded but freestanding is, probably an oversight since I want to have detailed debug info on my panic function on my Headstart bootloader.
The previous commit deleted the deprecated API and then made all the
follow-up changes; this commit reverts only the breaking API changes.
This commit can be reverted once 0.12.0 is tagged.
This adds the *std.Build owner to LazyPath so that lazy paths returned
from a dependency can be used in the application without friction or
footguns.
closes#19313
This allows `std.Uri.resolve_inplace` to properly preserve the fact
that `new` is already escaped but `base` may not be. I originally tried
just moving `raw_uri` around, but it made uri resolution unmanagably
complicated, so I instead added per-component information to `Uri` which
allows extra allocations to be avoided when constructing uris with
components from different sources, and in some cases, deferring the work
all the way to when the uri is printed, where an allocator may not even
be needed.
Closes#19587
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-185.pdf
This adds useful standard SHA3-based constructions from the
NIST SP 800-185 document:
- cSHAKE: similar to the SHAKE extensible hash function, but
with the addition of a context parameter.
- KMAC: SHAKE-based authentication / keyed XOF
- TupleHash: unambiguous hashing of tuples
These are required by recent protocols and specifications.
They also offer properties that none of the currently available
constructions in the stdlib offer, especially the ability to safely
hash tuples.
Other keyed hash functions/XOFs will fall back to using HMAC, which
is suboptimal from a performance perspective, but fine from a
security perspective.
