Commit graph

120 commits

Author SHA1 Message Date
Igor Anić
087ee497d4 io_uring: add zero-copy send operation
`send_zc` tries to avoid making intermediate copies of data. Zerocopy execution
is not guaranteed and may fall back to copying.

The flags field of the first struct io_uring_cqe may likely contain
IORING_CQE_F_MORE , which means that there will be a second completion event /
notification for the request, with the user_data field set to the same value.
The user must not modify the data buffer until the notification is posted. The
first cqe follows the usual rules and so its res field will contain the number
of bytes sent or a negative error code. The notification's res field will be set
to zero and the flags field will contain IORING_CQE_F_NOTIF. The two step model
is needed because the kernel may hold on to buffers for a long time, e.g.
waiting for a TCP ACK, and having a separate cqe for request completions allows
userspace to push more data without extra delays. Note, notifications are only
responsible for controlling the lifetime of the buffers, and as such don't mean
anything about whether the data has atually been sent out or received by the
other end. Even errored requests may generate a notification, and the user must
check for IORING_CQE_F_MORE rather than relying on the result.

Available since kernel 6.0.

References:
https://man7.org/linux/man-pages/man3/io_uring_prep_send_zc.3.html
https://man7.org/linux/man-pages/man2/io_uring_enter.2.html
2023-11-10 16:44:18 +01:00
Igor Anić
715e5f757f fix io_uring tests on kernel 5.4
Unsupported tests are now skipped on kernel 5.4:
```
uname -a
Linux d20 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:31:37 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
zig test lib/std/std.zig --zig-lib-dir lib --main-mod-path lib/std 2>&1 | cat
709/2616 test.nop... OK
710/2616 test.readv... OK
711/2616 test.writev/fsync/readv... OK
712/2616 test.write/read... SKIP
713/2616 test.splice/read... SKIP
714/2616 test.write_fixed/read_fixed... OK
715/2616 test.openat... SKIP
716/2616 test.close... SKIP
717/2616 test.accept/connect/send/recv... SKIP
718/2616 test.sendmsg/recvmsg... OK
719/2616 test.timeout (after a relative time)... SKIP
720/2616 test.timeout (after a number of completions)... OK
721/2616 test.timeout_remove... SKIP
722/2616 test.accept/connect/recv/link_timeout... SKIP
723/2616 test.fallocate... SKIP
724/2616 test.statx... SKIP
725/2616 test.accept/connect/recv/cancel... SKIP
726/2616 test.register_files_update... SKIP
727/2616 test.shutdown... SKIP
728/2616 test.renameat... SKIP
729/2616 test.unlinkat... SKIP
730/2616 test.mkdirat... SKIP
731/2616 test.symlinkat... SKIP
732/2616 test.linkat... SKIP
733/2616 test.provide_buffers: read... SKIP
734/2616 test.remove_buffers... SKIP
735/2616 test.provide_buffers: accept/connect/send/recv... SKIP
736/2616 test.accept multishot... SKIP
```
2023-11-08 18:52:03 +01:00
Igor Anić
7a1dbbd4f9 fix io_uring timeout_remove test on kernel 5.4
There is no grantee that `copy_cqes` will return exactly wait_nr number of cqes.
If there are ready cqes it can return > 0 but < wait_nr number of cqes.
2023-11-08 18:25:29 +01:00
Igor Anić
bec36aa7c0 io_uring: add multishot accept operation
Server networking application typically accept multiple connections. Multishot
accept simplifies handling these situations. Applications submits once and
receives CQE whenever a new connection request comes in.

Multishot is active until it is canceled or experience error. While active, and
further notification are expected CQE completion will have IORING_CQE_F_MORE set
in the flags. If this flag isn't set, the application must re-arm this request
by submitting a new one.

Reference: [io_uring and networking in 2023](https://github.com/axboe/liburing/wiki/io_uring-and-networking-in-2023#multi-shot)
2023-11-06 00:12:39 +02:00
Jacob Young
3b0dce8ebd
Merge pull request #17716 from jacobly0/x86_64
x86_64: pass more tests
2023-10-27 08:29:24 -04:00
Jacob Young
98cd378208 x86_64: fix behavior of getValue
Old behavior renamed to `getValueIfFree`.
2023-10-26 21:45:58 -04:00
Jacob Young
5e83441096 x86_64: implement @divFloor and @mod for i128 2023-10-26 21:45:57 -04:00
joadnacer
28b848e3f0 std.io_uring: Improve splice implementation 2023-10-27 00:31:55 +01:00
joadnacer
0005e7e70b std.linux: Update io_uring structs and consts for kernel 6.3.8 2023-10-27 00:31:52 +01:00
Jacob Young
8f69e977f1 x86_64: implement 128-bit builtins
* `@clz`
 * `@ctz`
 * `@popCount`
 * `@byteSwap`
 * `@bitReverse`
 * various encodings used by std
2023-10-23 22:42:18 -04:00
Jacob Young
27fe945a00 Revert "Revert "Merge pull request #17637 from jacobly0/x86_64-test-std""
This reverts commit 6f0198cadb.
2023-10-22 15:46:43 -04:00
Andrew Kelley
6f0198cadb Revert "Merge pull request #17637 from jacobly0/x86_64-test-std"
This reverts commit 0c99ba1eab, reversing
changes made to 5f92b070bf.

This caused a CI failure when it landed in master branch due to a
128-bit `@byteSwap` in std.mem.
2023-10-22 12:16:35 -07:00
Jacob Young
32e85d44eb x86_64: disable failing tests, enable test-std testing 2023-10-21 10:55:41 -04:00
Jacob Young
2e6e39a700 x86_64: fix bugs and disable erroring tests 2023-10-21 10:55:41 -04:00
joadnacer
89aad1b89c linux.io_uring: Implement splice functions 2023-10-21 05:02:17 +03:00
Jim Calabro
ec5a068ac1
Make a Couple Syscall Comments Consistent (#17066)
* Make a Couple Syscall Comments Consistent

* linux.diet.net -> man7.org
2023-09-04 09:09:11 +02:00
mllken
dc24835168 io_uring: let the OS choose a free local port for all tests
closes #14907
2023-07-24 20:12:41 -07:00
Zachary Raineri
d82b359010 Use builtin inference over @as where possible 2023-07-24 10:23:51 -07:00
mlugg
f26dda2117 all: migrate code to new cast builtin syntax
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:

* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
2023-06-24 16:56:39 -07:00
Eric Joldasov
50339f595a all: zig fmt and rename "@XToY" to "@YFromX"
Signed-off-by: Eric Joldasov <bratishkaerik@getgoogleoff.me>
2023-06-19 12:34:42 -07:00
r00ster91
2593156068 migration: std.math.{min, min3, max, max3} -> @min & @max 2023-06-16 13:44:09 -07:00
Andrew Kelley
125221cce9 std: update to use @memcpy directly 2023-04-28 13:24:43 -07:00
Andrew Kelley
6261c13731 update codebase to use @memset and @memcpy 2023-04-28 13:24:43 -07:00
Andrew Kelley
e102adc863 work around an intermittent io_uring test failure
See #15362
2023-04-19 22:49:51 -07:00
bing
64214b1ca6
Change ordering of prep provide buffers args 2023-03-30 16:51:57 +00:00
Andrew Kelley
cdda395590 std lib tests: avoid cwd races by using std.testing.tmpDir 2023-03-15 10:48:15 -07:00
Andrew Kelley
22d94eaf32 disable std lib unit tests that hard code port numbers
See tracking issue #14907
2023-03-15 10:48:15 -07:00
Veikka Tuominen
b2c85464be std.os.linux.io_uring: fix ABI error
`register_files_update` was passing a pointer to a Zig struct to the kernel.
2023-01-16 22:25:34 +02:00
Jacob Young
f5f28e0d2c io_uring: ignore SOCK_NONEMPTY for reproducible tests
Fixes #12670
2022-10-13 14:58:06 +02:00
mllken
42a3b60c33 io_uring: allow for nullable arguments 2022-10-11 17:49:30 +02:00
Vincent Rischmann
47c58cba59
Fix io_uring tests (#12134)
* io_uring: fix the timeout_remove test

The test does a IORING_OP_TIMEOUT followed with a IORING_OP_TIMEOUT_REMOVE
and assumed we would get the CQEs in the same order.

Linux v5.18 changed how this works and we now get them in the reverse order.

The documentation doesn't explicitly say which CQE we should get first
so just make the test work with both cases.

* io_uring: fix the remove_buffers test

The original test was buggy but accidentally worked with kernels < 5.18

The test assumed that IORING_OP_REMOVE_BUFFERS removed from the start of
but in fact the documentation doesn't specify which buffer is removed,
only that a certain number of buffers are removed.

Starting with the kernel 5.18 the check for the `used_buffer_id` fails.
Turns out that previous kernels removed buffers in such a way that the
remaining buffer for this read would always be 0, however this isn't
true anymore.

Instead of checking a specific value just check that the `used_buffer_id`
corresponds to a valid ID.
2022-07-16 10:05:11 -05:00
Andrew Kelley
c34392b24e std.os.linux.io_uring: work around LLVM bug
See #12014
2022-07-05 21:54:05 -07:00
Jonathan Marler
523fae420b add const to msghdr_const iov and control pointers
alongside the typical msghdr struct, Zig has added a msghdr_const
type that can be used with sendmsg which allows const data to
be provided.  I believe that data pointed to by the iov and control
fields in msghdr are also left unmodified, in which case they can
be marked const as well.
2022-06-07 15:23:44 -04:00
Veikka Tuominen
15ec55406d std: fix ambiguous references 2022-06-06 13:11:50 -07:00
Vincent Rischmann
3c58d3e281 io_uring: replace the readv method with a read on a new ReadBuffer type
readv() is essentially identical to read() except for the buffer type,
this simplifies the API for the caller at the cost of not clearly mapping to the liburing C API.
2022-05-25 13:53:09 +02:00
Vincent Rischmann
acb8af468f io_uring: add tests for automatic buffer selection 2022-05-25 13:53:09 +02:00
Vincent Rischmann
d8798ef0cd io_uring: use the socket test harness 2022-05-25 13:53:09 +02:00
Vincent Rischmann
456716b30d io_uring: add a test harness for server/client TCP socket tests 2022-05-25 13:53:09 +02:00
Vincent Rischmann
270a5039d4 io_uring: change recv() to use a RecvBuffer instead
RecvBuffer is equivalent to ReadBuffer but tailored for recv only.
2022-05-25 13:53:09 +02:00
Vincent Rischmann
52dd468cc3 io_uring: change read() to use a ReadBuffer instead
Reads can be done in two ways with io_uring:
* using a simple buffer
* using a automatic buffer selection which requires the user to have
provided a number of buffers before

ReadBuffer let's the caller choose where the data should be read.
2022-05-25 13:53:09 +02:00
Vincent Rischmann
7b3e5ce0b3 io_uring: add provide_buffers and remove_buffers
These functions are needed to implement automatic buffer selection.

This maps to the IORING_OP_PROVIDE_BUFFERS and IORING_OP_PROVIDE_BUFFERS ops.
2022-05-25 13:53:09 +02:00
Brian Gold
52651ae7a0 io_uring cancel test must tolerate CQ reordering
Fixes #11382. Tested on ArchLinux 5.17.5-arch1-1.
2022-05-10 11:49:41 +02:00
Hiroaki Nakamura
3605dd307f
os/linux/io_uring: add recvmsg and sendmsg (#10212)
* os/linux/io_uring: add recvmsg and sendmsg

* Use std.os.iovec and std.os.iovec_const

* Remove msg_ prefix in msghdr and msghdr_const in arm64 etc

* Strip msg_ prefix in msghdr and msghdr_const for linux arm-eabi

* Copy msghdr and msghdr_const from i386 to mips

* Add sockaddr to lib/std/os/linux/mips.zig

* Copy msghdr and msghdr_const from x86_64 to riscv64
2022-03-03 14:13:54 -06:00
Vincent Rischmann
6630a5ede5 io_uring: improve IO_Uring.copy_cqe
copy_cqes() is not guaranteed to return as many CQEs as provided in the
`wait_nr` argument, meaning the assert in `copy_cqe` can trigger.

Instead, loop until we do get at least one CQE returned.

This mimics the behaviour of liburing's _io_uring_get_cqe.
2022-01-04 02:13:41 -05:00
Vincent Rischmann
0662f1d522 io_uring: fix version check in tests
For renameat, unlinkat, mkdirat, symlinkat and linkat the error code
differs between kernel 5.4 which returns EBADF and kernel 5.10 which returns EINVAL.

Fixes #10466
2022-01-01 15:37:50 -05:00
Isaac Freund
9f9f215305
stage1, stage2: rename c_void to anyopaque (#10316)
zig fmt now replaces c_void with anyopaque to make updating
code easy.
2021-12-19 00:24:45 -05:00
Vincent Rischmann
f9b8808d74 os/linux/io_uring: implement linkat 2021-12-12 18:49:17 +01:00
Vincent Rischmann
5dd53c1986 os/linux/io_uring: implement symlinkat 2021-12-12 18:43:31 +01:00
Vincent Rischmann
1fd0542bee os/linux/io_uring: implement mkdirat 2021-12-12 18:33:17 +01:00
Vincent Rischmann
4e647dee9f os/linux/io_uring: implement unlinkat 2021-12-12 18:32:08 +01:00