Occasionally the topic of asynchronous I/O on local files comes up, though while there are APIs for event-based processing they don't work on regular files, resulting in the use of worker threads to circumvent this restriction.
The inciting incident for me to look into this was the fact that one of my (external, spinning-disk) hard drives has a rather short timeout for spin-down, such that when pausing while browsing through a directory tree would often annoy me when the shell (or any other program) was completely blocked as the motor was being turned on again.
At least on Linux
(since version 2.5)
there is actually a kernel syscall interface for asynchronous file I/O.
Unfortunately it is not being exposed via the libc at all, requiring custom
wrappers in order to trigger the proper syscalls. Apart from scheduling
asynchronous read and write requests it also supports exposing the
corresponding events via an
evenfd
queue, which then allows us to use epoll and friends.
SBCL being the ultimate breadboard that's not particularly hard though. Using CFFI and IOLib for convenience it's straightforward to port the C examples while writing a minimal amount of C code. The code is of course not very high-level, but can be plugged straight into the IOLib event loop as there's now a file descriptor available to listen on.
Grovelling & wrapping
The groveller can be used quite nicely to prevent us from having to drop down to C completely. Of course we're also using ASDF, so everything's peachy.
Now what's required? CFFI provides additional components for ASDF, namely
:CFFI-GROVEL-FILE and :CFFI-WRAPPER-FILE, which make the process seamless
and don't require us to write and code related to compiling and loading the C
wrapper:
;; -*- mode: lisp; syntax: common-lisp; coding: utf-8-unix; package: cl-user; -*-
(in-package #:cl-user)
(asdf:defsystem #:example
:defsystem-depends-on (#:cffi-grovel)
#+asdf-unicode :encoding #+asdf-unicode :utf-8
:depends-on (#:iterate
#:iolib
#:cffi
#:osicat)
:serial T
:components ((:module "src"
:components
((:file "package")
(:file "wrapper")
(:cffi-grovel-file "linux-aio-grovel")
(:cffi-wrapper-file "linux-aio-wrapper")
(:file "linux-aio")))))The package definition is probably not very interesting at this point:
(in-package #:cl-user)
(defpackage #:example
(:use #:cl #:iterate #:iolib #:cffi))I've added IOLib and CFFI, usually also ITERATE for convenience.
Next we grovel a couple of definitions related to the kernel API for the
asynchronous requests and for eventfd. This is the linux-aio-grovel file
mentioned above:
(in-package #:example)
(include "stdio.h" "unistd.h" "sys/syscall.h" "linux/aio_abi.h" "inttypes.h"
"signal.h" "sys/eventfd.h")
(ctype aio-context-t "aio_context_t")
(cenum iocb-cmd-t
((:pread "IOCB_CMD_PREAD"))
((:pwrite "IOCB_CMD_PWRITE"))
((:fsync "IOCB_CMD_FSYNC"))
((:fdsync "IOCB_CMD_FDSYNC"))
((:noop "IOCB_CMD_NOOP"))
((:preadv "IOCB_CMD_PREADV"))
((:pwritev "IOCB_CMD_PWRITEV")))
(constantenum iocb-flags-t
((:resfd "IOCB_FLAG_RESFD")))
(cstruct iocb "struct iocb"
(aio-data "aio_data" :type :uint64)
;; #-little-endian
;; (aio-reserved1 "aio_reserved1" :type :uint32)
(aio-key "aio_key" :type :uint32)
;; #+little-endian
;; (aio-reserved1 "aio_reserved1" :type :uint32)
(aio-lio-opcode "aio_lio_opcode" :type iocb-cmd-t)
(aio-fildes "aio_fildes" :type :uint32)
(aio-buf "aio_buf" :type :uint64)
(aio-nbytes "aio_nbytes" :type :uint64)
(aio-offset "aio_offset" :type :int64)
;; (aio-reserved2 "aio_reserved2" :type :uint64)
(aio-flags "aio_flags" :type :uint32)
(aio-resfd "aio_resfd" :type :uint32))
(cstruct io-event "struct io_event"
(data "data" :type :uint64)
(obj "obj" :type :uint64)
(res "res" :type :int64)
(res2 "res" :type :int64))
(cenum eventfd-flags-t
((:cloexec "EFD_CLOEXEC"))
((:nonblock "EFD_NONBLOCK"))
((:semaphore "EFD_SEMAPHORE")))Note that this not a complete list and a couple of reserved members are commented out as they're primarily used to provide space for further expansion. Fortunately offsets for the rest of the struct aren't affected by leaving out parts in the Lisp-side definition.
The enums are easy enough, even though they both represent flags, so should be or-ed together, which might be necessary to do manually or by finding a way to let CFFI do the coercion from a list of flags perhaps.
In order to have nice syscall wrappers we'd normally use defsyscall from
IOLib. Unfortunately we also want to use defwrapper from CFFI-GROVEL. This
is an example of bad composability of macros, requiring copy and paste of
source code. Of course with enough refactoring or an optional parameter this
could be circumvented. This is the wrapper file from the ASDF definition.
;; groan
cffi-grovel::
(define-wrapper-syntax defwrapper/syscall* (name-and-options rettype args &rest c-lines)
;; output C code
(multiple-value-bind (lisp-name foreign-name options)
(cffi::parse-name-and-options name-and-options)
(let ((foreign-name-wrap (strcat foreign-name "_cffi_wrap"))
(fargs (mapcar (lambda (arg)
(list (c-type-name (second arg))
(cffi::foreign-name (first arg) nil)))
args)))
(format out "~A ~A" (c-type-name rettype)
foreign-name-wrap)
(format out "(~{~{~A ~A~}~^, ~})~%" fargs)
(format out "{~%~{ ~A~%~}}~%~%" c-lines)
;; matching bindings
(push `(iolib/syscalls:defsyscall (,foreign-name-wrap ,lisp-name ,@options)
,(cffi-type rettype)
,@(mapcar (lambda (arg)
(list (symbol* (first arg))
(cffi-type (second arg))))
args))
*lisp-forms*))))The only change from DEFWRAPPER is the use of IOLIB/SYSCALLS:DEFSYSCALL
instead of DEFCFUN, which then performs additional checks with respect to the
return value, raising a IOLIB/SYSCALLS:SYSCALL-ERROR that we can then catch
rather than having to check the return value ourselves.
Lastly, the actual wrappers. Note that some inline C is used to define the
function bodies. This is linux-aio-wrapper from the ASDF definition:
(define "_GNU_SOURCE")
(include "stdio.h" "unistd.h" "sys/syscall.h" "linux/aio_abi.h" "inttypes.h"
"signal.h")
(defwrapper/syscall* "io_setup" :int
((nr :unsigned-int)
(ctxp ("aio_context_t*" (:pointer aio-context-t))))
"return syscall(__NR_io_setup, nr, ctxp);")
(defwrapper/syscall* "io_destroy" :int
((ctx aio-context-t))
"return syscall(__NR_io_destroy, ctx);")
(defwrapper/syscall* "io_submit" :int
((ctx aio-context-t)
(nr :long)
(iocbpp ("struct iocb**" (:pointer (:pointer (:struct iocb))))))
"return syscall(__NR_io_submit, ctx, nr, iocbpp);")
(defwrapper/syscall* "io_cancel" :int
((ctx aio-context-t)
(iocbp ("struct iocb*" (:pointer (:struct iocb))))
(result ("struct io_event*" (:pointer (:struct io-event)))))
"return syscall(__NR_io_cancel, ctx, iocbp, result);")
(defwrapper/syscall* "io_getevents" :int
((ctx aio-context-t)
(min-nr :long)
(max-nr :long)
(events ("struct io_event*" (:pointer (:struct io-event))))
(timeout ("struct timespec*" (:pointer (:struct iolib/syscalls::timespec)))))
"return syscall(__NR_io_getevents, ctx, min_nr, max_nr, events, timeout);")Looping
Now that we have all definitions in place, let's translate a moderately complex example of reading from an existing file.
Also EVENTFD is defined here as the C function is already defined in the libc
and doesn't have to generated.
(iolib/syscalls:defsyscall eventfd :int
(initval :unsigned-int)
(flags eventfd-flags-t))
(defun linux-aio-test (pathname &key (chunk-size 4096))
(with-foreign-object (context 'aio-context-t)
(iolib/syscalls:memset context 0 (foreign-type-size 'aio-context-t))
(let ((eventfd (eventfd 0 :nonblock)))
(unwind-protect
(with-open-file (stream pathname :element-type '(unsigned-byte 8))
(let* ((length (file-length stream))
(chunks (ceiling length chunk-size)))
(with-foreign-object (buffer :uint8 length)
(with-event-base (event-base)
(io-setup 1 context) ; set up with number of possible operations
(with-foreign-object (iocbs '(:struct iocb) chunks)
(iolib/syscalls:memset iocbs 0 (* (foreign-type-size '(:struct iocb)) chunks))
;; set up array of operations
(dotimes (i chunks)
(let ((iocb (mem-aptr iocbs '(:struct iocb) i)))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-lio-opcode) :pread)
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-buf) (pointer-address (mem-aptr buffer :uint8 (* i chunk-size))))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-nbytes) (if (eql i (1- chunks)) (- length (* i chunk-size)) chunk-size))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-fildes) (sb-sys:fd-stream-fd stream))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-offset) (* i chunk-size))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-flags) (foreign-enum-value 'iocb-flags-t :resfd))
(setf (foreign-slot-value iocb '(:struct iocb) 'aio-resfd) eventfd)))
;; set up array of pointers to operations
(with-foreign-object (iocbp '(:pointer (:struct iocb)) chunks)
(dotimes (i chunks)
(setf (mem-aref iocbp '(:pointer (:struct iocb)) i) (mem-aptr iocbs '(:struct iocb) i)))
;; submit as many operations as possible
(let ((submitted (io-submit (mem-ref context 'aio-context-t) chunks iocbp)))
;; keep track of how many operations completed total
(let ((total-events-read 0))
(flet ((get-events () ; named to be able to RETURN-FROM
(with-foreign-object (events '(:struct io-event) 3)
(loop
(handler-case
(with-foreign-object (available-buffer :uint64)
(iolib/syscalls:read eventfd available-buffer 8)
(let ((available (mem-ref available-buffer :uint64)))
(dotimes (i available)
(let ((events-read (io-getevents (mem-ref context 'aio-context-t) 0 3 events (null-pointer))))
(when (eql events-read 0)
(return))
(incf total-events-read events-read))
(when (eql total-events-read chunks)
(return-from linux-aio-test)))))
;; in case reading would block
(iolib/syscalls:syscall-error ()
(when (< submitted chunks)
(let ((more-submitted (io-submit (mem-ref context 'aio-context-t) chunks (mem-aptr iocbp '(:pointer (:struct iocb)) submitted))))
(incf submitted more-submitted)))
(return-from get-events)))))))
(set-io-handler
event-base eventfd :read
(lambda (fd event exception)
(declare (ignore fd event exception))
(get-events)))
(event-dispatch event-base))))))))))
(io-destroy (mem-ref context 'aio-context-t))
(iolib/syscalls:close eventfd)))))Relatively straightforward. Complexity comes from accurately submitting chunks, reading the number of available events on demand and submitting a new batch of chunks as long as there are some remaining ones.
Insert FORMAT statement as you like. Tuning the values would need to be
considered in order to keep memory consumption in check. Finally
Outlook
We still can't do many local file operations asynchronously. The whole reason to jump through these hoops is of course to integrate potentially blocking operations into an event loop, so some care still needs to be taken to do some work ahead of time or in separate threads as to not block the main part of the program from issuing the I/O requests.
