Recent Content

Byte code verification and ABCL
posted on 2017-06-01 19:51:41

A feature of JVM byte code that I knew about, but didn't concern me too much when I previously hacked on ABCL was interesting to see in a particular bug report, where a function, when compiled, got rejected by the byte code validator:

(compile NIL (lambda (list) (nth (lambda ()) list)))
;; => Compiled function can't be loaded

This is true for any class loader, so COMPILE-FILE with LOAD would show the same behaviour.

The reason here is inlining: In order to give better performance when some information about the types is known, the call to NTH will actually be optimised here. One option to see this is to set a debug option to dump the generated byte code while the compilation runs:

(setf jvm::*compiler-debug* T)

(Part) of the output will look like this:

  0 GETSTATIC      #24 <Field LispObject 42c1_92be_e5b48c1894c6.LFUN1812724>  
  1 ALOAD          (1)  
  2 SWAP             
  3 INVOKEVIRTUAL  #30 <Method LispObject LispObject.NTH(int)> -1 
  4 ARETURN          

Using DISASSEMBLE is not an option precisely because the byte code can't be loaded at all (that's actually a nice idea as another function to dump not only the compiled FASL content, but also disassemble the contained byte code too).

With that option set (and perhaps setting a few BREAK statements in the compiler), it's somewhat easy to debug the compiled byte code and to see that the validator notices that the Java method that implements the NTH function requires an integer (fixnum) argument, but the LispObject (the lambda) doesn't match that signature. This is not a problem if this was a regular call. In fact, with inlining disabled the NTH function will still raise an error for an argument with the wrong type!

Finally the fix is to check for the derived type in the "P2" transformation function for NTH, COMPILE-NTH:

(define-inlined-function compile-nth (form target representation)
  ((check-arg-count form 2))
  (let* ((index-form (second form))
         (list-form (third form))

    ;;; new check here
         (index-type (derive-compiler-type index-form)))
    (unless (fixnum-type-p index-type)
      (compile-function-call form target representation)
      (return-from compile-nth))
    ;;; till here

    (with-operand-accumulation
        ((compile-operand index-form :int)
         (compile-operand list-form nil)
         (maybe-emit-clear-values index-form list-form))
      (emit 'swap)
      (emit-invokevirtual +lisp-object+ "NTH" '(:int) +lisp-object+))
    (fix-boxing representation nil) ; FIXME use derived result type
    (emit-move-from-stack target representation)))

Note the falling back to a "general" function call using COMPILE-FUNCTION-CALL here in case the type is not known in advance, or not a fixnum type (though that could also raise a warning here already).

Again, compiling the above function again looks a bit different in the general case:

  0 GETSTATIC      #29 <Field Symbol 4faf_99d9_62a0381d4d65.SYM1814566>  
  1 GETSTATIC      #33 <Field LispObject 4faf_99d9_62a0381d4d65.LFUN1814565>  
  2 ALOAD          (1)  
  3 INVOKEVIRTUAL  #39 <Method LispObject LispObject.execute(LispObject,LispObject)> -2 
  4 ARETURN          

If instead a fixnum constant is used, (nth 1 list), this simplifies a lot:

  0 ICONST_1         
  1 ALOAD          (1)  
  2 SWAP             
  3 INVOKEVIRTUAL  #24 <Method LispObject LispObject.NTH(int)> -1 
  4 ARETURN          

Compare that to adding a (declare (type fixnum ...)) declaration - not as good as a constant argument, but still directly calling the Java method:

  0 ALOAD          (1)  
  1 INVOKEVIRTUAL  #24 <Method int LispObject.intValue()> 0 
  2 ISTORE         (1)  
  3 ILOAD          (1)  
  4 ALOAD          (2)  
  5 SWAP             
  6 INVOKEVIRTUAL  #28 <Method LispObject LispObject.NTH(int)> -1 
  7 ARETURN          

Note here that the type error (e.g. for again supplying the (lambda ())) will be raised at the caller site already!

Lastly, a good idea would also be to generally add more hints, as e.g. SBCL does, to debug other issues ("failed to inline because ..."), but that's for another day.

Memory reordering in Common Lisp and Kotlin
posted on 2016-11-23 21:43:41

I've been reading about multi-threaded programming out of curiosity over lock-free algorithms.

The first thing I've implemented for that was actually something completely different, namely a short example of memory reordering ported over from a C++ program.

In Common Lisp this relies on SBCL because of the availability of memory barriers and the Linux-specific API to set the thread affinity to a single CPU core (sb-cpu-affinity). Of course, given the appropriate compability libraries this could easily be done on other implementations too.

(in-package #:cl-user)

(defun main (&key single-cpu-p barrierp)
  (let ((begin-semaphore-1 (sb-thread:make-semaphore))
        (begin-semaphore-2 (sb-thread:make-semaphore))
        (end-semaphore (sb-thread:make-semaphore))
        (iterations 0)
        (detected 0)
        (x 0)
        (y 0)
        (r1 0)
        (r2 0)
        done)
    (declare (fixnum x y r1 r2))
    (declare (boolean done))
    (declare (optimize (speed 3) (safety 0) (space 0) (debug 0) (compilation-speed 0)))
    (labels ((single-cpu ()
               (when single-cpu-p
                 sb-cpu-affinity::
                 (with-cpu-affinity-mask (mask :save T)
                   (clear-cpu-affinity-mask mask)
                   (setf (cpu-affinity-p 0 mask) T))))
             (thread-1 ()
               (single-cpu)
               (loop
                 (when done
                   (return))
                 (sb-thread:wait-on-semaphore begin-semaphore-1)
                 (setf x 1)
                 (when barrierp
                   (sb-thread:barrier (:memory)))
                 (setf r1 y)
                 (sb-thread:signal-semaphore end-semaphore)))
             (thread-2 ()
               (single-cpu)
               (loop
                 (when done
                   (return))
                 (sb-thread:wait-on-semaphore begin-semaphore-2)
                 (setf y 1)
                 (when barrierp
                   (sb-thread:barrier (:memory)))
                 (setf r2 x)
                 (sb-thread:signal-semaphore end-semaphore))))
      (let ((thread-1 (sb-thread:make-thread #'thread-1))
            (thread-2 (sb-thread:make-thread #'thread-2)))
        (unwind-protect
             (loop
               (setf x 0 y 0)
               (sb-thread:signal-semaphore begin-semaphore-1)
               (sb-thread:signal-semaphore begin-semaphore-2)
               (sb-thread:wait-on-semaphore end-semaphore)
               (sb-thread:wait-on-semaphore end-semaphore)
               (when (and (eql r1 0) (eql r2 0))
                 (incf detected)
                 (format T "~D reorders detected after ~D iterations, every ~D on average~%" detected iterations (floor iterations detected)))
               (incf iterations))
          (setf done T)
          (sb-thread:signal-semaphore begin-semaphore-1)
          (sb-thread:signal-semaphore begin-semaphore-2)
          (sb-thread:join-thread thread-1)
          (sb-thread:join-thread thread-2))))))

On the JVM, respectively Kotlin, the code looks again remarkably similar. I haven't yet looked at thread affinity, but what's interesting here are two aspects related to the JVM memory model. Using volatile is, in contrast to C, viable, as it generates the necessary memory barriers.

import java.util.concurrent.Semaphore
import kotlin.concurrent.thread

class HelloWorld {
    var beginSempahore1 = Semaphore(0)
    var beginSempahore2 = Semaphore(0)
    var endSemaphore = Semaphore(0)

    var detected = 0
    var iterations = 0

    @Volatile var x = 0
    @Volatile var y = 0
    var r1 = 0
    var r2 = 0

    fun run() {
        thread {
            while (true) {
                beginSempahore1.acquire()
                x = 1
                r1 = y
                endSemaphore.release()
            }
        }

        thread {
            while (true) {
                beginSempahore2.acquire()
                y = 1
                r2 = x
                endSemaphore.release()
            }
        }

        while (true) {
            iterations += 1
            x = 0
            y = 0
            beginSempahore1.release()
            beginSempahore2.release()
            endSemaphore.acquire()
            endSemaphore.acquire()
            if (r1 == 0 && r2 == 0) {
                detected++
                println("$detected reorders detected after $iterations iterations")
            }
        }
    }
}

fun main(args: Array<String>) {
    HelloWorld().run()
}

Also, when using a JVM plugin, the generated assembly code can be dumped during compilation, which allows us to confirm whether those instructions actually have been generated; for the first loop:

  0x00007f9ad4f503c9: mov    (%rcx),%r11d       ;*getfield this$0
                                                ; - HelloWorld$run$1::invoke@11 (line 21)

  0x00007f9ad4f503cc: test   %r11d,%r11d
  0x00007f9ad4f503cf: je     0x00007f9ad4f507a9
  0x00007f9ad4f503d5: movl   $0x1,0x14(%r12,%r11,8)
  0x00007f9ad4f503de: lock addl $0x0,(%rsp)     ;*putfield x
                                                ; - HelloWorld::setX@2 (line 12)
                                                ; - HelloWorld$run$1::invoke@15 (line 21)

  0x00007f9ad4f503e3: mov    (%rcx),%r10d       ;*getfield this$0
                                                ; - HelloWorld$run$1::invoke@19 (line 22)

  0x00007f9ad4f503e6: test   %r10d,%r10d
  0x00007f9ad4f503e9: je     0x00007f9ad4f507cd  ;*invokevirtual getY
                                                ; - HelloWorld$run$1::invoke@26 (line 22)

  0x00007f9ad4f503ef: mov    0x18(%r12,%r10,8),%r11d  ;*getfield y
                                                ; - HelloWorld::getY@1 (line 13)
                                                ; - HelloWorld$run$1::invoke@26 (line 22)

  0x00007f9ad4f503f4: mov    %r11d,0x1c(%r12,%r10,8)  ;*putfield r1
                                                ; - HelloWorld::setR1@2 (line 14)
                                                ; - HelloWorld$run$1::invoke@29 (line 22)

And for the second loop:

  0x00007f9ad4f47269: mov    (%rcx),%r11d       ;*getfield this$0
                                                ; - HelloWorld$run$2::invoke@11 (line 30)

  0x00007f9ad4f4726c: test   %r11d,%r11d
  0x00007f9ad4f4726f: je     0x00007f9ad4f47629
  0x00007f9ad4f47275: movl   $0x1,0x18(%r12,%r11,8)
  0x00007f9ad4f4727e: lock addl $0x0,(%rsp)     ;*putfield y
                                                ; - HelloWorld::setY@2 (line 13)
                                                ; - HelloWorld$run$2::invoke@15 (line 30)

  0x00007f9ad4f47283: mov    (%rcx),%r10d       ;*getfield this$0
                                                ; - HelloWorld$run$2::invoke@19 (line 31)

  0x00007f9ad4f47286: test   %r10d,%r10d
  0x00007f9ad4f47289: je     0x00007f9ad4f4764d  ;*invokevirtual getX
                                                ; - HelloWorld$run$2::invoke@26 (line 31)

  0x00007f9ad4f4728f: mov    0x14(%r12,%r10,8),%r11d  ;*getfield x
                                                ; - HelloWorld::getX@1 (line 12)
                                                ; - HelloWorld$run$2::invoke@26 (line 31)

  0x00007f9ad4f47294: mov    %r11d,0x20(%r12,%r10,8)  ;*putfield r2
                                                ; - HelloWorld::setR2@2 (line 15)
                                                ; - HelloWorld$run$2::invoke@29 (line 31)

And the main loop:

  0x00007fd383813838: lock addl $0x0,(%rsp)     ;*putfield x
                                                ; - HelloWorld::run@58 (line 38)

  0x00007fd38381383d: mov    $0x0,%edx
  0x00007fd383813842: mov    %edx,0x18(%rsi)
  0x00007fd383813845: lock addl $0x0,(%rsp)     ;*putfield y
                                                ; - HelloWorld::run@63 (line 39)

  0x00007fd38381384a: mov    0x24(%rsi),%edi

Note that in contrast to the original and the Common Lisp version there's one additional memory barrier here that's unnecessary.

EverythingDB
posted on 2016-11-23 21:40:04

How do you track everything you know? Computers and mobile phones in turn have become external memory, however data is still fragmented, stored in different formats and usable in different programs.

While there are partial solutions to this issue, none seem to be all-encompassing enough.

For example, PIMs come close for communication and contacts, but it's hard to extend and customise them. You should be able to associate custom tags and key-value pairs with "documents"/entities. This goes straight into ontologies since the question how to model many kinds of relations will be a problem to deal with.

Lastly tracking all data only makes sense if there are reasonable search facilities and if exchange with other databases can be done with security in mind, especially if even metadata is sensitive.

Asynchronous file I/O
posted on 2016-11-11 01:19:28

Occasionally the topic of asynchronous I/O on local files comes up, though while there are APIs for event-based processing they don't work on regular files, resulting in the use of worker threads to circumvent this restriction.

The inciting incident for me to look into this was the fact that one of my (external, spinning-disk) hard drives has a rather short timeout for spin-down, such that when pausing while browsing through a directory tree would often annoy me when the shell (or any other program) was completely blocked as the motor was being turned on again.

At least on Linux (since version 2.5) there is actually a kernel syscall interface for asynchronous file I/O. Unfortunately it is not being exposed via the libc at all, requiring custom wrappers in order to trigger the proper syscalls. Apart from scheduling asynchronous read and write requests it also supports exposing the corresponding events via an evenfd queue, which then allows us to use epoll and friends.

SBCL being the ultimate breadboard that's not particularly hard though. Using CFFI and IOLib for convenience it's straightforward to port the C examples while writing a minimal amount of C code. The code is of course not very high-level, but can be plugged straight into the IOLib event loop as there's now a file descriptor available to listen on.

Grovelling & wrapping

The groveller can be used quite nicely to prevent us from having to drop down to C completely. Of course we're also using ASDF, so everything's peachy.

Now what's required? CFFI provides additional components for ASDF, namely :CFFI-GROVEL-FILE and :CFFI-WRAPPER-FILE, which make the process seamless and don't require us to write and code related to compiling and loading the C wrapper:

;; -*- mode: lisp; syntax: common-lisp; coding: utf-8-unix; package: cl-user; -*-

(in-package #:cl-user)

(asdf:defsystem #:example
  :defsystem-depends-on (#:cffi-grovel)
  #+asdf-unicode :encoding #+asdf-unicode :utf-8
  :depends-on (#:iterate
               #:iolib
               #:cffi
               #:osicat)
  :serial T
  :components ((:module "src"
                :components
                ((:file "package")
                 (:file "wrapper")
                 (:cffi-grovel-file "linux-aio-grovel")
                 (:cffi-wrapper-file "linux-aio-wrapper")
                 (:file "linux-aio")))))

The package definition is probably not very interesting at this point:

(in-package #:cl-user)

(defpackage #:example
  (:use #:cl #:iterate #:iolib #:cffi))

I've added IOLib and CFFI, usually also ITERATE for convenience.

Next we grovel a couple of definitions related to the kernel API for the asynchronous requests and for eventfd. This is the linux-aio-grovel file mentioned above:

(in-package #:example)

(include "stdio.h" "unistd.h" "sys/syscall.h" "linux/aio_abi.h" "inttypes.h"
         "signal.h" "sys/eventfd.h")

(ctype aio-context-t "aio_context_t")

(cenum iocb-cmd-t
  ((:pread "IOCB_CMD_PREAD"))
  ((:pwrite "IOCB_CMD_PWRITE"))
  ((:fsync "IOCB_CMD_FSYNC"))
  ((:fdsync "IOCB_CMD_FDSYNC"))
  ((:noop "IOCB_CMD_NOOP"))
  ((:preadv "IOCB_CMD_PREADV"))
  ((:pwritev "IOCB_CMD_PWRITEV")))

(constantenum iocb-flags-t
  ((:resfd "IOCB_FLAG_RESFD")))

(cstruct iocb "struct iocb"
  (aio-data "aio_data" :type :uint64)
  ;; #-little-endian
  ;; (aio-reserved1 "aio_reserved1" :type :uint32)
  (aio-key "aio_key" :type :uint32)
  ;; #+little-endian
  ;; (aio-reserved1 "aio_reserved1" :type :uint32)
  (aio-lio-opcode "aio_lio_opcode" :type iocb-cmd-t)
  (aio-fildes "aio_fildes" :type :uint32)
  (aio-buf "aio_buf" :type :uint64)
  (aio-nbytes "aio_nbytes" :type :uint64)
  (aio-offset "aio_offset" :type :int64)
  ;; (aio-reserved2 "aio_reserved2" :type :uint64)
  (aio-flags "aio_flags" :type :uint32)
  (aio-resfd "aio_resfd" :type :uint32))

(cstruct io-event "struct io_event"
  (data "data" :type :uint64)
  (obj "obj" :type :uint64)
  (res "res" :type :int64)
  (res2 "res" :type :int64))

(cenum eventfd-flags-t
  ((:cloexec "EFD_CLOEXEC"))
  ((:nonblock "EFD_NONBLOCK"))
  ((:semaphore "EFD_SEMAPHORE")))

Note that this not a complete list and a couple of reserved members are commented out as they're primarily used to provide space for further expansion. Fortunately offsets for the rest of the struct aren't affected by leaving out parts in the Lisp-side definition.

The enums are easy enough, even though they both represent flags, so should be or-ed together, which might be necessary to do manually or by finding a way to let CFFI do the coercion from a list of flags perhaps.

In order to have nice syscall wrappers we'd normally use defsyscall from IOLib. Unfortunately we also want to use defwrapper from CFFI-GROVEL. This is an example of bad composability of macros, requiring copy and paste of source code. Of course with enough refactoring or an optional parameter this could be circumvented. This is the wrapper file from the ASDF definition.

;; groan

cffi-grovel::
(define-wrapper-syntax defwrapper/syscall* (name-and-options rettype args &rest c-lines)
  ;; output C code
  (multiple-value-bind (lisp-name foreign-name options)
      (cffi::parse-name-and-options name-and-options)
    (let ((foreign-name-wrap (strcat foreign-name "_cffi_wrap"))
          (fargs (mapcar (lambda (arg)
                           (list (c-type-name (second arg))
                                 (cffi::foreign-name (first arg) nil)))
                         args)))
      (format out "~A ~A" (c-type-name rettype)
              foreign-name-wrap)
      (format out "(~{~{~A ~A~}~^, ~})~%" fargs)
      (format out "{~%~{  ~A~%~}}~%~%" c-lines)
      ;; matching bindings
      (push `(iolib/syscalls:defsyscall (,foreign-name-wrap ,lisp-name ,@options)
                 ,(cffi-type rettype)
               ,@(mapcar (lambda (arg)
                           (list (symbol* (first arg))
                                 (cffi-type (second arg))))
                         args))
            *lisp-forms*))))

The only change from DEFWRAPPER is the use of IOLIB/SYSCALLS:DEFSYSCALL instead of DEFCFUN, which then performs additional checks with respect to the return value, raising a IOLIB/SYSCALLS:SYSCALL-ERROR that we can then catch rather than having to check the return value ourselves.

Lastly, the actual wrappers. Note that some inline C is used to define the function bodies. This is linux-aio-wrapper from the ASDF definition:

(define "_GNU_SOURCE")

(include "stdio.h" "unistd.h" "sys/syscall.h" "linux/aio_abi.h" "inttypes.h"
         "signal.h")

(defwrapper/syscall* "io_setup" :int
  ((nr :unsigned-int)
   (ctxp ("aio_context_t*" (:pointer aio-context-t))))
  "return syscall(__NR_io_setup, nr, ctxp);")

(defwrapper/syscall* "io_destroy" :int
  ((ctx aio-context-t))
  "return syscall(__NR_io_destroy, ctx);")

(defwrapper/syscall* "io_submit" :int
  ((ctx aio-context-t)
   (nr :long)
   (iocbpp ("struct iocb**" (:pointer (:pointer (:struct iocb))))))
  "return syscall(__NR_io_submit, ctx, nr, iocbpp);")

(defwrapper/syscall* "io_cancel" :int
  ((ctx aio-context-t)
   (iocbp ("struct iocb*" (:pointer (:struct iocb))))
   (result ("struct io_event*" (:pointer (:struct io-event)))))
  "return syscall(__NR_io_cancel, ctx, iocbp, result);")

(defwrapper/syscall* "io_getevents" :int
  ((ctx aio-context-t)
   (min-nr :long)
   (max-nr :long)
   (events ("struct io_event*" (:pointer (:struct io-event))))
   (timeout ("struct timespec*" (:pointer (:struct iolib/syscalls::timespec)))))
  "return syscall(__NR_io_getevents, ctx, min_nr, max_nr, events, timeout);")

Looping

Now that we have all definitions in place, let's translate a moderately complex example of reading from an existing file.

Also EVENTFD is defined here as the C function is already defined in the libc and doesn't have to generated.

(iolib/syscalls:defsyscall eventfd :int
  (initval :unsigned-int)
  (flags eventfd-flags-t))

(defun linux-aio-test (pathname &key (chunk-size 4096))
  (with-foreign-object (context 'aio-context-t)
    (iolib/syscalls:memset context 0 (foreign-type-size 'aio-context-t))
    (let ((eventfd (eventfd 0 :nonblock)))
      (unwind-protect
           (with-open-file (stream pathname :element-type '(unsigned-byte 8))
             (let* ((length (file-length stream))
                    (chunks (ceiling length chunk-size)))
               (with-foreign-object (buffer :uint8 length)
                 (with-event-base (event-base)
                   (io-setup 1 context) ; set up with number of possible operations
                   (with-foreign-object (iocbs '(:struct iocb) chunks)
                     (iolib/syscalls:memset iocbs 0 (* (foreign-type-size '(:struct iocb)) chunks))
                     ;; set up array of operations
                     (dotimes (i chunks)
                       (let ((iocb (mem-aptr iocbs '(:struct iocb) i)))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-lio-opcode) :pread)
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-buf) (pointer-address (mem-aptr buffer :uint8 (* i chunk-size))))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-nbytes) (if (eql i (1- chunks)) (- length (* i chunk-size)) chunk-size))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-fildes) (sb-sys:fd-stream-fd stream))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-offset) (* i chunk-size))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-flags) (foreign-enum-value 'iocb-flags-t :resfd))
                         (setf (foreign-slot-value iocb '(:struct iocb) 'aio-resfd) eventfd)))
                     ;; set up array of pointers to operations
                     (with-foreign-object (iocbp '(:pointer (:struct iocb)) chunks)
                       (dotimes (i chunks)
                         (setf (mem-aref iocbp '(:pointer (:struct iocb)) i) (mem-aptr iocbs '(:struct iocb) i)))
                       ;; submit as many operations as possible
                       (let ((submitted (io-submit (mem-ref context 'aio-context-t) chunks iocbp)))
                         ;; keep track of how many operations completed total
                         (let ((total-events-read 0))
                           (flet ((get-events () ; named to be able to RETURN-FROM
                                    (with-foreign-object (events '(:struct io-event) 3)
                                      (loop
                                        (handler-case
                                            (with-foreign-object (available-buffer :uint64)
                                              (iolib/syscalls:read eventfd available-buffer 8)
                                              (let ((available (mem-ref available-buffer :uint64)))
                                                (dotimes (i available)
                                                  (let ((events-read (io-getevents (mem-ref context 'aio-context-t) 0 3 events (null-pointer))))
                                                    (when (eql events-read 0)
                                                      (return))
                                                    (incf total-events-read events-read))
                                                  (when (eql total-events-read chunks)
                                                    (return-from linux-aio-test)))))
                                          ;; in case reading would block
                                          (iolib/syscalls:syscall-error ()
                                            (when (< submitted chunks)
                                              (let ((more-submitted (io-submit (mem-ref context 'aio-context-t) chunks (mem-aptr iocbp '(:pointer (:struct iocb)) submitted))))
                                                (incf submitted more-submitted)))
                                            (return-from get-events)))))))
                             (set-io-handler
                              event-base eventfd :read
                              (lambda (fd event exception)
                                (declare (ignore fd event exception))
                                (get-events)))
                             (event-dispatch event-base))))))))))
        (io-destroy (mem-ref context 'aio-context-t))
        (iolib/syscalls:close eventfd)))))

Relatively straightforward. Complexity comes from accurately submitting chunks, reading the number of available events on demand and submitting a new batch of chunks as long as there are some remaining ones.

Insert FORMAT statement as you like. Tuning the values would need to be considered in order to keep memory consumption in check. Finally

Outlook

We still can't do many local file operations asynchronously. The whole reason to jump through these hoops is of course to integrate potentially blocking operations into an event loop, so some care still needs to be taken to do some work ahead of time or in separate threads as to not block the main part of the program from issuing the I/O requests.

Integrated tools #2
posted on 2016-10-15 23:53:14

In order to integrate any (interactive) application the following three parts need to be available, otherwise the integrator will have trouble one way or another:

  • objects, representing the domain model; these would be inspectable in order to get the contained information,
  • behaviours, allows the external user to invoke them,
  • notifications, to react to state changes

E.g. not only does an application need to expose its windows, but also it needs to allow to close or open a window and finally send updates if a window was closed or opened. Notifications in particular need to be in a channel that allows to poll multiple of them at the same time - if communication is done over a socket select and friends would allow the other side to control and react to multiple applications at the same time.

Debugging Java with ABCL
posted on 2016-09-10 00:05:24

So after adding a dispatch mechanism into the DISASSEMBLE function of ABCL I also want to show a neat way of disassembling arbitrary Java classes using one of the provided disassemblers.

First of, make sure that you have the ObjectWeb ASM library in your classpath (everything up from version 3 should work at least, perhaps even lower versions), note that in future releases you might be able to load it via Maven and one of the optional contribs as well, likely via (require '#:asm-all).

Next up, try (disassemble #'list) and confirm that some output is shown.

Now to show bytecode for arbitrary Java classes we've got to jump through some additional hoops - though perhaps at some point this could become a stable API as well:

(system::objectweb-disassemble (#"getResourceAsStream" (java:jclass "java.lang.Object") "/java/util/Random.class"))

I haven't tried this via the JAD disassembler, but it seems likely that a similar approach should work for it too.

Java integration woes
posted on 2016-08-18 21:26:13

Just a quick note on the JVM ecosystem since I've been wrestling with getting several different technologies to work together: It's a mess really.

The specific setup in this case is a mostly Java based project, sprinkled with some Kotlin code (which I only expect to grow in the future), using Maven as the build system. Added to that some Kotlin annotations (in lieu of using Kotlin in the first place).

Todays (and yesterdays) adventure was trying to get the Error Prone checker integrated with the existing system, which proved quite impossible, due to the fact that it's using a modified compiler(!) which conflicts with the use of Lombok annotation processing.

There are workarounds in the sense that Lombok can also be used to produce processed Java files (instead of byte code generation), however it seems like that process is less capable than the IDEA / internal processing and would have me remove a lot of val instances that didn't get their type inferred properly, making it an arduous process.

Summing this up, the fact that these tools integrate on different levels of the "stack", while also making tinkering with it relatively hard due to byte code generation, complicates this endeavour greatly. In the end I resolved to drop the Error Prone integration in favour of the much easier to setup SonarQube platform. I also hope that annotation processing for Lombok will improve such that we don't need workarounds in case of "invisible" getters anymore, for example.

Setting up an ABCL mirror from SVN to Git
posted on 2016-08-03 12:03:02

Copies of copies

As it so happens the ABCL main repository is held in Subversion, but since I'm more productive with Git I usually use a conversion to Git via the git-svn program. I've based my own copy off of slryus' one, however I had to fetch and update my own local copy since neither of us had done any fetching in quite some time. I'm writing down some notes below to make sure I (or others) can repeat the process in the future.

First we need a copy of an existing Git conversion.

git clone https://github.com/Ferada/abcl.git abcl-git
# or in case you have SSH access
git clone git@github.com:Ferada/abcl.git abcl-git

The master branch should be a direct mirror of the SVN repository, i.e. only have commits with git-svn annotations, like git-svn-id: http://abcl.org/svn/trunk/abcl@14851 1c010e3e-....

Next we want to initialise the SVN remote and fetch the new commits.

git svn init --prefix=abcl/ http://abcl.org/svn/trunk/abcl
git svn fetch -r 14791:14851

Note that the first revision in the fetch command is the last one in the Git master branch and the other one is the current HEAD of the SVN repository.

This process will take just a few moments, however while all the new commits will be based off of the master branch in Git, the first commit will be a duplicate and have more changes than the existing commit in Git.

2015-08-31 20:55 mevenson           │ │ o │ │ │ ansi-test: reference new git repository
2015-07-01 04:16 mevenson           o─┴─│─┴─┴─┘ abcl-asdf: fix usage with local repository   <<<<
2015-07-01 04:16 mevenson           │ I─┘ abcl-asdf: fix usage with local repository         <<<<
2015-06-30 18:42 mevenson           o abcl-asdf: correct metadata

The newly created first commit (14791 here) has lots of changes.

.dir-locals.el
CHANGES
COPYING
MANUAL
README
abcl.asd
abcl.bat.in
...

While the second, pre-existing one, has way less (and is just showing the same changes that are in the SVN commit).

contrib/abcl-asdf/abcl-asdf.asd
contrib/abcl-asdf/abcl-asdf.lisp
contrib/abcl-asdf/maven-embedder.lisp

To correct that and continue with a linear history I'm using the interactive rebase command.

git co -b test
git reset --hard abcl/git-svn
git rebase -i <commit _before_ the duplicated one>

Copy the hash of the correct, smaller commit and replace the pick ... line (at the top) that contains the duplicated commit with the one from the cloned-from Git repository, then exit the editor. Once the rebase is done we just need to update the ref for git-svn.

git update-ref refs/remotes/abcl/git-svn test

Note the syntax. If done correctly it will be updated in e.g. tig, if not there'll be a new entry showing up in git show-ref.

Lastly we want to rebase master to get the new commits.

git co master
git co -D test
git rebase abcl/git-svn master

Which will look somewhat like this:

2016-06-13 08:06 mevenson           o [master] {abcl/git-svn} {github/master} doc: note changes for abcl-1.3.4
2016-05-16 23:05 mevenson           o Update to asdf-3.1.7
2016-05-16 21:43 mevenson           o Update to jna-4.2.2

And push the updates.

git push origin master
Setting up ABCL and LIRE
posted on 2015-11-11 20:34:29

Intro

The purpose of this article is to examine how using ABCL with existing libraries (arguably the main point of using ABCL at the moment) actually looks like in practice. Never mind integration with Spring, or other more involved frameworks, this will only touch a single library and won't require us to write from-Java-callable classes.

In the process of refining this I'm hoping to also get ideas about the requirements for building a better DSL for the Java FFI, based on the intended "look" of the code (that is, coding by wishful thinking).

Setup

Ensuring the correct package is somewhat optional:

(in-package #:cl-user)

Generally using JSS is a bit nicer than the plain Java FFI. After the contribs are loaded, JSS can be required and used:

(require '#:abcl-contrib)
(require '#:jss)

(use-package '#:jss)

Next, we need access to the right libraries. After building LIRE from source and executing the mvn dist command we end up with a JAR file for LIRE and several dependencies in the lib folder. All of them need to be on the classpath:

(progn
  (add-to-classpath "~/src/LIRE/dist/lire.jar")
  (mapc #'add-to-classpath (directory "~/src/LIRE/dist/lib/*.jar")))

Prelude

Since we're going to read pictures in a couple of places, a helper to load one from a pathname is a good start:

(defun read-image (pathname)
  (#"read" 'javax.imageio.ImageIO (new 'java.io.File (namestring pathname))))

To note here is the use of NEW from JSS with a symbol for the class name, the conversion of the pathname to a regular string, since the Java side doesn't expect a Lisp object and the #"" reader syntax from JSS to invoke the method read in a bit of a simpler way than using the FFI calls directly.

JSS will automatically "import" Java names, so the same function can simply be the following instead (provided that the names aren't ambiguous):

(defun read-image (pathname)
  (#"read" 'ImageIO (new 'File (namestring pathname))))

The names will be looked up again on every call though, so this option isn't the best performing one.

For comparison, the raw FFI would be a bit more verbose, but explicitely specifies all names:

(defun read-image (pathname)
  (jstatic "read" "javax.imageio.ImageIO" (jnew "java.io.File" (namestring pathname))))

Though with a combination of JSS and cached lookup it could be nicer, even though the setup is more verbose:

(defvar +image-io+ (jclass "javax.imageio.ImageIO"))
(defvar +file+ (jclass "java.io.File"))

(defun read-image (pathname)
  (#"read" +image-io+ (jnew +file+ (namestring pathname))))

At this point without other improvements (auto-coercion of pathnames, importing namespaces) it's about as factored as it will be (except moving every single call into its own Lisp wrapper function).

Building an index

To keep it simple building the index will be done from a list of pathnames in a single step while providing the path of the index as a separate parameter:

(defun build-index (index-name pathnames)
  (let ((global-document-builder
          (new 'GlobalDocumentBuilder (find-java-class 'CEDD)))
        (index-writer (#"createIndexWriter"
                       'LuceneUtils
                       index-name
                       +true+
                       (get-java-field 'LuceneUtils$AnalyzerType "WhitespaceAnalyzer"))))
    (unwind-protect
         (dolist (pathname pathnames)
           (let ((pathname (namestring pathname)))
             (format T "Indexing ~A ..." pathname)
             (let* ((image (read-image pathname))
                    (document (#"createDocument" global-document-builder image pathname)))
               (#"addDocument" index-writer document))
             (format T " done.~%")))
      (#"closeWriter" 'LuceneUtils index-writer))))

Note: This code won't work on current ABCL as is, because the lookup is disabled for for nested classes (those containing the dollar character). Because of this, the AnalyzerType class would have to be looked up as follows:

(jfield "net.semanticmetadata.lire.utils.LuceneUtils$AnalyzerType" "WhitespaceAnalyzer")

All in all nothing fancy, JSS takes care of a lot of typing as the names are all unique enough.

The process is simply creating the document builder and index writer, reading all the files one by one and adding them to the index. There's no error checking at the moment though.

To note here is that looking up the precise kind of a Java name is a bit of a hassle. Of course intuition goes a long way, but again, manually figuring out whether a name is a nested class or static/enum field is annoying enough since it involves either repeated calls to JAPROPOS, or reading more Java documentation.

Apart from that, this is mostly a direct transcription. Unfortunately written this way there's no point in creating a WITH-OPEN-* macro to automatically close the writer, however, looking at the LuceneUtils source this could be accomplished by directly calling close on the writer object instead - a corresponding macro might this then:

(defmacro with-open ((name value) &body body)
  `(let ((,name ,value))
     (unwind-protect
          (progn ,@body)
       (#"close" ,name))))

It would also be nice to have auto conversion using keywords for enum values instead of needing to look up the value manually.

Querying an index

The other way round, looking up related pictures by passing in an example, is done using an image searcher:

(defun query-index (index-name pathname)
  (let* ((image (read-image pathname))
         (index-reader (#"open" 'DirectoryReader
                                (#"open" 'FSDirectory
                                         (#"get" 'Paths index-name (jnew-array "java.lang.String" 0))))))
    (unwind-protect
         (let* ((image-searcher (new 'GenericFastImageSearcher 30 (find-java-class 'CEDD)))
                (hits (#"search" image-searcher image index-reader)))
           (dotimes (i (#"length" hits))
             (let ((found-pathname (#"getValues" (#"document" index-reader (#"documentID" hits i))
                                                 (get-java-field 'builders.DocumentBuilder "FIELD_NAME_IDENTIFIER"))))
               (format T "~F: ~A~%" (#"score" hits i) found-pathname))))
      (#"closeReader" 'LuceneUtils index-reader))))

To note here is that the get call on java.nio.file.Paths took way more time to figure out than should've been necessary: Essentially the method is using a variable number of arguments, but the FFI doesn't help in any way, so the array (of the correct type!) needs to be set up manually, especially if the number of variable arguments is zero. This is not obvious at first and also takes unnecessary writing.

The rest of the code is straightforward again. At least a common wrapper for the length call would be nice, but since the result object doesn't actually implement a collection interface, the point about having better collection iteration is kind of moot here.

A better DSL

Considering how verbose the previous examples were, how would the "ideal" way look like?

There are different ways which are more, or less intertwined with Java semantics. On the one end, we could imagine something akin to "Java in Lisp":

(defun read-image (pathname)
  (ImageIO/read (FileInputStream. pathname)))

Which is almost how it would look like in Clojure. However, this is complicating semantics. While importing would be an extension to the package mechanism (or possibly just a file-wide setting), the Class/field syntax and Class. syntax are non-trivial reader extensions, not from the actual implementation point of view, but from the user point of view. They'd basically disallow a wide range of formerly legal Lisp names.

(defun read-image (pathname)
  (#"read" 'ImageIO (new 'FileInputStream pathname)))

This way is the middle ground that we have now. The one addition here could be that name lookup is done at macro expansion / compilation time, so they are fixed one step before execution, whereas at the moment the JSS reader macro will allow for very late bound name lookup instead.

The similarity with CLOS would be the use of symbols for class names, but the distinction is still there, since there's not much in terms of integrating CLOS and Java OO yet (which might not be desirable anyway?).

Auto-coercion to Java data types also takes place in both cases. Generally this would be appropriate, except for places where we'd really want the Java side to receive a Lisp object. Having a special variable to disable conversion might be enough for these purposes.

If we were to forego the nice properties of JSS by requiring a function form, the following would be another option:

(defun read-image (pathname)
  $(read 'ImageIO (new 'FileInputStream pathname)))

Where $(...) would be special syntax indicating a Java method call. Of course the exact syntax is not very relevant, more importantly static properties could be used to generate a faster, early bound call by examining the supplied arguments as a limited form of type inference.

Summary

After introducing the necessary steps to start using ABCL with "native" Java libraries, we transcribed two example programs from the library homepage.

Part of this process was to examine how the interaction between the Common Lisp and Java parts looks like, using the "raw" and the simplified JSS API. In all cases the FFI is clunkier than needs be. Especially the additional Java namespaces are making things longer than necessary. The obvious way of "importing" classes by storing a reference in a Lisp variable is viable, but again isn't automated.

Based on the verbose nature of the Java calls an idea about how a more concise FFI DSL could look like was developed next and discussed. At a future point in time this idea could now be developed fully and integrated (as a contrib) into ABCL.

Limits of Unix shell
posted on 2015-08-20 14:54:10

Let's go back to the GNU Coreutils list of tools. ls for example. Usually the user will have set some alias to ls instead of the plain invocation, either to enable highlighting (--color), sorting (--sort), or to add more information than just the filenames (e.g. --format). There is even integration with Emacs (--dired).

The question then is: How much of the functionality of ls is actually devoted to secondary formatting instead of listing files? And shouldn't this functionality be moved into separate tools? Since output is intended for multiple kinds of recipients, additional data creeps in and complicate tools a lot.

Alternatively, we could imagine using ls only to get unformatted and unsorted output. Which would then be passed through to a sort command and a fmt command of sorts. Of course this all takes some more time, re-parsing of output etc., so it's understandable in the interest of performance not to do this in the traditional Unix shell.

However, let's assume a more sophisticated shell. Assuming ls is limited to listing files, then the user will alias ls to a pipeline instead, namely something akin to ls | sort | fmt. Then again, formatting is part of the user interface, not the functionality, so it should rather be part of the internal shell formatting, possibly exposed as separate filters as well.

The result of ls is a (possibly nested) directory listing. Regardless of post-processing, this "object" should still be available for further investigation. Which means that while sorting may be applied destructively, formatting may not, unless specifically requested, in which case the result would be a kind of "formatted" object (text, GUI widget) instead.

In other terms, the user should be able to refer to the last results immediately, instead of rerunning the whole pipeline. E.g. coming from Common Lisp, variables like * to *** will store the last three results for interactive use. In the shell then, ls would set * to the generated directory listing; since the listing is also most likely printed to the screen, the full listing will also be stored (in that object) to be used again if e.g. * is requested again. Rerunning the command, on the other hand, will possibly generate a different directory listing as files may have been changed, so there is an immediate difference between the two forms.

Examples

The pipeline ls | wc -l is (at least for me) often used to get the number of files in the (current) directory. Unfortunately there is no direct way to get this number directly except to enumerate the entries in a directory (under Linux that is).

Previous Next

This blog covers work, unix, tachikoma, postgresql, lisp, kotlin, java, hardware, git, emacs

View content from 2014-08, 2014-11, 2014-12, 2015-01, 2015-02, 2015-04, 2015-06, 2015-08, 2015-11, 2016-08, 2016-09, 2016-10, 2016-11, 2017-06, 2017-07, 2017-12, 2018-04, 2018-07, 2018-08


Unless otherwise credited all material Creative Commons License by Olof-Joachim Frahm