Content from 2018-12

Did you know Scala has macros? Coming from Common Lisp they serve pretty much the same purpose, doing things that the (plethora of) other language features don't support and to shortcut the resulting boilerplate code. And even the S-expressions can be had when macro debugging is turned on, though the pretty-printed Scala code is arguably much more useful here.

Why would you realistically use them then? Turns out I had to deal with some auto-generated code dealing with Protobuf messages. The generated classes for any message look something like this (in Java syntax since that's what the generated code is):

public interface ExampleResponseOrBuilder
  extends com.google.protobuf.MessageOrBuilder;

public static final class ExampleResponse
  extends com.google.protobuf.GeneratedMessageV3
  implements ExampleReponseOrBuilder {

  public static Builder newBuilder();

  public static final class Builder
    extends com.google.protobuf.GeneratedMessageV3.Builder<Builder>
    implements ExampleResponseOrBuilder;
}

That is, we have one interface, two classes, one of them conveniently gives you a builder for new objects of the class. That's used like this (back to Scala here):

val builder: ExampleResponse.Builder = Example.newBuilder()
builder.mergeFrom(stream)
val result: ExampleResponse = builder.build()

If you try and make a generic builder here, you'll quickly notice that this is rather hard as the generic types don't really express the relationship between ExampleResponse and ExampleResponse.Builder well.

As an aside, you want to have a generic builder parametrised on the return type to be able to write something like this:

val result = build[ExampleResponse](stream)

Without ever having to pass through the type as a value. Better even if you just specify the result type and the type parameter for build is then automatically derived.

These builders look something like this then:

trait ProtobufBuilder[T <: Message] {
  def underlying(): Message.Builder

  def build(string: String)(implicit parser: JsonFormat.Parser): T = {
    val builder = underlying()
    parser.merge(string, builder)
    builder.build().asInstanceOf[T]
  }
}

class ExampleResponseBuilder() extends ProtobufBuilder[ExampleResponse] {
  override def underlying(): ExampleResponse.Builder =
    ExampleResponse.newBuilder()
}

This then allows us to use some implicit magic to pass these through to the decoder (sttp's framework in this case) to correctly decode the incoming data.

But, we've to 1. write one class for each type, 2. instantiate it. This is roughly five lines of code per type depending on the formatting.

Macros to the rescue!

Inspired by the circe derivation API I finally got all the pieces together to create such a macro:

def deriveProtobufBuilder[T <: Message]: ProtobufBuilder[T] = macro deriveProtobufBuilder_impl[T]

def deriveProtobufBuilder_impl[T <: Message: c.WeakTypeTag](
    c: blackbox.Context): c.Expr[ProtobufBuilder[T]] = {
  import c.universe._

  val messageType   = weakTypeOf[T]
  val companionType = messageType.typeSymbol.companion

  c.Expr[ProtobufBuilder[T]](q"""
    new ProtobufBuilder[$messageType] {
      override def underlying(): $companionType.Builder = $companionType.newBuilder()
    }
   """)
}

Used then like this:

private implicit val exampleResponseBuilder: ProtobufBuilder[ExampleResponse] = deriveProtobufBuilder

That's one or two lines and the types are only mentioned once (the variable name can be changed). Unfortunately getting rid of the variable name doesn't seem to be possible.

Easy, wasn't it? Unfortunately all of this is hampered by the rather undocumented APIs, you really have to search for existing code or Stackoverflow questions to figure this out.

One thing that helped immensly was the -Ymacro-debug-lite option, which prints the expanded macro when used in sbt via compile.

I wrote down a little scenario for myself for the next iteration of act:

• mark "http://www.google.de" with mouse (selection -> goes to x11 buffer-cut)
• listen on those changes and show current context and choices
• press+release "act primary" key -> runs primary action immediately
• (or) press "act" key once -> opens buffer for single character input to select choice
• buffer captures keyboard, after single character action is taken, esc aborts, "act" again should expand the buffer to full repl
• mouse -> selection of source? history?
• context -> focused window? -> lookup for program gives cwd, etc.
• since that's not foolproof an obvious addition to the protocol would be to store a reference to the source of the clipboard (or the program that will be queried for it)
• you should always be able to interrogate windows about their origin and capabilities, including their scripting interface
• pattern matching url -> rule based decisions
• primary / secondary actions -> rule of three basically, except we err on the side of caution and just bind two actions two keys
• special handling for clipboard types that aren't text? allow for example to override a rule based on whether a picture would be available

Now, there are several problems with this on a regular X11-based desktop already:

• How do you identify the program / PID belonging to a particular window?
• How do you get information about that program, e.g. its current working directory?

In addition, since I'm using tmux inside a terminal emulator there are some more problems:

• How do you know which tmux session is currently active?
• How do you know which program is currently "active" in a tmux session?

Basically it's a recursive lookup for the "current state" of what's being displayed to the user. Not only that, but for things like browsers, image editors, video editors, anything document based it's still the same problem at another level, namely, what the current "context" is, like the currently open website, picture, video, scene, what have you.

Coming back to earlier thoughts about automation, there's no way for most of these to be accurately determined at this time. Neither is e.g. DBUS scripting common enough to "just use it" for scripting, there are also several links missing in the scenario described above and some can likely never be fixed to a sufficient degree to not rely on heuristics.

Nevertheless, with said heuristics it's still possible to get to a state where a productivity plus can be achieved with only moderate amount of additional logic to step through all the indirections between processes and presentation layers.

Now let me list a few answers to the questions raised above:

• The PID for a particular window can be derived from an X11 property, together with xdotool this gives us an easy pipeline to get this value: ``.
• Information about the running process can then be retrieved via the proc filesystem, e.g. readlink /proc/$PID/cwd for the current working directory. Of course this has limited value in a multi-threaded program or any environment that doesn't rely on the standard filesystem interface (but uses its own defaults).
• I do not have an answer for the currently active tmux session yet, presumably you should somehow be able to get from a PID to a socket and thus to the session?
• For tmux, the currently active program in a session is a bit more complex, ``, which we'll also have to filter for the active values.

For scripting interfaces, many programs have their own little implementation of this, but most problematic here is that you want to go from a X11 window / PID to the scripting interface, not through some workaround by querying for interfaces!

For programs like Emacs and any programmable environment we can likely script something together, but again it's not a very coherent whole by any means.

Intro

Regardless of your position on DBus, sometimes you might need to interact with it. Common Lisp currently has at least two libraries ready for you, one of them is in Quicklisp, https://github.com/death/dbus/.

Setup

Have it loaded and create a package to use it, then change into it.

;; (asdf:load-system '#:dbus)
;; or
;; (ql:quickload "dbus)

(defpackage #:example
  (:use #:cl #:dbus))

(in-package #:example)

For reference, I'm going to refer to a (very old) polkit example in Python. For reference I'm reproducing it here (it still works in current Python 3 without any changes except the print):

import dbus

bus = dbus.SystemBus()
proxy = bus.get_object('org.freedesktop.PolicyKit1', '/org/freedesktop/PolicyKit1/Authority')
authority = dbus.Interface(proxy, dbus_interface='org.freedesktop.PolicyKit1.Authority')

system_bus_name = bus.get_unique_name()

subject = ('system-bus-name', {'name' : system_bus_name})
action_id = 'org.freedesktop.policykit.exec'
details = {}
flags = 1            # AllowUserInteraction flag
cancellation_id = '' # No cancellation id

result = authority.CheckAuthorization(subject, action_id, details, flags, cancellation_id)

print(result)

So, how does this look in Common Lisp? Mostly the same, except that at least at the moment you have to specify the variant type explicitly! This was also the reason to document the example, it's quite hard to understand what's wrong if there's a mistake, including the socket connection just dying on you and other fun stuff.

(with-open-bus (bus (system-server-addresses))
  (with-introspected-object (authority bus
                                       "/org/freedesktop/PolicyKit1/Authority"
                                       "org.freedesktop.PolicyKit1")
    (let* ((subject `("system-bus-name" (("name" ((:string) ,(bus-name bus))))))
           (action-id "org.freedesktop.policykit.exec")
           (details ())
           (flags 1)
           (cancellation-id "")
           (result
                (authority "org.freedesktop.PolicyKit1.Authority" "CheckAuthorization"
                           subject action-id details flags cancellation-id)))
      (format T "~A~%" result))))

Note the encoding of the dictionary: The type of the whole argument is specified as (sa{sv}), a structure of a string and a dictionary of strings to variants - we're spelling out the variant type here, compared to what's automatically done by the Python library.