For the longest time I've been using quite a number of Firefox extensions. The known problem with that is a steady slowdown, which is only amplified by my habit of soft bookmarks, i.e. having hundreds of open tabs with their corresponding state (which is the whole reason to do that).
However seeing that a lot of state is captured in a very inconvenient form, that is, it's hard to modify a long list of tabs, I want to make both this and incidentally also sharing of state between sessions and even browsers much easier.
The idea is to separate part of the browser state into a separate component, namely a database server for cookies (and other local storage), tabs, sessions and bookmarks. This way and by having a coarse control over loading of sessions the process of migrating state between sessions and browsers should be much easier.
Fortunately most of the browser extensions APIs seem to be usable enough to make this work for at least Firefox and Chrome, so at the moment I'm prototyping the data exchange. Weird as it is for Chrome you have to jump through some conversion hoops (aka local native extensions via a local process exchanging data via stdio), so it seems that the Firefox APIs, since they allow socket connections, are a bit friendlier to use. That said, the exchange format for Chrome, Pascal string encoded JSON, seems like a good idea with the exception of forcing local endianess, which is completely out of the question for a possibly network enabled system (which is to say, I'm definitely going to force network byte order instead).
The title sounds a bit too majestic too be true. However the though just occured to me, that much of what I've been doing over the last months often involved taking "dev ops" code, e.g. configuration code that lives in the deployment scripts, and putting it into a reusable format, i.e. an actual (Python) module.
That usually happens because what was once a depencency for an application (or service if you like), is now needed to be accessible from (Python) code for testing purposes, or because the setup routine isn't actually just setup any more, but happens more regularly as part of the application cycle.
Of course doing this has some major downsides, as the way scripts are written, using a specific library to access remote hosts, without much error handling, is fundamentally different from how a application code works, that is usually with a more powerful database interface, without any shell scripting underlying the commands (which will instead be replaced by "native" file manipulation commands) and with more proper data structures.
That leaves me with some real annoying busy work just to transform code from
one way of writing it to another. Maybe the thing to take away here is that
configuration code isn't and application code will sooner or later become
library code as well -- aka. build everything as reusable modules. This of
course means that using certain configuration (dev ops) frameworks is
prohibited, as they work from the outside in, e.g. by providing a wrapper
application (for Fabric that would be
fab) that invokes "tasks" written in a
Properly done, that would mean that configuration code would be a wrapper so thin that the actual code could still be reused from application code and different projects later on. The difference between configuration and business logic would then be more of a distinction between where code is used, not how and which framework it was written against.
After years of annoyance with the X11 behaviour for clipboard and selection handling with regards to terminal applications, I managed to find a good compromise via some additional shortcurts in my window manager of choice, dwm and terminal multiplexer, tmux.
C-o p (in tmux) pastes from X11 primary selection,
C-o P from
the clipboard and
Linux-z (meaning the key formerly known is Windows key)
exchanges clipboard and primary selection, so no more awkward pasting and
selecting with the mouse in order to get the correct string in the correct
# to switch clipboard and primary selection PRIMARY=`xsel -op`; xsel -ob | xsel -ip; echo "$PRIMARY" | xsel -ib # to paste from primary selection in tmux bind p run "tmux set-buffer \"$(xsel -op)\"; tmux paste-buffer
Other useful keybindings in tmux would copying into the clipboard etc. and there is a useful SO post explaning that.
As before this is public at the customisations branch on Github; I still have to upload the tmux part though.
After some time thinking about and rewriting the library in a subtly
looks good to me as a version one.
I've removed all mentions of generic functions for now, as first of all
I'm unsure if functionality to dynamically rebind methods is even
necessary, and second, because doing that is complicated by the details
of that protocol. Which means that specifying which method to override
is a bit hairy and I really want a good syntax before I let that stuff
loose. So it'll have to wait until I figure out a good way to do that.
Since it should be easily added to the existing frontend, it will very
probably be done with some overloading of existing functions / macros
(e.g. with a
:method specifier or so).
I'm hoping to test all of this and possibly investigate the generic
function issue on some other library. At the moment my single more
complex example is a replacement for the
HTTP-REQUEST call, which
worked surprisingly well and might even make it into a new test suite.
The benefit is obviously the improved reliability of not having to have
a running network connection for testing libraries against a (HTTP)
Since it came up on Hacker News I thought I can write a little bit about that topic as well.
I started to look for alternatives to the distribution default desktop environment relatively soon after arriving on Linux (Fedora if I remember correctly). At that point the options included Fluxbox and a couple of smaller ones like I3 and wmii. I also tried twm, but honestly, without any effort spend in themeing that was basically not really viable.
So after Fluxbox, which was great, but still leaves you with too much to do with your mouse, my conclusion was that I basically don't need a regular desktop. Having all those messy icons, menus, widgets lying around the screen is just way too distracting for me.
If you then remove all that decoration, you are left with a very bare bones look. Still, after starting to get the hang of Vim (with which I started) and later Emacs, the disadvantage of constantly having to deal with window positions became apparent.
I think the next step was to use wmii or one variant of that. Tiling leaves your mouse free to interact with the main point, your running program. No more juggling windows, aligning borders and so on. For me this isn't about a pretty and flashy screen, it's about the most comfortable environment to work in.
To the present day: I'm no converted to dwm from the awesome people of suckless.org. It's basically a single C file, you configure it with a header and additionally with a custom patch set and that's it. You'd be hard pressed to find a smaller, less resource intensive window manager. And on the flip side it has many amazing features which just work really well together.
Combined with tmux for terminal multiplexing, Emacs buffers for editing multiplexing and dwm for desktop and screen multiplexing this is just the right amount of flexibility to arrange and move around a lot of context.
Obviously this depends on each person, but since you can (and frankly, should) configure every aspect of this, with just a few keypresses you can switch to every part of your running programs and back, be it in the terminal, on a remote system, or graphical.
To be honest, until there is a better alternative to keyboards, I think I'll keep using this approach, maybe adding more scripting capabilities in the same line as in previous blog posts.
What different components would a daemon expose via the message bus, respectively knowledge store?
For one, methods for remote control. Since RPCs aren't very flexible, the communication style should use message passing. Waiting for response needs to be both aware of possible timeouts and communication failures.
Passed objects can't be too complex, since every interacting language/environment needs to be able to access, create, possibly manipulate them, as well as to minimize the amount of overhead during serialisation. At the same time the schema of messages isn't fixed, so there is an obvious problem for very strict type systems, in that either a fixed-schema wrapper can be used, which would need to be updated at some point, or be kept backwards-compatible, or a very dynamic representation with runtime checks would have to be implemented.
Comparable string based messages from e.g. Plan9 are too simple on the one hand, whereas a protocol like DBus might be overengineered(?).
An important point to consider is the in(tro)spection features of the
global system. It should be absolutely possible to easily see and edit
messages as well as stored data in either a text-based or convertible
format. Also, commands and objects should have runtime documentation in
the system, so that a hypothetical call like
describe object would
display text- (in the terminal), or hypertext based documentation (in
Naming clashes have to be solved by a one or multi level package system. Since the naming is both shared and global, typical approaches include reversed domain-names, UUIDs and prefixing of identifiers.
Session management would include saving and reloading of state, even with different session names. A separate session manager would take care of initialising applications based on the saved state.
For both text- and graphical UIs methods to switch to the active screen/window need to be provided. Copy & paste functionality might still be done via the window system, or additionally via messaging, which would allow connected system, as well as text and graphical applications to exchange content without any problems.
What kind of applications aren't easily possible without a central message bus / knowledge store? Or rephrased as what kind of applications would benefit very much from the use of a central message bus / knowledge store.
To answer this, let's see how IPC usually works between applications: If a program wants to communicate with other parts of the system, there are several options, including writing data to a regular file, pipe/fifo, or socket, as well as directly executing another program. In some cases the communication style is uni-directional, in some bi-directional.
So depending on the setup you can pretty much exactly define how messages are routed between components. However, the details of this communication are hidden from the outside. If you wanted react to one message from another part of the system, you're out of luck. This direct coupling between components doesn't lend itself very well to interception and rerouting.
Unless the program of your choice is very scriptable, you then have no good way to e.g. run a different editor for a file to be opened. Since programs typically don't advertise their internal capabilities to outside use (like CocoaScript (?) allows you to a degree), you also don't have a chance to react to internal events of a program.
Proposed changes to browsers would include decoupling of bookmark handling, cookies/session state, notifications and password management. Additionally it would be useful to expose all of the scripting interface to allow for external control of tabs and windows, as well as possible hooking into website updates, but I think that part is just a side-effect of doing the rest.
Proposed changes to IRC clients / instant messengers would include decoupling of password management and notifications. Additionally the same argument to expose channels/contacts/servers to external applications applies.
Now let's take a look at the knowledge store. In the past I've used a similar Blackboard system to store sensor data and aggregate knowledge from reasoners. The idea behind that is the decoupling of different parts of the program from the data they work on, reacting to input data if necessary and outputting results for other programs to work on.
I imagine that this kind of system relieves programs from creating their own formats for storing data, as well as the need to explicitly specify where to get data from. Compared to a RDBMS to downside is obviously the lack of a hard schema, so the same problems from document based data-stores apply here. Additionally the requirement to have triggers in order to satisfy the subscriptions of clients makes the overall model more complex and harder to optimise.
What is then possible with such a system? Imagine having a single command to switch to a specific buffer regardless of how many programs are open and whether they use a MDI or just a single window. In general scripting of all running programs will be easier.
The knowledge store on the other hand could be used to hold small amounts of data like contacts, the subjects of the newest emails, notifications from people and websites. All of that context data is then available for other programs to use.
Assuming such data was then readily available, using ML to process at least some of the incoming data to look for important bits of information (emails/messages from friends/colleagues, news stories) can then create an additional database of "current events". How this is displayed is again a different problem. The simplest approach would simply be a ticker listening on a specific query, the most complex would maybe consist of whole graphical dashboard.
Security is obviously a problem in such a share-all approach. It should be possible though to restrict access to data similarly to how user accounts in regular DBM systems work and for scripting interactions the system would still have to implement restrictions based on the originating user and group on a single system, as well as the host in a distributed environment.
Yay, found time to give my tumble log a little theme upgrade. And while I'm doing that, I thought I could write down some thoughts on the current design. I might even update it later.
In contrast to this Coleslaw blog I've made a small static renderer to HTML/ATOM for the tumble log, so the interface is mostly console vim to text file with S-expressions. Since at the moment there is only a single file I might just show the syntax for that one:
;; -*- mode: lisp; coding: utf-8-unix; -*- (video :id 1 :date @2013-05-23T00:00:00+01:00 :tags (music) :url #U"https://www.youtube.com/watch?v=QNppCfa_oRs" :title "I Charleston Tel Aviv" :via #U"http://n0p.tumblr.com/post/51100501191/i-charleston-tel-aviv") ... (image :id 3 :date @2014-08-16T01:01:00+01:00 :url #U"http://n0p.tumblr.com/post/94859150275" :src #P"data/tumblr_n17lclGKp51qdewwao1_1280.jpg" :alt "Scene from the mange BLAME!") ... (text :id 22 :date @2014-08-28T21:22:58.047809+01:00 :tags (emacs lisp) :text #"`(global-set-key (kbd "H-1") ...)` Hyper Hyper! In other words, bind `Caps-Lock` to `Hyper` and rejoice!"#) ... (link :id 88 :date @2014-11-20T17:27:05.107413+01:00 :url #U"http://htsql.org/" :title "HTSQL")
Clearly this would be viable to store in a DBMS as well, but this is good
enough as they say. Featuring
PURI, a few other syntax extensions, (my fork
CL-WHO (for XML extensions) and
CL-MARKDOWN for text rendering. The
Markdown rendering could use a few fixes for edge cases, but in general it
works pretty well and fast. Tags are also written to separate files (both HTML
and ATOM), so readers could actually restrict themselves to a few subsets, even
though it's not a by any stretch a full-fledged query system.
Still, by rendering to static files a lot of problems go away and the whole site can be served very efficiently via a small web server (I'm leaving Hunchentoot/teepeedee2 for more involved projects).
Continuing that thought, I feel that the tools we use are very much disconnected from each other. Of course that is the basic tenet, tools which do one thing well. However, tools serve a specific purpose, which in itself can be characterised as providing a specific service. It seems strange that basically the only widely applicable way to connect very different tools is the shell, respectively calling out to other programs.
I would argue that using some another paradigms could rather help to make the interplay between services in this sense much easier. For example, if we were to use not only a message bus, line in Plan9, but also a shared blackboard-like database to exchange messages in a common format, then having different daemons react on state changes would be much easier.
Of course, this would already be possible if the one point, a common exchange format, was satisfied. In that case, using file system notifications together with files, pipes, fifos etc. would probably be enough to have largely the same system, but based on the file system abstraction.
Imagine now, that programs actually were actually reusing functionality from
other services. My favourite pet-peeve would be a shared bookmark list, cookie
store, whatever browsers typically reimplement. It should be no surprise that
very focused browsers like
uzbl actually do just that, allowing for the
combination of different programs rather then the reimplementation of basic
It's unfortunate that tools like
xdg-open aren't very
integrated into shell workflow. Even though names like
muscle-memory for me, having unified generic nomenclature for actions like
open, edit would be very helpful as a first step into a more programmable
environment. Plan9-like decision-making for how to respond to these requests,
or like the aforementioned blackboard/messaging architecture all go into the
If we now compare how command-line tools respond to queries, so for an abstract
help it would naturally be
foo --help (and shame on you if
your tool then outputs "Use
git does it right fortunately), then
we can see how we can procede from here. Not only should we adapt these kind
of queries, but reflectivity is absolutely necessary to provide a more
Even though "worse is better", it hinders the tighter linking of tools, as everything first goes to serialised form, then back through the OS layers. Despite dreaming of a more civilised age (you know, Lisp Machines), a compromise based on light-weight message-passing in a service-based approach, relying on deamons as the implementation detail to provide functionality could be viable. Essentially, what can't be done in a single global memory space we emulate.
Sounds awfully like CORBA and DBUS, right? Well I think an important consideration is the complexity of the protocol versus the effort spent on integration. If it is very easy to link existing applications up to the message bus and if that system allows not only message passing, but also provides styleguides and an implementation of the basic interaction schemes (RPC call, two-way communication, ...), then what sounds otherwise like a lot of reimplementation might actually be more like leveraging existing infrastructure instead.
This blog post on LispCast is a pretty good start to get thinking about the intricacies of the interaction between Lisp (Machine) ideas and the current Unix environment. Of course that includes plan9 on the one side and Emacs on the other.
There is scsh, but it's not really what I'm looking for.
Using emacs as login shell (with the
eshell package) comes closest to it
regarding both with existing commands and integration of Lisp-based ones.
However, while pipes work as expected with
eshell, data is still passed
around as (formatted) text. There doesn't seem to be an easy way to pass
around in-memory objects, at least while staying in Emacs itself. That would
of course mean to reimplement some (larger?) parts of that system.
This all ties in to the idea that unstructured text isn't the best idea to
represent data between processes. Even though Unix pipes are extremely useful,
the ecosystem of shell and C conventions means that the obvious way isn't
completely correct, meaning that there are edge cases to consider. The best is
something as innocent as
ls | wc -l, which will break, depending on the shell
settings, with some (unlikely) characters in filenames, i.e. newlines.
One of the problems is obviously that in order to pass around structured data, i.e. objects, all participants have to understand their format. Passing references won't work without OS support though.
Instead of having unstructured streams, use streams of (data) objects. The distinction here is Plain Old Objects (PODs) instead of objects with an associated behaviour.
Let's take a look at standard Unix command line tools (I'm using GNU Coreutils here) in order to reproduce the behaviour and/or intent behind them:
Output of entire files
The first command here is
cat. Although GNU
cat includes additional
transformations, this command concatenates files. Similar to the description,
we can image a
CAT to perform a similar operation on streams of objects.
It doesn't make much sense to concatenate a HTML document and an MP3 file
(hence you won't do it in most cases anyway). However, since files are
cat can work on them.
Although you can call commands individually on files, some of them form an ad-hoc service interface already: The C compiler, along with the toolchain forms one such interface, where you're required to use the same interface if you want to seamlessly replace one part of the toolchain.
Same goes for the Coreutils: As long as you honour the interface, programs can be replaced with different implementations.
Emacs has a special form
interactive to indicate whether a command can be
directly called via the command prompt. There is also special handling there
to accomodate the use of interactive arguments. This is something that can be
generalised to the OS. An example of where this already happens is the
.mailcap file and the
.desktop files from desktop suites.
Threading and job control
Unfortunately getting proper job control to work will be a bit of a problem in any Lisp implementation, since the best way to implement the concurrent jobs is using threads, which are not particularly suited for handling the multitude of signals and other problems associated with them. Even without job control pipelines implemented in Lisp require shared in-memory communication channels, so something like (object-based) streams, mailboxes, or queues are necessary to move IO between different threads.