Continuing from an earlier post, what might the semantics be that we'd like to have for a more useful shell integrated with a Lisp environment?

Syntax

Of course we can always keep the regular Common Lisp reader; however it's not best suited for interactive shell use. In fact I'd say interactive use period requires heavy use of editing commands to juggle parens. Which is why some people use one of the simplifications of not requiring the outermost layer of parens on the REPL.

So, one direction would be to have better S-expression manipulation on the REPL, the other to have a syntax that's more incremental than S-expressions.

E.g. imagine the scenario that,

1. as a terminal user,
2. I'm navigating directories,
3. listing files,
4. then looking for how many files there were,
5. then grepping for a particular pattern,
6. then opening one of the matches.

In sh terms that's something the lines of

$ cd foo
$ ls
...
$ ls | wc -l
42
$ ls | grep bar
...
$ vim ...

In reality I'm somewhat sure no one's going as far as doing the last two steps in an iterative fashion, like via ls | grep x | grep y | xargs vim, as most people will have a mouse readily available to select the desired name. There are some terminal widgets which allow the user to select from e.g. one of the input lines in a command line dialog, but again it's not a widespread pattern and still requires manual intervention in such a case.

Also note that instead of reusing the computation the whole expression keeps being reused, which also makes this somewhat inefficient. The notion of the "current" thing being worked on ("this", "self") also isn't expressed directly here.

In the new shell I'd like to see part of explored. We already have the three special variables *, ** and *** (and / etc.!) in Common Lisp - R, IPython and other environments usually generalise this to a numbered history, which arguably we might want to add here as well - so it stands to reason that these special variables make sense for a shell as well.

$ cd foo
$ ls
...
$ * | wc
42
$ grep ** bar
...
$ vim *

(Disregarding the need for a different globbing character ...)

There's also the very special exit status special variable in shells that we need to replicate. This will likely be similar to Perl special variables that keep track of one particular thing instead of reusing the * triad of variables for this too.

Pipelines

The expression compiler should be able to convert from a serial to a concurrent form as necessary, that is, by converting to the required form at each pipeline step.

ls | wc -l | vim

Here, ls will be the built-in LIST-DIRECTORY, which is capable of outputting e.g. a stream of entries. wc -l might be compiled to simply LENGTH, which, since it operates on fixed sequences, will force ls to be fully evaluated (disregarding a potential optimisation of just counting the entries instead of producing all filenames). The pipe to vim will then force the wc output to text format, since vim is an unknown native binary.

These would be the default conversions. We might force a particular interpretation by adding explicit (type) conversions, or adding annotations to certain forms to explain that they accept a certain input/output format; for external binaries the annotations necessarily would be more extensive.

For actual pipelines it's extremely important that each step progresses as new data becomes available. Including proper backpressure this will ensure good performance and keep the buffer bloat to a minimum. Again this might be a tunable behaviour too.

I/O

The shell should be convenient. As such any output and error output should be captured as a standard behaviour, dropping data if the buffers would get too big. This will then allow us to refer to previous output without having to reevaluate any expression / without having to run a program again (which in both cases might be expensive).

Each datatype and object should be able to be rendered and parsed in more than a single format too. E.g. a directory listing itself might be an object, but will typically be rendered to text/JSON/XML, or to a graphical representation even.