Programmable Space — Research Update

January 20, 2019

Previously, I was working on tools to tell a more complete story of the making process and tools for seeing live data. But I heard of Dynamicland and thought it addressed some limitations of the tools I had worked on.

Dynamicland is a communal computer, designed for agency, not apps, where people can think like whole humans. Dynamicland puts the computer into the room instead in a tiny box of pixels. In my seeing tools, it felt wrong to put a tool meant to help people see on a screen when the primary work was going on in the room. Also I wondered how people would make custom visualizations. I have a background in data visualization so I could quickly write code to make a custom graph of data but generally this would be challenging. There was a bad jump in complexity and knowledge if you wanted to extend the system. I was inspired by Dynamicland's generic programming model made it equally easy to make graphics, robots, and improve the system itself.

A tiny text editor inside of premade visualizations in my previous project the Robot "Seeing Tool"

I wanted to judge the merit of these ideas for myself but Dynamicland is a building in Oakland, CA where the physical building and the coding environment are one in the same. Dynamicland is not an app that can be downloaded and run in a new space. A new space with new people should develop a system that works best for them.

I tried out a couple open-source projects related to "programmable spaces" — Paper Programs by JP Posma and Living Room by the Recurse Center. Both were impressively complete but I felt both were too closely tied to JavaScript and using a web browser. Programming was still done on a computer or in a web browser. I also felt both had too strong of a difference between the programs you could make with the system, and the system code itself.

Therefore, I started to make my own "programmable space" where everything in the system would be done in the room without needing to switch back to a traditional programming experience on a GUI operating system.

An early prototype of a programmable space using a projector and camera. A text editor paper is editing a line graph paper. Other papers measure their angle or the number of tokens on them as an input to the line graph paper.

Initially I tried to use a Dynamicland-like setup with cameras and projectors pointing at papers with colored dots. Each paper was an independent program. Papers in the room would have code written on them. If the paper was out in the room, then anyone in the room could look at the paper and read what it does. Likewise, if the paper was out in the room then the cameras could see it (using the colored dots) and cause the paper's code to run. The one-to-one mapping between what people can see and what programs were running helps the system be understandable. Projectors would project the output of the programs on the paper as nonintrusive display. The "text editor" used to edit papers was itself another paper that edited the code of the paper it was pointing at. Programs communicate by writing and reading to a global "tuplespace" — I think of it as a whiteboard, just another object in the room. See the Appendix for more details about this and the code.

This setup worked but was hard to make reliable and keep physically rearranged. The cameras and projectors never felt invisible. The point of the programmable room was to forget about them like you forget about the lights in your ceiling. I was also spending too much time on the code to run the camera and projectors and not spending enough time on the thing I cared about - learning what it's like to work in a programmable room.

To refocus, I scaled back and made a limited but complete demo of a programmable room with a complete editing experience outside a traditional GUI OS.

The room had an always on text-editor and a grid of 9 editable programs. The papers were fixed on the wall so I avoided needing a camera to detect papers. The fixed papers also made the projector easy to set up. Having 9 pre-made programs avoided the issue of updating and creating the physical papers.

After playing around for a few days, I discovered that the room did give promising looking at what it would be like to have a programmable room.

• The always-on text editor, live reloading programs, and a easy and extensible language for communicating between things in the room felt like the right format to orchestrate everything happening in the room. It was satisfying to see something in the room respond within a couple seconds just as easily as I had done only on a screen in the past.
• The room was successfully a stable and complete editing experience away from a GUI OS. It was refreshing to remove all distracting apps.
• I am happy that the 9 editable programs are very similar to and live in the same extensible system as the programs used to bootstrap the system. Although the boot papers weren't editable in this version, they exist, and in the future there can be a gentle transition between making fun programs and editing the system itself. I think this is the key idea to making the space work best for the specific place and people that work inside it.

But some limitations hurt the experience:

• The code is too closely tied to existing programming languages. It reads and looks like code instead of prose. There is no clear separation between the systems language (how things in the room talk) and the logic-level code.
• Having the papers fixed to one wall made the papers loose their essential quality — a cheap and common material that could show the code, be written on, and most importantly be moved around the room. Having a fixed output (the 9 papers) next to a text editor reminded me too much of the side-by-side text editing experience done on a flat computer screen. The "programs" in a room need to be easily moved and rearranged.
• Errors in the code weren't displayed. I got by because I was familiar with the system, the language, the code, and because the programs were short. I would often get the wording of facts or subscriptions wrong. ("Wish light strand was blue" vs "Wish lights are blue") This is what happens when the interface is not "self-revealing" as Mel Conway puts it.
• The room was set up with information displayed on the walls and people sitting in the middle looking at the wall. This is the "command center" setup. To make the system more social and collaborative it should be the opposite: information in the middle with people crowded around it in small groups.

I'll be looking for ways to address these limitations while keeping the setup as simple as possible. If you have thoughts or questions, feel free to email or reach out to me on Twitter.

Appendix

Programs interact by:

• Stating facts about the world. Facts are any normal sentence like The temperature is 20 C. Facts are stored in a global tuplespace. This idea is similar to the LINDA coordination language. I inherited this metaphor from the Living Room project.
• Subscribing to facts in the world stated by other programs using fill-in-the-blank sentences like When the temperature is X then run some code using the variable X. This way of matching data is similar to Datalog.

Below I describe most of programs that bootstrap this programmable room demo, including the programs I made while playing in the room.

Broker:

• A persistent program that all programs connect to.
• Manages updates to the tuplespace via claims.
• Notifies programs about changes to the tuplespace based on their subscriptions.

Boot programs:

• Claim that 9 papers exist at a certain coordinates in the room, claim that the text editor exists at certain coordinates in the room, start all the other boot papers.
• Boot papers communicate by claiming and subscribing to facts, but also interact with the Linux operating system to do things like start/stop code and draw graphics.
• One important boot paper listens for claims about graphics and displays them on the projector.

Inputs:

• Keyboard claims last key pressed was X
  • The keyboard is connected via bluetooth to a Raspberry Pi that makes the claim.
• Keypad claims last keypad button pressed was X
  • The keypad is wired to a Particle Photon that sends a HTTP request to a computer that makes the claim.

Editable programs that extend the language:

• Claim time is X every 1/2 second
• Listen for claims from program Y like wish I was labeled X then the wrap the text lines and claim that the text graphics should be drawn on Y
• Listen for claims from program Y like wish I was highlighted X then claim that rectangle graphics with color X should be drawn on the paper Y

Basic use of extended language:

• Claim wish I was highlighted yellow

Programs that use the inputs:

• Subscribe to last key pressed was X and then claim wish I was labeled LAST KEY PRESSED: X
• Subscribe to last keypad button pressed was X and then claim wish I was labeled Editing program X and draw the red rectangle graphics on the border of the program being edited

Text editor:

• Subscribe to the source code of each paper.
• Subscribe to last keypad button pressed was X to load the source code for the corresponding program X.
• Subscribe to last key pressed was X to apply that key to the source code. On "Control-S" claim that the temporary source code should be the new source code for the program and claim that the program should be restarted.
• Claim that new text graphics should be drawn to match the source code any time there is a change in source code, keypad button press, or keyboard button press.
• Do text editor logic like wrapping lines, editing and deleting characters.

The text editor is a prime example of how the system is bootstrapped. It is not a program that itself manages a GUI window or directly listens to keyboard events. It simply uses existing claims and wishes that certain things would happen. Its ~200 lines of code are close to minimum viable description of what a "text editor" does. Because the editor itself is within the system you can improve the editing experience using the editor itself. The concept of a"text editor" also takes on a new meaning not as something that is given to you but a set of capabilities that could be achieved in many ways through any program that makes the appropriate claims.

Programs that use the claim about time:

• When time is X then claim wish I was labeled X
• When time is X, if the number of seconds is even then claim wish I was highlighted red and claim it is go time
• When time is X claim that circle and line graphics should be drawn at certain angles to make it look like a clock

<< Home