We are big fans of distributed teams. They give an organization a lot of additional hiring options, and give employees meaningful flexibility benefits. Uprooting and moving house for a new software job should be a thing of the past. Of course, working with folks who aren’t co-located needs some extra effort.
Like many software teams, we practice pair programming. This can be challenging in a distributed context. Here’s a setup that works really well for us. If you are willing to place a few limitations on your dev environment, what you gain is a low latency pairing setup where you can quickly switch off who is driving. In fact, once we got used to this setup, we used it even when we were sitting next to each other. There are dozens of tools to facilitate remote pairing, with new products appearing every month. Our preferred setup is far from new technology, but it is reliable, secure, high fidelity, and has little or no asymmetry.
In short, we recommend Emacs (or Vim, or any other capable command line editor), tmux, and SSH tunnels.
To explain how all this works, this is an example of Eve hosting a pairing session for Joe. Want to build a connection between two developer’s computers, each with an internet connection. To connect these machines we use a third, internet-visible server, on which we have root access. A low-tier AWS instance works great. We call this server pika. Here’s an overview of what the final pairing setup will look like:
Eve has some setup steps to do before Joe can pair with her:
- Set up pairing user profile on pika
- Distribute Joe’s public key to pika and eve-pc
Eve needs to set up the eve-pair login profile on pika. Knowing the SSH port to be used, the name of the tmux socket and the session that she will open, we tweak the login profile scripts to replace the standard interactive shell on eve-pair with an SSH connection over port 7771 to the tmux session on Eve’s host computer:
$ cat ~eve-pair/.profile
# All this user gets to do is automatically run ssh-eve-computer:
The script to connect back to Eve’s computer looks like this:
$ cat /usr/bin/ssh-eve-computer
echo "Connecting to tmux with session '-L eve.tmux -t evepair'"
ssh -t -p 7771 eve@localhost "tmux -L eve.tmux attach -t evepair"
We also want to prevent the eve-pair login on pika from being able to do much other than this. Logins have two modes; the interactive login is already taken care of, but by tweaking the .bashrc file we can prevent any non-interactive commands sessions (e.g. scp).
$ cat ~eve-pair/.bashrc
# Disable non-login shells for this user:
Eve ensures Joe’s public key is in eve-pair’s ~/.ssh/authorized_keys on pika, and in Eve’s own ~/.ssh/authorized_keys on eve-pc.
Start a pairing session
Eve opens a reverse SSH tunnel from her host machine to pika on port 7771:
$ ssh -nNT -R 7771:localhost:22 eve@pika # Starts a reverse SSH tunnel
…and in a new terminal window, starts up a named tmux session using a named socket file:
$ tmux -L eve.tmux new-session -s evepair
Connect to the pairing session
Joe connects to pika with ssh -A eve-pair@pika, and is automatically connected to Eve’s tmux session (the -A option enables forwarding of Joe’s authentication agent, allowing secure key-based authentication without needing passwords). Once the pairing session is done, all Eve needs to do to close everything down is stop the SSH tunnel command (just Ctrl-c in that terminal window), and perhaps shut down the tmux session.
The good and the bad
We like this setup because each person gets almost exactly the same experience. There are no screen scaling or compression artifact issues, network lag has minimal effect, and with a similar or identical editor configuration this is a highly productive setup for both programmers.
Disadvantages of this are: one person needs shell access to a server; you need some familiarity with SSH, key pairs, and tmux; and it only works with terminal-based editors. Most programmers will be comfortable with the first two points, but terminal editors are not universally used, and you can’t share GUIs (e.g. a web browser window to view web development work).
For us, the disadvantages are minimal and easily accommodated. We haven’t found another pairing setup that is as equal for us as SSH, tmux, and Emacs. With this setup we’ve been able to work effectively across four time zones, giving people the flexibility they need to live their lives and get a lot of work done. If you were able to hire folks anywhere in the US, would that increase the size of your candidate pool?