simple is not small

# simple is not small Source: [https://jyn.dev/simple-is-not-the-same-as-small](https://jyn.dev/simple-is-not-the-same-as-small) > **@notjack\.space**: UIs with too many buttons confuse and alarm me @**sixfold\-origami\.com**: ah, this is the unix thing\! **@notjack\.space**: no, because I want cross device sync to actually work ## Do we need simplicity?[https://jyn.dev/simple-is-not-the-same-as-small#do-we-need-simplicity](https://jyn.dev/simple-is-not-the-same-as-small#do-we-need-simplicity) Recently, I gave a talk titled["Precise, consistent, and reliable code coverage"](https://www.youtube.com/watch?v=P9lmSc4oLFs&list=PL8Q1w7Ff68DBpmF38rcIAf8Z9Gj2TnlgM&index=27)\. It's about a truly gnarly bug that took my company 9 months to debug\. At the end, my friend[Predrag](https://predr.ag/)asks: > How would you recommend that we think about building tools such that these epic debugging stories aren't as necessary? and I answer him: > We need to prioritize simplicity\. If you go back to my coverage pipeline, there are a*lot*of nodes in this diagram\. \[\.\.\.\] The tooling's complicated\. We need to rethink how our computing works\. I'm not satisfied with that answer\. --- ## Unix pipelines are not simple[https://jyn.dev/simple-is-not-the-same-as-small#unix-pipelines-are-not-simple](https://jyn.dev/simple-is-not-the-same-as-small#unix-pipelines-are-not-simple) Consider two programs to calculate the frequency of words of a file\. First, a small unix pipeline: ``` cat README.md \ | tr --complement --squeeze-repeats '[:alpha:]' '\n' \ | tr A-Z a-z \ | sort \ | uniq --count \ | sort --reverse --numeric-sort ``` This says "read README\.md, translate each word boundary into a newline, collapsing multiple newlines, convert uppercase to lowercase, count the number of occurrences of each word, then show them in frequency order"\. I think this is what most people think of when they think of "simple": each program is small, they're designed to be joined together ad\-hoc in this way, it's concise and somewhat easy to read\. Next, consider a Clojure program: ``` (->> (slurp "README.md") (re-seq #"[a-zA-Z]+") (map str/lower-case) frequencies (sort-by val >) ; for every (word, count) pair in the sequence, call an anonymous function that prints it. (run! (fn [[word count]] (println count word)))) ``` This does the same thing, with a few more names and higher\-order functions thrown in\. Now, let's say we want to make a small change: show the output in the original file order\. In Clojure, this is fairly straightforward: store an ordered sequence of the words in`word\_seq`, store a map from each word to its frequency in`freq\_map`, iterate over the sequence, and look up each word in the map: ``` (let [word_seq (->> (slurp "README.md") (re-seq #"[a-zA-Z]+") (map str/lower-case)) freq_map (frequencies word_seq)] (->> (distinct word_seq) ; for every distinct word, in original order, print its frequency (from our `freq` map) and the word itself (run! (fn [w] (println (freq_map w) w))))) ``` In Bash you need a bunch of temp files and ugly opaque regexes, sorts, and joins: ``` tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \ | grep . > words sort words \ | uniq --count \ | sed --regexp-extended 's/^ *([0-9]+) (.*)/\2 \1/' \ | sort > counts nl --body-numbering=a words \ | sort --key=2,2 --key=1,1n \ | uniq --skip-fields=1 \ | sort --key=2,2 > firstseen join -1 2 -2 1 -o 1.1,2.2,1.2 firstseen counts \ | sort --numeric-sort \ | cut --delimiter=' ' --field=2,3 ``` That's because our original program was*small*but not*simple*\. --- ## What is simplicity?[https://jyn.dev/simple-is-not-the-same-as-small#what-is-simplicity](https://jyn.dev/simple-is-not-the-same-as-small#what-is-simplicity) In[Simple Made Easy](https://www.youtube.com/watch?v=SxdOUGdseq4)\([transcript](https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md)\), Rich Hickey defines "simple" from its root, "sim\-plex": having only one braid\. He contrasts this to "com\-plex": braiding multiple things together\. In this post I'll use "coupled" as a synonym for "complex" to avoid ambiguity\.  And that gives us a language to talk about what's going on with our first Unix pipeline: it's*small*but it's*coupled*\. Let's look at exactly what makes it that way\. ``` tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \ | tr A-Z a-z \ | sort \ | uniq --count \ | sort --reverse --numeric-sort ``` There are a bunch of little things here I could nitpick, but the main thing that's coupled \(*braided together*\) is the`sort \| uniq \-\-count`\. If we look at`uniq`'s man page, it says this: > Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first\. There's no native Unix equivalent to`frequencies`, this`sort \| uniq \-c`is the closest we can get\. Not only is it less performant \(it has to collect the full input into memory before continuing\), but it ties aggregation to ordering\. This is exactly the thing that makes "separate ordering from aggregation" so hard; we end up having to do this weird dance with table\-joins\-through\-text\-files\. You might have heard the phrase "Write programs that do one thing and do it well" in reference to Unix systems\. Maybe you heard it called the[Unix Philosophy](https://archive.org/details/aquartercenturyofunixpeterh.salus_201910/page/n65/mode/2up)\. I think "do one thing" is commonly understood to be about simplicity, but in practice it's actually about*size*\. Unix tools are*small*but they are not*simple*\. ## Large is not the same as coupled[https://jyn.dev/simple-is-not-the-same-as-small#large-is-not-the-same-as-coupled](https://jyn.dev/simple-is-not-the-same-as-small#large-is-not-the-same-as-coupled) Now, let's consider the opposite end\. Say you have Google Drive for Desktop running on your computer\. This is a*massively*large program: it depends on platform\-specific file watchers, "all of Google3", a streaming and syncing network client, and conflict resolution logic\. But to the*user*it feels quite simple: Install the program, tell it which folder you want it to watch, tell it whether to keep the files locally or primarily on Google's infra\. It does all the rest\.  ## Decoupling[https://jyn.dev/simple-is-not-the-same-as-small#decoupling](https://jyn.dev/simple-is-not-the-same-as-small#decoupling) When I think about complex programs, I think about*coupling*\. Programs are complex when different features are*coupled*to each other, even when they don't have to be\. Let's take one small example\. In Rust, you can associate names to values with a map, or with a struct: ``` struct HttpResponse { status: u16, } let strukt = HttpResponse { status: 200 }; let mut map = HashMap::new(); map.insert("status", 200); println!("map: {}", map.get("status").unwrap()); println!("struct: {}", strukt.status); ``` It's very clear from this that a struct gets you*known present fields*\. For the map, we have to call`unwrap\(\)`, because the type checker doesn't know what keys are in a map\. For the struct it does, so we can just directly access the value\. What might not be clear about this is that a struct*loses runtime information*\. If you want to iterate a map, that's easy: call`for \(key, val\) in map \{ \.\.\.`\. If you want to iterate a struct \.\.\. get fucked? write a proc\-macro? The reason for this is that in Rust, a struct*couples*type\-checking to a fixed data representation\. You can't get one without the other\. Contrast this to Clojure, where you*can*\. In Clojure, structs*are*maps: rather than defining a type, you annotate which fields a map is allowed to have\. If we wanted to translate our`struct HttpResponse`, we could write this: ``` ; bind the name `http-response` to a list of keywords (interned strings). ; this is a normal list that is created and manipulated at runtime, it is not special in any way. (def http-response [:map [:status :int]]) ; bind the name `print-resp` to a function. ; `^{}` is a "metadata" map that will be associated with that name. ; metadata on bindings can be retrieved at runtime. (defn ^{:malli/schema [:=> [:cat http-response] :nil]} print-resp [map] (println "status:" (:status map))) ``` Here, we've created a type annotation that's checked at runtime with the function[`\(malli/instrument\!\)`](https://github.com/metosin/malli/blob/master/docs/function-schemas.md#instrumentation)\. Notably, this is checked*with a library*\(Malli\), not by a compiler; and the annotation is*inspectable*\. You can, for example, write a`schema\-\>md`function that acts as your own little mini Rustdoc, without needing to integrate with compiler APIs\. And all of this works without giving up type safety, reflection, or iteration over the values of the map\. This works because Clojure*decouples*data representations from type checking\. Typed Racket does a similar trick, but using macros so that the type checking happens at compile time instead of runtime\. ## When is it useful to be small?[https://jyn.dev/simple-is-not-the-same-as-small#when-is-it-useful-to-be-small](https://jyn.dev/simple-is-not-the-same-as-small#when-is-it-useful-to-be-small) Being small makes sense when you as the maintainer don't have a lot of resources to dedicate to your program\. Maybe you're Brian Kernighan and your program is running on a literal PDP\-11\. Maybe you're an open source maintainer with only a couple hours a month to dedicate to your project\. Maybe you work in an environment where doing*anything*is a victory and you can only get support for a small subset of the features you actually want to build\. All of these are good reasons to keep your program small\. But small is not the same as simple\. The answer to "when should your program be simple?" is:**always**\. There is very little advantage to introducing coupling to parts of your program; it makes it harder for you as a developer to maintain the program, and is less flexible for your users\. ## How do we make simple programs?[https://jyn.dev/simple-is-not-the-same-as-small#how-do-we-make-simple-programs](https://jyn.dev/simple-is-not-the-same-as-small#how-do-we-make-simple-programs) Ah, now this is the hard part\. To write simple programs, you need to have a good[mental model](https://jyn.dev/theory-building-without-a-mentor/)of your program\. You also need to have*good taste*, which is something I don't yet know how to teach\. Sometimes, you also need to Suck It Up And Write The Hard Thing\. CSS and SQL are highly decoupled \([mostly](https://www.scattered-thoughts.net/writing/against-sql/)\): you write a declarative specification of what you want the program to do, and the browser engine or database runtime figure out how to do it\. This is really really hard\! SQLite alone has had centuries of person\-years put into making it work reliably\. Blink \(Chrome's renderer\) has probably had tens of thousands of person\-years put into it\. In some domains, that's what it takes to let you write programs that are decoupled\. It doesn't always make sense to spend that much time on a program\.[Crunchy](https://drmaciver.com/2023/08/two-types-of-work/)technical work can be a[mothlamp problem](https://unfoldingdiagrams.leaflet.pub/3mft6olldos26): it attracts a certain kind of person who loves dreaming about how code*might*,*should*,*could*work\. Sometimes it's better to put down your tools and take a nap in the sun instead\. But when it does work— If we go back to the start of the post, the coverage pipeline I describe actually got*larger*after I fixed it, not smaller\. But at the same time it got*simpler*, because there were fewer hidden dependencies between parts of the dataflow graph\. ## What next?[https://jyn.dev/simple-is-not-the-same-as-small#what-next](https://jyn.dev/simple-is-not-the-same-as-small#what-next) I hope this post encourages you to write programs that are simple, not small, and to look for tools that you use that are unnecessarily coupled\. In a future post, I hope to extend these ideas: how to develop your sense of taste; how programs can be vertically integrated while still being decoupled; and how to build large systems without making them complex\.