Plumbers, chains, and famous painters: The history of the pipe operator in R

# Plumbers, chains, and famous painters: The (updated) history of the pipe operator in R | Adolfo Álvarez Source: [http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/) ***Note:****This post is an updated version of another I published in 2014, in a[previous version of this site](https://web.archive.org/web/20150225020645/http://adolfoalvarez.cl/plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/)*\. Our story starts as many other R related stories… in[stackoverflow](http://stackoverflow.com/)\. On[January 17th, 2012](http://stackoverflow.com/questions/8896820/how-to-implement-fs-forward-pipe-operator-in-r), an anonymous user “user4” asked the following question: > How can you implement F\#’s forward pipe operator in R? The operator makes it possible to easily chain a sequence of calculations\. For example, when you have an input data and want to call functions foo and bar in sequence, you can write: > data \|\> foo \|\> bar [Ben Bolker](https://twitter.com/bolkerb)answered[same day](http://stackoverflow.com/a/8897778)and gives what can be considered the first pipe in R: ``` "%>%" <- function(x,f) do.call(f,list(x)) pi %>% sin %>% cos ``` ``` ## [1] 1 ``` Although probably the first formal introduction of a pipe function in an R package occured on[April 5, 2012](https://github.com/crowding/vadr/commit/9509884b1e88e1310d964acab7c6d9838df9a639#diff-d725d41bc2e847af038b2b15e28d8520831f4f69fcc0f87a6afe97bbb5df9c68), when[Peter Meilstrup](https://twitter.com/perceptophore)started the package`ptools`as a way to collect “various data manipulation and programming utilities”\. One of those utilities was the function`chain`, implementing a way to pipe the arguments\. The package was later renamed to[`vadr`](https://github.com/crowding/vadr)where the`chain`function can still be found\. ## dplyr\.[http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#dplyr](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#dplyr) But soon one of the packages that contributed to increase the popularity of the pipe use will be created: On[October 28th, 2012](https://github.com/tidyverse/dplyr/commit/80dc69b144711ec095db1d62cf0b73e09560eaf0),[Hadley Wickham](https://twitter.com/hadleywickham)started the`dplyr`project on[github](https://github.com/tidyverse/dplyr/)as an evolution of his data analysis package[`plyr`](https://github.com/hadley/plyr)\(Initially the package was indeed called ‘plyr2’\)\. Originally, the dplyr package didn’t incorporate a chain function, but this changed almost a year later, in[October 9, 2013](https://github.com/tidyverse/dplyr/commit/57f76ed560ac52590f7ded3d013db8fb19f7d627), when the first pipe in dplyr appeared\. The function was also denominated`chain`, but the package introduced its first operator for the pipe:`%\.%`\. The idea behind the introduction of the chain was simplify notation for applying several functions to a data frame\. Without the chain function, in dplyr you needed to read the verbs from inside out: ``` library(hflights) library(dplyr) filter( summarise( select( group_by(hflights, Year, Month, DayofMonth), Year:DayofMonth, ArrDelay, DepDelay ), arr = mean(ArrDelay, na.rm = TRUE), dep = mean(DepDelay, na.rm = TRUE) ), arr > 30 | dep > 30 ) ``` ``` ## # A tibble: 14 × 5 ## # Groups: Year, Month [10] ## Year Month DayofMonth arr dep ## <int> <int> <int> <dbl> <dbl> ## 1 2011 2 4 44.1 47.2 ## 2 2011 3 3 35.1 38.2 ## 3 2011 3 14 46.6 36.1 ## 4 2011 4 4 38.7 27.9 ## 5 2011 4 25 37.8 22.3 ## 6 2011 5 12 69.5 64.5 ## 7 2011 5 20 37.0 26.6 ## 8 2011 6 22 65.5 62.3 ## 9 2011 7 29 29.6 31.9 ## 10 2011 9 29 39.2 32.5 ## 11 2011 10 9 61.9 59.5 ## 12 2011 11 15 43.7 39.2 ## 13 2011 12 29 26.3 30.8 ## 14 2011 12 31 46.5 54.2 ``` But when the chain function was introduced this was changed first to: ``` chain( hflights, group_by(Year, Month, DayofMonth), select(Year:DayofMonth, ArrDelay, DepDelay), summarise( arr = mean(ArrDelay, na.rm = TRUE), dep = mean(DepDelay, na.rm = TRUE) ), filter(arr > 30 | dep > 30) ) ``` And with the %\.% pipe like this: ``` hflights %.% group_by(Year, Month, DayofMonth) %.% select(Year:DayofMonth, ArrDelay, DepDelay) %.% summarise( arr = mean(ArrDelay, na.rm = TRUE), dep = mean(DepDelay, na.rm = TRUE) ) %.% filter(arr > 30 | dep > 30) ``` ## magrittr[http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#magrittr](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#magrittr) Nevertheless, the`%\.%`pipe would not stay in`dplyr`package for long time, on[December 29th, 2013](http://stackoverflow.com/revisions/20828881/1),[Stefan Bache](http://www.stefanbache.dk/)revisited the old stackoverflow question proposing an alternative to the original answer: > How about ``` `%>%` <- function(e1, e2) { cl <- match.call() e <- do.call(substitute, list(cl[[3]], list(. = cl[[2]]))) eval(e) } ``` > which allows a chain like: ``` iris %>% subset(., Species == "setosa", select = -Species) %>% colMeans(.) ``` ``` ## Sepal.Length Sepal.Width Petal.Length Petal.Width ## 5.006 3.428 1.462 0.246 ``` Stefan continued working on this pipe operation, and on[December 30th, 2013](http://stackoverflow.com/revisions/20828881/4), he implemented in github the`plumbr`package which included the`%\>%`operator\.[Two days later](http://stackoverflow.com/revisions/20828881/5),`plumbr`would be renamed as[`magrittr`](https://github.com/smbache/magrittr), its current name, in a clear reference of the famous painting[“The Treachery of Images”](http://en.wikipedia.org/wiki/The_Treachery_of_Images)of the Belgian painter René Magritte\.  The`dplyr`package was being developed in parallel but these two developments were intended to converge\. In the first comment of the[announcing post](http://blog.rstudio.org/2014/01/17/introducing-dplyr/)of`dplyr`by the RStudio blog in January 2014, Stefan Bache let Hadley Wickham know about the`magrittr`operator\. After that, on[March 19th, 2014](https://github.com/tidyverse/dplyr/commit/3c91ba370286025c31f624317ebce421d1c70caa), the`chain`function was deprecated on dplyr, and finally on[April 14th, 2014](https://github.com/tidyverse/dplyr/commit/89aaa9a8b386d7e3705ffb682d89dc5bebe823c0),`dplyr`incorporated the`%\>%`operator of magrittr, recommending it in substitution of the original`%\.%`, because the former is more easy to type holding down the Shift key\. On[August 1st, 2014](https://github.com/tidyverse/dplyr/commit/67183e2771f24f59330fdaae1bb58b627a58a8ff),`%\.%`was deprecated\. At the same time, on[July 31th, 2014](https://github.com/rstudio/rstudio/commit/861d4e36ffd0233ee41bfb80c712a635c70c1b3c), the Rstudio IDE version v0\.98\.987 incorporated a shortcut for the`dplyr`/`magrittr`pipe operator`%\>%`to make even more easy its use \(Shift\+Alt\+\.\), but this was[soon changed](https://github.com/rstudio/rstudio/commit/0fb46fbc114a91217fae106bac42e82f60a02477)to Ctrl\+Shift\+M, its current form\. At this point of the story I recommend you to visit a blog post from August 5th, 2014, where Stefan told about[how he started magrittr](https://www.r-statistics.com/2014/08/simpler-r-coding-with-pipes-the-present-and-future-of-the-magrittr-package/)\. ## pipeR[http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#piper](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#piper) Another iteration of the pipe implementation in R started on April 7th, 2014, when[Kun Ren](https://twitter.com/renkun_ken)published on github[the`pipeR`package](https://github.com/renkun-ken/pipeR/), incorporating a different pipe operator`%\>\>%`to add more flexibility to the piping process\. The package incorporates several features for the pipe, such as first argument of a function, as argument in an expression \(Using`\.`\), by using a formula, or to save intermediate results \(using`~`\), or to extract element by names \(using`\(\)`\): ``` library(pipeR) mtcars %>>% (lm(mpg ~ wt + cyl, data = .)) %>>% (~ lm_mtcars) %>>% summary %>>% (r.squared) ``` ``` ## [1] 0.8302274 ``` Currently pipeR has not been updated since 2016, but is still available on CRAN and github, in case you want to give it a try\. ## wrapr[http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#wrapr](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#wrapr) On[July 2018](https://github.com/WinVector/wrapr/commit/6bb774f090c6aec457c472703c4759e98cc25e38), the[package`wrappr`](https://winvector.github.io/wrapr/)introduced the[“dot arrow pipe”](https://winvector.github.io/wrapr/reference/dot_arrow.html)`%\.\>%`where the notation`a %\.\>% b`is nearly equivalent to`\. <\- a; b`\. For example ``` library(wrapr) 1:10 %.>% mean(.) ``` ``` ## [1] 5.5 ``` And the lhs is still stored in the environment: ``` ## [1] 1 2 3 4 5 6 7 8 9 10 ``` [More details here](https://journal.r-project.org/archive/2018/RJ-2018-042/RJ-2018-042.pdf) ## Base R[http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#base-r](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#base-r) Pipes in R changed completely the way how we code in R, making it more simple and readable\. Therefore was a matter of time they were incorporated to the base R code\. One of the first attempts was developed by[Lionel Henry](https://twitter.com/_lionelhenry), who started testing a native pipe in base R on[February, 2016](https://github.com/lionel-/r-source/tree/pipe)\. He wrote[a very recommended post](https://lionel-.github.io/2016/02/15/ideas-for-an-updated-r-syntax/)about this and other proposals for the R core, imagining R 4\.0 with a native pipe operator\. One of the reasons for such implementation was that “native handling of the pipe would improve error recovery” because of the dependence of an external package \(`magrittr`\)\. At this point we are probably missing part of the story\. It seems that during the next 3 years there was no progress on this topic, although probably internal discussions happened at the R Core team\. On October 5th, 2019, Antoine F\. asked to the r\-devel list:[should base R have a piping operator ?](https://stat.ethz.ch/pipermail/r-devel/2019-October/078520.html)\. That question indeed provoked[several replies](https://stat.ethz.ch/pipermail/r-devel/2019-October/thread.html#78520)from R developers, including Lionel Henry, Duncan Murdoch, or Luke Tierney among others\. It was actually[Luke Tierney](https://stat.uiowa.edu/people/luke-tierney), who in July 8, 2020, in his and Martin Mächler’s keynote at[UseR\! 2020](https://user2020.r-project.org/)spoke about new developments in R\. A[section of the keynote](https://www.youtube.com/watch?v=X_eDHNVceCU&t=4082s)was dedicated to the pipe, where he advanced that thanks to the ideas of Lionel Henry and Jim Hester, a base pipe`\|\>`was under development\. Finally, at the end of 2020,`\|\>`was born\. The native pipe was[officially announced in the devel version of R](https://developer.r-project.org/blosxom.cgi/R-devel/NEWS/2020/12/04), with immediate reactions in the[r\-devel mailing list](https://stat.ethz.ch/pipermail/r-devel/2020-December/thread.html#80173)and in the rest of the R community \(examples[here](https://win-vector.com/2020/12/07/my-opinion-on-rs-upcoming-pipe/),[here](https://news.ycombinator.com/item?id=25316608),[here](https://www.jumpingrivers.com/blog/new-features-r410-pipe-anonymous-functions/), and[here](https://www.infoworld.com/article/3621369/use-the-new-r-pipe-built-into-r-41.html)\)\. The pipe finally landed in the R stable version 4\.1\.0, which was released and officially announced on[May 18, 2021](https://stat.ethz.ch/pipermail/r-announce/2021/000670.html) Now, our first example taken from the dplyr documentation can we written as: ``` hflights |> group_by(Year, Month, DayofMonth) |> select(Year:DayofMonth, ArrDelay, DepDelay) |> summarise( arr = mean(ArrDelay, na.rm = TRUE), dep = mean(DepDelay, na.rm = TRUE) ) |> filter(arr > 30 | dep > 30) ``` ``` ## # A tibble: 14 × 5 ## # Groups: Year, Month [10] ## Year Month DayofMonth arr dep ## <int> <int> <int> <dbl> <dbl> ## 1 2011 2 4 44.1 47.2 ## 2 2011 3 3 35.1 38.2 ## 3 2011 3 14 46.6 36.1 ## 4 2011 4 4 38.7 27.9 ## 5 2011 4 25 37.8 22.3 ## 6 2011 5 12 69.5 64.5 ## 7 2011 5 20 37.0 26.6 ## 8 2011 6 22 65.5 62.3 ## 9 2011 7 29 29.6 31.9 ## 10 2011 9 29 39.2 32.5 ## 11 2011 10 9 61.9 59.5 ## 12 2011 11 15 43.7 39.2 ## 13 2011 12 29 26.3 30.8 ## 14 2011 12 31 46.5 54.2 ``` In parallel, the`magrittr`package was still evolving but converging to the native pipe\. After a transition to the tidyverse family of packages in 2016, on November 2020, the[2\.0 version was released](https://www.tidyverse.org/blog/2020/11/magrittr-2-0-is-here/)\. As noticed in the announcement post, this version of`magrittr`incorporated a new version of the %\>% pipe written in C, allowing a significant increase in speed\. A deeper benchmark analysis with similar results can also be found[here](https://michaelbarrowman.co.uk/post/the-new-base-pipe/)\. As remarked by[this blog post of Jumping Rivers on the announcement of R 4\.1](https://www.jumpingrivers.com/blog/new-features-r410-pipe-anonymous-functions/)there are some differences between`magrittr`pipe and the native one\. For example, with`magrittr`you don’t need to write the parenthesis at the end of the function names, while this is compulsory in the native pipe ``` library(magrittr) 1:10 %>% mean() ``` ``` ## [1] 5.5 ``` ``` ## [1] 5.5 ``` ``` ## [1] 5.5 ``` ``` ## Error: The pipe operator requires a function call as RHS ``` Another probably more important difference is the dot \(\.\) replacement\. By default, the pipes are passing the lhs \(left\-hand side\) as a first argument to the rhs \(right\-hand side\), as in the example above, where 1:10 is used as the first argument for mean\(\)\. Using`magrittr`pipe, the lhs can be used as any other argument of the rhs using a dot \(\.\) where the lhs should be placed\. For example, while this is working with magrittr[1](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#fn:1): ``` library(palmerpenguins) library(rpart) library(rpart.plot) penguins %>% rpart(species ~ ., method = "class", data = .) %>% prp(extra = 4) ```  This is not: ``` penguins |> rpart(species ~ ., method = "class", data = .) |> prp(extra = 4) ``` ``` ## Error in model.frame.default(formula = penguins, data = ., weights = species ~ : 'data' must be a data.frame, environment, or list ``` To make it work we need to use anothe new feature of R 4\.1\.0, the shorthand inline function notation`\\\(x\)`: ``` penguins |> {\(x) rpart(species ~ ., method = "class", data = x)}() |> prp(extra = 4) ```  While there were some ideas to introduce[a new pipe in R base](https://stat.ethz.ch/pipermail/r-devel/2021-January/080396.html)`=\>`to pass the lhs to other arguments \(see also[this](https://twitter.com/rdataberlin/status/1396144404033949700)\), it is still[not officially supported](https://stat.ethz.ch/pipermail/r-devel/2021-April/080645.html)\.[Previous options](https://www.youtube.com/watch?v=X_eDHNVceCU&t=3099s)included also using`\_`as a replacement for the`\.`placeholder\. We need to wait to see if that second pipe notation will be adopted, or they will choose to stay with the`\|\>`pipe and the`\\\(x\)`solution\. 9 years later, and as a result of the efforts and time from many R users and developers, our`user4`has finally her`data \|\> foo \|\> bar`fully implemented in R, changing for many of us the way we are coding\. I am willing to know what versions 5, 6, or 7 will bring us, thank you for reading\! - The`magrittr`package implements[other three pipes](https://data-and-the-world.onrender.com/posts/magrittr-pipes/)\(`%T\>%`,`%$%`, and`%<\>%`\) - The`data\.table`package does not include a general use pipe, but[you can pipe](https://www.gl-li.com/2017/07/25/compare-data.table-pipes-and-magrittr-pipes/)functions with`\]\[` - Maybe another difference between the`magrittr`pipe and the base R one is pronunciation: While[Hadley Wickham stated](https://twitter.com/hadleywickham/status/507643720401842176)that the`%\>%`pipe should be pronounced as “then”, we can clearly hear Luke Tierney on his[User\! keynote](https://www.youtube.com/watch?v=X_eDHNVceCU&t=4082s)read`\|\>`as “goes into”\. - The pipe`\|\>`is also implemented in other languages such as[Elixir](https://elixirschool.com/en/lessons/basics/pipe-operator/),[Julia](https://syl1.gitbook.io/julia-language-a-concise-tutorial/useful-packages/pipe),[Javascript](https://stackoverflow.com/questions/55143363/what-does-the-operator-do-in-javascript)and[F\#](https://theburningmonk.com/2011/09/fsharp-pipe-forward-and-pipe-backward/)\. - The pipe`\|\>`was originally proposed by Tobias Nipkow[in May 1994](https://web.archive.org/web/20190217164203/https://blogs.msdn.microsoft.com/dsyme/2011/05/17/archeological-semiotics-the-birth-of-the-pipeline-symbol-1994/)for the proof assistant[Isabelle/ML](https://isabelle.in.tum.de/)\. --- 1. Notice that`penguins`is placed in the data argument, while a dot in the first argument,`species ~ \.`, is ignored since a dot in a formula means something different\.[↩︎](http://adolfoalvarez.cl/blog/2021-09-16-plumbers-chains-and-famous-painters-the-history-of-the-pipe-operator-in-r/#fnref:1) Posted on:September 16, 2021Length:10 minute read, 2101 wordsCategories:[R](http://adolfoalvarez.cl/categories/r)See Also: