TXR: An Original, New Programming Language for Convenient Data Munging

# TXR: an Original, New Programming Language for Convenient Data Munging Source: [https://www.nongnu.org/txr/](https://www.nongnu.org/txr/) ## Kaz Kylheku`<kaz@kylheku\.com\>` ## Quick Links - [Reference Manual](https://www.nongnu.org/txr/txr-manpage.html), also in[PDF](https://www.nongnu.org/txr/txr-manpage.pdf) - TXR 302 installers for[32 bit Windows](https://www.kylheku.com/txr-downloads/txr-302/txr-302-Win32.exe)and[64 bit Windows](https://www.kylheku.com/txr-downloads/txr-302/txr-302-Win64.exe) - [Rosetta Code Solutions in TXR](https://www.nongnu.org/txr/rosetta-solutions.html) - TXR is known to packaged and kept up\-to\-date by these operating system distributions or package installation systems:- [GNU Guix](https://guix.gnu.org/)\. - [NixOS](https://nixos.org/)\. - [Chimera Linux](https://chimera-linux.org/)\. - [Alpine Linux](https://alpinelinux.org/)\. - [Void Linux](https://voidlinux.org/)\. - [EasyBuild](https://easybuild.io/)\. - [T2 System Development Environment](https://t2sde.org/)\. - Homebrew: do not use TXR via Homebrew\. See Warning near the bottom of this page\. - Accepted TXR\-based solutions to non\-TXR\-related questions on StackOverflow:- [Customized Sort based on Multiple Columns of CSV](https://stackoverflow.com/questions/10050656/perform-highly-customized-sort-based-on-multiple-columns-of-a-csv-file/10050995#10050995) - [Extract Text from HTML Table](https://stackoverflow.com/questions/10055385/extract-text-from-html-table/10055851#10055851) - [Using Awk to Store Values from Columns from Multiple Files](https://stackoverflow.com/questions/9731461/using-awk-to-store-values-from-column-reads-for-multiple-files/9764676#9764676) - [Complex XML Editing Scenario](https://stackoverflow.com/questions/32523087/xml-editing-is-it-possible-somehow/34421108#34421108) - [AWK to Parse Input with Multiple Conditions](https://stackoverflow.com/questions/29079870/awk-to-parse-input-with-multiple-conditions/31058002#31058002) - [Efficient Shift\-Invariant Transform of Text](https://stackoverflow.com/questions/39920702/efficient-shift-invariant-feature-transform-for-text/39925664#39925664) - [Get the Current Status of a`git bisect`](https://stackoverflow.com/questions/37667324/how-do-i-get-the-current-status-of-a-git-bisect/37668768?s=13|0.8110#37668768) - [Get n\-th Field of Each Create Referring to Another File](https://stackoverflow.com/questions/37585956/how-to-get-nth-field-of-each-create-table-refering-to-another-file-under-linux-s/37666937?s=15|0.7987#37666937) - [`git add`Only Lines Matching a Pattern](https://stackoverflow.com/questions/36932309/how-do-i-git-add-only-lines-matching-a-pattern/37336801#37336801) - [Metaprogramming in awk, convert file to html table format](https://stackoverflow.com/questions/41684176/metaprogramming-in-awk-convert-file-to-html-table-format/43440063#43440063) - Accepted TXR\-based solutions on Unix StackExchange:- [Find largest recurring group of lines](https://unix.stackexchange.com/questions/97634/os-x-linux-one-liner-script-to-find-the-largest-recurring-group-of-lines-in-a-te/97852#97852) - New\!All answers matching a query for`TXR`:- [In StackOverflow](https://stackoverflow.com/search?q=is%3aanswer%20TXR) - [In the Unix StackExchange](https://unix.stackexchange.com/search?q=is%3Aanswer+TXR) - [GIT Repository/Source Downloads](https://kylheku.com/cgit/txr) - [Release Notes](https://www.kylheku.com/cgit/txr/tree/RELNOTES) - Piotr Klibert's[Emacs Mode](https://github.com/piotrklibert/txr-mode)for TXR - [Binary Downloads](https://www.kylheku.com/txr-downloads/txr-302/)of TXR 302 for a number of platforms - Discussion, Help, Updates:- [Mailing List](https://kylheku.com/txr-users)—old school: you can post without the nuisance of subscribing\! - [Libera IRC Channel](https://kiwiirc.com/client/irc.libera.chat/?nick=Guest&#txr) - Follow[TXR](https://fosstodon.org/@txr)on Mastodon\. - Follow[Kaz](https://mstdn.ca/@Kazinator)on Mastodon \(mstdn\.ca, more active\)\. - Follow[Kaz](https://fosstodon.org/@Kazinator)on Mastodon \(fosstodon\.org, less active\)\. ## What is it? TXR is a pragmatic, convenient tool ready to take on your daily hacking challenges with its dual personality: its whole\-document pattern matching and extraction language for scraping information from arbitrary text sources, and its powerful data\-processing language to slice through problems like a hot knife through butter\. Many tasks can be accomplished with TXR "one liners" directly from your system prompt\. TXR is relatively new: the project started in 2009\. It is difficult to give a small introduction to TXR because it is no longer a small language\. The PDF rendition of the reference manual, which takes the form of a large Unix man page, is778881900951 pages long, excluding any index or table of contents\. There are many ways to solve a given data processing problem with TXR: many skills and techniques can be used\. ## Random Testimonials > *"TXR Lisp: making coding fun & snappy\! 💥 Concise, yet powerful, it’s like having a cheat sheet for the language of programming\! 😎"* \(A[Twitter/X post](https://x.com/AITechtree/status/1733192711740801136)by user AI TechTree\) > *"The best lisp for text processing is TXR Lisp\. \[\.\.\.\] But \[it is\] a rabbit hole that goes so deep, you'd better have nothing but free time if you want to grok it all"* \(anonymous[comment](https://boards.4chan.org/g/thread/102488015/lisp-general#p102516300)on 4Chan\) > *"So far, txr\-lisp felt most ergonomic to me when I tested various languages for their suitability to implement MAL in"* \([comment](https://snailgeist.com/irc-logs/?message_id=189174)by user wasamasa in a \#Lisp IRC channel\) > *"My bank statements are pdf's which can be converted to ascii using pdftotext, however this destroys the structure of the documents which makes extracting data using regexps \(even pcre's\) very difficult, but much easier using txr\."* \([comment](https://news.ycombinator.com/item?id=38600248)by user vcdimension on HackerNews\) > *"I recently discovered TXR\. I'm very impressed, especially for a relatively young language\. And the man page/manual is really good\>\."\>* \([comment](https://github.com/raxod502/dotman/issues/2#issuecomment-394177167)by user alphapapa on GitHub\) ## Small, efficient; low dependencies TXR Lisp supports compilation to code for a register\-based virtual machine code\. Individual functions can be compiled as well as files \(`\.tl`to`\.tlo`\)\. Compiled files may be catenated together to`load`\. Individual compiled files, as well as catenated files, may be compressed with`gzip`\. Though not native, the compiler is optimizing: it performs optimizations like jump threading, dead code elimination, constant folding, and data flow optimizations\. The compiler is actively being improved\. Application deployment is possible\. The`save\-exe`function creates a copy of the TXR executable, under a name of your choosing, and containing an expression that is executed at startup, typically used to load the rest of the application relative to the same directory\. The executable just has to be accompanied by all the needed library modules; the[details](https://www.nongnu.org/txr/txr-manpage.html#N-0161CE31)are in the reference manual\. TXR is light in terms of resources\. The executable is some 1\.7 megabytes of code \(a little more than twice the size of GNU Awk\), and the satellite library modules add up to another megabyte and a half\. It has no external dependencies other than`libffi`\. The entire project is easy to build; it just requires GNU Make and GCC or Clang, and a few shell utilities needed by the`configure`script\. There are some generated sources, which are shipped and so the tools for them are not required\. TXR has a small memory footprint\. When the executable is started up to its interactive prompt, the memory use is similar to GNU Bash\. When compiling its standard library of TXR Lisp code, including complex files such as the compiler, TXR requires a peak of only around 18 megabytes\. ## Innovative, but with traditional roots TXR is a fusion of many different ideas, a few of which are original, and it is influenced by many languages, such as Common Lisp, Scheme, Awk, M4, POSIX Shell, Prolog, Ruby, Python, Arc, Clojure, S\-Lang and others\. TXR consists of two languages, which can be used separately or tangled together: the[TXR Pattern Language](https://www.nongnu.org/txr/txr-pattern-language.html), and[TXR Lisp](https://www.nongnu.org/txr/txr-lisp.html)\. A comparison may be drawn between the[TXR Pattern Language](https://www.nongnu.org/txr/txr-pattern-language.html)and the Unix utility Awk\. Both provide an implicit, convenient way of scanning input\. Whereas Awk implicitly reads a file, breaking it into records and fields which are accessible as positional variables,**TXR has quite a different way of making input handling implicit: namely via a nested, recursive pattern matching notation which binds variables\.**This approach still handles delimited fields with relative convenience, but generalizes into handling messy, loosely structured data, or data which exhibits different regularities in different sections, etc\. Constructs in TXR \(the pattern language\) aren't imperative statements, but rather pattern\-matching directives: each construct terminates by matching, failing, or throwing an exception\. Searching and backtracking behaviors are implicit\. It has features like structured named blocks with nonlocal exits, structured exception handling, named pattern matching functions, and numerous other features\. TXR's pattern language is powerful enough to parse grammars, yet simple to use in an ad\-hoc way on trivial tasks\. Speaking of Awk, TXR in fact[contains an implementation of Awk](https://www.nongnu.org/txr/txr-manpage.html#N-000264BC), in the form of a Lisp macro, which brings us to the next topic\. The other language in TXR is[TXR Lisp](https://www.nongnu.org/txr/txr-lisp.html)\. This is not an implementation of an existing Common Lisp or Scheme, but a new dialect, which contains many new ideas\. TXR Lisp is feature\-rich, and oriented toward succinct, convenient expressiveness\. While staying completely true to the Lisp heritage, it takes cues from new scripting and functional languages\. Users of mainstream Lisp will find that skills transfer well to and from TXR\. There will be features users will miss from TXR Lisp when using others Lisps, and likely vice versa\. The TXR project values brevity in programs: programs should be short and clear\. If you're struggling in coming up with a nice solution to a problem, the TXR project wants to hear from you; give a shout to the mailing list\. If a program is significantly clearer and shorter in another language, that is of interest to the project; TXR may be able to absorb the technique or something equivalent\. ## Help Needed The TXR project is looking for hackers to develop features TXR has clean, easy to understand and maintain internals that are a pleasure to work with\. Be sure to read the[HACKING guide](https://www.kylheku.com/cgit/txr/plain/HACKING)\. ## Examples Here is a collection of[TXR Solutions](https://www.nongnu.org/txr/rosetta-solutions.html)to a number of problems from Rosetta Code\. ## Byte Code Optimization Case Study This is an[article](https://www.nongnu.org/txr/txr-opt.html)highlighting some of the capabilities of TXR Lisp's optimizing compiler\. ## Make a Donation *TXR is truly free software*because it is distributed under the two\-clause[BSD license](https://www.kylheku.com/cgit/txr/tree/LICENSE)which allows every conceivable use, commercial and non\-commercial\. If you find TXR to be a valuable tool in your arsenal, here is one way to show your appreciation and support\! Developing stuff like this takes countless hours\. ## Warning: Regarding Homebrew WARNING: Do not use TXR packaged by Homebrew\!The TXR project has received reports that Homebrew packages of TXR contain an unstable, unreliable executable\. There are reports from users that they are not even able to build the Homebrew TXR package themselves \(in order to investigate into the problem\)\. The Homebrew build formula`txr\.rb`does not run`make tests`, so when the build does succeed, it is not verified to be good\. It is suspected that Homebrew may be using compiler code generation features that cause instability\.