@QingQ77: A command-line CSV processing tool written in Rust ("CSV Magician"), capable of extremely fast, low-memory processing of CSV files of several GB, with built-in expression language, terminal visualization, and social science-oriented extensions such as dictionary statistics, graph theory, and even scraping. https://github.com/medialab/…

A command-line CSV processing tool written in Rust (“CSV magician”), capable of handling GB-sized CSV files extremely quickly and with low memory usage, featuring its own expression language, terminal visualization, and social science-oriented extensions like lexicometry, graph theory, and even scraping. https://t.co/eh5iZEbjBN https://t.co/MKctM459xB — # medialab/xan Source: https://github.com/medialab/xan Build Status (https://github.com/medialab/xan/actions) DOI (https://doi.org/10.5281/zenodo.15310200) # xan, the CSV magician xan is a command line tool that can be used to process CSV files directly from the shell. It has been written in Rust to be as fast as possible, use as little memory as possible, and can very easily handle large CSV files (Gigabytes). It leverages a novel SIMD (https://en.wikipedia.org/wiki/Single_instruction,_multiple_data) CSV parser (https://docs.rs/simd-csv) and is also able to parallelize some computations (through multithreading) to make some tasks complete as fast as your computer can allow. It can easily preview, filter, slice, aggregate, sort, join CSV files, and exposes a large collection of composable commands that can be chained together to perform a wide variety of typical tasks. xan also offers its own expression language so you can perform complex tasks that cannot be done by relying on the simplest commands. This minimalistic language has been tailored for CSV data and is way faster than evaluating typical dynamically-typed languages such as Python, Lua, JavaScript etc. Note that this tool is originally a fork of BurntSushi (https://github.com/BurntSushi)’s xsv (https://github.com/BurntSushi/xsv), but has been nearly entirely rewritten at that point, to fit SciencesPo’s médialab (https://github.com/medialab) use-cases, rooted in web data collection and analysis geared towards social sciences (you might think CSV is outdated by now, but read our love letter to the format before judging too quickly). xan therefore goes beyond typical data manipulation and expose utilities related to lexicometry, graph theory and even scraping. Beyond CSV data, xan is able to process a large variety of CSV-adjacent data formats from many different disciplines such as web archival (.cdx) or bioinformatics (.vcf, .gtf, .sam, .bed etc.). xan is also able to convert to & from many data formats such as json, excel files, numpy arrays etc. using xan to and xan from. See this section for more detail. Finally, xan can be used to display CSV files in the terminal, for easy exploration, and can even be used to draw basic data visualisations: |view command|flatten command| |:—:|:—:| |view|flatten| |categorical histogram|scatterplot| |categ-hist|correlation| |categorical scatterplot|histograms| |scatter|hist| |parallel processing|time series| |parallel|series| |small multiples (facet grid)|grouped view| |small-multiples|view-grid| |correlation matrix heatmap|heatmap| |small-multiples|view-grid| ## Summary * How to install * Cargo * Scoop (Windows) * Homebrew (macOS) * Arch Linux * NetBSD * Nix * Pixi * Conda Forge * Pre-built binaries * Installing completions * Quick tour * Learning * Available commands * General flags and IO model * Getting help * Regarding input & output formats * Working with headless CSV file * Regarding stdin * Regarding stdout * Supported file formats * Compressed files * Regarding color * Expression language reference * News * How to cite? * Frequently Asked Questions ## How to install ### Cargo xan can be installed using cargo (it usually comes with Rust (https://www.rust-lang.org/tools/install)): bash cargo install xan --locked Optional features Some features are not built by default because they cost too much in compilation time or in executable size. Here is a list of those optional features: * parquet: enables xan from -f parquet You can specify which optional features you want thusly: bash # To enable all optional features cargo install xan --locked --all-features # To enable only specific features cargo install xan --locked --features parquet You can also tweak the build flags to make sure the Rust compiler is able to leverage all your CPU’s features: bash CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' cargo install xan --locked You can also install the latest dev version thusly: bash cargo install --git https://github.com/medialab/xan --locked ### Scoop (Windows) xan can be installed using Scoop (https://scoop.sh/) on Windows: powershell scoop bucket add extras scoop install xan ### Homebrew (macOS) xan can be installed with Homebrew (https://brew.sh/) on macOS thusly: bash brew install xan ### Arch Linux You can install xan from the extra repository (https://archlinux.org/packages/extra/x86_64/xan/) using pacman: bash sudo pacman -S xan ### NetBSD A package is available from the official repositories. To install xan simply run: pkgin install xan ### Nix xan is packaged for Nix, and is available in Nixpkgs as of 25.05 release. To install it, you may add it to your environment.systemPackages as pkgs.xan or use nix-shell to enter an ephemeral shell. bash nix-shell -p xan ### Pixi (Linux, macOS, Windows) xan can be installed in Linux, macOS, and Windows using the Pixi (https://pixi.sh/latest/) package manager: bash pixi global install xan ### Conda Forge xan can be installed through conda-forge (https://conda-forge.org/) thusly: bash conda install conda-forge::xan ### Pre-built binaries Pre-built binaries can be found attached to every GitHub releases (https://github.com/medialab/xan/releases/latest). Currently supported targets include: - x86_64-apple-darwin - x86_64-unknown-linux-gnu - x86_64-unknown-linux-musl - x86_64-pc-windows-msvc - aarch64-apple-darwin - aarch64-unknown-linux-gnu ppc64le targets are not built by the CI yet but prebuilt binaries can still be found in the conda-forge package’s files (https://anaconda.org/conda-forge/xan/files) if you need them. Feel free to open a PR to improve the CI by adding relevant targets. ### Installing completions Note that xan also exposes handy automatic completions for command and header/column names that you can install through the xan completions command. Run the following command to understand how to install those completions: bash xan completions -h For zsh, you can add the completion file to ~/.zfunc: bash mkdir -p ~/.zfunc xan completions zsh > ~/.zfunc/_xan Then add this before compinit in your ~/.zshrc: zsh fpath=(~/.zfunc $fpath) autoload -Uz compinit compinit ## Quick tour Let’s learn about the most commonly used xan commands by exploring a corpus of French medias: ### Downloading the corpus bash curl -LO https://github.com/medialab/corpora/raw/master/polarisation/medias.csv ### Displaying the file’s headers bash xan headers medias.csv 0 webentity_id 1 name 2 prefixes 3 home_page 4 start_pages 5 indegree 6 hyphe_creation_timestamp 7 hyphe_last_modification_timestamp 8 outreach 9 foundation_year 10 batch 11 edito 12 parody 13 origin 14 digital_native 15 mediacloud_ids 16 wheel_category 17 wheel_subcategory 18 has_paywall 19 inactive ### Counting the number of rows bash xan count medias.csv 478 ### Previewing the file in the terminal bash xan view medias.csv Displaying 5/20 cols from 10 first rows of medias.csv ┌───┬───────────────┬───────────────┬────────────┬───┬─────────────┬──────────┐ │ - │ name │ prefixes │ home_page │ ... │ has_paywall │ inactive │ ├───┼───────────────┼───────────────┼────────────┼───┼─────────────┼──────────┤ │ 0 │ Acrimed.org │ http://acrim... │ http://ww... │ ... │ false │ │ │ 1 │ 24matins.fr │ http://24mat... │ https://w... │ ... │ false │ │ │ 2 │ Actumag.info │ http://actum... │ https://a... │ ... │ false │ │ │ 3 │ 2012un-Nouve... │ http://2012u... │ http://ww... │ ... │ false │ │ │ 4 │ 24heuresactu... │ http://24heu... │ http://24... │ ... │ false │ │ │ 5 │ AgoraVox │ http://agora... │ http://ww... │ ... │ false │ │ │ 6 │ Al-Kanz.org │ http://al-ka... │ https://w... │ ... │ false │ │ │ 7 │ Alalumieredu... │ http://alalu... │ http://al... │ ... │ false │ │ │ 8 │ Allodocteurs... │ http://allod... │ https://w... │ ... │ false │ │ │ 9 │ Alterinfo.net │ http://alter... │ http://ww... │ ... │ │ true │ │ ... │ ... │ ... │ ... │ ... │ ... │ ... │ └───┴───────────────┴───────────────┴────────────┴───┴─────────────┴──────────┘ On unix, don’t hesitate to use the -p flag to automagically forward the full output to an appropriate pager and skim through all the columns. ### Reading a flattened representation of the first row bash # NOTE: drop -c to avoid truncating the values xan flatten -c medias.csv Row n°0 ─────────────────────────────────────────────────────────────────────────────── webentity_id 1 name Acrimed.org prefixes http://acrimed.org|http://acrimed69.blogspot... home_page http://www.acrimed.org start_pages http://acrimed.org|http://acrimed69.blogspot... indegree 61 hyphe_creation_timestamp 1560347020330 hyphe_last_modification_timestamp 1560526005389 outreach nationale foundation_year 2002 batch 1 edito media parody false origin france digital_native true mediacloud_ids 258269 wheel_category Opinion Journalism wheel_subcategory Left Wing has_paywall false inactive Row n°1 ─────────────────────────────────────────────────────────────────────────────── webentity_id 2 ... ### Searching for rows bash xan search -s outreach internationale medias.csv | xan view Displaying 4/20 cols from 10 first rows of ┌───┬──────────────┬────────────────────┬───┬─────────────┬──────────┐ │ - │ webentity_id │ name │ ... │ has_paywall │ inactive │ ├───┼──────────────┼────────────────────┼───┼─────────────┼──────────┤ │ 0 │ 25 │ Businessinsider.fr │ ... │ false │ │ │ 1 │ 59 │ Europe-Israel.org │ ... │ false │ │ │ 2 │ 66 │ France 24 │ ... │ false │ │ │ 3 │ 220 │ RFI │ ... │ false │ │ │ 4 │ 231 │ fr.Sott.net │ ... │ false │ │ │ 5 │ 246 │ Voltairenet.org │ ... │ true │ │ │ 6 │ 254 │ Afp.com /fr │ ... │ false │ │ │ 7 │ 265 │ Euronews FR │ ... │ false │ │ │ 8 │ 333 │ Arte.tv │ ... │ false │ │ │ 9 │ 341 │ I24News.tv │ ... │ false │ │ │ ... │ ... │ ... │ ... │ ... │ ... │ └───┴──────────────┴────────────────────┴───┴─────────────┴──────────┘ ### Selecting some columns bash xan select foundation_year,name medias.csv | xan view Displaying 2 cols from 10 first rows of ┌───┬─────────────────┬───────────────────────────────────────┐ │ - │ foundation_year │ name │ ├───┼─────────────────┼───────────────────────────────────────┤ │ 0 │ 2002 │ Acrimed.org │ │ 1 │ 2006 │ 24matins.fr │ │ 2 │ 2013 │ Actumag.info │ │ 3 │ 2012 │ 2012un-Nouveau-Paradigme.com │ │ 4 │ 2010 │ 24heuresactu.com │ │ 5 │ 2005 │ AgoraVox │ │ 6 │ 2008 │ Al-Kanz.org │ │ 7 │ 2012 │ Alalumieredunouveaumonde.blogspot.com │ │ 8 │ 2005 │ Allodocteurs.fr │ │ 9 │ 2005 │ Alterinfo.net │ │ ... │ ... │ ... │ └───┴─────────────────┴───────────────────────────────────────┘ ### Sorting the file bash xan sort -s foundation_year medias.csv | xan view -s name,foundation_year Displaying 2 cols from 10 first rows of ┌───┬────────────────────────────────────┬─────────────────┐ │ - │ name │ foundation_year │ ├───┼────────────────────────────────────┼─────────────────┤ │ 0 │ Le Monde Numérique (Ouest France) │ │ │ 1 │ Le Figaro │ 1826 │ │ 2 │ Le journal de Saône-et-Loire │ 1826 │ │ 3 │ L'Indépendant │ 1846 │ │ 4 │ Le Progrès │ 1859 │ │ 5 │ La Dépêche du Midi │ 1870 │ │ 6 │ Le Pélerin │ 1873 │ │ 7 │ Dernières Nouvelles d'Alsace (DNA) │ 1877 │ │ 8 │ La Croix │ 1883 │ │ 9 │ Le Chasseur Francais │ 1885 │ │ ... │ ... │ ... │ └───┴────────────────────────────────────┴─────────────────┘ ### Deduplicating the file on some column bash # Some medias of our corpus have the same ids on mediacloud.org xan dedup -s mediacloud_ids medias.csv | xan count && xan count medias.csv 457 478 Deduplicating can also be done while sorting: bash xan sort -s mediacloud_ids -u medias.csv ### Computing frequency tables bash xan frequency -s edito medias.csv | xan view Displaying 3 cols from 5 rows of ┌───┬───────┬────────────┬───────┐ │ - │ field │ value │ count │ ├───┼───────┼────────────┼───────┤ │ 0 │ edito │ media │ 423 │ │ 1 │ edito │ individu │ 30 │ │ 2 │ edito │ plateforme │ 14 │ │ 3 │ edito │ agrégateur │ 10 │ │ 4 │ edito │ agence │ 1 │ └───┴───────┴────────────┴───────┘ ### Printing a histogram bash xan frequency -s edito medias.csv | xan hist Histogram for edito (bars: 5, sum: 478, max: 423): media |423 88.49%|━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━| individu | 30 6.28%|━━━╸ | plateforme | 14 2.93%|━╸ | agrégateur | 10 2.09%|━╸ | agence | 1 0.21%|╸ | ### Computing descriptive statistics bash xan stats -s indegree,edito medias.csv | xan transpose | xan view -I Displaying 2 cols from 14 rows of ┌─────────────┬───────────────────┬────────────┐ │ field │ indegree │ edito │ ├─────────────┼───────────────────┼────────────┤ │ count │ 463 │ 478 │ │ count_empty │ 15 │ 0 │ │ type │ int │ string │ │ types │ int|empty │ string │ │ sum │ 25987 │ │ │ mean │ 56.12742980561554 │ │ │ variance │ 4234.530197929737 │ │ │ stddev │ 65.07326792108829 │ │ │ min │ 0 │ │ │ max │ 424 │ │ │ lex_first │ 0 │ agence │ │ lex_last │ 99 │ plateforme │ │ min_length │ 0 │ 5 │ │ max_length │ 3 │ 11 │ └─────────────┴───────────────────┴────────────┘ ### Evaluating an expression to filter a file bash xan filter 'batch > 1' medias.csv | xan count 130 To access the expression language’s cheatsheet, run xan help cheatsheet. To display the full list of available functions, run xan help functions. ### Evaluating an expression to create a new column based on other ones bash xan map 'fmt("{} ({})", name, foundation_year) as key' medias.csv | xan select key | xan slice -l 10 key Acrimed.org (2002) 24matins.fr (2006) Actumag.info (2013) 2012un-Nouveau-Paradigme.com (2012) 24heuresactu.com (2010) AgoraVox (2005) Al-Kanz.org (2008) Alalumieredunouveaumonde.blogspot.com (2012) Allodocteurs.fr (2005) Alterinfo.net (2005) To access the expression language’s cheatsheet, run xan help cheatsheet. To display the full list of available functions, run xan help functions. ### Transform a column by evaluating an expression bash xan transform name 'split(name, ".") | first | upper' medias.csv | xan select name | xan slice -l 10 name ACRIMED 24MATINS ACTUMAG 2012UN-NOUVEAU-PARADIGME 24HEURESACTU AGORAVOX AL-KANZ ALALUMIEREDUNOUVEAUMONDE ALLODOCTEURS ALTERINFO To access the expression language’s cheatsheet, run xan help cheatsheet. To display the full list of available functions, run xan help functions. ### Performing custom aggregation bash xan agg 'sum(indegree) as total_indegree, mean(indegree) as mean_indegree' medias.csv | xan view -I Displaying 1 col from 1 rows of ┌────────────────┬───────────────────┐ │ total_indegree │ mean_indegree │ ├────────────────┼───────────────────┤ │ 25987 │ 56.12742980561554 │ └────────────────┴───────────────────┘ To access the expression language’s cheatsheet, run xan help cheatsheet. To display the full list of