RankShift: In-Database Detection and Explanation of Categorical Shifts

# RankShift: In-Database Detection and Explanation of Categorical Shifts
Source: [https://arxiv.org/html/2608.28922](https://arxiv.org/html/2608.28922)
###### Abstract

A login service can receive its usual number of failed sign\-ins while one source grows from 2% to 30% of them\. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable\. These events change which categories are active without changing how many events occur\. RankShift detects such changes inside the analytical database that stores the data\. It compares each window’s category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change\. The same query returns the score, calibrated alert, and largest increasing contributions\.

We evaluate RankShift on HDFS, BGL, and Thunderbird\. It matches the count\-vector autoencoder within 0\.001 AUROC on HDFS \(0\.999 versus 1\.000\) and leads on Thunderbird \(0\.983 versus 0\.949\)\. In a controlled fixed\-volume experiment, RankShift detects rare\-category shifts that are invisible to event\-count monitoring, reaching 0\.787 AUROC compared with 0\.771 for the autoencoder\. Across all three corpora, observed false\-alarm rates track the requested operating levels\. RankShift requires no model training or inference service, and the autoencoder’s deployed state is 137×\\timeslarger\.

###### Index Terms:

categorical data, anomaly detection, distribution shift, data streams, in\-database analytics

## IIntroduction

Monitoring systems often summarize each time window by request rate, error count, or latency\. These measures can remain stable while the shares of source addresses, routes, event templates, or error codes change sharply\. Figure[1](https://arxiv.org/html/2608.28922#S1.F1)illustrates this pattern in a failed\-sign\-in stream\. Changes in categorical composition can reveal an emerging source, a failing component, or a new workload pattern that aggregate monitoring misses\. A useful monitor must detect the change and identify which categories gained or lost share\.

Existing log detectors require a preprocessing, fitting, and scoring pipeline\. Count\-vector methods first group parsed events into windows and fit a model over the resulting vectors\[[1](https://arxiv.org/html/2608.28922#bib.bib6)\]\. The autoencoder evaluated here learns normalization statistics from benign fitting windows, trains an encoder\-decoder, and scores later windows by reconstruction error\. Sequence detectors construct ordered event\-template sequences and train next\-event or masked\-event predictors\[[2](https://arxiv.org/html/2608.28922#bib.bib7),[3](https://arxiv.org/html/2608.28922#bib.bib8)\]\. Deployment retains preprocessing statistics, learned weights, and an inference runtime\.

Distribution\-change methods use different operational state\. QuantTree estimates a partition from reference observations, while MStream maintains hash sketches and decayed counts\[[4](https://arxiv.org/html/2608.28922#bib.bib12),[5](https://arxiv.org/html/2608.28922#bib.bib13)\]\. They run as dedicated streaming algorithms outside the analytical event table\.

This paper makes three contributions:

- •Monitoring method\.RankShift uses category\-level Pearson contributions to score a window and identify the categories driving the change\. Benign score ranks calibrate the alert\.
- •Evaluation\.We evaluate RankShift on three real log datasets and use fixed\-total shifts to show when it leads or trails a count\-vector autoencoder\.
- •Native execution\.It implements equivalent NumPy, DuckDB SQL, and ADX KQL semantics and measures correctness, event\-volume scaling, setup, online cost, and deployed state\.

Fig\. 1:Illustrative fixed\-volume shift in failed sign\-ins\. \(a\) Each minute contains 100 failed sign\-ins, while source x rises from 2% to 30% of the total\. A monitor of total volume remains flat\. \(b\) At minute 8, source x accounts for 98% of the RankShift score \(3\.92 of 4\.00\)\. The signs show whether each source gained or lost share\.
## IIRelated Work

Distribution monitoring\.Pearson’s chi\-square goodness\-of\-fit test compares observed and theoretical frequencies across groups\[[6](https://arxiv.org/html/2608.28922#bib.bib1)\]\. Its statistic sums one normalized squared deviation for each group, allowing the total discrepancy to be traced to individual groups\. Lakhina et al\. analyzed distributions of source addresses, destination addresses, and ports in network flows\[[7](https://arxiv.org/html/2608.28922#bib.bib2)\]\. They summarized these distributions with entropy, detected anomalies beyond those found by volume\-based methods, and clustered the results for unsupervised anomaly classification\. QuantTree learns a multivariate histogram with prescribed bin probabilities from stationary reference data, then tests later batches using the number of observations entering each bin\[[4](https://arxiv.org/html/2608.28922#bib.bib12)\]\. RankShift retains operational category names, such as source addresses and event templates, and compares their shares so a uniform change in total count leaves the score unchanged\.

Log and stream detectors\.Log detectors differ in the relationships they learn\. Xu et al\. represent each window as event\-template counts and use PCA to learn correlations among event types\[[1](https://arxiv.org/html/2608.28922#bib.bib6)\]\. DeepLog learns which event template should follow a preceding sequence, while LogBERT learns normal sequence context through masked\-event prediction and a one\-class objective\[[2](https://arxiv.org/html/2608.28922#bib.bib7),[3](https://arxiv.org/html/2608.28922#bib.bib8)\]\. Stream detectors operate on other structures\. MStream captures correlations across the categorical and numerical attributes of each record, whereas MIDAS detects sudden bursts of repeated source\-destination edges\[[5](https://arxiv.org/html/2608.28922#bib.bib13),[8](https://arxiv.org/html/2608.28922#bib.bib14)\]\. RankShift addresses changes in the relative frequencies within one named categorical field\. Event order, cross\-field relationships, and graph structure lie outside this scope\.

Explaining anomalies\.Existing methods usually explain a result produced by another detector\. SHAP attributes a model prediction to its input features, with computation and exactness determined by the model and chosen explainer\[[9](https://arxiv.org/html/2608.28922#bib.bib18)\]\. ACE and DeepAID similarly explain anomaly scores after a detector has produced them\[[10](https://arxiv.org/html/2608.28922#bib.bib17),[11](https://arxiv.org/html/2608.28922#bib.bib19)\]\. MacroBase instead identifies attribute values that are enriched among records already classified as outliers, while weighted log odds ranks terms that distinguish two corpora\[[12](https://arxiv.org/html/2608.28922#bib.bib15),[13](https://arxiv.org/html/2608.28922#bib.bib16)\]\. RankShift requires no separate explainer\. Its category contributions are computed as part of the score, sum exactly to that score, and retain operational values such as source addresses or event templates\.

In\-database analytics\.MADlib brings statistical analysis and machine learning into relational databases, allowing model fitting and inference to run where the data is stored\[[14](https://arxiv.org/html/2608.28922#bib.bib20)\]\. DuckDB provides an embedded analytical database that executes SQL directly within an application process\[[15](https://arxiv.org/html/2608.28922#bib.bib21)\]\. These systems establish the value of performing analytics without moving data into a separate processing service\. RankShift applies this principle to categorical monitoring and returns both an alert and its category\-level decomposition from relational operations\.

Alert calibration\.An anomaly score does not by itself define when to raise an alert\. Split\-conformal methods compare a new score with scores from held\-out calibration examples and convert its rank into a p\-value\[[16](https://arxiv.org/html/2608.28922#bib.bib3),[17](https://arxiv.org/html/2608.28922#bib.bib4),[18](https://arxiv.org/html/2608.28922#bib.bib5)\]\. When benign calibration and future windows are exchangeable, alerting at levelα\\alphacontrols the marginal false\-alarm probability atα\\alpha\. RankShift uses benign calibration windows to set this operating level without assuming a parametric distribution for its scores\.

## IIIRankShift

### III\-ACategories and reference shares

RankShift monitors one categorical field in timestamped events grouped into fixed time windows\. Before monitoring begins, every field value is mapped to one ofCCcategories\. A field with a known vocabulary, such as an HTTP status code, can use its values directly\. For unstructured logs, a parser and template dictionary can be learned from historical data and then frozen; later messages that match no template map toOTHER\. A fixed\-size hash mapping is another option for large vocabularies\. The mapping determines how precisely RankShift can identify a driver: values merged intoOTHERor the same hash bucket receive one combined contribution\.

Benign history establishes the expected category mix\. Letrir\_\{i\}be the number of reference events assigned to categoryii, and letR=∑iriR=\\sum\_\{i\}r\_\{i\}be the total number of reference events\. A category absent from this history has a raw share of zero, which cannot be used in the Pearson score because the score divides by the reference share\. RankShift therefore spreads a small pseudocount across allCCcategories:

qi=ri\+τ/CR\+τ,τ\>0\.q\_\{i\}=\\frac\{r\_\{i\}\+\\tau/C\}\{R\+\\tau\},\\qquad\\tau\>0\.\(1\)The resulting reference sharesqiq\_\{i\}are positive and sum to one\. Assigningτ/C\\tau/Cto each category keeps the total smoothing equal toτ\\tauregardless of vocabulary size\. We useτ=1\\tau=1throughout, adding one count of total mass divided equally across the category set\.

### III\-BWindow score and category contributions

For a window containingN\>0N\>0events, letsis\_\{i\}be the number assigned to categoryii\. Its observed category share ispi=si/Np\_\{i\}=s\_\{i\}/N, whereN=∑isiN=\\sum\_\{i\}s\_\{i\}\. RankShift compares the observed sharesppwith the reference sharesqq:

ci=\(pi−qi\)2qi,X=∑i=1Cci\.c\_\{i\}=\\frac\{\(p\_\{i\}\-q\_\{i\}\)^\{2\}\}\{q\_\{i\}\},\\qquad X=\\sum\_\{i=1\}^\{C\}c\_\{i\}\.\(2\)The totalXXis the Pearson divergence between the observed and reference compositions\. Each termcic\_\{i\}measures how much categoryiicontributes to that difference\. For the same change in share, a category that was rare in the reference receives a larger contribution becauseqiq\_\{i\}appears in the denominator\. Every contribution is nonnegative, and the complete set sums exactly toXX\. Squaring removes direction, so RankShift also records the sign ofpi−qip\_\{i\}\-q\_\{i\}: positive values gained share and negative values lost share\.

For triage, RankShift reports thekklargest contributions among categories whose shares increased\. Categories that lost share and contributions below rankkkremain available in the full table but do not appear in this ranked view\.

RankShift normalizes each window by its total event count before scoring\. If every category count grows or shrinks by the same factor, the observed sharespip\_\{i\}remain unchanged, and so do the category contributions and total scoreXX\. The score therefore isolates changes in category mix from uniform changes in traffic volume\. It can detect a redistribution while the total event rate remains stable, as in Figure[1](https://arxiv.org/html/2608.28922#S1.F1)\. A surge or drop that preserves the same category shares produces no RankShift signal and requires a separate volume monitor\.

### III\-CAlert calibration

Once the reference shares are fixed, a separate set of benign windows defines the normal range of scores\. LetT1,…,TnT\_\{1\},\\ldots,T\_\{n\}be the scores ofn≥1n\\geq 1calibration windows\. For a new window with scoreXX, RankShift computes

p^=1\+\|\{j:Tj≥X\}\|n\+1\.\\widehat\{p\}=\\frac\{1\+\|\\\{j:T\_\{j\}\\geq X\\\}\|\}\{n\+1\}\.\(3\)The numerator counts calibration windows whose scores are at least as large asXX\. The additional one includes the new score in its finite\-sample rank and prevents a p\-value of zero\. A smallp^\\widehat\{p\}means that few benign calibration windows produced a score as large as the new window\.

RankShift raises an alert whenp^≤α\\widehat\{p\}\\leq\\alpha\. If the calibration scores and future benign scores are exchangeable after the reference is fixed, the marginal false\-alarm probability is at mostα\\alpha\[[16](https://arxiv.org/html/2608.28922#bib.bib3),[18](https://arxiv.org/html/2608.28922#bib.bib5)\]\. The≥\\geqcomparison counts tied calibration scores against the alert\. A deployment with a fixedα\\alphacan store the corresponding score cutoff instead of the complete calibration sample\.

## IVDatabase Execution

After the category mapping is fixed, RankShift’s deployed state consists ofCCreference shares and eithernnsorted calibration scores or a fixed alert cutoff and comparison rule\. Its scoring and calibration procedure has no model training phase, learned weights, model\-inference runtime, or GPU requirement\. While a window is open, each event increments one category count\. For a window withaaactive categories, the sparse score requiresO⁡\(a\)O\(a\)work, and selecting the largestkkincreasing contributions requiresO⁡\(a​log⁡k\)O\(a\\log k\)work\. A sorted calibration sample supports p\-value lookup inO⁡\(log⁡n\)O\(\\log n\)time; a fixed alert cutoff reduces the decision toO⁡\(1\)O\(1\)\.

RankShift begins by counting events by cohort, window, and category\. The database materializes this grouped table once, allowing reference construction, window scoring, calibration, and driver ranking to reuse the same counts without rescanning the raw events\. A dense implementation would createCCrows for every window, including categories with zero events\. RankShift instead uses the identity

X=∑i:pi\>0pi2qi−1,X=\\sum\_\{i:p\_\{i\}\>0\}\\frac\{p\_\{i\}^\{2\}\}\{q\_\{i\}\}\-1,which evaluates the exact score using only categories present in the window\. Appendix[A](https://arxiv.org/html/2608.28922#A1)gives the derivation\. Reference counts produce the smoothed sharesqiq\_\{i\}\. When a present category has no reference count, the query assigns it the smoothed floor\(τ/C\)/\(R\+τ\)\(\\tau/C\)/\(R\+\\tau\)from Eq\. \([1](https://arxiv.org/html/2608.28922#S3.E1)\)\.

After scoring each test window, the query counts calibration scores at least as large as the test score to produce the conformal p\-value and alert\. It then keeps categories whose shares increased, orders them by contribution and category identifier, and returns the largestkk\. Each result contains the window score, p\-value, alert, selected category values, and their contributions\. Dividing the sum of the returned contributions byXXgives the fraction of the score represented by the ranked view\. Appendix[B](https://arxiv.org/html/2608.28922#A2)prints the complete DuckDB SQL executed in the experiments\.

Conformal alerts depend on the ordering of window scores\. Small floating\-point differences between query engines can change a tie or move a score across the alert boundary\. RankShift therefore separates the reported score from the value used for decisions\. It returns the unrounded scoreXX, but rounds a nonnegative copy to six decimal places, with exact halves rounded upward, before comparing calibration and test scores\. DuckDB and ADX use the same rule\.

We compared both query engines with the NumPy reference implementation on a 30\-window BGL fixture\. The maximum absolute score difference was3×10−83\\times 10^\{\-8\}for DuckDB and8×10−98\\times 10^\{\-9\}for ADX\. Both engines produced identical p\-values, alerts, contribution values, and category ordering\. Additional tests at exact rounding boundaries confirmed the same decision rule\.

## VEvaluation

The experiments test different parts of the RankShift result\. HDFS, BGL, and Thunderbird measure anomaly detection and false\-alarm calibration on natural log windows\. A fixed\-total experiment changes category shares while preserving every window’s event count, isolating composition from volume\. DuckDB and ADX measure native query execution\. CTU\-13 tests whether a bounded category mapping preserves enough information to identify raw attack sources\. In every comparison, the category mapping is fixed before the test period, and RankShift and its baseline receive the same count matrix and test windows\. Table[I](https://arxiv.org/html/2608.28922#S5.T1)summarizes these protocols\.

TABLE I:Datasets and evaluation protocols\. Category vocabularies are fixed before testing\. Alert evaluation uses disjoint benign reference, calibration, and test cohorts\.### V\-AData and protocol

HDFS, BGL, and Thunderbird are public system\-log datasets distributed through Loghub\[[19](https://arxiv.org/html/2608.28922#bib.bib9)\]\. HDFS records operations on Hadoop file\-system blocks\. Events associated with the same block form one sample, producing 575,061 block samples over 29 event identifiers; the published block label defines whether each sample is anomalous\. BGL and Thunderbird contain timestamped supercomputer logs with message\-level anomaly labels\. We group BGL into 3,619 one\-hour windows and the first ten million Thunderbird messages into 30,120 one\-minute windows\. A time window is anomalous when it contains at least one labeled anomalous message\.

Category vocabularies use only information available before testing\. For BGL, Drain learns templates from the first half of the log and is then frozen\[[20](https://arxiv.org/html/2608.28922#bib.bib10)\]\. Its 643 learned templates plusOTHERproduce 644 categories\. For Thunderbird, the vocabulary retains the 1,024 most frequent benign templates from the first half and maps every remaining template toOTHER\. Both frozen mappings are applied unchanged throughout the second\-half tests\.

Detection comparisons use the same benign fitting data and test samples for both methods\. HDFS uses five paired random splits\. In each split, one benign half defines the RankShift reference and trains the autoencoder; the remaining benign samples and all anomalous samples form the shared test set\. BGL and Thunderbird preserve time order: first\-half benign windows define the reference and train the autoencoder, and every second\-half window is tested\. Autoencoder training is repeated with five random seeds\.

The autoencoder receives the same category\-count vectors as RankShift\. Its preprocessing applieslog⁡\(1\+x\)\\log\(1\+x\)and estimates each feature’s mean and standard deviation from the benign fitting windows only\. The encoder contains at most 128 hidden units and a 32\-unit bottleneck, followed by a symmetric decoder\. Training minimizes reconstruction error on benign windows, and mean reconstruction error scores each test window\.

Alert calibration is evaluated separately from detection ranking\. HDFS benign samples are divided into disjoint reference, calibration, and test cohorts\. For BGL and Thunderbird, the category mapping is learned from the first half, then the later benign windows are divided into the three cohorts\. This measures false alarms under the later operating distribution without allowing any test label to influence the reference or calibration scores\.

### V\-BDetection and alert calibration on labeled logs

Table[II](https://arxiv.org/html/2608.28922#S5.T2)compares RankShift and the autoencoder on the same test samples\. RankShift and the autoencoder both separate HDFS anomalies almost perfectly, reaching 0\.999 and 1\.000 AUROC\. On the complete second half of BGL, RankShift reaches 0\.699 compared with 0\.864 for the autoencoder\. On the complete second half of Thunderbird, RankShift reaches 0\.983 compared with 0\.949\. The opposing BGL and Thunderbird results motivate the fixed\-total experiment below, which tests how performance changes when shifted activity enters rare or common categories\.

TABLE II:Detection AUROC on shared test samples\. HDFS reports mean \(standard deviation\) over five paired splits\. BGL and Thunderbird use first\-half benign windows for setup and the complete second half for testing; autoencoder values report mean \(standard deviation\) over five training seeds\.Table[III](https://arxiv.org/html/2608.28922#S5.T3)reports mean alert rates over five random partitions\. Within each partition, the benign reference, calibration, and test cohorts are disjoint\. At the requested 0\.05 false\-alarm level, the observed rates are 0\.041 on HDFS, 0\.038 on BGL, and 0\.046 on Thunderbird\. HDFS detects every anomaly at all three alert levels\. BGL detection rises from 0\.490 atα=0\.01\\alpha=0\.01to 0\.760 atα=0\.05\\alpha=0\.05, then changes little atα=0\.10\\alpha=0\.10\. Thunderbird detection rises from 0\.024 to 0\.213 and 0\.527 across the same levels\.

Thunderbird’s 0\.983 AUROC and its 0\.213 detection rate atα=0\.05\\alpha=0\.05measure different properties\. AUROC evaluates score ordering across all possible thresholds\. The detection rate measures recall at one threshold chosen to limit benign false alarms\. Strong overall ordering can therefore coexist with low recall at a strict operating point\.

TABLE III:Calibrated alert rates averaged over five random partitions\. Within each partition, the benign reference, calibration, and test cohorts are disjoint\. False\-alarm rate is measured on benign test windows; detection rate is measured on anomalous windows\.RankShift trails the autoencoder on BGL and leads on Thunderbird\. The next subsection tests whether this contrast is related to how often the changing categories appeared in benign history\.

### V\-CControlled synthetic redistribution

For each of five seeds, we randomly divide benign BGL windows from the second half of the log into two equal groups after the parser has been frozen\. The setup group defines RankShift’s reference shares and trains the autoencoder\. From the evaluation group, we select 400 windows containing at least 20 events\.

Each selected window remains unchanged and serves as a benign example\. We create a synthetic anomaly by copying the window and moving some of its event counts between categories\. Categories observed in the setup group are ranked by their total counts\. The least frequent 10% form the rare\-recipient pool, and the most frequent 10% form the common\-recipient pool\. For each window, we sample five recipient categories from each pool\.

We attempt to move 30% of the window’s events\. Counts are removed from its largest active categories outside the selected recipients and reassigned to the recipients\. The fraction sent to common recipients varies from 0% to 100% in 25\-point steps; the remainder goes to rare recipients\. Limited donor counts and integer rounding produce an average realized transfer of 18\.5%\. No events are added or removed, so every synthetic anomaly has exactly the same total as its original benign window\. A volume\-only detector therefore assigns identical scores to both classes and has 0\.500 AUROC\.

Fig\. 2:Detection under controlled fixed\-total redistribution on BGL\. The horizontal axis gives the fraction of transferred events assigned to common recipient categories; the remainder enters rare recipient categories\. The upper panel reports mean AUROC over five paired seeds\. The lower panel reports RankShift minus autoencoder AUROC, with positive values favoring RankShift\. Error bars show one standard deviation\. Every synthetic anomaly has the same event count as its original benign window, giving the volume baseline 0\.500 AUROC\.At 0% common allocation, when all transferred events enter rare recipients, RankShift reaches 0\.787 AUROC compared with 0\.771 for the autoencoder\. The methods remain within 0\.016 AUROC through 75% common allocation\. At 100%, RankShift falls to 0\.398 while the autoencoder reaches 0\.594\. RankShift emphasizes changes in rare categories because each contribution is divided by the category’s benign reference shareqiq\_\{i\}\. The same absolute change therefore produces a larger contribution for a rare category than for a common one\. The 0\.398 result means that synthetic anomalies tend to receive lower RankShift scores than the original benign windows when all transferred events enter common recipients\. Under this controlled intervention, RankShift responds most strongly when activity moves into categories that were rare in benign history and loses sensitivity as the redistribution becomes confined to common categories\.

### V\-DExecution and cost

The execution benchmark measures the complete RankShift query, including the score, conformal p\-value, alert, and five largest increasing contributions\. Event rows are generated and loaded before timing begins\. The benchmark varies the number of windows while holding the workload structure fixed: 5,000 supported categories, 80 active categories per window, and four events per active category\. Each reported value is the median of five warm query executions\.

On the measured DuckDB virtual machine, median query time rises from 0\.428 seconds for 0\.64 million events to 12\.2 seconds for 64 million events\. The largest run processes approximately 5\.2 million events per second\. Every query profile contains one scan of the event table\.

ADX median query time rises from 0\.900 seconds for 0\.64 million events to 11\.000 seconds for 64 million events\. The largest run processes approximately 5\.8 million events per second\. DuckDB and ADX run on different infrastructure\. Their measurements characterize scaling within each engine and do not support a direct speed ranking between them\.

Fig\. 3:Complete RankShift query time as event volume increases\. Lines report median client wall time over five warm runs; upper bars report p95\. DuckDB and ADX complete workloads through 64 million events\.TABLE IV:Native\-query scaling under the same logical workload\. Client entries are median \(p95\) seconds over five warm runs\. Peak memory is the maximum observed across those runs; ADX memory is per node\.At 64 million events, DuckDB uses 5\.09 GiB of peak buffer memory and ADX uses 9\.00 GiB of peak query memory per node\.

A separate benchmark isolates method cost from database infrastructure\. It runs RankShift and the autoencoder in the same process on the 644\-category BGL representation\. Both methods return a score, conformal p\-value, alert, and five ranked categories\. Values are medians over five complete runs; each online measurement contains two warmups and five timed repetitions\.

TABLE V:Matched same\-process cost on reference\-frozen BGL\. Times are median \(p95\) over five complete runs\.RankShift scores a window in a median of 19\.5 microseconds, compared with 43\.5 microseconds for the autoencoder\. Its median setup time is 0\.009 seconds, compared with 2\.279 seconds\. RankShift retains 5,160 bytes of deployed state, while the autoencoder retains 706,264 bytes, making RankShift’s state 137×\\timessmaller\. Online and setup times vary across complete runs; the retained\-state sizes are fixed by the measured implementations\.

## VIOperational Scope

RankShift is designed to detect changes in category composition\. A proportional increase or decrease across all categories leaves the score unchanged because the category shares remain unchanged\. A deployment that must also detect changes in total traffic should pair RankShift with a volume monitor\.

Window size determines how much category shares vary under normal conditions\. For example, a 30\-to\-70 split observed across 10 events is less stable than the same split observed across 10,000 events\. Reference, calibration, and monitored windows should therefore use the same duration and represent similar event\-volume ranges\. The calibration scores then reflect the normal sampling variation of the deployed windows\.

The benign category distribution can also change over time\. On BGL, RankShift reaches 0\.923 AUROC under a random held\-out split and 0\.699 when the first half defines the reference and the complete second half is tested\. This decline shows the effect of evaluating against an older reference\. Reference shares and calibration scores should be refreshed when sustained benign changes alter the operating distribution\. The conformal guarantee requires future benign scores to be exchangeable with the calibration scores\.

The category vocabulary determines the precision of the reported drivers\. In eight CTU\-13 captures, every labeled attack source is absent from the 2,048\-address reference vocabulary and maps toOTHER\[[21](https://arxiv.org/html/2608.28922#bib.bib11)\]\. Across 1,909 later attack windows, RankShift recovers 0\.228 of true sources among its first five candidates, compared with 0\.644 for current\-count ranking\. RankShift computes an exact contribution forOTHERas a group\. It cannot identify an individual address within that group\. Future versions can address this limitation with a hierarchical vocabulary that refines an anomalous group into progressively narrower subgroups and raw values while retaining bounded top\-level state\.

## VIIConclusion

RankShift turns a classical Pearson comparison into an operational monitor for categorical streams\. Its per\-category contributions produce the anomaly score and identify which values gained or lost share\. Benign calibration converts the score into an alert\. The complete calculation runs inside the analytical database without a trained detector or separate explanation system\.

The evaluation defines RankShift’s operating range\. It responds strongly when activity moves into categories that were rare in benign history and loses sensitivity when redistribution remains among common categories\. Its reported drivers are exact at the resolution of the deployed vocabulary; values merged intoOTHERrequire a finer or hierarchical representation\. RankShift targets internet\-scale categorical telemetry where model training, inference, and movement of raw events to a separate detector are infeasible\. It executes inside the analytical database’s optimized counting path, providing low\-latency monitoring together with category\-level drivers\. Changes in total activity remain a separate monitoring channel\.

## Appendix ASupporting Identities

The database calculates the exact score without creating rows for categories absent from a window\. Expanding Eq\. \([2](https://arxiv.org/html/2608.28922#S3.E2)\) gives

X\\displaystyle X=∑i\(pi−qi\)2qi\\displaystyle=\\sum\_\{i\}\\frac\{\(p\_\{i\}\-q\_\{i\}\)^\{2\}\}\{q\_\{i\}\}\(4\)=∑i\(pi2qi−2​pi\+qi\)\\displaystyle=\\sum\_\{i\}\\left\(\\frac\{p\_\{i\}^\{2\}\}\{q\_\{i\}\}\-2p\_\{i\}\+q\_\{i\}\\right\)=∑i:pi\>0pi2qi−1,\\displaystyle=\\sum\_\{i:p\_\{i\}\>0\}\\frac\{p\_\{i\}^\{2\}\}\{q\_\{i\}\}\-1,where the final equality uses∑ipi=∑iqi=1\\sum\_\{i\}p\_\{i\}=\\sum\_\{i\}q\_\{i\}=1\. Terms withpi=0p\_\{i\}=0disappear from the final sum\. The query therefore evaluates the exact score using only categories present in the current window\.

Vocabulary aggregation can hide changes among raw values\. For a groupGG, definePG=∑i∈GpiP\_\{G\}=\\sum\_\{i\\in G\}p\_\{i\}andQG=∑i∈GqiQ\_\{G\}=\\sum\_\{i\\in G\}q\_\{i\}\. Cauchy–Schwarz gives

\(PG−QG\)2QG≤∑i∈G\(pi−qi\)2qi\.\\frac\{\(P\_\{G\}\-Q\_\{G\}\)^\{2\}\}\{Q\_\{G\}\}\\leq\\sum\_\{i\\in G\}\\frac\{\(p\_\{i\}\-q\_\{i\}\)^\{2\}\}\{q\_\{i\}\}\.\(5\)The contribution of a merged category cannot exceed the sum of the original contributions\. Increases and decreases among values insideOTHERcan cancel before scoring\. RankShift remains exact for the deployed vocabulary, while the finer raw\-value representation can contain information lost through aggregation\. This result motivates hierarchical refinement of anomalous groups\.

## Appendix BDuckDB Implementation

Listing[1](https://arxiv.org/html/2608.28922#LST1)gives the complete DuckDB implementation evaluated in the paper\. Its parameters specify the smoothing mass, vocabulary size, score precision, alert level, and number of returned drivers\. The query constructs reference shares, calculates exact sparse scores, assigns conformal p\-values, produces alerts, and ranks increasing category contributions\.

Listing 1:Complete DuckDB query used by the experiments\.WITHcountsASMATERIALIZED\(

SELECTrole,w,cat,COUNT\(\*\)::DOUBLEcnt

FROMeventsGROUPBYrole,w,cat

\),totalsAS\(

SELECTrole,w,SUM\(cnt\)nFROMcounts

GROUPBYrole,w

\),reference\_countsAS\(

SELECTcat,SUM\(cnt\)rFROMcounts

WHERErole=’ref’GROUPBYcat

\),reference\_totalAS\(

SELECTSUM\(r\)r\_totalFROMreference\_counts

\),reference\_qAS\(

SELECTcat,\(r\+$tau/$category\_count\)

/\(\(SELECTr\_totalFROMreference\_total\)\+$tau\)q

FROMreference\_counts

\),observedASMATERIALIZED\(

SELECTc\.role,c\.w,c\.cat,c\.cnt/t\.np,

COALESCE\(q\.q,\($tau/$category\_count\)

/\(\(SELECTr\_totalFROMreference\_total\)\+$tau\)\)q

FROMcountscJOINtotalstUSING\(role,w\)

LEFTJOINreference\_qqUSING\(cat\)

WHEREc\.roleIN\(’cal’,’test’\)

\),scored\_rawASMATERIALIZED\(

SELECTrole,w,

GREATEST\(SUM\(p\*p/q\)\-1\.0,0\.0\)score

FROMobservedGROUPBYrole,w

\),scoredASMATERIALIZED\(

SELECTrole,w,score,

ROUND\(score,$score\_key\_decimals\)score\_key

FROMscored\_raw

\),rankedAS\(

SELECTw,cat,\(p\-q\)\*\(p\-q\)/qcell,

ROW\_NUMBER\(\)OVER\(PARTITIONBYw

ORDERBY\(p\-q\)\*\(p\-q\)/qDESC,cat\)driver\_rank

FROMobserved

WHERErole=’test’ANDp\>q

\),top\_driversAS\(

SELECTw,cat,cell,driver\_rankFROMranked

WHEREdriver\_rank<=$top\_k

\),calibrationAS\(

SELECTscore\_keycalibration\_score\_key

FROMscoredWHERErole=’cal’

\),test\_scoresAS\(

SELECTw,score,

\(1\.0\+\(SELECTCOUNT\(\*\)FROMcalibration

WHEREcalibration\_score\_key\>=scored\.score\_key\)\)

/\(\(SELECTCOUNT\(\*\)FROMcalibration\)\+1\.0\)p\_value

FROMscoredWHERErole=’test’

\)

SELECTs\.w,s\.score,s\.p\_value,

s\.p\_value<=$alphais\_alert,

d\.cat,d\.cell,d\.driver\_rank

FROMtest\_scoress

LEFTJOINtop\_driversdUSING\(w\)

ORDERBYs\.w,d\.driver\_rank;

## Acknowledgments

We thank Sudeep Agarwal, Mohit Suley, Greg Dunham, Mukul Sabharwal, and Euan Grant \(Microsoft\) for leadership support of this work\. We thank Zohair Shafi \(Northeastern University\) for writing mentorship and guidance\. We also thank Abhishek Gupta, Varun Chawla, and the Azure Front Door, Web Application Firewall, MAI/Bing Fundamentals, Bing Defense, and Bot Detection teams at Microsoft for production\-traffic access and operational feedback that informed this work\.

#### AI use disclosure\.

Portions of this manuscript and its accompanying artifacts \(code, empirical results, statistical tests, and figures\) were generated with assistance from a generative AI assistant\. The author directed all experiments, reviewed all outputs, and is responsible for the final content\.

#### Disclaimer\.

Some of the information in this document relates to pre\-released content which may be subsequently modified\. Microsoft makes no warranties, express or implied, with respect to the information provided here\. This document is provided “as\-is”\. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice\. © 2026 Microsoft\. All rights reserved\.

## References

- \[1\]W\. Xu, L\. Huang, A\. Fox, D\. Patterson, and M\. I\. Jordan\(2009\)Detecting large\-scale system problems by mining console logs\.InProceedings of the 22nd ACM Symposium on Operating Systems Principles,pp\. 117–132\.External Links:[Document](https://dx.doi.org/10.1145/1629575.1629587)Cited by:[§I](https://arxiv.org/html/2608.28922#S1.p2.1),[§II](https://arxiv.org/html/2608.28922#S2.p2.1)\.
- \[2\]M\. Du, F\. Li, G\. Zheng, and V\. Srikumar\(2017\)DeepLog: anomaly detection and diagnosis from system logs through deep learning\.InProceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security,pp\. 1285–1298\.External Links:[Document](https://dx.doi.org/10.1145/3133956.3134015)Cited by:[§I](https://arxiv.org/html/2608.28922#S1.p2.1),[§II](https://arxiv.org/html/2608.28922#S2.p2.1)\.
- \[3\]H\. Guo, S\. Yuan, and X\. Wu\(2021\)LogBERT: log anomaly detection via BERT\.In2021 International Joint Conference on Neural Networks,pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/IJCNN52387.2021.9534113)Cited by:[§I](https://arxiv.org/html/2608.28922#S1.p2.1),[§II](https://arxiv.org/html/2608.28922#S2.p2.1)\.
- \[4\]G\. Boracchi, D\. Carrera, C\. Cervellera, and D\. Macciò\(2018\)QuantTree: histograms for change detection in multivariate data streams\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 639–648\.External Links:[Link](https://proceedings.mlr.press/v80/boracchi18a.html)Cited by:[§I](https://arxiv.org/html/2608.28922#S1.p3.1),[§II](https://arxiv.org/html/2608.28922#S2.p1.1)\.
- \[5\]S\. Bhatia, A\. Jain, P\. Li, R\. Kumar, and B\. Hooi\(2021\)MStream: fast anomaly detection in multi\-aspect streams\.InProceedings of The Web Conference 2021,pp\. 3371–3382\.External Links:[Document](https://dx.doi.org/10.1145/3442381.3450023)Cited by:[§I](https://arxiv.org/html/2608.28922#S1.p3.1),[§II](https://arxiv.org/html/2608.28922#S2.p2.1)\.
- \[6\]K\. Pearson\(1900\)On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling\.The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science50\(302\),pp\. 157–175\.External Links:[Document](https://dx.doi.org/10.1080/14786440009463897)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p1.1)\.
- \[7\]A\. Lakhina, M\. Crovella, and C\. Diot\(2005\)Mining anomalies using traffic feature distributions\.InProceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications,pp\. 217–228\.External Links:[Document](https://dx.doi.org/10.1145/1080091.1080118)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p1.1)\.
- \[8\]S\. Bhatia, B\. Hooi, M\. Yoon, K\. Shin, and C\. Faloutsos\(2020\)MIDAS: microcluster\-based detector of anomalies in edge streams\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 3242–3249\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i04.5724)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p2.1)\.
- \[9\]S\. M\. Lundberg and S\. Lee\(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 4765–4774\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p3.1)\.
- \[10\]X\. Zhang, M\. Marwah, I\. Lee, M\. Arlitt, and D\. Goldwasser\(2019\)ACE – an anomaly contribution explainer for cyber\-security applications\.In2019 IEEE International Conference on Big Data \(Big Data\),pp\. 1991–2000\.External Links:[Document](https://dx.doi.org/10.1109/BigData47090.2019.9005989)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p3.1)\.
- \[11\]D\. Han, Z\. Wang, W\. Chen, Y\. Zhong, S\. Wang, H\. Zhang, J\. Yang, X\. Shi, and X\. Yin\(2021\)DeepAID: interpreting and improving deep learning\-based anomaly detection in security applications\.InProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security,pp\. 3197–3217\.External Links:[Document](https://dx.doi.org/10.1145/3460120.3484589)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p3.1)\.
- \[12\]P\. Bailis, E\. Gan, S\. Madden, D\. Narayanan, K\. Rong, and S\. Suri\(2017\)MacroBase: prioritizing attention in fast data\.InProceedings of the 2017 ACM International Conference on Management of Data,pp\. 541–556\.External Links:[Document](https://dx.doi.org/10.1145/3035918.3035928)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p3.1)\.
- \[13\]B\. L\. Monroe, M\. P\. Colaresi, and K\. M\. Quinn\(2008\)Fightin’ words: lexical feature selection and evaluation for identifying the content of political conflict\.Political Analysis16\(4\),pp\. 372–403\.External Links:[Document](https://dx.doi.org/10.1093/pan/mpn018)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p3.1)\.
- \[14\]J\. M\. Hellerstein, C\. Ré, F\. Schoppmann, D\. Z\. Wang, E\. Fratkin, A\. Gorajek, K\. S\. Ng, C\. Welton, X\. Feng, K\. Li, and A\. Kumar\(2012\)The MADlib analytics library: or MAD skills, the SQL\.Proceedings of the VLDB Endowment5\(12\),pp\. 1700–1711\.External Links:[Document](https://dx.doi.org/10.14778/2367502.2367510)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p4.1)\.
- \[15\]M\. Raasveldt and H\. Mühleisen\(2019\)DuckDB: an embeddable analytical database\.InProceedings of the 2019 International Conference on Management of Data,pp\. 1981–1984\.External Links:[Document](https://dx.doi.org/10.1145/3299869.3320212)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p4.1)\.
- \[16\]V\. Vovk, A\. Gammerman, and G\. Shafer\(2005\)Algorithmic learning in a random world\.Springer\-Verlag\.External Links:[Document](https://dx.doi.org/10.1007/b106715)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p5.1),[§III\-C](https://arxiv.org/html/2608.28922#S3.SS3.p2.1)\.
- \[17\]A\. N\. Angelopoulos and S\. Bates\(2023\)Conformal prediction: a gentle introduction\.Foundations and Trends in Machine Learning16\(4\),pp\. 494–591\.External Links:[Document](https://dx.doi.org/10.1561/2200000101)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p5.1)\.
- \[18\]S\. Bates, E\. Candès, L\. Lei, Y\. Romano, and M\. Sesia\(2023\)Testing for outliers with conformal p\-values\.The Annals of Statistics51\(1\),pp\. 149–178\.External Links:[Document](https://dx.doi.org/10.1214/22-AOS2244)Cited by:[§II](https://arxiv.org/html/2608.28922#S2.p5.1),[§III\-C](https://arxiv.org/html/2608.28922#S3.SS3.p2.1)\.
- \[19\]J\. Zhu, S\. He, P\. He, J\. Liu, and M\. R\. Lyu\(2023\)Loghub: a large collection of system log datasets for AI\-driven log analytics\.In2023 IEEE 34th International Symposium on Software Reliability Engineering,pp\. 355–366\.External Links:[Document](https://dx.doi.org/10.1109/ISSRE59848.2023.00071)Cited by:[§V\-A](https://arxiv.org/html/2608.28922#S5.SS1.p1.1)\.
- \[20\]P\. He, J\. Zhu, Z\. Zheng, and M\. R\. Lyu\(2017\)Drain: an online log parsing approach with fixed depth tree\.In2017 IEEE International Conference on Web Services,pp\. 33–40\.External Links:[Document](https://dx.doi.org/10.1109/ICWS.2017.13)Cited by:[§V\-A](https://arxiv.org/html/2608.28922#S5.SS1.p2.1)\.
- \[21\]S\. García, M\. Grill, J\. Stiborek, and A\. Zunino\(2014\)An empirical comparison of botnet detection methods\.Computers & Security45,pp\. 100–123\.External Links:[Document](https://dx.doi.org/10.1016/j.cose.2014.05.011)Cited by:[§VI](https://arxiv.org/html/2608.28922#S6.p4.1)\.