Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks

# Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks
Source: [https://arxiv.org/html/2606.06861](https://arxiv.org/html/2606.06861)
11institutetext:Department of Mathematics, Informatics and Technology, University of Applied Sciences Koblenz, Joseph\-Rovan\-Allee 2, 53424 Remagen, Germany
11email:\{jaekel, dellen\}@hs\-koblenz\.de22institutetext:Technical University of Munich, Munich, Germany
22email:ziyuan\.li@tum\.de###### Abstract

Understanding nonlinear feature interactions is crucial in science and engineering, yet standard multilayer perceptrons \(MLPs\) often capture such interactions only implicitly, leading to entangled representations that can impair robustness and interpretability\. We investigate product\-unit residual networks \(PURe\) that integrate multiplicative product units with residual connections to explicitly model cross\-feature couplings while stabilizing optimization\. We conduct a systematic evaluation on an interaction\-driven synthetic benchmark and two real\-world datasets, assessing predictive accuracy, robustness to Gaussian feature noise, and performance under limited training data, and we compare real\- and complex\-valued variants under a matched parameter budget\. Beyond accuracy, SHapley Additive exPlanations \(SHAP\)\-based interaction analyses show that PURe learns more concentrated and structurally coherent interaction patterns than MLP baselines\. Overall, PURe achieves competitive or improved performance, better robustness and sample efficiency in low\-data regimes, and enhanced interaction\-level interpretability\.

## 1Introduction

Modeling nonlinear relationships between input variables is a fundamental challenge occurring in scientific and engineering regression tasks, including physical system modeling, materials science, and socio\-economic prediction problems\. Often the target variable is not determined solely by individual features, but by nonlinear interactions among multiple factors\. Accurately capturing these nonlinear feature interactions is therefore essential for achieving reliable predictions and meaningful scientific insights\.

Multilayer perceptron networks \(MLP\) have become a standard tool for nonlinear regression due to their universal approximation capability and flexible function representation\[[3](https://arxiv.org/html/2606.06861#bib.bib5),[18](https://arxiv.org/html/2606.06861#bib.bib22)\]\. However, their underlying additive composition often leads to implicit and entangled representations of feature interactions, which can limit generalization performance, particularly in data\-scarce or noisy regimes\[[17](https://arxiv.org/html/2606.06861#bib.bib6),[22](https://arxiv.org/html/2606.06861#bib.bib7)\]\. Moreover, internal representations learned by conventional neural networks are typically difficult to interpret, making it challenging to assess whether meaningful interaction patterns have been captured or whether the model has learned spurious correlations\[[15](https://arxiv.org/html/2606.06861#bib.bib8),[14](https://arxiv.org/html/2606.06861#bib.bib9)\]\. Although recent studies have shown that neural networks can implicitly encode statistical feature interactions, such interactions are often entangled and difficult to disentangle without explicit structural constraints\[[19](https://arxiv.org/html/2606.06861#bib.bib15)\]\.

To address these limitations, multiplicative neural architectures based on product units have been proposed as an alternative modeling paradigm\[[12](https://arxiv.org/html/2606.06861#bib.bib11),[13](https://arxiv.org/html/2606.06861#bib.bib10),[4](https://arxiv.org/html/2606.06861#bib.bib1),[11](https://arxiv.org/html/2606.06861#bib.bib3),[5](https://arxiv.org/html/2606.06861#bib.bib2),[10](https://arxiv.org/html/2606.06861#bib.bib4),[6](https://arxiv.org/html/2606.06861#bib.bib16)\]\. By explicitly encoding multiplicative relationships, product\-unit networks provide a natural inductive bias for modeling nonlinear feature interactions that commonly arise in scientific data\. Despite their theoretical expressiveness, product\-unit models have to date received limited attention, presumably due to optimization difficulties, sensitivity to initialization, and training instability\[[10](https://arxiv.org/html/2606.06861#bib.bib4),[5](https://arxiv.org/html/2606.06861#bib.bib2)\], which are expected to be especially problematic in deeper architectures\.

Residual connections have been shown to substantially improve the optimization and stability of deep neural networks by facilitating gradient flow and reducing vanishing gradient effects\[[9](https://arxiv.org/html/2606.06861#bib.bib12)\]\. Residual architectures have been successfully applied to a wide range of learning problems, including regression tasks with complex nonlinear structures\[[20](https://arxiv.org/html/2606.06861#bib.bib13),[1](https://arxiv.org/html/2606.06861#bib.bib14)\]\. Recently, the use of residual connections in conjunction with multiplicative product\-unit networks has been proposed\[[13](https://arxiv.org/html/2606.06861#bib.bib10),[12](https://arxiv.org/html/2606.06861#bib.bib11)\]\. Although the results have been encouraging, it is unclear how residual connections influence predictive performance, generalization behavior, and interpretability when targeting explicit interaction modeling\.

In this work, we study product\-unit residual networks \(PURe\), a class of neural architectures that integrate multiplicative product units with residual connections to explicitly model nonlinear feature interactions while maintaining stable optimization, and conduct a systematic empirical evaluation on both synthetic and real\-world regression datasets, analyzing predictive accuracy, robustness to noise, and performance under limited training data\. Furthermore, an interaction\-based analysis is employed to assess whether the proposed architecture captures meaningful feature dependencies with improved structural clarity compared to standard MLP baselines\. Our results demonstrate that PURe achieves competitive performance, enhanced generalization, and improved interpretability, making it a promising modeling framework for computational\-science applications governed by complex nonlinear relationships\.

## 2Related Work

Product\-unit neural networks were introduced to explicitly encode multiplicative interactions between input features\[[6](https://arxiv.org/html/2606.06861#bib.bib16)\]\. Given an input vector𝐱=\(x1,…,xd\)\\mathbf\{x\}=\(x\_\{1\},\\dots,x\_\{d\}\), a classical product unit computes

y=∏i=1dxiwi,y=\\prod\_\{i=1\}^\{d\}x\_\{i\}^\{w\_\{i\}\},\(1\)wherewiw\_\{i\}denotes a learnable exponent associated with theii\-th input\. This formulation enables compact representations of high\-order feature interactions that may otherwise require substantially deeper additive networks\.

For numerical stability, product units are commonly implemented using a logarithmic reparameterization\[[5](https://arxiv.org/html/2606.06861#bib.bib2)\]:

y=exp⁡\(∑i=1dwi​log⁡xi\)\.y=\\exp\\left\(\\sum\_\{i=1\}^\{d\}w\_\{i\}\\log x\_\{i\}\\right\)\.\(2\)This log\-linear\-exponential form highlights the multiplicative inductive bias of product units and distinguishes them structurally from additive neural representations\. Despite their expressive advantages, previous studies have reported optimization difficulties and sensitivity to initialization, which presumably have limited the use of product\-unit networks in deeper architectures so far\[[5](https://arxiv.org/html/2606.06861#bib.bib2)\]\.

Product\-unit formulations are inherently restricted to positive\-valued inputs due to the logarithmic transformation involved in their implementation\. To accommodate signed inputs and to enable joint modeling of magnitude and phase information, product\-unit representations have been extended to the complex domain\[[4](https://arxiv.org/html/2606.06861#bib.bib1),[12](https://arxiv.org/html/2606.06861#bib.bib11)\]\. For a complex\-valued input𝐳∈ℂd\\mathbf\{z\}\\in\\mathbb\{C\}^\{d\}, a complex product unit can be defined as

y=exp⁡\(∑i=1dwi​log⁡\(zi\)\),wi∈ℂ,y=\\exp\\left\(\\sum\_\{i=1\}^\{d\}w\_\{i\}\\log\(z\_\{i\}\)\\right\),\\quad w\_\{i\}\\in\\mathbb\{C\},\(3\)where the complex logarithm is given bylog⁡\(z\)=log⁡\|z\|\+i​arg⁡\(z\)\\log\(z\)=\\log\|z\|\+\\mathrm\{i\}\\arg\(z\)\. This formulation preserves the multiplicative interaction structure while enabling joint modeling of amplitude and phase, which is difficult to achieve using real\-valued additive networks\. Existing work on complex\-valued product units has primarily focused on representational aspects, with limited investigation into their integration with modern deep architectural components\.

## 3Method

### 3\.1Product\-Unit Residual Block

Let𝐡∈ℝd\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\}denote the input of a residual block, whereddis the hidden dimension\. Our real\-valued product\-unit residual block is implemented as a three\-stage transformation followed by an identity skip connection

𝐡out=𝐡\+FC2​\(PU​\(FC1​\(𝐡\)\)\)\.\\mathbf\{h\}\_\{\\text\{out\}\}=\\mathbf\{h\}\+\\mathrm\{FC\}\_\{2\}\\\!\\left\(\\mathrm\{PU\}\\\!\\left\(\\mathrm\{FC\}\_\{1\}\(\\mathbf\{h\}\)\\right\)\\right\)\.\(4\)Here,FC1\\mathrm\{FC\}\_\{1\}andFC2\\mathrm\{FC\}\_\{2\}are affine mappings \(linear layers with bias\) fromℝd\\mathbb\{R\}^\{d\}toℝd\\mathbb\{R\}^\{d\}\.

Given an intermediate vector𝐚=FC1​\(𝐡\)∈ℝd\\mathbf\{a\}=\\mathrm\{FC\}\_\{1\}\(\\mathbf\{h\}\)\\in\\mathbb\{R\}^\{d\}, the product\-unit mapping is computed using a positivity\-enforced preprocessing with

𝐬=ReLU​\(𝐚\)\+𝟏,ReLU​\(⋅\)=max⁡\(⋅,0\),\\mathbf\{s\}=\\mathrm\{ReLU\}\(\\mathbf\{a\}\)\+\\mathbf\{1\},\\qquad\\mathrm\{ReLU\}\(\\cdot\)=\\max\(\\cdot,0\),\(5\)where𝟏∈ℝd\\mathbf\{1\}\\in\\mathbb\{R\}^\{d\}denotes the all\-ones vector, followed by a log\-linear\-exponential computation

PU​\(𝐚\)=exp⁡\(Wpu​log⁡\(𝐬\)\),\\mathrm\{PU\}\(\\mathbf\{a\}\)=\\exp\\\!\\left\(W\_\{\\mathrm\{pu\}\}\\log\(\\mathbf\{s\}\)\\right\),\(6\)whereWpu∈ℝd×dW\_\{\\mathrm\{pu\}\}\\in\\mathbb\{R\}^\{d\\times d\}is a learnable weight matrix \(without bias\), andlog⁡\(⋅\)\\log\(\\cdot\)andexp⁡\(⋅\)\\exp\(\\cdot\)are applied element\-wise\. We adopt𝐬=ReLU​\(𝐚\)\+𝟏\\mathbf\{s\}=\\mathrm\{ReLU\}\(\\mathbf\{a\}\)\+\\mathbf\{1\}to ensure𝐬\>𝟎\\mathbf\{s\}\>\\mathbf\{0\}, so thatlog⁡\(𝐬\)\\log\(\\mathbf\{s\}\)is well\-defined and zero\-valued factors are avoided\. Moreover, adding11provides a strictly positive multiplicative gate: whenak≤0a\_\{k\}\\leq 0,sk=1s\_\{k\}=1acts as a neutral element \(sincelog⁡1=0\\log 1=0\), while forak\>0a\_\{k\}\>0it produces a positive scaling factor, enabling the PU to modulate interactions without sign ambiguities\.

### 3\.2Overall Real\-Valued Network

Given an input𝐱∈ℝp\\mathbf\{x\}\\in\\mathbb\{R\}^\{p\}, the network first maps it to the hidden space and applies an element\-wise rectified linear unit:

𝐡\(0\)=ReLU​\(FCin​\(𝐱\)\)\.\\mathbf\{h\}^\{\(0\)\}=\\mathrm\{ReLU\}\\\!\\left\(\\mathrm\{FC\}\_\{\\text\{in\}\}\(\\mathbf\{x\}\)\\right\)\.\(7\)Then, two residual blocks are applied sequentially, namely

𝐡\(1\)=ℬ​\(𝐡\(0\)\),𝐡\(2\)=ℬ0​\(𝐡\(1\)\),\\mathbf\{h\}^\{\(1\)\}=\\mathcal\{B\}\\\!\\left\(\\mathbf\{h\}^\{\(0\)\}\\right\),\\qquad\\mathbf\{h\}^\{\(2\)\}=\\mathcal\{B\}\_\{0\}\\\!\\left\(\\mathbf\{h\}^\{\(1\)\}\\right\),\(8\)whereℬ\\mathcal\{B\}denotes the block in Eq\. \([4](https://arxiv.org/html/2606.06861#S3.E4)\), andℬ0\\mathcal\{B\}\_\{0\}denotes the same block with zero initialization for stability \(see implementation details\)\. Finally, the prediction is produced by a linear output layer with

y^=FCout​\(𝐡\(2\)\)\.\\hat\{y\}=\\mathrm\{FC\}\_\{\\text\{out\}\}\\\!\\left\(\\mathbf\{h\}^\{\(2\)\}\\right\)\.\(9\)

### 3\.3Complex\-Valued Product\-Unit Residual Block

To accommodate signed inputs and enable joint modeling of magnitude and phase information, we extend the block to the complex domain\. Let𝐳∈ℂd\\mathbf\{z\}\\in\\mathbb\{C\}^\{d\}be the input of a complex\-valued block\. The block structure mirrors the real\-valued case, i\.e\.,

𝐳out=𝐳\+FC2ℂ​\(PUℂ​\(FC1ℂ​\(𝐳\)\)\)\.\\mathbf\{z\}\_\{\\text\{out\}\}=\\mathbf\{z\}\+\\mathrm\{FC\}\_\{2\}^\{\\mathbb\{C\}\}\\\!\\left\(\\mathrm\{PU\}\_\{\\mathbb\{C\}\}\\\!\\left\(\\mathrm\{FC\}\_\{1\}^\{\\mathbb\{C\}\}\(\\mathbf\{z\}\)\\right\)\\right\)\.\(10\)
Given𝐚=FC1ℂ​\(𝐳\)\\mathbf\{a\}=\\mathrm\{FC\}\_\{1\}^\{\\mathbb\{C\}\}\(\\mathbf\{z\}\), we apply a numerically\-stable preprocessing by separately rectifying the real\- and imaginary parts and adding a constant offset, to obtain

𝐚~=\(ReLU​\(ℜ⁡\(𝐚\)\)\+c\)\+i​\(ReLU​\(ℑ⁡\(𝐚\)\)\+c\),c=0\.5,\\tilde\{\\mathbf\{a\}\}=\\big\(\\mathrm\{ReLU\}\(\\Re\(\\mathbf\{a\}\)\)\+c\\big\)\+\\mathrm\{i\}\\big\(\\mathrm\{ReLU\}\(\\Im\(\\mathbf\{a\}\)\)\+c\\big\),\\qquad c=\\sqrt\{0\.5\},\(11\)and compute

PUℂ​\(𝐚\)=exp⁡\(Wpuℂ​log⁡\(𝐚~\)\),\\mathrm\{PU\}\_\{\\mathbb\{C\}\}\(\\mathbf\{a\}\)=\\exp\\\!\\left\(W\_\{\\mathrm\{pu\}\}^\{\\mathbb\{C\}\}\\log\(\\tilde\{\\mathbf\{a\}\}\)\\right\),\(12\)whereWpuℂ∈ℂd×dW\_\{\\mathrm\{pu\}\}^\{\\mathbb\{C\}\}\\in\\mathbb\{C\}^\{d\\times d\}is a complex\-valued weight matrix \(without bias\), andlog⁡\(⋅\)\\log\(\\cdot\)andexp⁡\(⋅\)\\exp\(\\cdot\)denote the complex logarithm and exponential applied element\-wise\.

## 4Experiments

### 4\.1Experimental Setup

We evaluate the proposed PURe on three regression benchmarks of increasing complexity, covering synthetic interaction\-driven data, small\-scale real\-world engineering data, and large\-scale socio\-economic data\.

*Friedman 1*is a synthetic benchmark designed to explicitly assess a model’s ability to capture nonlinear feature interactions\[[7](https://arxiv.org/html/2606.06861#bib.bib17)\]\. It consists of1010independent input features, where the target variable is generated by known nonlinear interactions among a subset of features,

y=10​sin⁡\(π​x0​x1\)\+20​\(x2−0\.5\)2\+10​x3\+5​x4\+ε,y=10\\sin\(\\pi x\_\{0\}x\_\{1\}\)\+20\(x\_\{2\}\-0\.5\)^\{2\}\+10x\_\{3\}\+5x\_\{4\}\+\\varepsilon,\(13\)while the remaining features act as noise\. The dataset contains10,00010\{,\}000samples and is split into70%70\\%training,15%15\\%validation, and15%15\\%test sets\.

*Concrete Compressive Strength*is a real\-world engineering dataset involving nonlinear relationships between material composition and compressive strength\[[21](https://arxiv.org/html/2606.06861#bib.bib18)\]\. It includes88physical and chemical features and1,0301\{,\}030samples, split into60%60\\%training,20%20\\%validation, and20%20\\%test sets\.

*California Housing*is a large\-scale real\-world regression benchmark based on U\.S\. census data, featuring heterogeneous socio\-economic and geographic factors\[[16](https://arxiv.org/html/2606.06861#bib.bib19)\]\. It contains88input features and20,64020\{,\}640samples, split into80%80\\%training,10%10\\%validation, and10%10\\%test sets\.

We evaluated real\-valued and complex\-valued variants of both MLP and PURe under identical experimental settings\. For complex\-valued models, all layers operate in the complex domain, and the final prediction is obtained by applying a magnitude operation to the complex\-valued output in order to produce a real\-valued regression target\. This is appropriate here because all three benchmarks have non\-negative targets\. To ensure a fair comparison in terms of model capacity, we account for the fact that each complex\-valued parameter corresponds to two real\-valued degrees of freedom\. Accordingly, real\-valued networks use a hidden dimension of128128, while complex\-valued networks use a hidden dimension of6464, resulting in comparable total numbers of trainable parameters across real and complex architectures\.

All models are trained using the mean squared error loss, i\.e\.,

ℒ=1N​∑i=1N\(yi−y^i\)2,\\mathcal\{L\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(y\_\{i\}\-\\hat\{y\}\_\{i\}\\right\)^\{2\},\(14\)whereyiy\_\{i\}andy^i\\hat\{y\}\_\{i\}denote the ground\-truth target and model prediction, respectively\[[2](https://arxiv.org/html/2606.06861#bib.bib20)\]\.

The models are trained for100100epochs using the Adam optimizer with a learning rate of10−310^\{\-3\}and a batch size of128128\. Unless stated otherwise, all layers are initialized using Kaiming initialization\[[8](https://arxiv.org/html/2606.06861#bib.bib21)\]\. For complex\-valued layers, we apply Kaiming initialization independently to the real and imaginary parts\. For the PURe, the second residual block is initialized to zero to improve early\-stage training stability, while the remaining layers follow standard initialization\.

Each experimental configuration is repeated over five independent runs with different random seeds\. Results are reported as the mean and sample standard deviation across runs\. For evaluation on the test set, the model checkpoint achieving the lowest validation loss is selected\.

For interpretability analyses, including the SHapley Additive exPlanations \(SHAP\)\-based feature interaction studies, models are trained once using a fixed random seed\. These analyses are intended to qualitatively examine the structure and stability of learned feature interactions rather than to provide statistically aggregated performance estimates\.

To assess robustness, predictive performance is evaluated under both clean conditions and Gaussian noise corruption, where zero\-mean noise with standard deviationσ=0\.05\\sigma=0\.05is added to the input features at test time\.

### 4\.2Benchmark Performance and Robustness

We evaluate the proposed PURe with respect to predictive accuracy, robustness, and training dynamics\. Quantitative test set results are summarized in Table[1](https://arxiv.org/html/2606.06861#S4.T1), while training and validation loss trajectories are presented in Fig\.[1](https://arxiv.org/html/2606.06861#S4.F1)\.

Table 1:Test mean squared error \(MSE; mean±\\pmsample standard deviation over five runs\)\. We report results under clean inputs \(σ=0\.00\\sigma=0\.00\) and Gaussian input noise \(σ=0\.05\\sigma=0\.05\)\. The best result for each dataset and noise level is highlighted in bold\.![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/loss_friedman.png)

![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/loss_concrete.png)

![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/loss_california.png)

Figure 1:Training and validation loss curves \(log10 MSE\) on the three benchmark datasets: Friedman 1 \(top\), Concrete Compressive Strength \(middle\), and California Housing \(bottom\)\.On the Friedman 1 dataset, CV\-PURe eventually achieves the lowest test MSE as shown in Table[1](https://arxiv.org/html/2606.06861#S4.T1)\. However, the loss trajectories in Fig\.[1](https://arxiv.org/html/2606.06861#S4.F1)\(top\) reveal that while the MLP baselines converge more rapidly in the initial 20 epochs, the CV\-PURe model \(red solid line\) exhibits a more sustained and stable descent, eventually surpassing all other models after approximately 50 epochs\. This suggests that while the explicit multiplicative structure may require more iterations to resolve initially, it ultimately yields a more precise fit to the underlying interaction\-driven data\.

On the Concrete dataset, although CV\-PURe achieves the best average test performance \(Table[1](https://arxiv.org/html/2606.06861#S4.T1)\), the corresponding loss curves in Fig\.[1](https://arxiv.org/html/2606.06861#S4.F1)\(middle\) show that it plateaus at a slightly higher validation MSE compared to the RV\-based models during this specific run\. Notably, however, the CV\-PURe trajectory is significantly smoother, avoiding the erratic oscillations seen in the CV\-MLP and RV\-MLP curves\. This stability highlights the regularization benefit of the CV\-PURe architecture, even in regimes where the performance gains are marginal\.

On the California Housing dataset, the validation trajectories largely overlap, confirming that all models achieve comparable accuracy\. Nevertheless, the CV\-MLP model \(green solid line\) exhibits a prominent instability spike near the 90th epoch, whereas the PURe variants remain remarkably stable\. This reinforces the conclusion that explicit multiplicative modeling, particularly when combined with complex\-valued weights, enhances the robustness and training stability of the network, especially when dealing with the high\-variance geographic dependencies inherent in this dataset\.

Across the benchmark datasets, the product\-unit models, particularly CV\-PURe, demonstrate superior robustness to Gaussian input noise\. As detailed in Table[1](https://arxiv.org/html/2606.06861#S4.T1), CV\-PURe consistently maintains the lowest mean test MSE under both clean and noisy \(σ=0\.05\\sigma=0\.05\) conditions, showing significantly less performance degradation than the MLP baselines\. While the more expressive complex\-valued parameterization in CV\-PURe introduces slightly higher training variability \(standard deviation\) in certain cases like Friedman 1 and Concrete, it consistently yields the highest predictive accuracy\. Ultimately, CV\-PURe offers a superior trade\-off between expressiveness and noise resilience, effectively balancing model stability with the ability to capture complex nonlinear dependencies\.

### 4\.3Interpretability and Feature Interaction Analysis

To evaluate the interpretability and structural alignment of the models, we analyze pairwise feature interaction patterns using SHAP\-based, ranking\-derived interaction maps\. Figure[2](https://arxiv.org/html/2606.06861#S4.F2)shows the normalized interaction maps for RV\-MLP, RV\-PURe, CV\-MLP, and CV\-PURe across the three benchmark datasets\. These visualizations serve as a diagnostic of whether explicitly incorporating multiplicative units leads to interaction structures that are more concentrated and less diffuse than those produced by purely additive architectures\.

![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/shap_compare_all_friedman_run0.png)

![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/shap_compare_all_concrete_run0.png)

![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/shap_compare_all_california_run0.png)

Figure 2:SHAP\-based feature interaction comparisons on Friedman 1 \(top\), Concrete Compressive Strength \(middle\), and California Housing \(bottom\)\. For each dataset, interaction maps are shown for RV\-MLP, RV\-PURe, CV\-MLP, and CV\-PURe\. Color intensity indicates normalized interaction strength\.Table 2:Quantifying the concentration of SHAP interaction maps \(Fig\.[2](https://arxiv.org/html/2606.06861#S4.F2)\)\. We report the normalized entropyHnormH\_\{\\mathrm\{norm\}\}\(lower indicates more concentrated interactions\) and the Herfindahl\-Hirschman Index \(HHI; higher indicates more concentrated interactions\), both computed on the Top\-KKoff\-diagonal interaction pairs \(hereK=10K\{=\}10\)\. Specifically, we normalize the Top\-KKinteraction strengths to a probability distribution\{pk\}k=1K\\\{p\_\{k\}\\\}\_\{k=1\}^\{K\}and computeHnorm=−∑kpk​log⁡pk/log⁡KH\_\{\\mathrm\{norm\}\}=\-\\sum\_\{k\}p\_\{k\}\\log p\_\{k\}/\\log KandHHI=∑kpk2\\mathrm\{HHI\}=\\sum\_\{k\}p\_\{k\}^\{2\}\. Best values per dataset are highlighted in bold \(min forHnormH\_\{\\mathrm\{norm\}\}, max for HHI\)\. Metrics are computed on the same fixed\-seed run as Fig\.[2](https://arxiv.org/html/2606.06861#S4.F2)and are intended as descriptive summaries rather than aggregated estimates\.On the Friedman\-1 dataset, where the target is generated from specific nonlinear interactions \(most notably betweenx0x\_\{0\}andx1x\_\{1\}\), the interaction maps reveal clear differences in concentration\. In Fig\.[2](https://arxiv.org/html/2606.06861#S4.F2)\(top\), the MLP baselines tend to distribute interaction strength across a broader set of feature pairs, beyond the dominant\(x0,x1\)\(x\_\{0\},x\_\{1\}\)coupling\. In contrast, product\-unit models appear to display a more concentrated interaction landscape, with CV\-PURe exhibiting the strongest concentration\. This is corroborated by Table[2](https://arxiv.org/html/2606.06861#S4.T2): CV\-PURe achieves the lowest normalized entropy \(HnormH\_\{\\text\{norm\}\}\) and the highest HHI on Friedman, indicating that the Top\-KKinteraction signal is dominated by a small number of pairs, consistent with the ground\-truth\(x0,x1\)\(x\_\{0\},x\_\{1\}\)interaction\.

A similar trend can be observed for the Concrete Compressive Strength dataset \(middle panel\)\. The interaction maps of the MLP baselines are comparatively dense, suggesting a more diffuse allocation of interaction strength across many pairs of mixture components\. The PURe variants, especially CV\-PURe, show more localized high\-intensity regions indicating a smaller subset of dominant material couplings \(e\.g\., involving Cement, Water, and Age\), while reducing diffuse background structure\. Quantitatively, CV\-PURe again achieves the most concise interaction profile, as shown in Table[2](https://arxiv.org/html/2606.06861#S4.T2)\.

For the California Housing dataset \(bottom panel\), interaction structure is inherently more heterogeneous, and all models highlight the key spatial dependency between Latitude and Longitude\. However, PURe models, and in particular CV\-PURe, produce interaction patterns that are more regular and less fragmented than those of the additive baselines\. This is supported by the result shown in Table[2](https://arxiv.org/html/2606.06861#S4.T2), where CV\-PURe yields the lowestHnormH\_\{\\mathrm\{norm\}\}and the highest HHI, indicating a more concise interaction map\. Overall, across all three benchmarks, explicit multiplicative modeling leads to more structured and concentrated SHAP interaction patterns, and the complex\-valued PURe variant shows the strongest effect\.

### 4\.4Sample Efficiency on Concrete

To evaluate sample efficiency, we train all models on progressively smaller subsets of the Concrete dataset while keeping the validation and test sets fixed\. Specifically, we used training fractions of \{5%, 10%, 20%, 40%, 60%, 80%\} and repeated each configuration over five random seeds\. Figure[3](https://arxiv.org/html/2606.06861#S4.F3)shows the test MSE as a function of the available training data, and Table[3](https://arxiv.org/html/2606.06861#S4.T3)reports the corresponding numerical results \(mean±\\pmsample standard deviation\)\.

Table 3:Sample efficiency on Concrete: test MSE \(mean±\\pmsample standard deviation over five runs\)\. Best results at each training fraction are highlighted in bold\.![Refer to caption](https://arxiv.org/html/2606.06861v1/fig/sample_efficiency_all_concrete.png)Figure 3:Sample efficiency comparison on the Concrete dataset\. Test MSE \(mean±\\pmsample standard deviation over five runs\) versus training data percentage\.Across the entire range of training data fractions, the product\-unit models \(RV\-PURe and CV\-PURe\) consistently outperform the additive MLP baselines\. This advantage is most prominent in the extremely low\-data regime; at the 5% training fraction, CV\-PURe achieves a test MSE of 0\.2976, which is significantly lower than the 0\.4197 reported for RV\-MLP\. The visual gap between the PURe architectures \(red and orange lines\) and the MLP models \(blue and green lines\) in Figure[3](https://arxiv.org/html/2606.06861#S4.F3)confirms that explicit multiplicative modeling provides a much stronger inductive bias when supervision is severely limited\.

A detailed inspection of Table[3](https://arxiv.org/html/2606.06861#S4.T3)reveals that CV\-PURe attains the best overall performance at every training fraction\. Furthermore, at the 5% level, CV\-PURe exhibits the smallest standard deviation \(±0\.0393\\pm 0\.0393\), indicating that the complex\-valued parameterization not only improves accuracy, but also training stability when data is scarce\. While the RV\-PURe model also performs well and occasionally shows lower variability at mid\-range fractions, CV\-PURe remains the most efficient architecture, maintaining the lowest mean error throughout the experiment\.

As the available training data increases from 5% to 80%, the performance gap between the architectures gradually narrows, as illustrated by the converging trajectories in Figure[3](https://arxiv.org/html/2606.06861#S4.F3)\. However, even at the 80% fraction, the product\-unit models maintain a measurable lead, with CV\-PURe reaching a final test MSE of 0\.1232 compared to 0\.1355 for CV\-MLP\. This indicates that while additive networks can eventually approximate complex nonlinear relationships with sufficient data, PURe models extract more information from fewer samples, demonstrating superior sample efficiency for modeling complex material dependencies\.

### 4\.5Ablation Study: Disentangling PU and Residual Roles

To isolate the individual contributions of multiplicative product units and residual connections, we performed an ablation study on the Concrete dataset\. We compare four variants: the standard MLP baseline; a Res\-MLP, which incorporates residual connections into the MLP architecture in the same manner as the proposed PURe; a standalone product\-unit network \(PU\) that removes all residual connections and adds activation functions between adjacent fully connected layers; and the PURe\. Table[4](https://arxiv.org/html/2606.06861#S4.T4)reports the MSE for these configurations under both clean \(σ=0\.00\\sigma=0\.00\) and noisy \(σ=0\.05\\sigma=0\.05\) conditions\.

Table 4:Ablation study on the Concrete dataset\. Test MSE \(mean±\\pmsample standard deviation over five runs\)\.The results indicate that the explicit multiplicative bias provided by the product units is the primary driver of performance gains\. As shown in Table[4](https://arxiv.org/html/2606.06861#S4.T4), the PU variant without residual connections outperforms the baseline MLP, reducing the test MSE from 0\.1118 to 0\.1100 under clean inputs\. In contrast, the Res\-MLP variant actually results in a performance degradation compared to the baseline, with the error increasing to 0\.1209\. This suggests that for this specific regression task, adding residual connections to a standard additive architecture without the appropriate functional bias does not necessarily improve generalization\.

The combination of both components in PURe yields the best overall results, achieving the lowest test MSE of 0\.1095\. Furthermore, the introduction of residual connections to the product\-unit architecture significantly enhances training stability, as evidenced by the substantial reduction in standard deviation—from 0\.0040 in the PU model to 0\.0015 in PURe\. This pattern holds true under noisy conditions \(σ=0\.05\\sigma=0\.05\), where PURe maintains the highest accuracy and the lowest variability\. These findings demonstrate that while multiplicative units provide the necessary inductive bias to capture complex feature interactions, the residual framework is essential for stabilizing the learning process and achieving optimal predictive performance\.

## 5Analysis and Discussion

### 5\.1Interaction\-aligned inductive bias

The central empirical finding is that product\-unit models are beneficial when the target is governed by nonlinear feature interactions\. This confirms our intuition: product units directly parameterize multiplicative couplings, so the model does not need to synthesize interaction structure indirectly through many additive compositions\. As a result, the learned dependency structure tends to be focused on a small set of informative cross\-feature relations, rather than being spread across many weak and entangled interactions\. This observation is quantitatively supported by a lower normalized interaction entropy and higher concentration \(HHI\) calculated from the Top\-KK\(hereK=10K\{=\}10\) SHAP\-based, ranking\-derived interaction maps \(Table[2](https://arxiv.org/html/2606.06861#S4.T2)\)\.

### 5\.2Robustness under feature noise

With Gaussian perturbations on the inputs, product\-unit variants typically exhibit a smaller performance drop\. A plausible explanation is that, once the model relies on a few stable, high\-signal interactions, predictions become less sensitive to small, distributed perturbations across many weak interaction channels\. In contrast, diffuse interaction patterns can accumulate noise effects, leading to larger error increases\.

### 5\.3Complex\-valued modeling: expressiveness vs\. variability

Complex\-valued product units often improve mean performance under a matched parameter budget, potentially because magnitude and phase offer a richer parameterization for representing nonlinear dependencies\. The trade\-off is a slightly increased run\-to\-run variability on some datasets, which is expected given the larger effective hypothesis space and higher sensitivity to initialization\. Importantly, this variability manifests itself mainly around a stronger mean, rather than indicating systematic instability\.

### 5\.4Why residual connections matter here

Residual pathways are particularly helpful for multiplicative blocks, where activations can be amplified or attenuated sharply and gradients may become less predictable\. The skip connection provides a stable identity route that improves optimization reliability and reduces sensitivity to early training dynamics\. This also explains why residualization is not guaranteed to help a plain MLP in shallow tabular settings, but becomes beneficial when paired with multiplicative transformations\.

### 5\.5Interpretability and sample efficiency

Interaction analyses indicate that product\-unit models yield more concentrated and structurally coherent pairwise dependency patterns, emphasizing a small set of dominant interactions while reducing diffuse low\-strength couplings\. This aligns with the sample\-efficiency behavior: in low\-data regimes, an interaction\-aligned inductive bias effectively regularizes the hypothesis space, improving generalization when supervision is limited\.

## 6Conclusion

We introduced PURe, a product\-unit residual architecture for tabular regression, and investigated both real\- and complex\-valued parameterizations under matched capacity\. Across interaction\-driven synthetic data and real\-world benchmarks, product\-unit models generally achieved lower prediction error than MLP baselines, with the strongest gains observed on the Friedman 1 and California Housing datasets\. In particular, the complex\-valued PURe variant reduced test MSE by up to about 35% on the synthetic benchmark and by roughly 12–15% under some noisy settings, while improvements on the Concrete dataset were more modest\. In the low\-data regime, the advantage of explicit multiplicative modeling became more pronounced, yielding about 10–30% lower error depending on the training fraction\. The learned interaction patterns also appeared more concentrated and structurally coherent in the SHAP\-based analyses, suggesting that explicit multiplicative modeling provides a useful inductive bias for capturing nonlinear dependencies\. Complex\-valued PURe further improved mean performance in most settings, although it occasionally exhibited slightly higher variability across runs\. Future work will focus on stronger quantitative validation of interaction structure, improved numerical stabilization for complex product units, and broader robustness evaluation under distribution shift and missing\-feature settings\.

## References

- \[1\]Z\. Allen\-Zhu, Y\. Li, and Z\. Song\(2019\)On the convergence rate of training recurrent neural networks\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p4.1)\.
- \[2\]C\. M\. Bishop and N\. M\. Nasrabadi\(2006\)Pattern recognition and machine learning\.Springer\.Cited by:[§4\.1](https://arxiv.org/html/2606.06861#S4.SS1.p6.2)\.
- \[3\]G\. Cybenko\(1989\)Approximation by superpositions of a sigmoidal function\.Mathematics of control, signals and systems2\(4\),pp\. 303–314\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[4\]B\. Dellen, U\. Jaekel, P\. S\. Freitas, and J\. W\. Clark\(2024\)Predicting nuclear masses with product\-unit networks\.Physics Letters B852,pp\. 138608\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1),[§2](https://arxiv.org/html/2606.06861#S2.p3.1)\.
- \[5\]B\. Dellen, U\. Jaekel, and M\. Wolnitza\(2019\)Function and pattern extrapolation with product\-unit networks\.InComputational Science–ICCS 2019: 19th International Conference, Faro, Portugal, June 12–14, 2019, Proceedings, Part II 19,pp\. 174–188\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1),[§2](https://arxiv.org/html/2606.06861#S2.p2.1),[§2](https://arxiv.org/html/2606.06861#S2.p2.2)\.
- \[6\]R\. Durbin and D\. E\. Rumelhart\(1989\)Product units: a computationally powerful and biologically plausible extension to backpropagation networks\.Neural computation1\(1\),pp\. 133–142\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1),[§2](https://arxiv.org/html/2606.06861#S2.p1.1)\.
- \[7\]J\. H\. Friedman\(1991\)Multivariate adaptive regression splines\.The annals of statistics19\(1\),pp\. 1–67\.Cited by:[§4\.1](https://arxiv.org/html/2606.06861#S4.SS1.p2.1)\.
- \[8\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2015\)Delving deep into rectifiers: surpassing human\-level performance on imagenet classification\.InProceedings of the IEEE international conference on computer vision,pp\. 1026–1034\.Cited by:[§4\.1](https://arxiv.org/html/2606.06861#S4.SS1.p7.3)\.
- \[9\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p4.1)\.
- \[10\]L\. R\. Leerink, C\. L\. Giles, B\. G\. Horne, and M\. A\. Jabri\(1995\)Learning with product units\.Advances in Neural Information Processing Systems7,pp\. 537\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1)\.
- \[11\]Z\. Li, U\. Jaekel, and B\. Dellen\(2024\)Data\-driven 3d shape completion with product units\.InInternational Conference on Computational Science,pp\. 302–315\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1)\.
- \[12\]Z\. Li, U\. Jaekel, and B\. Dellen\(2025\)Advancing complex\-valued neural networks with product units for mri reconstruction\.InInternational Conference on Neural Information Processing,pp\. 540–554\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1),[§1](https://arxiv.org/html/2606.06861#S1.p4.1),[§2](https://arxiv.org/html/2606.06861#S2.p3.1)\.
- \[13\]Z\. Li, U\. Jaekel, and B\. Dellen\(2025\)Deep residual learning with product units\.arXiv preprint arXiv:2505\.04397\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p3.1),[§1](https://arxiv.org/html/2606.06861#S1.p4.1)\.
- \[14\]S\. M\. Lundberg and S\. Lee\(2017\)A unified approach to interpreting model predictions\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[15\]C\. Molnar, G\. Casalicchio, and B\. Bischl\(2020\)Interpretable machine learning–a brief history, state\-of\-the\-art and challenges\.InJoint European conference on machine learning and knowledge discovery in databases,pp\. 417–431\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[16\]R\. K\. Pace and R\. Barry\(1997\)Sparse spatial autoregressions\.Statistics & Probability Letters33\(3\),pp\. 291–297\.Cited by:[§4\.1](https://arxiv.org/html/2606.06861#S4.SS1.p4.5)\.
- \[17\]T\. Poggio, K\. Kawaguchi, Q\. Liao, B\. Miranda, L\. Rosasco, X\. Boix, J\. Hidary, and H\. Mhaskar\(2017\)Theory of deep learning iii: explaining the non\-overfitting puzzle\.arXiv preprint arXiv:1801\.00173\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[18\]D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams\(1986\)Learning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[19\]M\. Tsang, D\. Cheng, and Y\. Liu\(2017\)Detecting statistical interactions from neural network weights\.arXiv preprint arXiv:1705\.04977\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.
- \[20\]A\. Veit, M\. J\. Wilber, and S\. Belongie\(2016\)Residual networks behave like ensembles of relatively shallow networks\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p4.1)\.
- \[21\]I\. Yeh\(1998\)Modeling of strength of high\-performance concrete using artificial neural networks\.Cement and Concrete research28\(12\),pp\. 1797–1808\.Cited by:[§4\.1](https://arxiv.org/html/2606.06861#S4.SS1.p3.5)\.
- \[22\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\(2017\)Understanding deep learning requires rethinking generalization \(2016\)\.arXiv preprint arXiv:1611\.03530\.Cited by:[§1](https://arxiv.org/html/2606.06861#S1.p2.1)\.