Joint Utilization of Geospatial and census proxies for Autoencoder-Assisted Downscaling (JUGAAD) of socioeconomic indicators in India
Summary
Introduces JUGAAD, a deep learning framework that uses autoencoders and census/geospatial data to downscale socioeconomic indicators in India from coarse survey resolution to fine-grained village-cluster scale, validated against district-level NSSO data.
arXiv:2607.20559v1 Announce Type: new
Abstract: Monitoring poverty and food security indicators is imperative for addressing socioeconomic challenges in developing nations. A limitation is mismatches in scale between data sources: census data provide geographic coverage, while socioeconomic indicators are derived from infrequently conducted surveys at coarse resolutions, posing a methodological challenge. This study introduces a deep learning framework, JuGAAD, using Indian census and survey data from 2001 and 2011 as a case study. We employ a three-step process: census and geospatial data are averaged into intermediate village-cluster-scale tessellations to reduce noise and regularize administrative boundary changes; an autoencoder compresses high-dimensional National Sample Survey Office (NSSO) data into a low-dimensional latent representation; and a regression model maps upscaled census and geospatial data to this representation. This function is applied to fine-grained census data to generate high-resolution predictions, validated against ground-truth district-level NSSO indicators. Results confirm the methodology predicts socioeconomic indicators at fine scales with strong accuracy.
View Cached Full Text
Cached at: 07/24/26, 05:12 AM
# Joint Utilization of Geospatial and census proxies for Autoencoder-Assisted Downscaling (JUGAAD) of socioeconomic indicators in India
Source: [https://arxiv.org/html/2607.20559](https://arxiv.org/html/2607.20559)
\[1\]\\fnmAditya\\surDutt
\[1\]\\orgdivDepartment of Electrical and Computer Engineering,\\orgnameUniversity of Florida,\\orgaddress\\cityGainesville,\\postcode32603,\\stateFlorida,\\countryUSA
2\]\\orgdivDepartment of Engineering School of Sustainable Infrastructure & Environment,\\orgnameUniversity of Florida,\\orgaddress\\cityGainesville,\\postcode32603,\\stateFlorida,\\countryUSA
3\]\\orgdivCaesar Kleberg Wildlife Research Institute,\\orgnameTexas A&M University,\\orgaddress\\cityKingsville,\\postcode78363,\\stateTexas,\\countryUSA
###### Abstract
Monitoring poverty and food security indicators is imperative for addressing socioeconomic challenges in developing nations\. A limitation is mismatches in scale between data sources: census data provide geographic coverage, while socioeconomic indicators are derived from infrequently conducted surveys at coarse resolutions, posing a methodological challenge\. This study introduces a deep learning framework, JuGAAD, using Indian census and survey data from 2001 and 2011 as a case study\. We employ a three\-step process: census and geospatial data are averaged into intermediate village\-cluster\-scale tessellations to reduce noise and regularize administrative boundary changes; an autoencoder compresses high\-dimensional National Sample Survey Office \(NSSO\) data into a low\-dimensional latent representation; and a regression model maps upscaled census and geospatial data to this representation\. This function is applied to fine\-grained census data to generate high\-resolution predictions, validated against ground\-truth district\-level NSSO indicators\. Results confirm the methodology predicts socioeconomic indicators at fine scales with strong accuracy\.
###### keywords:
Autoencoders, spatial downscaling, socioeconomic indicators, poverty mapping, food security, census data, machine learning, regression modeling
## 1Introduction
Detailed, unit\-level demographic and socioeconomic data are the bedrock of effective, evidence\-based policymaking\. They allow governments to accurately assess population structures, monitor urbanization, and track migration patterns, all essential steps for designing targeted interventions in sectors like healthcare, education, and infrastructure\. In developing economies, these data are crucial for evaluating economic stability and pinpointing disparities in income, employment, and resource access, which in turn inform poverty reduction strategies and resource allocation\. Furthermore, they are vital for measuring progress toward international targets, such as the Sustainable Development Goals \(SDGs\), nearly half of which rely on such indicators\.
Despite their fundamental importance, high\-resolution socioeconomic data are frequently unavailable, costly to acquire, or confined to infrequent household surveys\. This results in a critical gap between data granularity and policy requirements\. For instance, in India, the National Sample Survey Office \(NSSO\) provides socioeconomic metrics at the district level, a resolution insufficient for localized policy action\.
Conventional efforts to address this scarcity have centered on Small Area Estimation \(SAE\) techniques\. Prominent SAE methods, including the Elbers, Lanjouw, and Lanjouw \(ELL\) approach\[das2015robust\], Empirical Bayes Prediction \(EBP\)\[molina\_small\_2010\], and M\-quantile \(MQ\) modeling\[chambers\_m\-quantile\_2006\], have become standard tools for integrating survey and census data to generate localized estimates\. While each method offers distinct benefits—such as the parametric simplicity of ELL, the non\-linearity handling of EBP, and the outlier robustness of MQ—they are uniformly constrained by a reliance on the assumption of spatial homogeneity\. This foundational premise posits that relationships between variables are stable across different geographic areas\. This assumption often compromises predictive power in developing regions, which are frequently characterized by the very spatial heterogeneity that these models fail to capture\.
Recognizing the limitations of conventional statistical models in capturing spatial complexity, the research community has increasingly shifted toward remote sensing and machine learning as scalable, data\-driven alternatives\. Among the earliest and most impactful geospatial indicators is night\-time lights \(NTL\) data, which became publicly available in the 1990s\. The underlying principle is that artificial illumination serves as a strong, quantifiable proxy for economic activity\[huang2021saturated\]\. Studies have confirmed robust correlations between NTL intensity and various economic metrics, including wage income in Sweden\[mellander2015night\]and asset\-based wealth across numerous African nations\[noor2008using,andersson2019data\]\.
However, a core limitation of NTL\-based methods is their difficulty in distinguishing between economically distinct areas with similar luminosity \(e\.g\., densely populated low\-income versus sparsely populated high\-income regions\)\[jean2016combining\]\. This critical shortcoming necessitated an evolution: researchers began augmenting NTL data with high\-resolution daytime imagery\. The incorporation of granular daytime features, such as building roofing materials and infrastructure, provides the necessary detail for more robust and accurate poverty prediction frameworks\[jean2016combining\]\. To extract meaningful spatial representations from high\-resolution satellite imagery, Convolutional Neural Networks \(CNNs\) serve as the methodological standard\[lecun2002gradient,daoud2023using\]\. A significant obstacle, however, is the scarcity of labeled satellite data, which is particularly acute for the data\-hungry deep learning architectures required to achieve generalizable features and high predictive performance\.
The prevailing strategy to mitigate data scarcity is transfer learning\. This paradigm adapts knowledge from models pre\-trained on massive, general\-purpose image datasets, such as ImageNet, to a more specialized domain\. By fine\-tuning these powerful, learned feature hierarchies, effective models can be developed even with limited task\-specific data\. Features from networks like Overfeat\[sermanet2013overfeat\], for example, have demonstrated efficacy in generalizing to diverse downstream tasks\[sharif2014cnn\]and have been applied to identify development proxies \(e\.g\., built structures, vegetation\) in satellite imagery from countries including Haiti, Cambodia, and India\.
A persistent challenge is the domain mismatch between the object\-centric, ground\-level perspective of ImageNet and the top\-down, aerial perspective of satellite imagery\. To bridge this gap, a pivotal innovation was the intermediate fine\-tuning step proposed by Xieet al\.\[xie2016transfer\]\. Their multi\-stage approach first adapts the ImageNet\-trained model to the aerial domain by training it to predict NTL intensity from daytime images—a well\-established proxy for economic activity\. This intermediate task compels the network to learn spatially relevant representations from an aerial viewpoint\. Methodologically, this is often implemented using a fully convolutional architecture to preserve image detail\. While powerful, a purely spatial analysis provides only a static snapshot; consequently, recent work has underscored the importance of incorporating the temporal dimension to capture the dynamics of development\[pettersson2023time,kakooei2024increasing\]\.
A parallel research stream seeks to augment remote sensing data with other auxiliary sources, such as human mobility patterns derived from mobile phone metadata and environmental conditions from meteorological records\[engstrom2015mapping,puttanapong\_predicting\_2020,xu\_poverty\_2021\]\. These approaches often share a deep learning foundation with imagery\-based techniques, exploiting spatial patterns in urbanization and land use\[tingzon\_mapping\_2019,ayush2021efficient,xie\_transfer\_2016\]\. However, these novel data streams face critical limitations, including uneven spatial coverage, reliance on proprietary data, and significant privacy concerns\.
In policy\-relevant domains, a critical consideration is model interpretability\. Deep neural networks, while excelling at modeling complex, high\-dimensional data, often function as “black boxes,” obscuring the decision\-making process\[rudin\_stop\_2019,zhang\_survey\_2021\]\. This presents a fundamental trade\-off: traditional transparent models like linear regression lack the capacity to capture the non\-linear relationships inherent in geospatial data\. Research is actively focused on developing inherently interpretable models for use with satellite data to reconcile the tension between predictive power and explainability\[ayush\_generating\_2020,abitbol2020interpretable,ledesma2020interpretable\]\.
In addition, another foundational challenge is the selection of an appropriate poverty metric, as poverty is inherently multidimensional\[alkire2015multidimensional,d2024multidimensional\]\. Metrics can vary by measurement scale: Absolute \(e\.g\., World Bank’s international poverty lines\) vs\. relative \(benchmarked against income distribution\)\[decerf2022unambiguous\]; be of a temporal nature: Transient vs\. Persistent\[chung2020understanding,dang2019poverty\], or may vary by the unit of Analysis: Individual vs\. Place\-based deprivation\[abascal2022domains\]\. For geospatial mapping, persistent, place\-based poverty is particularly salient, as it identifies regions with structural disadvantages that can be targeted for intervention\[hall2022review\]\. Approaches that eliminate predefining any poverty metric may therefore be attractive simply due to the flexibility in use for downstream users\.
In India, a significant body of research has applied machine learning frameworks to integrate traditional survey data, such as the National Family Health Survey \(NFHS\), with a diverse range of geospatial indicators: census records, satellite daytime imagery, NTL, climate data, and points of interest\[daoud2021measuring,mehta2025predicting\]\. This multi\-source strategy aims to overcome the limitations of survey data, which, despite their detail, can be biased or statistically insufficient in underrepresented regions \(e\.g\., NSSO samples in some districts contain fewer than 30 observations\[mehta2025predicting\]\)\.
While a consensus is emerging that integrating heterogeneous data sources is crucial for robust poverty estimation in India\[arya2025integrating,khare2024eyes\], significant methodological challenges persist\. Firstly, most existing models generate coarse predictions at the state or district level, leaving high\-resolution mapping unresolved\[subash2018satellite,mehta2025predicting\]\. Secondly, socioeconomic indicators are typically high\-dimensional and exhibit strong multicollinearity\. And finally, data misalignment can occur when administrative boundaries \(villages,taluks, districts\) are moved without generating corresponding lookup tables to reconcile databases across space and time\. Conventional dimensionality reduction techniques, such as Principal Component Analysis \(PCA\)\[wold\_principal\_1987\], assume linearity and often fail to capture the complex, nonlinear patterns in socioeconomic data\. A clear need exists for a framework that can inherently model non\-linear relationships within high\-dimensional data to produce granular, high\-resolution predictions\.
We proposeJuGAAD, a deep learning framework designed to directly address the challenges of high spatial resolution and data complexity in socioeconomic mapping, particularly for downscaling raw socioeconomic indicators\.’JuGAAD’also happens to be a colloquial Hindi term for finding a novel solution to a complex problem\. Our core methodology leverages the complementary strengths of two key datasets: 1\) High\-Resolution Inputs: geographically comprehensive but thematically sparse village\-level Indian Census data augmented with geospatial environmental variables, and 2\) Target rich data: Detailed NSSO indicators \(∼\\sim450, including debt, employment, housing condition, etc\.\) available only at a coarse quasi\-district\-level resolution\.
We first establish a temporally consistent spatial reference scheme based on a hexagonal tessellation \(sized by clusters of∼\\sim20 villages\) that allows data to be tracked across years\. We then generate a downscaling function that maps the high\-resolution input features to the detailed NSSO indicators\. By applying this learned relationship at the sub\-district level,JuGAADgenerates fine\-scale predictions of critical socioeconomic indicators where they were previously unavailable\. The result is a set of high\-resolution, temporally aligned maps that enable analysis of decadal changes in poverty and food security\.
To do this, we first employ an autoencoder to compress high\-dimensional and multicollinear NSSO data into a dense, low\-dimensional latent representation\. As an unsupervised neural network, the autoencoder learns to efficiently encode and reconstruct the input, preserving essential patterns while removing statistical redundancy\. Unlike linear methods such as PCA, autoencoders excel at modeling the complex, non\-linear relationships characteristic of socioeconomic data\[zhou\_anomaly\_2017,chung\_audio\_2016,pu\_variational\_2016\]\. With the compressed representation established, the second stage involves training a predictive regression model to estimate these latent features using readily available high\-resolution data\. The input feature set comprises fine\-grained census indicators and geographical variables\. Recognizing that administrative heterogeneity \(policy and governance\) can lead to divergent poverty levels across similar geographies, we augment this feature set with state\-level identifiers\. A neural network is then trained to map this comprehensive set of high\-resolution inputs to a low\-dimensional latent representation of NSSO indicators\. This regression model forms the core of our downscaling mechanism, enabling predictions at a granular level\. During the inference phase, the trained regression model is deployed using fine\-grained census and geographic data to produce sub\-district\-level socioeconomic estimates\. To rigorously validate the framework, high\-resolution predictions are aggregated back to the original district level and then compared against actual district level NSSO indicators\. Experimental results confirm that this framework successfully downscales socioeconomic indicators with high fidelity, providing a scalable, data\-efficient solution to generate granular socioeconomic data in regions where it is otherwise unavailable\.
## 2Results
This section presents the performance evaluation and analysis of the proposedJuGAADframework\. We first provide a quantitative assessment of the downscaled socioeconomic estimates to evaluate downscaled estimates of NSSO indicators\. Next, spatial visualizations of the predicted maps are examined to illustrate how the model captures fine\-scale variation across regions\. Finally, we analyze the resulting high\-resolution socioeconomic indicators with a focus on poverty\-related measures to demonstrate the framework’s interpretability and policy relevance\.
### 2\.1Quantitative Evaluation of Downscaling Results
Following the preprocessing steps, the final analytical dataset derived from the NSSO surveys comprises a total of 475 features distributed across six socioeconomic categories\. The feature counts for each category are as follows: Consumer Expenditure \(110\), Employment \(92\), Agriculture \(76\), Housing Conditions \(89\), Land and Livestock Holdings \(45\), and Debt and Investment \(83\)\. For the downscaling task, each of these categories was encoded into a lower\-dimensional latent space\. Specifically, the Agriculture and Land and Livestock Holdings categories were mapped to 16\-dimensional latent vectors, while the remaining four categories were each mapped to a 20\-dimensional latent vector\. This results in a consolidated latent representation with a total dimensionality of 112\.
To validate the dimensionality reduction process, a separate autoencoder was trained for each socioeconomic category using the combined district\-level data from 2001 and 2011\. The performance of these autoencoders was evaluated based on their ability to reconstruct the original high\-dimensional features from the compressed latent representations\. Table[1](https://arxiv.org/html/2607.20559#S2.T1)summarizes the reconstruction accuracy, reporting the Mean Squared Error \(MSE\), UniformR2R^\{2\}, and WeightedR2R^\{2\}for each category\.
The trained autoencoders demonstrated strong performance in reconstructing the original data from the compressed latent spaces, with most categories achieving a WeightedR2R^\{2\}value exceeding0\.850\.85\. The Land and Livestock Holdings and Housing Conditions categories yielded the highest reconstruction fidelity \(WeightedR2R^\{2\}=0\.89 and0\.870\.87, respectively\), suggesting a relatively stable and well\-defined feature structure\. The Consumer Expenditure and Agriculture categories also showed robust performance \(WeightedR2≈0\.87R^\{2\}\\approx 0\.87\)\. In particular, the model effectively captured the structure of the Agriculture data despite its higher intrinsic variation between districts\. The slightly lower reconstruction accuracy for the Debt and Investment, and Employment categories is likely attributable to greater heterogeneity in their constituent variables\.
To contextualize these results, we performed a comparative analysis against Principal Component Analysis \(PCA\), a standard linear dimensionality reduction technique\. For a direct comparison, PCA was applied to reduce the dimensionality of each category to match that of its autoencoder counterpart\. As detailed in Table[1](https://arxiv.org/html/2607.20559#S2.T1), the autoencoder consistently outperformed PCA, achieving higherR2R^\{2\}values and lower Mean Squared Error across all categories\. This result underscores the advantage of employing a nonlinear model to capture the complex relationships within the socioeconomic data, a departure from prior methodologies that have often relied on PCA\.
While the category\-level metrics indicate strong overall performance, a more granular, feature\-level analysis reveals important variations\. The scatter plots presented in Fig\.[1](https://arxiv.org/html/2607.20559#S2.F1), which compare original and reconstructed values for individual variables, illustrate this point\. Although the majority of features within each category were reconstructed with high fidelity, certain indicators, particularly within the Debt and Investment and Employment categories, exhibit greater variance and reconstruction error\. This finding suggests that while the category\-specific autoencoders effectively capture the dominant modes of variation, the reconstruction quality for specific, more heterogeneous indicators can vary at the individual level\.
\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
Figure 1:Feature Reconstruction Fidelity of the Autoencoder\. Autoencoder reconstruction scatter plots on training data compare the true values of the high\-dimensional NSSO input features \(x\-axis\) against the values reconstructed from the compressed latent space \(y\-axis\)\. Plots correspond to \(a\) Agriculture, \(b\) Consumer, \(c\) Debt, \(d\) Employment, \(e\) Housing, and \(f\) Land categories, using the combined20012001and20112011NSSO datasets\. The tight clustering of points along the diagonal line confirms the high fidelity of the autoencoder in capturing the non\-linear structure of the data\.Table 1:Performance comparison between autoencoder \(non\-linear\) and PCA \(linear\) for dimensionality reduction of NSSO data\. The table compares the fidelity of the reconstructed district\-level NSSO data after compression into a low\-dimensional latent space\. Metrics are presented for Mean Squared Error \(MSE\), UniformR2R^\{2\}, and WeightedR2R^\{2\}, combining data from 2001 and 2011\. The consistently lower MSE and higherR2R^\{2\}values validate the autoencoder’s capacity to reconstruct the complex, non\-linear structure of the socioeconomic data more effectively than the linear PCA\.Following the dimensionality reduction step, a regression model was trained to predict the category\-specific latent NSSO encodings at the district level\. The model’s inputs consisted of district\-level Census data augmented with auxiliary geographic features and one\-hot encoded state identifiers\. Separate regression models were trained for the 2001 and 2011 datasets, and their respective performance statistics are presented in Table[2](https://arxiv.org/html/2607.20559#S2.T2)\.
The results indicate strong predictive performance for most categories, with WeightedR2R^\{2\}values generally exceeding 0\.65\. The Consumer Expenditure and Housing Conditions categories demonstrated the highest correspondence between predicted and ground\-truth encodings\. In contrast, the Agriculture and Employment categories proved more challenging to predict, exhibiting lowerR2R^\{2\}values and higher MSE\. This is likely attributable to the presence of more idiosyncratic, region\-specific patterns or greater noise within these particular variables\.
Table 2:Regression model performance for predicting latent NSSO encodings from census data \(2001, 2011\)\. Mean Squared Error \(MSE\) andR2R^\{2\}during training for predicting NSSO encodings from Census data across socioeconomic categories, evaluated against ground truth NSSO encodings for20012001and20112011\. The highR2R^\{2\}values indicate the model successfully learned the non\-linear relationship between census data and highly compressed, low\-dimensional feature space\.Following the training phase, regression models were deployed for inference to predict the NSSO latent encodings at the cluster level, using the corresponding high\-resolution census and auxiliary features as input\. To validate these downscaled predictions, the predicted cluster\-level encodings were then aggregated to the district level, enabling a direct comparison with the ground\-truth district\-level encodings\. The summary statistics for this comparison are presented in Table[3](https://arxiv.org/html/2607.20559#S2.T3)\. Furthermore, Figs\.[2](https://arxiv.org/html/2607.20559#S2.F2)and[3](https://arxiv.org/html/2607.20559#S2.F3)present histograms of the resultingR2R^\{2\}scores for the years 2001 and 2011, respectively, illustrating the distribution of predictive accuracy across the districts\.
Table 3:Performance of cluster\-level encoding predictions aggregated to the district scale\. Summary statistics \(MSE,R2R^\{2\}\) quantify the model’s accuracy after generating fine\-scale cluster\-level predictions of latent NSSO encodings, aggregating them to the district level, and comparing against the ground\-truth NSSO latent representations for 2001 and 2011\.\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
Figure 2:Histograms ofR2R^\{2\}scores at the district level comparing aggregated cluster\-based encodings predictions with ground truth encodings, shown separately for each category\. The data is from 2001 for \(a\) Agriculture, \(b\) Consumer, \(c\) Debt, \(d\) Employment, \(e\) Housing, and \(f\) Land categories\.\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
Figure 3:Histograms ofR2R^\{2\}scores at the district level comparing aggregated cluster\-based encodings predictions with ground truth encodings, shown separately for each category\. The data is from 2011 for \(a\) Agriculture, \(b\) Consumer, \(c\) Debt, \(d\) Employment, \(e\) Housing, and \(f\) Land categories\.An analysis of the predictive performance across the different socioeconomic categories reveals significant variation by both theme and year\. The Consumer Expenditure category consistently exhibited the strongest performance, achieving WeightedR2R^\{2\}values of0\.82890\.8289\(2001\) and0\.82370\.8237\(2011\), which suggests its underlying latent structure is well\-captured by the available predictor variables\.
The Land and Livestock Holdings and Housing Conditions categories also demonstrated high predictive accuracy in 2001 \(WeightedR2\>R^\{2\}\>0\.74\); however, both experienced a notable decline in performance in 2011, with the Land category showing a substantial drop from0\.78860\.7886to0\.63440\.6344\. In contrast, the Debt and Investment and Employment categories showed more consistent, albeit moderate, performance across the two periods, withR2R^\{2\}values typically ranging between0\.550\.55and0\.700\.70\.
The Agriculture category was consistently the most challenging to predict, with its WeightedR2R^\{2\}falling to0\.50920\.5092in 2011\. This lower performance likely reflects a high degree of variability and noise inherent in agricultural indicators that are not fully explained by the model’s inputs\. Collectively, these results indicate that while the latent NSSO encodings can be effectively approximated from high\-resolution data, the model’s accuracy is heterogeneous, varying significantly by both socioeconomic category and year\.
The final stage of the methodology involves generating the full\-scale, high\-resolution NSSO indicators\. This was accomplished by passing the predicted cluster\-level latent encodings through their respective pre\-trained, category\-specific decoders\. To quantitatively validate these final reconstructed indicators, they were first spatially aggregated to the district level by averaging the predictions of all clusters within each district boundary\. This aggregation allows for a direct comparison against the ground\-truth district\-level NSSO data\. Table[4](https://arxiv.org/html/2607.20559#S2.T4)summarizes the performance of this final reconstruction, reporting the Mean Squared Error \(MSE\), the overallR2R^\{2\}, and the number of individual indicators achieving anR2≥0\.5R^\{2\}\\geq 0\.5for both years\. Additionally, the distributions of theR2R^\{2\}scores are visualized in the histograms presented in Figs\.[4](https://arxiv.org/html/2607.20559#S2.F4)and[5](https://arxiv.org/html/2607.20559#S2.F5)\.
The final evaluation assessed the correspondence between the fully reconstructed high\-resolution indicators \(aggregated to the district level\) and the ground\-truth NSSO data\. While the decoding process introduced an expected degree of information loss compared to the latent space prediction, the final indicators for most categories demonstrated a satisfactory correspondence with the ground truth\.
The Consumer Expenditure and Land and Livestock Holdings categories were the most accurately reconstructed, yielding WeightedR2R^\{2\}values of approximately0\.690\.69and0\.680\.68, respectively, in 2001, with only a modest decline in performance observed in 2011\. The robustness of the reconstruction for the Consumer Expenditure category is further evidenced by the fact that over85%85\\%of its constituent variables consistently achieved anR2≥0\.5R^\{2\}\\geq 0\.5in both years, indicating a reliable recovery of the primary underlying patterns\. In contrast, categories like Agriculture and Employment showed lowerR2R^\{2\}scores and higher MSE, particularly in 2011\. For example, Agriculture’s WeightedR2R^\{2\}dropped to0\.28870\.2887, with fewer than half of the variables meeting theR2≥0\.5R^\{2\}\\geq 0\.5threshold\. These results suggest that some fine\-grained details may be lost during encoding, prediction, and decoding—especially in categories with more variable or nonlinear distributions\.
In summary, the results demonstrate that the end\-to\-end pipeline can successfully reconstruct coherent socioeconomic patterns from the predicted latent encodings, although performance varies by category and year\. Crucially, this validates the central premise of our approach: that it is feasible to generate meaningful, high\-resolution estimates of complex survey\-based indicators using only census and geographic features as inputs during the inference stage, without recourse to the target survey data itself\.
Table 4:End\-to\-end predictive performance of downscaled socioeconomic indicators after full reconstruction\. The table shows the performance metrics \(MSE,R2R^\{2\}\) for the final, high\-resolution cluster\-level predictions after they have been aggregated back to the district level and compared against the original ground\-truth NSSO data for 2001 and 2011\. The column “Vars withR2≥0\.5R^\{2\}\\geq 0\.5” indicates the percentage of individual variables demonstrating adequate predictive correspondence\. Note the negativeR2R^\{2\}values for Agriculture, suggesting low predictive power for highly heterogeneous indicators after the full downscaling process\.\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
Figure 4:Histograms ofR2R^\{2\}scores at the district level comparing aggregated cluster\-based full\-scale decoded predictions with ground truth NSSO data, shown separately for each category\. The data is from 2001 for \(a\) Agriculture, \(b\) Consumer, \(c\) Debt, \(d\) Employment, \(e\) Housing, and \(f\) Land categories\.\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
Figure 5:Histograms ofR2R^\{2\}scores at the district level comparing aggregated cluster\-based full\-scale decoded predictions with ground truth NSSO data, shown separately for each category\. The data is from 2011 for \(a\) Agriculture, \(b\) Consumer, \(c\) Debt, \(d\) Employment, \(e\) Housing, and \(f\) Land categories\.
### 2\.2Spatial Visualization of Downscaled Predictions
Beyond the statistical performance metrics, a crucial evaluation of the downscaling model involves analyzing the spatial patterns of its predictions\. Visualizing the downscaled indicators on a map offers qualitative insights into regional trends that may not be apparent from aggregate statistics alone\. This approach enables a direct visual comparison between the fine\-grained predicted patterns and the coarse\-grained ground\-truth data, providing an assessment of the model’s capacity to generate spatially coherent results\.
#### 2\.2\.1Encoded variables visualization
The spatial analysis begins with a visualization of the predicted latent encodings\. While a detailed semantic interpretation of these latent dimensions is reserved for future investigation, their geographic distributions offer insight into the model’s learned representations\. We present these spatial distributions at both the high\-resolution cluster level and the aggregated district level\.
This dual visualization serves two purposes: first, it allows for a qualitative comparison between the district\-averaged predictions and the corresponding ground\-truth encodings; second, it enables an assessment of the spatial coherence between the fine\-grained and coarse\-grained predictions\. Figures[6](https://arxiv.org/html/2607.20559#S2.F6)and[7](https://arxiv.org/html/2607.20559#S2.F7)display these maps for the years 2001 and 2011, respectively, illustrating the patterns with representative latent vectors from the Consumer Expenditure, Debt and Investment, and Housing Conditions categories\.
\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
\(g\)
\(h\)
\(i\)
Figure 6:Spatial Coherence of Predicted Latent Socioeconomic Structures \(2001\)\. Each row represents one latent dimension of the compressed NSSO space\. Columns: Left \(a, d, g\) show ground\-truth district\-level values; Center \(b, e, h\) show predicted cluster values averaged to the district level; Right \(c, f, i\) show final high\-resolution cluster\-level predictions\. The visual alignment between the Center and Left columns illustrates the model’s qualitative agreement with the ground truth, while the Right column reveals plausible fine\-grained, intra\-district variations absent in the coarse input\.A key characteristic of the autoencoder’s latent space is that it represents a complex, nonlinear combination of the original input indicators\. A direct consequence of this is that a singular, human\-interpretable meaning—such as ‘income’ or ‘expenditure’—cannot be ascribed to any individual latent dimension\. However, despite this lack of direct semantic interpretability, a qualitative inspection of the mapped dimensions reveals that some components do appear to encode distinct geographic signatures\. These often manifest as an emphasis on particular regional patterns, such as concentrations in eastern or southern districts, alongside other more complex, mixed spatial signals\.
A central finding of this spatial analysis is that the high\-resolution cluster\-level predictions exhibit a strong spatial coherence with the district\-level ground\-truth data\. The overall geographic patterns observed in the predicted maps correspond closely to those in the original district data, while simultaneously revealing plausible, fine\-grained variations within each district that are absent in the coarse summaries\.
This alignment is confirmed both qualitatively through visual inspection and quantitatively by aggregating the cluster\-scale predictions to the district scale for direct comparison\. The model’s demonstrated ability to preserve both broad regional trends and localized patterns indicates that the latent encodings—despite their lack of direct semantic interpretability—successfully capture meaningful socioeconomic structures\.
\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
\(g\)
\(h\)
\(i\)
Figure 7:Spatial Coherence of Predicted Latent Socioeconomic Structures \(2011\)\. Each row represents one latent dimension of the compressed NSSO space\. Columns: Left \(a, d, g\) show ground\-truth district\-level values; Center \(b, e, h\) show predicted cluster\-scale values averaged to the district level; Right \(c, f, i\) show final high\-resolution cluster\-scale predictions\. The visual alignment between the Center and Left columns illustrates the model’s qualitative agreement with the ground truth, while the Right column reveals plausible fine\-grained, intra\-district variations absent in the coarse input\.The predicted high\-resolution maps effectively capture the temporal evolution of socioeconomic indicators between 2001 and 2011, revealing significant shifts in their spatial distributions over the decade\. The model successfully resolves these year\-to\-year dynamics while simultaneously preserving the broader, underlying regional patterns, demonstrating a capacity to model both change and stability\.
#### 2\.2\.2Decoded \(full\-scale\) variables visualization
Following the analysis of the learned latent representations, the examination now turns to the final, full\-scale indicators produced by the decoder\. These decoded variables represent the model’s best estimate of the original NSSO features and provide the basis for the subsequent quantitative error analysis\.
\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
\(g\)
\(h\)
\(i\)
Figure 8:Visualization of Decoded Full\-Scale Socioeconomic Indicators \(2001\)\. The figure highlights the model’s ability to reproduce broad spatial patterns while capturing fine\-scale detail\. Rows: \(a–c\) Coffee Expenditure, concentrated in southern states; \(d–f\) Tap Water Access, showing regional infrastructure gaps; and \(g–i\) Tube Well Usage, inversely related to tap water, indicating groundwater reliance in northern regions\. Columns: Left—ground\-truth district values; Center—predicted cluster\-scale values averaged to districts; Right—final high\-resolution predictions\.An examination of the decoded variables reveals that their spatial distributions often correspond to established regional characteristics\. A clear example is the expenditure on coffee powder, as depicted in the original district\-level data \(Fig\.[8\(a\)](https://arxiv.org/html/2607.20559#S2.F8.sf1)\)\. The map indicates that expenditure is highest in the southern states of Kerala, Tamil Nadu, and Karnataka, a pattern consistent with their status as India’s primary coffee\-producing and consuming regions due to both cultural and agricultural factors\. A visual comparison with the model’s high\-resolution cluster\-level prediction \(Fig\.[8\(c\)](https://arxiv.org/html/2607.20559#S2.F8.sf3)\) and the aggregated district\-level prediction \(Fig\.[8\(b\)](https://arxiv.org/html/2607.20559#S2.F8.sf2)\) confirms that our framework successfully reproduces this broad geographic trend\. However, while the primary regional pattern is preserved, a loss of some finer\-scale detail is evident in the aggregated map\. This attenuation is likely attributable to factors such as the limited number of training samples and the inherent uncertainty in the source NSSO data, which required imputation for missing indicators\. Nevertheless, the strong overall correspondence demonstrates the promise of this approach and provides a foundation for future methodological refinements\.
The model’s predictions for 2011 capture a significant temporal shift in coffee consumption patterns\. As illustrated in the corresponding map \(Fig\.[9](https://arxiv.org/html/2607.20559#S2.F9)\), expenditure on coffee had undergone a notable geographic diffusion by 2011\. In contrast to the pattern in 2001, where consumption was heavily concentrated in the southern states, the 2011 map reveals an expansion into other parts of the country, suggesting a broader cultural adoption of coffee as a beverage\.
A second set of examples is provided by indicators related to drinking water sources, which exhibit distinct and often inverse regional patterns\. For instance, the proportion of households relying on tube wells is notably high in northern states such as Uttar Pradesh, a reflection of widespread groundwater extraction practices in the region \(Fig\.[8\(g\)](https://arxiv.org/html/2607.20559#S2.F8.sf7)\)\. Conversely, the use of tap water as a principal source is more prevalent in other parts of the country, exhibiting a largely inverse spatial distribution \(Fig\.[8\(d\)](https://arxiv.org/html/2607.20559#S2.F8.sf4)\)\. The model’s cluster\-level and aggregated predictions for both the tube well indicator \(Figs\.[8\(i\)](https://arxiv.org/html/2607.20559#S2.F8.sf9),[8\(h\)](https://arxiv.org/html/2607.20559#S2.F8.sf8)\) and the tap water indicator \(Figs\.[8\(f\)](https://arxiv.org/html/2607.20559#S2.F8.sf6),[8\(e\)](https://arxiv.org/html/2607.20559#S2.F8.sf5)\) successfully reproduce these opposing spatial trends\. Crucially, the high\-resolution cluster\-scale maps for both indicators reveal plausible fine\-scale variations within individual districts, providing a more granular view of local water infrastructure patterns than is available in the original source data\.
\(a\)
\(b\)
\(c\)
\(d\)
\(e\)
\(f\)
\(g\)
\(h\)
\(i\)
Figure 9:Visualization of Decoded Full\-Scale Socioeconomic Indicators \(2011\)\. The figure highlights the model’s ability to reproduce broad spatial patterns while capturing fine\-scale detail\. Rows: \(a–c\) Coffee Expenditure, concentrated in southern states; \(d–f\) Tap Water Access, showing regional infrastructure gaps; and \(g–i\) Tube Well Usage, inversely related to tap water, indicating groundwater reliance in northern regions\. Columns: Left—ground\-truth district values; Center—predicted cluster\-scale values averaged to districts; Right—final high\-resolution predictions\.The temporal analysis of drinking water sources between 2001 and 2011 reveals divergent yet complementary trends\. The 2011 map for tap water as the principal drinking source \(Fig\.[9](https://arxiv.org/html/2607.20559#S2.F9)\) shows a marked increase in usage across numerous districts, which is indicative of an expansion of piped water infrastructure into urban and semi\-urban areas\.
Concurrently, tube well usage also intensified in certain regions, notably in the northern state of Uttar Pradesh, suggesting a continued and potentially growing reliance on local groundwater sources where formal supply may be insufficient\. In many areas, the spatial patterns of these two indicators exhibit a complementary relationship; regions with a rising prevalence of tap water access often show a corresponding decrease in tube well dependence\. This observable trend can be interpreted as a proxy for development, reflecting improved access to formal water infrastructure in those regions\.
### 2\.3Socioeconomic Indicators of Poverty
The NSSO dataset contains numerous key indicators that provide valuable insights into regional variations in socioeconomic conditions and food security\. The spatiotemporal visualization of these indicators facilitates the identification of geographic patterns, regional disparities, and temporal shifts between 2001 and 2011\. While the full dataset offers a wide range of variables for diverse analytical purposes, the following discussion will focus on a selection of representative indicators to illustrate the utility of our downscaled predictions\. Specifically, we examine patterns related to the type of housing structure, the primary cooking fuel used, and the number of meals consumed per day\.
The first indicator examined is the ‘type of structure,’ which categorizes housing into three classes:katcha\(structures made from temporary materials like mud and bamboo\),pucca\(structures made from durable materials like brick and concrete\), andsemi\-pucca\(a hybrid of the two\)\. The prevalence ofpuccahousing is a well\-established proxy for higher socioeconomic status\. Figure[10](https://arxiv.org/html/2607.20559#S2.F10)presents the maps of the original district\-level data alongside the high\-resolution cluster\-level predictions for thepuccaandsemi\-puccahousing types for the year 2001\.
A visual comparison of the maps reveals that the model successfully captures the distinct spatial distribution patterns for both housing types\. For instance, regions with a high proportion ofpuccahousing, such as parts of southern and western India, exhibit strong correspondence between the ground\-truth data and the model’s predictions\. Similarly, the higher prevalence ofsemi\-puccahousing in some northeastern and central regions is also faithfully reproduced\. Critically, the cluster\-level predictions further resolve fine\-scale variations within individual districts, elucidating localized housing conditions that are obscured by the coarse aggregation of the original district\-level data\.
\(a\)
\(b\)
\(c\)
\(d\)
Figure 10:Each row shows, left: district\-level ground‐truth, and right: cluster‐level predictions\. \(a\) and \(b\) shows the percentage of people who live in a partially permanent structure \(called asemi\-puccahouse in Hindi language\) with a mix of durable and temporary materials\. \(c\) and \(d\) shows the percentage of people who live in a permanent structure \(called apuccahouse in Hindi language\) made of durable materials like concrete and brick\. The data is from 2001\.The second illustrative indicator is the primary fuel used for cooking, with maps for 2001 presented in Fig\.[11](https://arxiv.org/html/2607.20559#S2.F11)\. An analysis of the spatial pattern for kerosene usage reveals a high prevalence in specific regions that correspond to areas historically characterized by higher levels of poverty and developmental challenges\. This elevated reliance on kerosene can be interpreted as a proxy for limited access to more modern cooking fuels, such as LPG or electricity, and a continued dependence on subsidized energy sources\.
This example demonstrates the utility of the downscaled data for regional disparity analysis\. The high\-resolution maps can help identify localized, sub\-district areas with limited access to clean fuels—a condition often correlated with underdevelopment and infrastructure gaps—thereby highlighting pockets of persistent socioeconomic vulnerability\.
\(a\)
\(b\)
Figure 11:The percentage of people who use kerosene as a fuel for cooking in 2001\. \(a\) the district\-level ground‐truth, \(b\) the cluster‐level predictions\.The final illustrative indicator is a direct measure of severe food insecurity, capturing the proportion of the population that experienced one or more days in the preceding month without a single meal due to an inability to afford food\. The spatial distribution of this indicator for 2001 is presented in Fig\.[12](https://arxiv.org/html/2607.20559#S2.F12), which shows both the original district\-level data and the corresponding high\-resolution cluster\-scale prediction\.
\(a\)
\(b\)
\(c\)
\(d\)
Figure 12:The percentage of people who experienced days within the past 30 days when they could not afford even a single meal\. \(a\) the district\-level ground‐truth from 2001, \(b\) the cluster‐level predictions from 2001, \(c\) the district\-level ground‐truth from 2011, and \(d\) the cluster‐level predictions from 2011\.The spatial distribution of this indicator of food insecurity in 2001 exhibits a dispersed pattern, likely reflecting localized pockets of acute poverty in districts with limited employment opportunities\. A temporal comparison reveals a marked improvement by 2011, with a significant reduction in the number of regions reporting high values for this indicator\. This trend suggests an overall decline in extreme food insecurity, potentially indicating broader improvements in access to basic necessities during this period\.
This type of spatiotemporal analysis exemplifies the utility of the downscaled dataset for temporal poverty tracking\. Such an approach can highlight districts experiencing improvements in food security over time and contribute to the evaluation of social welfare programs and policy interventions\.
## 3Method
This section describes the JuGAAD framework in detail\. JuGAAD integrates coarse socioeconomic indicators from NSSO with fine\-resolution census, geographical, and state\-level features to learn mappings that allow the generation of spatially detailed socioeconomic estimates\.
Figure 13:Conceptual overview of the proposed JuGAAD framework\. The model learns relationships between coarse but detailed NSSO socioeconomic indicators and fine\-scale predictors \(census, geographical, and state data\) at the district level and applies the learned mapping to generate high\-resolution detailed socioeconomic estimates\.The overall pipeline is summarized in Fig\.[13](https://arxiv.org/html/2607.20559#S3.F13)\. Subsequent subsections describe the data sources and the model design in detail\.
### 3\.1Data Sources
Data Sources and Spatial Framework This study integrates two principal sources of nationwide data from India: the National Sample Survey Office \(NSSO\) and the Indian Census, using datasets from both for 2001 and 2011\. Our methodology is designed to leleveragesgths of each dataset to produce high\-resolution, detailed socioeconomic indicators\.
1. 1\.NSSO Data:The NSSO provides extensive data on a wide range of socioeconomic topics through large\-scale household surveys\. These surveys are organized into six primary categories: Consumer Expenditure, Employment, Agriculture, Housing Conditions, Land and Livestock Holdings, and Debt and Investment\. The thematic richness of the NSSO data is a key strength, offering detailed indicators unavailable in the census, such as housing construction materials, specific agricultural practices, household dietary habits, and access to credit\. For the purposes of this study, the NSSO data is aggregated at the district level, as illustrated by the spatial variation in irrigated land shown in Fig\.[15\(a\)](https://arxiv.org/html/2607.20559#S3.F15.sf1)\.
2. 2\.Census Data:The Indian Census offers a comprehensive demographic and socioeconomic snapshot of the entire population\. While less thematically detailed than the NSSO surveys, the Census provides data at a much finer spatial resolution, including both district and village levels \(an example feature is shown in Fig\.[15\(b\)](https://arxiv.org/html/2607.20559#S3.F15.sf2)\)\. However, the coverage and availability of village\-level census data are inconsistent across different years, villages often change administrative units, or are added or removed causing gaps in the record\. To establish a uniform basis for our analyses, we constructed a series of hexagonal tessellations across the entire country using an iterative methodology: we aimed at finding a size that would be large enough to allow averaging of census and geospatial data to reduce noise, while being small enough to preserve the spatial representation of districts and subdistricts \(taluks/tehsils\)\. A vertex\-to\-vertex size of 15Km was found to be optimal \(~20 villages to a hexagon\), hexagons with fewer than 5 villages were dropped to avoid biasing data and extrapolating into sparsely populated areas\. Districts and subdistrict IDs were assigned to each hexagon using a simple majority rule for all village centroids falling within each hexagon\. Each hexagon therefore represents a village cluster and is a consistent spatial unit of analysis across years that is considerably smaller than a district and subdistrict, and allows the capture of the hierarchical spatial structure of administrative units\. This scheme also ensures that villages changing administrative boundaries or getting added/removed between survey periods do not unduly influence large\-scale inference\. Generating data for subsequent years is also made simple as the same aggregation scheme can be applied keeping the number of records \(the number of hexagons\) the same\. Figure[14](https://arxiv.org/html/2607.20559#S3.F14)illustrates the spatial schema \(see specifically Fig\.[14\(c\)](https://arxiv.org/html/2607.20559#S3.F14.sf3)\)\.
A central methodological challenge arises from the disparity between the datasets: the NSSO provides rich, nuanced indicators but only at a coarse spatial resolution \(districts\), while the Census offers high\-resolution data that lacks thematic detail\. This study addresses this gap by using the downscaled Census indicators as predictors to estimate the detailed NSSO indicators at the village cluster level\. The resulting predictions yield a novel, high\-resolution dataset that combines the spatial granularity of the Census with the thematic depth of the NSSO, enabling a more nuanced analysis of socioeconomic landscapes and their development over the study period\. The complete NSSO dataset utilized is available as supplementary material\.
\(a\)
\(b\)
\(c\)
Figure 14:The map of India showing \(a\) states and \(b\) different districts in each state\. \(c\) Uniform hexagons are used as fine\-scale village cluster\-scale prediction units\. Each district \(separated by black boundaries\) contains multiple hexagons, enabling localized analysis of socioeconomic indicators\.#### 3\.1\.1NSSO Data Preprocessing
A critical preprocessing phase was necessary to address two primary challenges inherent in the raw NSSO data: data completeness and feature dimensionality\. First, the prevalence of missing values across numerous states and districts necessitated the implementation of a robust gap\-filling procedure\. Second, the dataset comprises both continuous \(non\-negative\) and discrete \(compositional categories summing to 100%\) features\. Many discrete variables exhibit high dimensionality and significant sparsity, which can adversely affect model convergence and performance\. To mitigate this, a dimensionality reduction strategy was applied to generate a more compact feature representation, thereby improving both computational efficiency and model efficacy\. The complete preprocessing pipeline is detailed below\.
1. 1\.Gap\-filling strategy:To address the challenge of non\-uniformly distributed missing data, we developed a predictive imputation methodology\. A complete\-case analysis was deemed unsuitable as it would result in a substantial loss of geographic coverage\. Instead, our approach imputes missing values for each NSSO feature by training a model on a set of predictor variables from other thematic categories\. Prior to imputation, the predictor set is refined by removing highly correlated features and standardizing the remaining variables to a zero mean and unit variance\. The imputation model consists of a shallow regression neural network with a single hidden layer and a ReLU activation function on the output layer to enforce non\-negativity\. The network is trained using the Adam optimizer to minimize mean squared error loss\. The validity of each imputation was evaluated using the coefficient of determination \(R2R^\{2\}\), with an acceptance threshold ofR2≥0\.6R^\{2\}\\geq 0\.6\. If a feature could not be predicted with sufficient accuracy, alternative predictor categories were tested\. Features that failed to meet the performance threshold across all predictor combinations were removed from the entire dataset\. Finally, to ensure a consistent analytical framework, only features present across all districts post\-imputation were retained for subsequent analysis\.
2. 2\.Data cleaning:Following imputation, all the features of the various NSSO categories were consolidated into a unified analytical data set\. This data set was subjected to a rigorous feature selection process to enhance the performance and interpretability of the model\. Initially, we systematically removed uninformative predictors, specifically those exhibiting zero or near\-zero variance \(i\.e\., constant or quasi\-constant features\)\. To address multicollinearity, we then pruned features that were highly correlated with others\. However, an exception was made for a curated set of variables directly pertinent to food security and poverty; these were deliberately retained for downstream analysis, regardless of their correlation coefficients, due to their theoretical importance\. High\-dimensional categorical variables presented a distinct challenge, particularly within the Consumer Expenditure category, some of which comprised over 200 discrete classes\. Such features introduce significant sparsity, which can impede the learning capacity of neural networks\. Upon inspection, these high\-dimensional variables were found to possess minimal variance and were consequently eliminated from the dataset based on the criterion of near\-zero variance\. After this step, the NSSO dataset contains475475features, summarized in Table[5](https://arxiv.org/html/2607.20559#S3.T5)\.
\(a\)
\(b\)
\(c\)
Figure 15:Example of features from each data source: \(a\) Irrigated land map from NSSO data, \(b\) Forest area map from Census data, and \(c\) Soil pH map from geographic data\. Each map illustrates the spatial distribution of the respective variable across India\. All maps are based on data from20012001\.
#### 3\.1\.2Additional Data Sources: Geographic Information and State Identifiers
To enhance the predictive power of the model beyond the primary Census indicators, the feature space was augmented with a comprehensive set of auxiliary geospatial data\. We incorporated4848distinct geographical variables, including soil characteristics \(e\.g\., sand fraction, organic content, pH\), vegetation indices \(e\.g\., tree cover, NDVI\), climate data, and topographic information such as elevation\. The inclusion of these variables is justified by their well\-documented correlation with patterns of socioeconomic development and urbanization, thereby providing crucial geospatial context for the prediction task\. Figure[15\(c\)](https://arxiv.org/html/2607.20559#S3.F15.sf3)shows the spatial variation of soil pH across India\.
To control for unobserved heterogeneity arising from distinct administrative, economic, and political contexts at the state level, we incorporated state identifiers as categorical features in the model\. These identifiers were subsequently one\-hot encoded, transforming each state’s numeric label,ii, into a binary vector where theithi^\{\\text\{th\}\}index is set to 1 and all other elements are 0\. This final set of state\-level fixed effects, combined with the census and geographical variables, constitutes the complete feature space used to predict the NSSO indicators\.
The final input feature set for the prediction model is composed of5858census\-derived variables,4848geographical variables, and3535one\-hot encoded state and union territory identifiers, constituting a feature vector with a total dimensionality of141141\.
Table 5:Number of NSSO features per category after preprocessing\.
### 3\.2Architecture
The primary modeling objective is to simultaneously predict the full suite of NSSO indicators using the consolidated set of census, geographic, and state\-level features\. This task constitutes a multi\-output regression problem\. A significant methodological challenge arises from the fact that the dimensionality of the target variable space \(the NSSO indicators\) is considerably greater than that of the input feature space\. It is well\-established that the performance of such models tends to degrade as the number of target variables increases, particularly when outputs substantially outnumber inputs\. This imbalance can lead to reduced predictive accuracy, lower computational efficiency, and increased prediction variance\. To mitigate these issues, our approach first involves reducing the dimensionality of the target NSSO indicators themselves, for which we employ an autoencoder to learn a compact, low\-dimensional representation\.
This section outlines the proposed methodology, beginning with a formal definition of the data sources\. The target NSSO data is organized into six thematic categories: Consumer Expenditure, Employment, Agriculture, Housing Conditions, Land and Livestock Holdings, and Debt and Investment\. The data vector for any given categoryxxis denoted as𝐗nssox∈ℝDnx\\mathbf\{X\}\_\{nsso\}^\{x\}\\in\\mathbb\{R\}^\{D^\{x\}\_\{n\}\}, whereDnxD^\{x\}\_\{n\}represents its dimensionality\. The predictor variables consist of census data, denoted as𝐗cen∈ℝDc\\mathbf\{X\}\_\{cen\}\\in\\mathbb\{R\}^\{D\_\{c\}\}, and geographic features,𝐗geo∈ℝDg\\mathbf\{X\}\_\{geo\}\\in\\mathbb\{R\}^\{D\_\{g\}\}\.To prevent the model from learning a spurious ordinal relationship from arbitrary numerical assignments, state identifiers are one\-hot encoded into categorical vectors\. These are denoted as𝐗st∈ℝN\\mathbf\{X\}\_\{st\}\\in\\mathbb\{R\}^\{N\}, whereNNis the total number of states and union territories\. The complete methodology is detailed in the subsections that follow\.
#### 3\.2\.1Autoencoder: Efficiently compressing the NSSO socioeconomic indicators
To preserve the thematic integrity of the data, a separate autoencoder was trained independently for each of the six NSSO categories\. This approach ensures that the resulting latent variables are category\-specific, preventing the conflation of features from disparate domains and thereby enhancing the interpretability of the learned low\-dimensional representations\.
We begin by formally defining the data at our two primary spatial scales: the district and the village cluster\. At the district level, the available data for each of the six NSSO categories is denoted by𝐗nssodist,x\\mathbf\{X\}^\{dist,x\}\_\{nsso\}\. The corresponding predictor variables at this scale are the census data,𝐗cendist\\mathbf\{X\}\_\{cen\}^\{dist\}, and geographic data,𝐗geodist\\mathbf\{X\}\_\{geo\}^\{dist\}\. At the finer cluster level, only the predictor variables are available, denoted as𝐗cenhex\\mathbf\{X\}\_\{cen\}^\{hex\}and𝐗geohex\\mathbf\{X\}\_\{geo\}^\{hex\}, respectively\.
The central objective is to predict the NSSO indicators at the cluster level, for which data is unavailable\. Our methodology commences by first learning a compact representation of the known district\-level NSSO indicators\. As illustrated in Fig\.[16](https://arxiv.org/html/2607.20559#S3.F16), a category\-specific encoder,ex\(⋅\)\\textbf\{e\}\_\{x\}\(\\cdot\), is employed to reduce the dimensionality of𝐗nssodist,x\\mathbf\{X\}^\{dist,x\}\_\{nsso\}\. The latent space of the autoencoder is given by:
Figure 16:The autoencoder model for compressing the high\-dimensional district\-level NSSO features is shown\.zdist,x=ex\(Xnssodist,x\),\\textbf\{z\}^\{dist,x\}=\\textbf\{e\}\_\{x\}\\left\(\\textbf\{X\}^\{dist,x\}\_\{nsso\}\\right\),\(1\)
wherexxis the NSSO data category,𝐳dist,x∈ℝDzx\\mathbf\{z\}^\{dist,x\}\\in\\mathbb\{R\}^\{D^\{x\}\_\{z\}\},DzxD^\{x\}\_\{z\}is the dimensionality of the latent space, andDzx<DnxD^\{x\}\_\{z\}<D\_\{n\}^\{x\}\.
A decoder,dx\(⋅\)\\textbf\{d\}\_\{x\}\(\\cdot\), is used to reconstruct the NSSO data from categoryxx\. The reconstructed NSSO data,X~nssodist,x\\tilde\{\\textbf\{X\}\}\_\{nsso\}^\{dist,x\}, is denoted by:
X~nssodist,x=dx\(zdist,x\),\\tilde\{\\textbf\{X\}\}\_\{nsso\}^\{dist,x\}=\\textbf\{d\}\_\{x\}\\left\(\\textbf\{z\}^\{dist,x\}\\right\),\(2\)
whereX~nssodist,x\\tilde\{\\textbf\{X\}\}\_\{nsso\}^\{dist,x\}is the reconstructed NSSO data from thexthx^\{th\}category\.
#### 3\.2\.2Regression Model: Predicting compressed NSSO socioeconomic indicators using the census
The next stage of the methodology involves training a regression model to predict the compressed latent representation of the district\-level NSSO data,zdist,x\\textbf\{z\}^\{dist,x\}\. The input for this model is a consolidated feature vector,Xcendist\\textbf\{X\}\_\{cen\}^\{dist\}, formed by concatenating the district\-level census data, geographic features, and the one\-hot encoded state identifiers:
𝐗inpdist=𝐗cendist⊕𝐗geodist⊕𝐗st,\\mathbf\{X\}\_\{inp\}^\{dist\}=\\mathbf\{X\}\_\{cen\}^\{dist\}\\oplus\\mathbf\{X\}\_\{geo\}^\{dist\}\\oplus\\mathbf\{X\}\_\{st\},\(3\)
where⊕\\oplusdenotes the concatenation operator\.
The regression model is a fully connected neural network, the architecture of which is illustrated in Fig\.[17](https://arxiv.org/html/2607.20559#S3.F17)\. The network consists of a series of blocks, where each block comprises a fully connected transformation, a batch normalization step, and a rectified linear unit \(ReLU\) activation function\. The final fully connected layer is shaped by a hyperbolic tangent \(tanh\) activation function as the same activation is used for latent space in the autoencoder\. The output of this network is the predicted latent space for categoryxx, denoted asz~dist,x\\tilde\{\\textbf\{z\}\}^\{dist,x\}, and is formally defined in[Equations4](https://arxiv.org/html/2607.20559#S3.E4)to[7](https://arxiv.org/html/2607.20559#S3.E7)\.
Figure 17:The regression model for predicting the latent space of NSSO socioeconomic features is shown\. The district\-level census, geographical, and categorical state ID features are used as inputs to predict the latent space of district\-level NSSO socioeconomic features\. FC denotes fully\-connected layers\.h1=ReLU\(BN\(XinpdistW1\+b1\)\),\\textbf\{h\}\_\{1\}=\\textit\{ReLU\}\\left\(BN\\left\(\\textbf\{X\}\_\{inp\}^\{dist\}\\textbf\{W\}\_\{1\}\+\\textbf\{b\}\_\{1\}\\right\)\\right\),\(4\)
h2=ReLU\(BN\(h1W2\+b2\)\),\\textbf\{h\}\_\{2\}=\\textit\{ReLU\}\\left\(BN\\left\(\\textbf\{h\}\_\{1\}\\textbf\{W\}\_\{2\}\+\\textbf\{b\}\_\{2\}\\right\)\\right\),\(5\)
h3=ReLU\(BN\(h2W3\+b3\)\),\\textbf\{h\}\_\{3\}=\\textit\{ReLU\}\\left\(BN\\left\(\\textbf\{h\}\_\{2\}\\textbf\{W\}\_\{3\}\+\\textbf\{b\}\_\{3\}\\right\)\\right\),\(6\)
z~dist=tanh\(h3W4\+b4\),\\tilde\{\\textbf\{z\}\}^\{dist\}=tanh\\left\(\\textbf\{h\}\_\{3\}\\textbf\{W\}\_\{4\}\+\\textbf\{b\}\_\{4\}\\right\),\(7\)
where𝐳~dist∈ℝD\\tilde\{\\mathbf\{z\}\}^\{dist\}\\in\\mathbb\{R\}^\{D\}is the predicted latent space of district\-level NSSO data\.W1\\textbf\{W\}\_\{1\},W2\\textbf\{W\}\_\{2\},W3\\textbf\{W\}\_\{3\}, andW4\\textbf\{W\}\_\{4\}are the neural network weight matrices\.b1\\textbf\{b\}\_\{1\},b2\\textbf\{b\}\_\{2\},b3\\textbf\{b\}\_\{3\}, andb4\\textbf\{b\}\_\{4\}are the biases\.h1\\textbf\{h\}\_\{1\},h2\\textbf\{h\}\_\{2\}, andh3\\textbf\{h\}\_\{3\}are hidden layer outputs\.BNBNis the batch normalization layer\.
The operations in[Equations4](https://arxiv.org/html/2607.20559#S3.E4)to[7](https://arxiv.org/html/2607.20559#S3.E7)can be represented concisely using the operator,REG\(⋅\)\\textbf\{REG\}\(\\cdot\), for each categoryx, as shown in Eq\.[8](https://arxiv.org/html/2607.20559#S3.E8)\.
z~dist,x=REGx\(Xinpdist\)\.\\tilde\{\\textbf\{z\}\}^\{dist,x\}=\\textit\{REG\}\_\{x\}\\left\(\\textbf\{X\}\_\{inp\}^\{dist\}\\right\)\.\(8\)
The predicted NSSO latent space of categoryxxcan be reconstructed using the decoder,dx\(⋅\)\\textbf\{d\}\_\{x\}\(\\cdot\)\. This concludes the training phase of the models\.
#### 3\.2\.3Loss Function
The Huber loss function was selected for training both the autoencoder and the regression models\. This choice is motivated by the function’s inherent robustness to outliers, a common characteristic of socioeconomic data\. The Huber loss effectively combines the desirable properties of the mean squared error \(MSE\) for small errors with the lower outlier sensitivity of the mean absolute error \(MAE\) for large errors\. This makes it particularly well\-suited for modeling real\-world datasets that often contain noise and anomalous values\. The function is formally defined as:
ℒδ=\{12‖𝐲~−𝐲‖22,if‖𝐲~−𝐲‖≤δ,δ\(‖𝐲~−𝐲‖−12δ\),otherwise,\\mathcal\{L\}\_\{\\delta\}=\\begin\{cases\}\\frac\{1\}\{2\}\\left\\\|\\mathbf\{\\tilde\{y\}\}\-\\mathbf\{y\}\\right\\\|\_\{2\}^\{2\},&\\text\{if \}\\left\\\|\\mathbf\{\\tilde\{y\}\}\-\\mathbf\{y\}\\right\\\|\\leq\\delta,\\\\ \\delta\\left\(\\left\\\|\\mathbf\{\\tilde\{y\}\}\-\\mathbf\{y\}\\right\\\|\-\\frac\{1\}\{2\}\\delta\\right\),&\\text\{otherwise\},\\end\{cases\}\(9\)
whereinyydenotes the observed value,y~\\tilde\{y\}is the predicted value, and the hyperparameterδ\\deltamodulates the function’s robustness to outliers by defining the transition point between a mean\-squared\-error and a mean\-absolute\-error\-based penalty\.
For autoencoder training phase,y=𝐗nssodist,xy=\\mathbf\{X\}^\{\\text\{dist\},x\}\_\{nsso\}, andy~=𝐗~nssodist,x\\tilde\{y\}=\\tilde\{\\mathbf\{X\}\}^\{\\text\{dist\},x\}\_\{nsso\}\. For the regression model training phase,y=𝐳dist,xy=\\mathbf\{z\}^\{\\text\{dist\},x\}, andy~=𝐳~dist,x\\tilde\{y\}=\\tilde\{\\mathbf\{z\}\}^\{\\text\{dist\},x\}\.
#### 3\.2\.4Inference: Predicting the high\-resolution NSSO indicators
During inference, the model is deployed to predict the NSSO indicators at the cluster level\. Accordingly, the input to the regression network is formed by combining the downscaled census features, geographic attributes, and state identifiers as follows:
𝐗inphex=𝐗cenhex⊕𝐗geohex⊕𝐗st\.\\mathbf\{X\}\_\{inp\}^\{hex\}=\\mathbf\{X\}\_\{cen\}^\{hex\}\\oplus\\mathbf\{X\}\_\{geo\}^\{hex\}\\oplus\\mathbf\{X\}\_\{st\}\.\(10\)
This cluster\-level input vector,Xinphex\\textbf\{X\}\_\{inp\}^\{hex\}, is then passed through the trained regression network to predict the corresponding latent space for each NSSO categoryxx, denoted aszhex,x\\textbf\{z\}^\{hex,x\}\. This prediction is formulated as:
z~hex,x=REGx\(Xinphex\)\.\\tilde\{\\textbf\{z\}\}^\{hex,x\}=\\textit\{REG\}\_\{x\}\\left\(\\textbf\{X\}\_\{inp\}^\{hex\}\\right\)\.\(11\)
The predicted latent vectors are then transformed back to the original feature space via the decoder,d\(⋅\)\\textbf\{d\}\(\\cdot\)\. This reconstruction step produces the final estimate of the cluster\-level NSSO indicators, formulated as:
X~nssohex,x=dx\(𝐳~hex,x\),\\tilde\{\\textbf\{X\}\}\_\{nsso\}^\{hex,x\}=\\textbf\{d\}\_\{x\}\\left\(\\tilde\{\\mathbf\{z\}\}^\{hex,x\}\\right\),\(12\)
whereX~nssohex,x\\tilde\{\\textbf\{X\}\}\_\{nsso\}^\{hex,x\}is the cluster\-level NSSO data fromxthx^\{th\}category\. To qualitatively validate the downscaled predictions, the resulting cluster\-level NSSO indicators were mapped and visually compared against the original district\-level data\. This comparison allows for an assessment of whether the model preserves the broad spatial patterns present in the source data while introducing plausible fine\-scale variations\.
#### 3\.2\.5Evaluation
A quantitative validation was performed to assess model accuracy\. To facilitate a direct comparison with the source data, the cluster\-level predictions were first aggregated to the district scale via averaging\. The mean squared error \(MSE\) was then computed between these aggregated district\-level predictions and the actual district\-level NSSO indicators to measure overall performance\.
#### 3\.2\.6Uncertainty Estimation
To produce robust predictions and estimate epistemic uncertainty, an ensemble of models with identical architectures was trained\. Each model was initialized with different random weights, resulting in a distribution of predictions for each output indicator\. The final prediction is the mean of this distribution, while its standard deviation provides a direct measure of model uncertainty\. This uncertainty metric is crucial for end\-users to gauge the reliability of any given prediction\.
## 4Discussion
Principal Findings and Methodological Contribution\.This study demonstrates the efficacy of an autoencoder\-based framework for capturing and reconstructing complex socioeconomic patterns from coarse\-grained survey data\. The central contribution of our methodology is its ability to learn a compact, latent representation of diverse socioeconomic indicators at coarse scales and subsequently decode this representation to generate meaningful, high\-resolution predictions at a finer tesselation\. While the latent space itself is not directly interpretable, each dimension encodes a nonlinear combination of input features, enabling the generation of decoded maps that reflect key spatial and temporal patterns consistent with observations\.
The most significant outcome of this spatial downscaling method is the model’s capacity to reproduce not only broad regional trends but also plausible fine\-scale variations within districts\. Despite being trained on a single data point per district, the model’s predictions exhibit significant intra\-district heterogeneity, revealing subtle socioeconomic gradients that are obscured in the original coarse\-scale data\. This capacity for spatial decomposition allows for a more nuanced analysis of local socioeconomic dynamics than was previously possible\.
Temporal Dynamics and Thematic Exploration\.The utility of this approach is further underscored by temporal comparisons between the 2001 and 2011 datasets\. For instance, the predicted maps of coffee consumption illustrate a clear geographic expansion of the practice from its traditional concentration in southern states like Karnataka and Kerala to broader regions of the country over the decade, reflecting a documented shift in consumer behavior\. Similarly, observed increases in the use of tap water and tubewells as primary drinking sources, particularly in states like Uttar Pradesh, suggest patterns of infrastructure development and public health interventions\. While caution must be exercised in causal interpretation, these temporal shifts provide valuable insights into regional progress and socioeconomic change\. These examples represent only a fraction of the analytical possibilities; the rich variety of indicators within the NSSO data can be similarly explored to study topics ranging from regional disparities to evolving patterns in living conditions\.
Uncertainty Quantification and Data Dissemination\.In addition to the high\-resolution prediction maps, our framework provides corresponding uncertainty estimates to support more informed interpretation\. By employing an ensemble of models, we quantify the standard deviation across their predictions, which serves as a proxy for epistemic uncertainty arising from limitations in the model and data\. This information enables end\-users to assess the reliability of specific predictions and make decisions with an explicit understanding of the model’s confidence\. To facilitate further research and application, a cleaned version of the original district\-level NSSO dataset and the associated geographic data will be made publicly available alongside our predictions\.
Limitations\.Several limitations of the present study must be acknowledged\. First, the training dataset is constrained to a single sample per district, which limits the model’s learning capacity and increases the risk of overfitting\. Second, a potential distributional shift may exist between the training and inference phases; the decoder is trained exclusively on latent vectors derived from district\-level data but is tasked during inference with decoding representations aggregated from hexagon\-level features\. The statistical properties of these two sets of latent vectors may differ, posing a generalization challenge\. Third, the imputation procedure used to address missing values may have introduced noise\. Finally, the source survey data itself may contain inherent biases or inaccuracies from the collection process, which would propagate through the modeling pipeline\.
Conclusion and Future Directions\.In summary, this study introduces a flexible and broadly applicable framework for modeling and decoding latent socioeconomic structures from sparse, coarse\-grained data into fine\-grained spatial predictions\. Despite the noted limitations, the approach demonstrates how regional modeling, temporal tracking, and indicator\-specific analysis can be successfully performed even in data\-scarce environments\. The findings point to the broader utility of representation learning in enhancing the accessibility and analytical value of socioeconomic datasets across spatial and temporal scales\.
The framework can be readily extended to other geographical contexts where similar data structures exist\. Moreover, the learned latent features hold potential as inputs for a range of downstream tasks, such as classifying regions by poverty level, clustering for policy targeting, or as auxiliary inputs to hybrid models that also incorporate satellite imagery\. Ultimately, this form of representation learning offers researchers and policymakers a powerful tool to explore and interpret socioeconomic data at a finer resolution than is directly observable, unlocking new insights into the dynamics of human development\.
## 5Conclusion
This study presents a deep learning framework, JuGAAD, for generating high\-resolution socioeconomic maps from limited census survey data\. Using features derived from Census records, geographic attributes, and state identifiers, the model predicts district\-level indicators from the National Sample Survey Office \(NSSO\)\. During inference, the model enables fine\-grained predictions, capturing variations even within districts and facilitating more sophisticated and detailed socioeconomic analysis\.
The high\-resolution socioeconomic indicators preserve broad spatial trends while revealing localized variations that are not captured at the district level\. Temporal comparisons between 2001 and 2011 further highlight meaningful changes in key indicators, such as improvements in drinking water access and reductions in food insecurity\. To assess prediction confidence, an ensemble of regression models is used\. The standard deviation of their outputs is used to estimate epistemic uncertainty\. This uncertainty information allows users to identify which outputs are more reliable and to interpret prediction confidence spatially\.
The final dataset includes a cleaned version of district\-level \(or coarse\) indicators, predicted high\-resolution indicators, and uncertainty estimates\. By transforming sparse and irregular data into detailed spatial representations, this work provides a foundation for regional analysis, policy evaluation, and further research in socioeconomic development, particularly in data\-constrained contexts\.
Acknowledgments
This study was supported by funding from NASA South/Southeast Asia Research Initiative grants NNX17AK75G, and 80NSSC22K1363\.
## Appendix ANSSO Gap\-filling of Indicators
The gap\-filling procedure is summarised briefly in Section[3\.1\.1](https://arxiv.org/html/2607.20559#S3.SS1.SSS1)\. This appendix reports the performance of the imputation models across all six NSSO thematic categories\.
Table[6](https://arxiv.org/html/2607.20559#A1.T6)reports, for each category, the total number of variables, the number of districts with missing observations, and the meanR2R^\{2\}and MSE of the accepted imputation models\. Across all categories, the meanR2R^\{2\}exceeds0\.940\.94, indicating that the cross\-category predictor structure in the NSSO surveys provides sufficient information to reliably recover missing values\. Only two variables, one in Agriculture and one in Housing Conditions, could not be imputed with sufficient accuracy and were removed\.
The missing data is not uniformly distributed across categories\. Housing Conditions and Employment exhibit the highest gap fractions \(0\.1620\.162and0\.1120\.112, respectively\), indicating that these surveys had substantially lower geographic coverage, likely reflecting the administrative challenges of collecting labor and dwelling condition data across all districts\. The remaining categories, Consumer Expenditure, Debt and Investment, and Land and Livestock Holdings show near\-complete coverage, with gap fractions below0\.0030\.003\.
Table 6:Summary of NSSO gap\-filling performance by category\. Missing Districts is the maximum number of district\-level observations requiring imputation across indicators in the category\. Gap Fraction is the corresponding proportion of all districts\. MeanR2R^\{2\}and Mean MSE are averaged over all indicators\.
Figure 18:Distribution of cross\-validatedR2R^\{2\}values for the gap\-filling imputation models, shown separately for each NSSO thematic category\. Each bar represents the number of variables whose imputation model achieved the correspondingR2R^\{2\}value\. The heavy concentration near 1\.0 across all categories indicates strong and reliable imputation performance\.Figure[18](https://arxiv.org/html/2607.20559#A1.F18)shows the distribution of cross\-validatedR2R^\{2\}values for each category\. In all cases, the vast majority of variables achieveR2\>0\.95R^\{2\}\>0\.95, with the distribution heavily concentrated near 1\.0\. Consumer Expenditure shows the widest spread, with some variables in the 0\.7–0\.9 range, reflecting the greater heterogeneity across districts in expenditure patterns\. Employment, Land, and Debt and Investment show the most concentrated distributions, with nearly all variables exceedingR2=0\.94R^\{2\}=0\.94\.
## ReferencesSimilar Articles
Life Style Levels: Neighborhood Delineation using Geospatial Data
This paper proposes a scalable framework using satellite imagery and building morphology to map intra-urban affluence variation across Indian cities, validated with ground-level observations.
Scalable and Efficient Joint Spiking Embedding Predictive Architecture for Large-Scale Dynamic Graphs
Proposes SG-JEPA, a joint spiking embedding predictive architecture for large-scale dynamic graphs that partitions nodes into context and target sets along the temporal dimension to learn predictive embeddings, achieving competitive performance on node classification while scaling to graphs with 13 million edges and avoiding complex self-supervised mechanisms.
Reconstructing and forecasting disease trajectories of patients with Alzheimer's disease using routine data in resource-constrained settings
This paper introduces GNOVA, a GRU-Neural ODE Variational Autoencoder framework for reconstructing and forecasting Alzheimer's disease cognitive trajectories from routine clinical data without expensive neuroimaging or biomarkers, achieving low error and uncertainty estimation on the ADNI dataset.
Geometry-Aware Infrastructure-Anchored Denoiser for UWB Sensing and Work-Zone Reconstruction
This paper presents GAIA, a geometry-aware learning framework for UWB denoising and work-zone reconstruction that couples temporal range modeling with latent anchor-layout estimation. Evaluated on real-world outdoor data, GAIA reduces range MSE by 18.4% and improves polygon IoU by 15.5% over baselines, demonstrating effective boundary-level reconstruction under NLOS conditions.
AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modeling
This paper introduces AeroJEPA, a Joint-Embedding Predictive Architecture for scalable 3D aerodynamic field modeling. It addresses limitations in current surrogate models by predicting semantic latent representations of flow fields, enabling efficient high-fidelity analysis and design optimization.