Low accuracy (~50%) with SSL (BYOL/MAE/VICReg) on hyperspectral crop stress data — what am I missing? [R]

I’m working on a hyperspectral dataset of cabbage crops for nitrogen deficiency detection. The dataset has 3 classes: Healthy Mild nitrogen stress Severe nitrogen stress I’m trying to use self-supervised learning (SSL) for representation learning and then fine-tune for classification. What I’ve done: Tried multiple SSL methods: BYOL, MAE, VICReg Used data augmentation (spectral noise, masking, scaling, etc.) Fine-tuned with a classifier head Evaluated using accuracy and F1-score Problem: No matter what I try, the performance is stuck around: Accuracy: \~45–50% F1-score: also low (\~0.5) This is barely better than random (since 3 classes ≈ 33%). My setup: Hyperspectral data (hundreds of bands) 1D/patch-based model (ViT-style) SSL pretraining → fine-tuning pipeline Tried k-NN and linear probe as well (still weak) What I suspect: Classes might not be well separable spectrally SSL methods designed for RGB may not adapt well Augmentations might be hurting instead of helping Model not capturing spectral-specific patterns What I’m looking for: Would really appreciate suggestions on: Better SSL methods for hyperspectral data Is VICReg actually the best choice here? Should I try masked spectral modeling instead? Feature engineering Should I include vegetation indices (NDVI, etc.)? PCA before training? Model architecture 1D CNN vs ViT vs hybrid? Any proven architectures for hyperspectral? Evaluation Best way to validate SSL representations? Any tricks to improve linear probe results? General advice Anyone worked on plant stress / hyperspectral classification? Common