OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

OmniSpectra is a wavelength-native masked-autoencoder foundation model that encodes spectra at continuous physical log-wavelength coordinates instead of pixel indices, letting one 12.2M-parameter model pretrain on eight surveys and match or beat larger instrument-specific baselines without resampling.

Standard sequence models encode position by token rank, so the same physical feature (e.g. H-alpha) receives a different positional code on each instrument's sampling grid. This coordinate mismatch forces practitioners to destructively resample continuous physical signals onto a common grid, blurring narrow absorption and emission lines and injecting interpolation artifacts. Instruments in the paper's corpus differ by up to 25x in sampling density, so per-instrument models or lossy homogenization are the only alternatives under index-based encoding.

Spectra are split into fixed P=32-pixel patches; within each patch a linear embedding plus depthwise convolution is attention-pooled by a shared learned query into content tokens that carry no patch rank or instrument identifier. Each token is given a positional code computed as a continuous sin/cos function of its masked mean log-wavelength relative to a fixed global anchor, with log-spaced frequencies, and continuous 1D RoPE over log-wavelength supplies relative distances in attention. Because redshift is additive in log-lambda, the shared offset cancels in RoPE's relative term, reducing positional distance to the physical wavelength ratio. Reconstruction is done by a cross-attention Fourier decoder that queries every native coordinate with a Fourier basis gated by exp(-0.5*(omega_k*sigma_LSF)^2) using the instrument's reported line-spread-function width, attenuating frequencies above the effective resolution. The model is pretrained with 0.5 per-patch masking for 200 epochs on 4 A100 GPUs and evaluated with a frozen encoder plus a lightweight cross-attention pooling head.

• On APOGEE stellar parameter estimation OmniSpectra is best on all three targets, including [Fe/H] where the margin over AstroCLIP is largest. §Stellar Parameter Estimation
• On DESI PROVABGS galaxy properties OmniSpectra has the highest R2 for sSFR, mass-weighted age and metallicity. §Galaxy Property Estimation
• In per-class source classification on SDSS, OmniSpectra has the best F1 for every class, with the largest gap on quasars and stars. §Source Classification
• On DESI PROVABGS redshift estimation OmniSpectra reaches the best R2 and ties GaSNet-III on robust dispersion while having a nearly 5x lower catastrophic outlier rate. §Redshift Estimation
• The leave-one-component-out ablation shows the full configuration is best on every reported task; removing the positional code hurts APOGEE most and removing LSF gating hurts source classification. §Ablation Study
• Model size is small relative to baselines: 12.2M parameters versus 370M for AstroCLIP and 314M for AION-1 counting only parameters active on spectral input. §Inference Efficiency

• The positional encoding is a continuous function of log-wavelength and could be evaluated outside the training wavelength union, but its period range is tuned to approximately 3,600-18,000 Angstrom and performance at substantially different wavelengths has not been evaluated.
• The learned representations are unimodal; extending them to other modalities, e.g. via CLIP-style alignment, is stated as future work.
• The LSF gate is an inductive bias rather than a strict band-limit guarantee, because the final prediction passes through a nonlinear MLP (stated in the decoder section).
• The ablation reports that component contributions are task-dependent and non-additive, and that some partial configurations underperform the Base configuration, so the design is not independently justified component-by-component.

• Positional code is a continuous sin/cos function of masked mean log-wavelength with log-spaced frequencies, evaluable at any wavelength rather than a lookup table keyed by integer rank. §Log-Wavelength Positional Encoding
• Cosmological redshift is additive in log-lambda, so a shared offset cancels in continuous RoPE and positional distance reduces to log(lambda_A/lambda_B), the physical ratio of wavelengths. §Log-Wavelength Positional Encoding
• A learned global query attention-pools each patch's depthwise-convolved pixel embeddings, concentrating the token on narrow features that mean pooling would dilute with the continuum. §Attention-Pooled Patch Tokenizer
• Because patch size P is fixed in pixels rather than in wavelength, the number of patches varies with instrument resolution; cross-instrument consistency comes from anchoring positional codes and reconstruction queries to lambda. §Attention-Pooled Patch Tokenizer
• The decoder cross-attends over encoder tokens at the patch centroid and then evaluates a resolution-gated Fourier basis at each native pixel coordinate, providing anti-aliasing from the instrument's reported LSF. §Implicit Resolution-Gated Fourier Decoder
• Pretraining uses 6.4M spectra from eight surveys spanning resolving power R of about 1,500-34,000 and 3,865-16,384 native pixels per spectrum, with no resampling or interpolation. §Experimental Setup
• When pretrained on SDSS alone and evaluated on DESI and APOGEE with no further pretraining, OmniSpectra transfers with little degradation while index-based Perceiver degrades sharply on redshift. §Cross-Survey Generalization
• Native-pixel patching scales encoder sequence length with the observed spectrum, giving OmniSpectra the highest throughput among compared models with far less memory than per-pixel and fixed-length baselines. §Inference Efficiency