Modular Mixture of Experts for Multi-Sensor Remote Sensing

Designing and benchmarking a plugin-style MoE architecture with independent sensor-specific encoders for multi-sensor remote sensing, as an alternative to monolithic foundation models.

Requirements

  • M.Sc. in Data Science, Computer Science, Artificial Intelligence, Mathematics, or similar
  • Strong knowledge of Python and deep learning frameworks (PyTorch, Lightning)
  • Solid software engineering skills (modular design, clean interfaces, testing)
  • Understanding of deep learning architectures (Transformers, ViTs, MoE)
  • Basic knowledge of remote sensing and multi-sensor satellite data

Description

Current remote sensing foundation models such as SkySense, Prithvi, and Clay are predominantly monolithic: a single large model ingests data from multiple sensors through a unified encoder. While effective, this design has practical drawbacks. Adding a new sensor requires retraining or fine-tuning the entire model. Inference always loads the full model even when only one sensor is available. The codebase tends to grow into complex, tightly-coupled systems that are difficult to extend and maintain.

Mixture of Experts (MoE) architectures have recently gained traction in this space. RingMoE (14.7B parameters, 9 satellites), MAPEX (modality-aware expert pruning), and CSMoE (cross-sensor soft MoE) all demonstrate that sparse expert routing can reduce computational cost while handling multiple modalities. However, these approaches still train as monolithic systems and apply pruning or routing post-hoc. The experts are not truly independent modules.

This thesis proposes a different design philosophy: a plugin-style MoE where sensor-specific encoder modules are genuinely independent. Each expert is a self-contained encoder for a specific sensor type (e.g., Sentinel-2 optical, Sentinel-1 SAR, Landsat thermal) with a well-defined interface. A lightweight router determines which experts to activate based on the available input, and a shared fusion module combines their outputs. Experts can be trained independently, added or removed without retraining the system, and loaded selectively at inference time depending on what data is available.

The thesis combines two contributions: (1) designing and implementing this modular framework with clean, extensible interfaces, and (2) benchmarking it against existing monolithic and MoE approaches on standard multi-sensor RS tasks to understand the trade-offs between modularity, accuracy, and computational cost.

This is a crowded research area, and the thesis should not aim to compete on scale with RingMoE’s 14.7B parameters. Instead, the focus is on practical modularity: can a system built from independent, reusable sensor experts match the accuracy of monolithic models while being more flexible, easier to maintain, and cheaper to deploy?

Known challenges include MoE training instability (load balancing across experts, expert collapse), defining clean sensor boundaries when sensors overlap spectrally, and designing a router that generalizes to unseen sensor combinations.

The research will address the following objectives:

  • Architecture Design: Defining the expert interface, router mechanism, and fusion module for a plugin-style multi-sensor MoE.
  • Framework Implementation: Building a modular, extensible codebase where sensor experts can be independently trained, loaded, and composed.
  • Expert Training: Training sensor-specific experts for at least 3 sensor types (e.g., Sentinel-2, Sentinel-1, Landsat).
  • Benchmarking: Comparing against monolithic FMs (Prithvi, Clay) and existing MoE approaches (MAPEX, CSMoE) on multi-sensor RS benchmarks.
  • Ablation Studies: Analyzing the impact of router design, number of experts, and selective loading on accuracy and computational cost.

Main Activities

  • Reviewing the literature on MoE architectures, multi-sensor fusion, and modular deep learning systems
  • Designing the expert interface, router, and fusion module
  • Implementing the modular framework with clean, testable interfaces
  • Training sensor-specific expert encoders (Sentinel-2, Sentinel-1, Landsat)
  • Benchmarking against monolithic and MoE baselines on standard RS tasks
  • Ablation studies on modularity vs. accuracy trade-offs
  • Analysis of selective loading, inference cost, and extensibility

Contacts