Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

A framework that aligns self-supervised respiratory encoders with medical terminology in a shared latent space to enable zero-shot inference. To compensate for limited paired data, a medical LLM generates structured reports from metadata, providing semantic anchors for contrastive learning. The approach combines a sigmoid-based contrastive loss with the encoder’s SSL objective and targeted negative sampling, achieving strong zero-shot performance across multiple tasks and datasets.

The study introduces a method to ground respiratory sound encoders in clinical semantics by leveraging an LLM to produce structured medical reports from available metadata, forming semantically rich anchors for training.

Its training blends a sigmoid-based contrastive loss with the encoder’s self-supervised objective and similarity-aware negative sampling, enabling sharper discrimination of pathological vs. non-pathological states.

Evaluated across nine tasks on six datasets, the method reports a 61.3% mean zero-shot AUC, outperforming prior models and achieving the highest linear probing AUC (71.6%) using only 43% of the data used by larger baselines.

Source: cs.CL updates on arXiv.org — https://arxiv.org/abs/2609.00055