How Granary-HiFi Turns Wild Speech Into a High-Fidelity Resource for European Languages

GPU ① Speaker Diarization & Clustering

Problem: A single audio segment may contain multiple speakers, or the same speaker may appear across many segments without a shared identity.

Solution: Speaker diarization separates overlapping or sequential speakers within segments. Speaker-embedding clustering then links the same voice across an entire recording session, enabling same-speaker context retrieval downstream.

Why it matters: A high-fidelity segment has to belong to one speaker. If another voice is present, even briefly, later use of that segment mixes two identities.

GPU ② Sound Event Detection (SED)

Problem: Source audio can contain non-speech events. Music beds, applause, traffic, keyboard clicks, and other non-speech events are common.

Solution: A sound-event detection model annotates each segment with the presence and type of background events. Segments with persistent non-speech interference are flagged or excluded.

Why it matters: A high-fidelity subset cannot leave music, applause, and other non-speech in the recording and expect every later user to ignore them.

GPU ③ Perceptual Quality & Bandwidth Annotation

Problem: Audio quality is multidimensional—a segment can be intelligible but reverberant, clean but low-bandwidth, or high-fidelity but clipped.

Solution: Multiple quality signals are computed: UTMOS for perceptual naturalness, bandwidth detection for sample-rate adequacy, and speech-rate estimation. These remain as separate metadata fields rather than being collapsed into a single accept/reject score.

Why it matters: No single quality metric captures all failure modes. A 4.2 UTMOS score tells you the audio sounds good. It doesn’t tell you the sentence was cut off mid-word.

GPU ④ Early cut-off verification (CTC–MFA Cascade)

Problem: ASR segmenters optimize for recognition accuracy, not utterance completeness. Words are routinely truncated at segment boundaries.

Solution: A two-stage cascade combines CTC-based alignment with Montreal Forced Aligner (MFA) to verify that segment boundaries coincide with complete words.

Critical nuance: MFA doesn’t support all 22 languages equally. When a language is unsupported, the result is a “review” state—not evidence of truncation.

GPU ⑤ Same-Speaker Context Retrieval

Problem: Many speech tasks need a reference clip of the same speaker.

Solution: Using the speaker clusters from Stage ①, the pipeline retrieves a context clip from the same speaker that meets its own acoustic quality thresholds.

Why it matters: The clip has to be as clean as the segment it represents. A noisy or mismatched clip ties that speaker’s identity to artifacts.

CPU ⑥ Language-Aware Phonemization

Each segment is phonemized using language-appropriate tools, producing IPA annotations aligned with the original text. IPA, the International Phonetic Alphabet, is a standard notation for pronunciation. Spelling rules differ across languages, so these symbols record pronunciation explicitly instead of leaving later users to infer it from the written text.

FILTER Quality Gate

Every segment must pass a composite filter requiring: valid raw and normalized text, valid IPA, valid context text, no early cutoff detected, and context text overlap below 0.5. Segments that fail any criterion are rejected.

GPU ⑦ Speech Enhancement

Segments that pass the quality gate are enhanced with the Studio Voice or RE USE model. It restores the retained speech toward a studio-quality signal.

Key design decision: The pipeline enriches JSONL manifests with metadata at each stage without rewriting the source audio files, dramatically reducing I/O cost and making the pipeline resumable.