Skip to main content

Audio datasets

An AudioDataset is a lightweight, parquet-backed collection of audio samples that TrackGenerator, SceneGenerator, and SceneCollection consume as source content. It separates the metadata index (a Polars DataFrame with columns such as id, path, speaker_id, transcript, and length_s) from the audio bytes, which are loaded on demand through an audio loader.

You can create an AudioDataset in three ways:

  • from a cloud repository such as HuggingFace, using from_huggingface() with a matching audio loader
  • directly via the AudioDataset constructor, when you already have a parquet manifest and a loader instance
  • from local WAV files on disk, using from_local_wav_directory() or from_local_speech_directory()

All three paths write or consume a parquet manifest. This manifest is the portable unit you can save, reload across sessions, move between machines, or share with colleagues — as long as the WAV files travel with it at the same relative paths.

The SDK ships built-in audio loaders for common public datasets. Pass one of these as audio_loader_class when calling from_huggingface(), or instantiate one directly when constructing an AudioDataset manually:

  • LibriSpeechAudioLoader — LibriSpeech and LibriSpeech-derived datasets with embedded audio bytes and a text transcript column
  • AudioSetAudioLoader — AudioSet with embedded bytes or HuggingFace-hosted paths, identified by a configurable column that defaults to video_id
  • CommonVoiceAudioLoader — Common Voice 17.0 with embedded audio and a sentence transcript column
  • GigaSpeechAudioLoader — GigaSpeech with embedded audio and begin_time/end_time columns for automatic duration computation
  • VoxPopuliAudioLoader — VoxPopuli with embedded audio and a normalized_text transcript column
  • TedliumAudioLoader — TED-LIUM Release 3 with embedded audio and a text transcript column
  • MultilingualLibriSpeechAudioLoader — Multilingual LibriSpeech with embedded audio and an audio_duration column
  • AudioPathAudioLoader — general-purpose loader for datasets where each row carries a file path to a local WAV file

If none of these match your dataset, see Custom audio loaders.

Basic examples​

Dataset from Hugging Face​

Use from_huggingface() to load speech datasets hosted on the HuggingFace Hub. The method discovers parquet shards via the HuggingFace API, downloads and caches them locally, and instantiates the appropriate audio loader.

The following example loads LibriSpeech, a dataset derived from OpenSLR's LibriSpeech ASR corpus, using LibriSpeechAudioLoader:

import treble_tsdk.scene as sc

speech_dataset = sc.AudioDataset.from_huggingface(
    repo_id="treble-technologies/librispeech_asr_sliced",
    split="test",
    audio_loader_class=sc.LibriSpeechAudioLoader,
    schema_mapping={"transcript": "text"},
    drop_columns=["audio"],
)

schema_mapping maps logical column names used by the SDK (such as transcript) to the actual column names in the parquet. drop_columns drops the embedded-audio column from the metadata index after the audio loader is in place, since the loader itself keeps a lazy handle on the parquet and reads the underlying audio bytes on demand.

Use max_parquet_files to limit how many shards are loaded. This is useful for large datasets during development.

Use filter_collection() to apply a Polars expression filter after loading and narrow the dataset down to matching rows. The following example loads AudioSet and filters it down to clips labeled as air conditioning noise:

import polars as pl

hvac_dataset = sc.AudioDataset.from_huggingface(
    repo_id="agkphysics/AudioSet",
    config="full",
    split="bal_train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "video_id"},
    drop_columns=["audio"],
    max_parquet_files=1,
)
hvac_dataset = hvac_dataset.filter_collection(
    pl.col("human_labels").list.join(" ").str.to_lowercase().str.contains(r"air conditioning|air conditioner")
)

See Public dataset loaders for examples covering the other built-in loaders.

Direct construction from a parquet manifest​

Use the AudioDataset constructor directly when you already have a parquet manifest and a loader instance and don't need the scanning or discovery logic provided by the factory methods. This is the lowest-level path and gives you full control over every parameter.

The constructor accepts:

  • parquet_url: a single path or URL, or a list of paths/URLs for multi-shard datasets
  • audio_loader: a loader instance responsible for resolving sample identifiers to audio
  • schema_mapping: a dict mapping logical column names to the actual column names in the parquet
  • drop_columns: a list of column names to discard when loading

Pass a single path and a pre-configured loader:

from pathlib import Path
parquet_path = Path("<path_to_dataset>") / "dataset.parquet"

dataset = sc.AudioDataset(
    parquet_url=parquet_path,
    audio_loader=sc.AudioPathAudioLoader(parquet_path),
    schema_mapping={"transcript": "text"},
)

Speech dataset from a WAV directory​

Use from_local_speech_directory() when you need per-clip speaker identity, duration, and transcripts. This dataset type works directly with ConversationRules and talker_identifier in TrackGenerator.

You can provide transcripts in two ways:

  • as sibling .txt files placed next to each WAV, read automatically during the scan
  • from an external source such as a CSV file, added after the dataset is created using add_column() — see Transcripts from an external CSV

Organize the directory with one subfolder per talker. The subfolder name becomes the talker identifier. Place a sibling .txt file next to each WAV to provide its transcript:

speech_dir/
    alice/
        001.wav
        001.txt    # transcript for 001.wav
        002.wav
        002.txt
    bob/
        001.wav
        001.txt

Create the dataset by pointing from_local_speech_directory() at this layout. This writes a parquet manifest to parquet_path as a side effect, so once that file exists, load it directly instead of rescanning the directory:

speech_dir = Path("<path_to_speech_directory>")
parquet_path = speech_dir / "speech_dataset.parquet"

if parquet_path.exists():
    speech_dataset = sc.AudioDataset(
        parquet_url=parquet_path,
        audio_loader=sc.AudioPathAudioLoader(parquet_path),
    )
else:
    speech_dataset = sc.AudioDataset.from_local_speech_directory(
        speech_dir=speech_dir,
        parquet_path=parquet_path,
        load_transcripts=True,
    )

Pass this dataset directly to TrackGenerator, using the same talker_identifier string (default: "speaker_id"):

conversation = sc.TrackGenerator(
    audio_dataset=speech_dataset,
    rules=sc.ConversationRules(
        block_duration_range=(1, 15),
        overlap_range=(0.2, 0.5),
        in_track_level_range_db_spl=(64, 70),
    ),
    talker_identifier="speaker_id",
)

By default, WAV files without a sibling .txt file receive an empty transcript and a warning is logged. Set load_transcripts=True to raise an error instead when any transcript is missing.

caution

from_local_speech_directory() and from_local_wav_directory() overwrite parquet_path on every call. Guard the call as shown above to reload the dataset if the parquet file is already constructed.

Other audio from a WAV directory​

Use from_local_wav_directory() when your WAVs are flat or loosely organized and you don't need per-clip speaker labels or transcripts. This is the typical path for music, background noise, or other non-speech content.

The method scans a directory for *.wav files (recursively by default), writes a parquet manifest recording relative paths, and returns a ready-to-use AudioDataset backed by AudioPathAudioLoader. As with from_local_speech_directory(), reload the parquet directly once it exists instead of rescanning the directory on every run:

music_dir = Path("<path_to_wav_directory>")
parquet_path = music_dir / "music_dataset.parquet"

if parquet_path.exists():
    music_dataset = sc.AudioDataset(
        parquet_url=parquet_path,
        audio_loader=sc.AudioPathAudioLoader(parquet_path),
    )
else:
    music_dataset = sc.AudioDataset.from_local_wav_directory(
        wav_dir=music_dir,
        parquet_path=parquet_path,
    )

Pass search_subdir=False to limit the scan to the top level of the directory, ignoring subdirectories.

Advanced examples​

Transcripts from an external CSV​

If transcripts are stored separately from the WAVs — for example, in a single CSV file — create the dataset without transcripts, then populate the column using add_column().

add_column() only updates the in-memory DataFrame; it doesn't modify the parquet file written by from_local_speech_directory(). Call dataset.dataframe.write_parquet() afterward to persist the enriched manifest, so future sessions can reload it without repeating the enrichment step.

import csv

speech_dir = Path("<path_to_speech_directory>")
parquet_path = speech_dir / "speech_dataset.parquet"

speech_dataset = sc.AudioDataset.from_local_speech_directory(
    speech_dir=speech_dir,
    parquet_path=parquet_path
)

# Build a lookup from the CSV: {sample_id -> transcript}
with open("<path_to_transcripts_csv>", newline="", encoding="utf-8") as f:
    transcript_lookup = {row["id"]: row["transcript"] for row in csv.DictReader(f)}

speech_dataset.add_column(
    name="transcript",
    mapper=lambda ref: transcript_lookup.get(ref.id, ""),
)

# Persist the enriched manifest so future sessions can reload it directly.
speech_dataset.dataframe.write_parquet(parquet_path)

add_column() accepts any callable that takes an AudioSampleReference and returns a scalar value. You can also use it to add custom metadata columns for downstream filtering with Polars expressions.

Dataset reload from parquet​

Once the parquet manifest has been written, you can skip rescanning and reload the dataset directly. Use this path in subsequent sessions or after moving the dataset to another machine.

Call the AudioDataset constructor with the parquet path and an AudioPathAudioLoader pointing at the same file:

speech_dataset = sc.AudioDataset(
    parquet_url=parquet_path,
    audio_loader=sc.AudioPathAudioLoader(parquet_path),
)

AudioPathAudioLoader resolves audio paths relative to the parquet file's parent directory. This means the dataset's portable: copy the parquet and the WAV files to a new location and they'll load correctly, as long as the relative layout between them stays the same.

Column mapping and dropping​

Use schema_mapping when the parquet uses different column names from the logical names the SDK expects (such as id, transcript, length_s). Use drop_columns to strip large columns — such as embedded audio bytes — that the loader doesn't need:

dataset = sc.AudioDataset(
    parquet_url="<path_or_url_to_parquet>",
    audio_loader=MyCustomAudioLoader("<path_or_url_to_parquet>"),
    schema_mapping={"id": "clip_id", "transcript": "text"},  # logical_name -> actual_column_name
    drop_columns=["audio"],
)
caution

schema_mapping is applied before columns are available for filtering or indexing. If a mapped source column is absent from the parquet schema, the mapping is silently ignored rather than raising an error.

Metadata cache​

When the constructor reads many shards, it reassembles the full DataFrame on every instantiation. Pass metadata_cache_path to write the assembled DataFrame to a local parquet file on first load and read from that cache on subsequent calls:

cache_path = Path("<path_to_cache_dir>") / "assembled_metadata.parquet"

dataset = sc.AudioDataset(
    parquet_url=["<shard_url_1>", "<shard_url_2>"],
    audio_loader=MyCustomAudioLoader("<shard_url_1>"),
    metadata_cache_path=cache_path,
)

On subsequent runs, the constructor reads cache_path directly and skips the per-shard scan. Delete the cache file to force a rebuild.

Public dataset loaders​

Each built-in audio loader targets one public dataset's schema and pairs with from_huggingface() to download, cache, and index that dataset. The examples below cover the remaining loaders introduced at the top of this page. See Dataset from Hugging Face for the LibriSpeechAudioLoader example, and Direct construction from a parquet manifest for AudioPathAudioLoader.

AudioSet​

Use AudioSetAudioLoader for AudioSet samples. The loader accepts either embedded audio bytes or a HuggingFace-hosted path in the audio column, and selects rows by an identifier column. Set this by passing id_column in audio_loader_kwargs to from_huggingface(), or by mapping id in schema_mapping.

The example below uses agkphysics/AudioSet, one compatible hosting of the dataset on HuggingFace:

audio_set_dataset = sc.AudioDataset.from_huggingface(
    repo_id="agkphysics/AudioSet",
    config="full",
    split="bal_train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "video_id"},
    drop_columns=["audio"],
)

The following example loads BoJack/MMAE, which already uses id as its identifier column:

noise_dataset = sc.AudioDataset.from_huggingface(
    repo_id="BoJack/MMAE",
    split="train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "id"},
    drop_columns=["audio"],
)

Common Voice​

Use CommonVoiceAudioLoader for Common Voice 17.0 samples. Each row is identified by path, and the sentence column holds the transcript:

common_voice_dataset = sc.AudioDataset.from_huggingface(
    repo_id="fixie-ai/common_voice_17_0",
    audio_loader_class=sc.CommonVoiceAudioLoader,
    config="en",
    split="train",
    schema_mapping={"id": "path", "transcript": "sentence"},
    drop_columns=["audio"],
)

GigaSpeech​

Use GigaSpeechAudioLoader for GigaSpeech samples, identified by segment_id. Map begin_time and end_time so AudioDataset computes length_s automatically:

gigaspeech_dataset = sc.AudioDataset.from_huggingface(
    repo_id="fixie-ai/gigaspeech",
    audio_loader_class=sc.GigaSpeechAudioLoader,
    config="xl",
    split="train",
    schema_mapping={
        "id": "segment_id",
        "transcript": "text",
        "begin_time": "begin_time",
        "end_time": "end_time",
    },
    drop_columns=["audio"],
)

VoxPopuli​

Use VoxPopuliAudioLoader for VoxPopuli samples, identified by audio_id. The normalized_text column holds the transcript:

voxpopuli_dataset = sc.AudioDataset.from_huggingface(
    repo_id="facebook/voxpopuli",
    audio_loader_class=sc.VoxPopuliAudioLoader,
    config="en",
    split="train",
    schema_mapping={"id": "audio_id", "transcript": "normalized_text"},
    drop_columns=["audio"],
)

TED-LIUM​

Use TedliumAudioLoader for TED-LIUM Release 3 samples. The text column holds the transcript:

tedlium_dataset = sc.AudioDataset.from_huggingface(
    repo_id="sanchit-gandhi/tedlium-data",
    audio_loader_class=sc.TedliumAudioLoader,
    split="train",
    schema_mapping={"transcript": "text"},
    drop_columns=["audio"],
)

Multilingual LibriSpeech​

Use MultilingualLibriSpeechAudioLoader for Multilingual LibriSpeech samples. Map length_s to the dataset's audio_duration column so clip durations are available without decoding audio:

mls_dataset = sc.AudioDataset.from_huggingface(
    repo_id="facebook/multilingual_librispeech",
    audio_loader_class=sc.MultilingualLibriSpeechAudioLoader,
    config="english",
    split="train",
    schema_mapping={"length_s": "audio_duration"},
    drop_columns=["audio"],
)

Free Music Archive​

Use FMAAudioLoader for Free Music Archive samples, identified by url:

fma_dataset = sc.AudioDataset.from_huggingface(
    repo_id="benjamin-paine/free-music-archive-full",
    audio_loader_class=sc.FMAAudioLoader,
    split="train",
    schema_mapping={"id": "url"},
    drop_columns=["audio"],
)

Custom audio loaders​

If your dataset doesn't match any built-in loader, subclass BaseAudioLoader. The SDK requires three things from the subclass: a unique loader_key string, a loader_url pointing to the parquet, and a __call__ method that returns an AudioSignal for a given sample id.

The following template shows the minimum structure:

import soundfile as sf
import io
import polars as pl
from treble_tsdk.scene.audioloaders import BaseAudioLoader
from treble_tsdk.results.audio_signal import AudioSignal


class MyCustomAudioLoader(BaseAudioLoader):
    def __init__(self, parquet_url: str, cache_directory=None):
        # The first argument is the loader_key.
        # It must be unique across all loader classes in your project.
        super().__init__("MyCustomAudioLoader", parquet_url, cache_directory)

    def __call__(self, id: str, metadata: dict | None = None):
        # Call the parent to resolve and materialize the parquet.
        super().__call__(id, metadata)

        # Look up the row by id. Adjust the filter column to match your parquet schema.
        row = self._df.filter(pl.col("id") == id).row(0, named=True)

        # Decode audio from embedded bytes. Adjust to match your parquet structure.
        audio, sampling_rate = sf.read(io.BytesIO(row["audio"]["bytes"]))
        if audio.ndim == 2:
            audio = audio.T[0]  # take the first channel if stereo
        audio_signal = AudioSignal(audio, sampling_rate)

        # Apply trimming if the scene system has stored crop markers in metadata.
        if metadata is not None and "audio_signal_trim" in metadata:
            return audio_signal.crop(metadata["audio_signal_trim"][0], metadata["audio_signal_trim"][1])
        return audio_signal

Pass this loader to the AudioDataset constructor, or use it with from_huggingface() by passing it as audio_loader_class.