Audio datasets

An AudioDataset is a lightweight, parquet-backed collection of audio samples that TrackGenerator, SceneGenerator, and SceneCollection consume as source content. It separates the metadata index (a Polars DataFrame with columns such as id, path, speaker_id, transcript, and length_s) from the audio bytes, which are loaded on demand through an audio loader.

You can create an AudioDataset in three ways:

• from a cloud repository such as HuggingFace, using from_huggingface() with a matching audio loader
• directly via the AudioDataset constructor, when you already have a parquet manifest and a loader instance
• from local WAV files on disk, using from_local_wav_directory() or from_local_speech_directory()

All three paths write or consume a parquet manifest. This manifest is the portable unit you can save, reload across sessions, move between machines, or share with colleagues — as long as the WAV files travel with it at the same relative paths.

The SDK ships built-in audio loaders for common public datasets. Pass one of these as audio_loader_class when calling from_huggingface(), or instantiate one directly when constructing an AudioDataset manually:

• LibriSpeechAudioLoader — LibriSpeech and LibriSpeech-derived datasets with embedded audio bytes and a text transcript column
• AudioSetAudioLoader — AudioSet with embedded bytes or HuggingFace-hosted paths, identified by a configurable column that defaults to video_id
• CommonVoiceAudioLoader — Common Voice 17.0 with embedded audio and a sentence transcript column
• GigaSpeechAudioLoader — GigaSpeech with embedded audio and begin_time/end_time columns for automatic duration computation
• VoxPopuliAudioLoader — VoxPopuli with embedded audio and a normalized_text transcript column
• TedliumAudioLoader — TED-LIUM Release 3 with embedded audio and a text transcript column
• MultilingualLibriSpeechAudioLoader — Multilingual LibriSpeech with embedded audio and an audio_duration column
• FMAAudioLoader — Free Music Archive with embedded audio, identified by url
• RowIndexAudioLoader — datasets with only an embedded audio column and no natural identifier; rows are addressed positionally
• AudioPathAudioLoader — general-purpose loader for datasets where each row carries a file path to a local WAV file

If none of these match your dataset, see Custom audio loaders.

Basic examples

Dataset from Hugging Face

Use from_huggingface() to load speech datasets hosted on the HuggingFace Hub. The method discovers parquet shards via the HuggingFace API, downloads and caches them locally, and instantiates the appropriate audio loader.

The following example loads LibriSpeech, a dataset derived from OpenSLR's LibriSpeech ASR corpus, using LibriSpeechAudioLoader:

tip

Prefer this pre-sliced treble-technologies/librispeech_asr_sliced repository over the raw openslr/librispeech_asr corpus for speech content. It's split into short utterances rather than long-form recordings, so it already matches the clip-length guideline in Speech dataset from a WAV directory, and it carries length_s without an extra enrichment pass.

import treble_tsdk.scene as sc

speech_dataset = sc.AudioDataset.from_huggingface(
    repo_id="treble-technologies/librispeech_asr_sliced",
    split="test",
    audio_loader_class=sc.LibriSpeechAudioLoader,
    schema_mapping={"transcript": "text"},
    drop_columns=["audio"],
)

schema_mapping maps logical column names used by the SDK (such as transcript) to the actual column names in the parquet. drop_columns drops the embedded-audio column from the metadata index after the audio loader is in place, since the loader itself keeps a lazy handle on the parquet and reads the underlying audio bytes on demand.

Use max_parquet_files to limit how many shards are loaded. This is useful for large datasets during development.

from_huggingface() also looks up the repository's license from its HuggingFace metadata and exposes it as dataset.license, so you know under which terms the audio you simulate with may be used without having to check the repository page manually:

print(speech_dataset.license)

Use filter_collection() to apply a Polars expression filter after loading and narrow the dataset down to matching rows. The following example loads AudioSet and filters it down to clips labeled as air conditioning noise:

import polars as pl

hvac_dataset = sc.AudioDataset.from_huggingface(
    repo_id="agkphysics/AudioSet",
    config="full",
    split="bal_train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "video_id"},
    drop_columns=["audio"],
    max_parquet_files=1,
)
hvac_dataset = hvac_dataset.filter_collection(
    pl.col("human_labels").list.join(" ").str.to_lowercase().str.contains(r"air conditioning|air conditioner")
)

See Public dataset loaders for examples covering the other built-in loaders.

Direct construction from a parquet manifest

Use the AudioDataset constructor directly when you already have a parquet manifest and a loader instance and don't need the scanning or discovery logic provided by the factory methods. This is the lowest-level path and gives you full control over every parameter.

The constructor accepts:

• parquet_url: a single path or URL, or a list of paths/URLs for multi-shard datasets
• audio_loader: a loader instance responsible for resolving sample identifiers to audio
• schema_mapping: a dict mapping logical column names to the actual column names in the parquet
• drop_columns: a list of column names to discard when loading

Pass a single path and a pre-configured loader:

from pathlib import Path
parquet_path = Path("<path_to_dataset>") / "dataset.parquet"

dataset = sc.AudioDataset(
    parquet_url=parquet_path,
    audio_loader=sc.AudioPathAudioLoader(parquet_path)
)

Speech dataset from a WAV directory

Use from_local_speech_directory() when you need per-clip speaker identity, duration, and transcripts. This dataset type works directly with ConversationRules and talker_identifier in TrackGenerator.

tip

ConversationRules works best when each talker's clips are short — seconds, not multi-minute recordings. Slice long source audio into utterance-length clips before loading rather than after; loading long-form recordings as-is produces one long block per talker instead of the turn-taking ConversationRules is meant to generate.

You can provide transcripts in two ways:

• as sibling .txt files placed next to each WAV, read automatically during the scan
• from an external source such as a CSV file, added after the dataset is created using add_column() — see Transcripts from an external CSV

Organize the directory with one subfolder per talker. The subfolder name becomes the talker identifier. Place a sibling .txt file next to each WAV to provide its transcript:

speech_dir/
    alice/
        001.wav
        001.txt    # transcript for 001.wav
        002.wav
        002.txt
    bob/
        001.wav
        001.txt

Create the dataset by pointing from_local_speech_directory() at this layout. This writes a parquet manifest to parquet_path as a side effect, so once that file exists, load it directly instead of rescanning the directory:

speech_dir = Path("<path_to_speech_directory>")
parquet_path = speech_dir / "speech_dataset.parquet"

if parquet_path.exists():
    speech_dataset = sc.AudioDataset(
        parquet_url=parquet_path,
        audio_loader=sc.AudioPathAudioLoader(parquet_path),
    )
else:
    speech_dataset = sc.AudioDataset.from_local_speech_directory(
        speech_dir=speech_dir,
        parquet_path=parquet_path,
        load_transcripts=True,
    )

Pass this dataset directly to TrackGenerator, using the same talker_identifier string (default: "speaker_id"):

conversation = sc.TrackGenerator(
    audio_dataset=speech_dataset,
    rules=sc.ConversationRules(
        block_duration_range=(1, 15),
        overlap_range=(0.2, 0.5),
        in_track_level_range_db_spl=(64, 70),
    ),
    talker_identifier="speaker_id",
)

By default, WAV files without a sibling .txt file receive an empty transcript and a warning is logged. Set load_transcripts=True to raise an error instead when any transcript is missing.

caution

If parquet_path already exists, from_local_speech_directory() and from_local_wav_directory() load the existing manifest (with a warning) instead of rescanning the directory. Delete the parquet file first if the directory contents have changed and the manifest must be rebuilt.

Other audio from a WAV directory

Use from_local_wav_directory() when your WAVs are flat or loosely organized and you don't need per-clip speaker labels or transcripts. This is the typical path for music, background noise, or other non-speech content.

The method scans a directory for *.wav files (recursively by default), writes a parquet manifest recording relative paths, and returns a ready-to-use AudioDataset backed by AudioPathAudioLoader. As with from_local_speech_directory(), reload the parquet directly once it exists instead of rescanning the directory on every run:

music_dir = Path("<path_to_wav_directory>")
parquet_path = music_dir / "music_dataset.parquet"

if parquet_path.exists():
    music_dataset = sc.AudioDataset(
        parquet_url=parquet_path,
        audio_loader=sc.AudioPathAudioLoader(parquet_path),
    )
else:
    music_dataset = sc.AudioDataset.from_local_wav_directory(
        wav_dir=music_dir,
        parquet_path=parquet_path,
    )

Pass search_subdir=False to limit the scan to the top level of the directory, ignoring subdirectories.

Advanced examples

Transcripts from an external CSV

If transcripts are stored separately from the WAVs — for example, in a single CSV file — create the dataset without transcripts, then populate the column using add_column().

add_column() only updates the in-memory DataFrame; it doesn't modify the parquet file written by from_local_speech_directory(). Call dataset.dataframe.write_parquet() afterward to persist the enriched manifest, so future sessions can reload it without repeating the enrichment step.

import csv

speech_dir = Path("<path_to_speech_directory>")
parquet_path = speech_dir / "speech_dataset.parquet"

speech_dataset = sc.AudioDataset.from_local_speech_directory(
    speech_dir=speech_dir,
    parquet_path=parquet_path
)

# Build a lookup from the CSV: {sample_id -> transcript}
with open("<path_to_transcripts_csv>", newline="", encoding="utf-8") as f:
    transcript_lookup = {row["id"]: row["transcript"] for row in csv.DictReader(f)}

speech_dataset.add_column(
    name="transcript",
    mapper=lambda ref: transcript_lookup.get(ref.id, ""),
)

# Persist the enriched manifest so future sessions can reload it directly.
speech_dataset.dataframe.write_parquet(parquet_path)

add_column() accepts any callable that takes an AudioSampleReference and returns a scalar value. You can also use it to add custom metadata columns for downstream filtering with Polars expressions.

Dataset reload from parquet

Once the parquet manifest has been written, you can skip rescanning and reload the dataset directly. Use this path in subsequent sessions or after moving the dataset to another machine.

Call the AudioDataset constructor with the parquet path and an AudioPathAudioLoader pointing at the same file:

speech_dataset = sc.AudioDataset(
    parquet_url=parquet_path,
    audio_loader=sc.AudioPathAudioLoader(parquet_path),
)

AudioPathAudioLoader resolves audio paths relative to the parquet file's parent directory. This means the dataset's portable: copy the parquet and the WAV files to a new location and they'll load correctly, as long as the relative layout between them stays the same.

Column mapping and dropping

Use schema_mapping when the parquet uses different column names from the logical names the SDK expects (such as id, transcript, length_s). Use drop_columns to strip large columns — such as embedded audio bytes — from AudioDataset's own metadata table, so filtering and indexing don't hold those bytes in memory.

drop_columns does not affect audio loading. The audio loader keeps its own independent, lazy handle on the same parquet and reads audio bytes from that on demand — it never receives drop_columns at all. Dropping the embedded audio column from the metadata table is therefore safe even when the loader still needs those bytes.

dataset = sc.AudioDataset(
    parquet_url="<path_or_url_to_parquet>",
    audio_loader=MyCustomAudioLoader("<path_or_url_to_parquet>"),
    schema_mapping={"id": "clip_id", "transcript": "text"},  # logical_name -> actual_column_name
    drop_columns=["audio"],
)

Any parquet column you don't map keeps its original name in the resulting dataframe — schema_mapping only needs entries for columns whose logical name differs from their actual name.

Logical columns expected by TrackGenerator, SceneGenerator, and AudioScene

schema_mapping exists to align your parquet's own column names with the logical column names that TrackGenerator, SceneGenerator, and AudioScene read by convention. Map a logical name only when your parquet's column is named differently; an unmapped column already named correctly needs no entry.

Logical name	Required when	Meaning
id	Always	Unique identifier for each sample, used to look up audio through the audio loader. If omitted from schema_mapping and not already present, AudioDataset synthesizes one from the parquet shard and row position.
transcript	Reading transcripts via AudioScene.transcript(), or any other transcript-dependent workflow	The text spoken in the sample. Optional otherwise — datasets without it simply have no transcript.
length_s	Using ConversationRules (TrackGenerator filters candidate blocks by block_duration_range against this column)	Clip duration in seconds. If both begin_time and end_time are present as columns (mapped or not), AudioDataset computes length_s from them automatically and drops the two source columns. Otherwise, map it directly to an existing duration column, as shown for MultilingualLibriSpeechAudioLoader's audio_duration column.

enrich_with_spl() adds two further columns, spl and spl_octave_bands, used only for SNR prediction (enrich_with_predicted_snr() on a scene or SceneCollection — see the Scene collections how-to). These are fixed output column names, not something you map through schema_mapping.

Beyond these, SceneGenerator and SourceGroup don't read any further logical columns from the audio dataset directly. Any other column your parquet carries needs no entry in schema_mapping unless you also want to rename it.

Metadata cache

When the constructor reads many shards, it reassembles the full DataFrame on every instantiation. Pass metadata_cache_path to write the assembled DataFrame to a local parquet file on first load and read from that cache on subsequent calls:

cache_path = Path("<path_to_cache_dir>") / "assembled_metadata.parquet"

dataset = sc.AudioDataset(
    parquet_url=["<shard_url_1>", "<shard_url_2>"],
    audio_loader=MyCustomAudioLoader("<shard_url_1>"),
    metadata_cache_path=cache_path,
)

On subsequent runs, the constructor reads cache_path directly and skips the per-shard scan. Delete the cache file to force a rebuild.

Public dataset loaders

Each built-in audio loader targets one public dataset's schema and pairs with from_huggingface() to download, cache, and index that dataset. The examples below cover the remaining loaders introduced at the top of this page. See Dataset from Hugging Face for the LibriSpeechAudioLoader example, and Direct construction from a parquet manifest for AudioPathAudioLoader.

AudioSet

Use AudioSetAudioLoader for AudioSet samples. The loader accepts either embedded audio bytes or a HuggingFace-hosted path in the audio column, and selects rows by an identifier column. Set this by passing id_column in audio_loader_kwargs to from_huggingface(), or by mapping id in schema_mapping.

tip

You can usually omit audio_loader_kwargs entirely. AudioSetAudioLoader implements get_default_loader_kwargs, which infers id_column from schema_mapping.get("id", "video_id") automatically. Pass audio_loader_kwargs explicitly only when you need to override that inference — for example, when a repo uses an identifier column that can't be expressed through schema_mapping. Loaders that don't implement get_default_loader_kwargs always require explicit audio_loader_kwargs when their constructor needs extra arguments.

The example below uses agkphysics/AudioSet, one compatible hosting of the dataset on HuggingFace:

audio_set_dataset = sc.AudioDataset.from_huggingface(
    repo_id="agkphysics/AudioSet",
    config="full",
    split="bal_train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "video_id"},
    drop_columns=["audio"],
)

The following example loads BoJack/MMAE, which already uses id as its identifier column:

noise_dataset = sc.AudioDataset.from_huggingface(
    repo_id="BoJack/MMAE",
    split="train",
    audio_loader_class=sc.AudioSetAudioLoader,
    schema_mapping={"id": "id"},
    drop_columns=["audio"],
)

Common Voice

Use CommonVoiceAudioLoader for Common Voice 17.0 samples. Each row is identified by path, and the sentence column holds the transcript:

common_voice_dataset = sc.AudioDataset.from_huggingface(
    repo_id="fixie-ai/common_voice_17_0",
    audio_loader_class=sc.CommonVoiceAudioLoader,
    config="en",
    split="train",
    schema_mapping={"id": "path", "transcript": "sentence"},
    drop_columns=["audio"],
)

GigaSpeech

Use GigaSpeechAudioLoader for GigaSpeech samples, identified by segment_id. Map begin_time and end_time so AudioDataset computes length_s automatically:

gigaspeech_dataset = sc.AudioDataset.from_huggingface(
    repo_id="fixie-ai/gigaspeech",
    audio_loader_class=sc.GigaSpeechAudioLoader,
    config="xl",
    split="train",
    schema_mapping={
        "id": "segment_id",
        "transcript": "text",
        "begin_time": "begin_time",
        "end_time": "end_time",
    },
    drop_columns=["audio"],
)

VoxPopuli

Use VoxPopuliAudioLoader for VoxPopuli samples, identified by audio_id. The normalized_text column holds the transcript:

voxpopuli_dataset = sc.AudioDataset.from_huggingface(
    repo_id="facebook/voxpopuli",
    audio_loader_class=sc.VoxPopuliAudioLoader,
    config="en",
    split="train",
    schema_mapping={"id": "audio_id", "transcript": "normalized_text"},
    drop_columns=["audio"],
)

TED-LIUM

Use TedliumAudioLoader for TED-LIUM Release 3 samples. The text column holds the transcript:

tedlium_dataset = sc.AudioDataset.from_huggingface(
    repo_id="sanchit-gandhi/tedlium-data",
    audio_loader_class=sc.TedliumAudioLoader,
    split="train",
    schema_mapping={"transcript": "text"},
    drop_columns=["audio"],
)

Multilingual LibriSpeech

Use MultilingualLibriSpeechAudioLoader for Multilingual LibriSpeech samples. Map length_s to the dataset's audio_duration column so clip durations are available without decoding audio:

mls_dataset = sc.AudioDataset.from_huggingface(
    repo_id="facebook/multilingual_librispeech",
    audio_loader_class=sc.MultilingualLibriSpeechAudioLoader,
    config="english",
    split="train",
    schema_mapping={"length_s": "audio_duration"},
    drop_columns=["audio"],
)

Free Music Archive

Use FMAAudioLoader for Free Music Archive samples, identified by url:

fma_dataset = sc.AudioDataset.from_huggingface(
    repo_id="benjamin-paine/free-music-archive-full",
    audio_loader_class=sc.FMAAudioLoader,
    split="train",
    schema_mapping={"id": "url"},
    drop_columns=["audio"],
)

Datasets with no identifier column

Use RowIndexAudioLoader for a dataset that carries only an embedded audio column and no column suited as an identifier. Omit an id entry from schema_mapping so AudioDataset synthesizes one from the row's position instead:

noise_dataset = sc.AudioDataset.from_huggingface(
    repo_id="<repo_id>",
    split="train",
    audio_loader_class=sc.RowIndexAudioLoader,
    drop_columns=["audio"],
)

Custom audio loaders

If your dataset doesn't match any built-in loader, subclass BaseAudioLoader. The SDK requires three things from the subclass: a unique loader_key string, a loader_url pointing to the parquet, and a __call__ method that returns an AudioSignal for a given sample id.

tip

Before writing one, confirm the dataset actually contains audio to load. Not every dataset that sounds audio-related has any — for example, a dataset of captions describing sounds, with no embedded bytes and no path or URL column pointing at a media file, isn't an audio dataset in this sense, no matter what loader logic is written. Check the repository's dataset viewer on Hugging Face, or inspect a shard's schema directly with pl.scan_parquet(url).collect_schema(), for an audio-typed or path/URL column before assuming a custom loader is the right tool.

The following template shows the minimum structure:

import soundfile as sf
import io
import polars as pl
from treble_tsdk.scene.audioloaders import BaseAudioLoader
from treble_tsdk.results.audio_signal import AudioSignal


class MyCustomAudioLoader(BaseAudioLoader):
    def __init__(self, parquet_url: str, cache_directory=None):
        # The first argument is the loader_key. It must be unique across every loader
        # class used in the project — it's the key BaseAudioLoader registers itself
        # under, and a collision raises ValueError.
        super().__init__("MyCustomAudioLoader", parquet_url, cache_directory)
        self._id_column = "id"  # enables an O(1) id -> row lookup, set after super().__init__()

    def __call__(self, id: str, metadata: dict | None = None):
        # Always call the parent first — it lazily downloads/reads the parquet, cached to disk.
        super().__call__(id, metadata)

        # Uses _id_column; O(1) once the parquet is materialized, unlike filtering by hand.
        row = self._row_for_id(id)

        # Decode audio from embedded bytes. Adjust to match your parquet structure.
        audio, sampling_rate = sf.read(io.BytesIO(row["audio"]["bytes"]))
        if audio.ndim == 2:
            audio = audio.T[0]  # sources are mono — downmix stereo rather than keeping both channels
        audio_signal = AudioSignal(audio, sampling_rate)

        # Required, not optional: every loader must honor a trim window set on the row's
        # metadata — see "Trimming samples during loading" below.
        if metadata is not None and "audio_signal_trim" in metadata:
            return audio_signal.crop(*metadata["audio_signal_trim"])
        return audio_signal

Pass this loader to the AudioDataset constructor, or use it with from_huggingface() by passing it as audio_loader_class.

A few points where the template above matters beyond the minimum structure:

• Set self._id_column and look up rows with self._row_for_id(), not self._df.filter(pl.col(...) == id).row(0, named=True) by hand. BaseAudioLoader.__call__ builds an id-to-row-index map from _id_column once, during materialization; skipping it falls back to a linear scan per sample, which is fine for a quick test and slow at dataset scale.
• Downmix stereo to mono. Every built-in loader except LibriSpeechAudioLoader does this — sources in a scene are mono, with any stereo or spatial effect applied later by room or device rendering, not baked into the source recording.
• Match the I/O pattern to the closest built-in loader rather than inventing a new one: embedded bytes in the parquet (LibriSpeechAudioLoader), an embedded-bytes-or-remote-URL fallback (AudioSetAudioLoader, RowIndexAudioLoader), a path column resolved relative to the parquet's own directory (AudioPathAudioLoader), or a remote URL downloaded and cached to disk on first access (AnechoicSoundsAudioLoader).
• Store extra constructor arguments (for example, AudioSetAudioLoader's id_column, or AudioPathAudioLoader's path_column) in self._registry_loader_kwargs, so get_registry_loader_kwargs() can reconstruct an equivalent loader instance later. This is required for a SceneCollection or AudioDataset using the loader to round-trip through parquet — without it, reloading loses whatever those extra arguments configured.