Model Architecture

Purpose and Scope

This document describes the neural network architecture used in Basic Pitch's pitch detection system. It explains the model's components, how audio signals are transformed and processed through the network, and how the three primary outputs (pitch contours, notes, and onsets) are generated. For information about how the model is loaded and used for inference, see Model Loading and Initialization.

Model Overview

Basic Pitch uses a convolutional neural network architecture that processes audio spectrograms to detect musical notes, their onsets, and fine-grained pitch contours (including pitch bends). The model employs a multi-task learning approach with three output heads to simultaneously predict different aspects of musical content.

Output Interpretations

Audio Input
(22050Hz Mono)

CQT Calculation

Harmonic Stacking

Contour Branch

Note Branch

Onset Branch

Contour Posteriorgram
(264x430 matrix)

Note Posteriorgram
(88x430 matrix)

Onset Posteriorgram
(88x430 matrix)

Pitch contours
with pitch bends

Note activations
at semitone level

Note onset
detection

Sources: basic_pitch/models.py173-299 basic_pitch/constants.py18-63

Input Processing and CQT Transformation

Basic Pitch processes raw audio input using a Constant-Q Transform (CQT), which is a time-frequency representation particularly well-suited for music analysis as it provides logarithmically-spaced frequency bins that align with musical pitch intervals.

get_cqt Function

Audio Input
(batch, AUDIO_N_SAMPLES, 1)

FlattenAudioCh Layer

CQT Layer

NormalizedLog Layer

BatchNormalization Layer

Processed CQT
(batch, n_freq_bins, time_frames, 1)

The CQT implementation uses the following key parameters:

Sources: basic_pitch/models.py137-170 basic_pitch/constants.py22-45 basic_pitch/layers/nnaudio.py455-493

Harmonic Stacking

After the CQT transformation, the model applies a Harmonic Stacking layer that captures harmonic relationships by stacking frequency-shifted versions of the spectrogram. This layer helps the model identify fundamental pitches by providing explicit access to harmonic patterns.

Harmonic Shifts

Stack channels

CQT Input
(batch, time, freq, 1)

HarmonicStacking Layer

Original (×1)

Octave Below (×0.5)

First Harmonic (×2)

Second Harmonic (×3)

...

Nth Harmonic (×N)

Output
(batch, time, N_FREQ_BINS_CONTOURS, n_harmonics)

The harmonic stacking layer:

• Takes frequency bins from the CQT output
• Creates frequency-shifted versions for each harmonic
• Stacks these shifted versions along the channel dimension
• By default, uses harmonics [0.5, 1, 2, 3, 4, 5, 6, 7, 8] (subharmonic + fundamental + 7 overtones)

Sources: basic_pitch/nn.py26-85 basic_pitch/models.py193-204

Model Branches

The model uses a multi-headed architecture with three main branches:

1. Contour Branch

The contour branch predicts fine-grained pitch activation with 3 bins per semitone, enabling the detection of pitch bends and microtonal information.

if not no_contours

Harmonic Stacked Input

Conv2D (5×5)
n_filters_contour=32

BatchNormalization + ReLU

Conv2D (3×39)
8 filters

BatchNormalization + ReLU

Conv2D (5×5)
1 filter
Sigmoid Activation

FlattenFreqCh
'contour' output

Input to Note Branch

Sources: basic_pitch/models.py207-243

2. Note Branch

The note branch produces semitone-level note activations that indicate which notes are being played, with 1 bin per semitone (standard piano note resolution).

Pre-flattened

Contour Output
or Intermediate Features

Conv2D (7×7)
n_filters_notes=32
stride=(1,3)

ReLU Activation

Conv2D (7×3)
1 filter
Sigmoid Activation

FlattenFreqCh
'note' output

Input to Onset Branch

Sources: basic_pitch/models.py247-267

3. Onset Branch

The onset branch detects note onsets, which are the beginning points of musical notes.

Harmonic Stacked Input

Conv2D (5×5)
n_filters_onsets=32
stride=(1,3)

BatchNormalization + ReLU

Note Pre-output

Concatenate

Conv2D (3×3)
1 filter
Sigmoid Activation

FlattenFreqCh
'onset' output

Sources: basic_pitch/models.py272-295

Complete Model Architecture

The following diagram shows the complete architecture with detailed layer dimensions:

Audio Input
(batch, 43264, 1)

CQT + Log + BatchNorm
(batch, 264, 430, 1)

Harmonic Stacking
(batch, 264, 430, n_harmonics)

Contour Conv2D
(batch, 264, 430, 32)

BatchNorm + ReLU

Contour Conv2D
(batch, 264, 430, 8)

BatchNorm + ReLU

Contour Conv2D
(batch, 264, 430, 1)

FlattenFreqCh
(batch, 430, 264)

Conv2D
(batch, 264, 430, 1)

Conv2D stride(1,3)
(batch, 264, 144, 32)

ReLU

Note Conv2D
(batch, 264, 144, 1)

FlattenFreqCh
(batch, 144, 88)

Onset Conv2D stride(1,3)
(batch, 264, 144, 32)

BatchNorm + ReLU

Concatenate

Onset Conv2D
(batch, 88, 144, 1)

FlattenFreqCh
(batch, 144, 88)

Contour Output
(batch, 430, 264)

Note Output
(batch, 144, 88)

Onset Output
(batch, 144, 88)

Loss Functions

The model is trained using binary cross entropy loss functions for each output head:

The model uses label smoothing to prevent overconfident predictions, which is particularly important for pitch detection where the boundaries between notes may not be perfectly defined.

Sources: basic_pitch/models.py39-126

Model Parameters

The default model configuration has the following parameters:

Sources: basic_pitch/models.py173-178

Implementation Details

The model implementation uses several custom techniques to improve performance:

1. Kernel Initialization: Uses variance scaling initialization for stable training
2. Kernel Constraints: Applies unit norm constraints to control filter magnitudes
3. Early Downsampling: For efficient CQT computation on lower frequencies
4. Multi-resolution Output: Different frequency resolutions for different tasks (3 bins/semitone for contours, 1 bin/semitone for notes)

The model is designed to be lightweight while still achieving high-quality transcription results, with a particular focus on accurate pitch contour detection that enables capturing expressive pitch characteristics like vibrato and pitch bends.

Sources: basic_pitch/models.py129-136 basic_pitch/layers/nnaudio.py30-135