# Model Architecture ## Purpose and Scope This document describes the neural network architecture used in Basic Pitch's pitch detection system. It explains the model's components, how audio signals are transformed and processed through the network, and how the three primary outputs (pitch contours, notes, and onsets) are generated. For information about how the model is loaded and used for inference, see [Model Loading and Initialization](/content/spotify/basic-pitch/2.1-model-loading-and-initialization/index.html). ## Model Overview Basic Pitch uses a convolutional neural network architecture that processes audio spectrograms to detect musical notes, their onsets, and fine-grained pitch contours (including pitch bends). The model employs a multi-task learning approach with three output heads to simultaneously predict different aspects of musical content. ``` Output Interpretations Audio Input (22050Hz Mono) CQT Calculation Harmonic Stacking Contour Branch Note Branch Onset Branch Contour Posteriorgram (264x430 matrix) Note Posteriorgram (88x430 matrix) Onset Posteriorgram (88x430 matrix) Pitch contours with pitch bends Note activations at semitone level Note onset detection ``` Sources: [basic_pitch/models.py173-299](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L173-L299) [basic_pitch/constants.py18-63](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/constants.py#L18-L63) ## Input Processing and CQT Transformation Basic Pitch processes raw audio input using a Constant-Q Transform (CQT), which is a time-frequency representation particularly well-suited for music analysis as it provides logarithmically-spaced frequency bins that align with musical pitch intervals. ``` get_cqt Function Audio Input (batch, AUDIO_N_SAMPLES, 1) FlattenAudioCh Layer CQT Layer NormalizedLog Layer BatchNormalization Layer Processed CQT (batch, n_freq_bins, time_frames, 1) ``` The CQT implementation uses the following key parameters: | Parameter | Value | Description | |---------------|------------|------------------------------------------------------------| | Sample Rate | 22050 Hz | Audio input sampling rate | | Hop Length | 256 | Step size between analysis frames | | Base Frequency | 27.5 Hz | Lowest detectable frequency (A0 on piano) | | Bins Per Octave| 36 | 3 bins per semitone for contours, 1 for notes | | Total Bins | 264 | 88 semitones × 3 bins per semitone | Sources: [basic_pitch/models.py137-170](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L137-L170) [basic_pitch/constants.py22-45](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/constants.py#L22-L45) [basic_pitch/layers/nnaudio.py455-493](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/layers/nnaudio.py#L455-L493) ## Harmonic Stacking After the CQT transformation, the model applies a Harmonic Stacking layer that captures harmonic relationships by stacking frequency-shifted versions of the spectrogram. This layer helps the model identify fundamental pitches by providing explicit access to harmonic patterns. ``` Harmonic Shifts Stack channels CQT Input (batch, time, freq, 1) HarmonicStacking Layer Original (×1) Octave Below (×0.5) First Harmonic (×2) Second Harmonic (×3) ... Nth Harmonic (×N) Output (batch, time, N_FREQ_BINS_CONTOURS, n_harmonics) ``` The harmonic stacking layer: - Takes frequency bins from the CQT output - Creates frequency-shifted versions for each harmonic - Stacks these shifted versions along the channel dimension - By default, uses harmonics [0.5, 1, 2, 3, 4, 5, 6, 7, 8] (subharmonic + fundamental + 7 overtones) Sources: [basic_pitch/nn.py26-85](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/nn.py#L26-L85) [basic_pitch/models.py193-204](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L193-L204) ## Model Branches The model uses a multi-headed architecture with three main branches: ### 1. Contour Branch The contour branch predicts fine-grained pitch activation with 3 bins per semitone, enabling the detection of pitch bends and microtonal information. ``` if not no_contours Harmonic Stacked Input Conv2D (5×5) n_filters_contour=32 BatchNormalization + ReLU Conv2D (3×39) 8 filters BatchNormalization + ReLU Conv2D (5×5) 1 filter Sigmoid Activation FlattenFreqCh 'contour' output Input to Note Branch ``` Sources: [basic_pitch/models.py207-243](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L207-L243) ### 2. Note Branch The note branch produces semitone-level note activations that indicate which notes are being played, with 1 bin per semitone (standard piano note resolution). ``` Pre-flattened Contour Output or Intermediate Features Conv2D (7×7) n_filters_notes=32 stride=(1,3) ReLU Activation Conv2D (7×3) 1 filter Sigmoid Activation FlattenFreqCh 'note' output Input to Onset Branch ``` Sources: [basic_pitch/models.py247-267](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L247-L267) ### 3. Onset Branch The onset branch detects note onsets, which are the beginning points of musical notes. ``` Harmonic Stacked Input Conv2D (5×5) n_filters_onsets=32 stride=(1,3) BatchNormalization + ReLU Note Pre-output Concatenate Conv2D (3×3) 1 filter Sigmoid Activation FlattenFreqCh 'onset' output ``` Sources: [basic_pitch/models.py272-295](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L272-L295) ## Complete Model Architecture The following diagram shows the complete architecture with detailed layer dimensions: ``` Audio Input (batch, 43264, 1) CQT + Log + BatchNorm (batch, 264, 430, 1) Harmonic Stacking (batch, 264, 430, n_harmonics) Contour Conv2D (batch, 264, 430, 32) BatchNorm + ReLU Contour Conv2D (batch, 264, 430, 8) BatchNorm + ReLU Contour Conv2D (batch, 264, 430, 1) FlattenFreqCh (batch, 430, 264) Conv2D (batch, 264, 430, 1) Conv2D stride(1,3) (batch, 264, 144, 32) ReLU Note Conv2D (batch, 264, 144, 1) FlattenFreqCh (batch, 144, 88) Onset Conv2D stride(1,3) (batch, 264, 144, 32) BatchNorm + ReLU Concatenate Onset Conv2D (batch, 88, 144, 1) FlattenFreqCh (batch, 144, 88) Contour Output (batch, 430, 264) Note Output (batch, 144, 88) Onset Output (batch, 144, 88) ``` ## Loss Functions The model is trained using binary cross entropy loss functions for each output head: | Output Head | Loss Function | Description | |-------------|---------------------------------|--------------------------------------------------------------------| | Contour | `transcription_loss` | Standard binary cross entropy with label smoothing | | Note | `transcription_loss` | Standard binary cross entropy with label smoothing | | Onset | `weighted_transcription_loss` | Binary cross entropy with positive sample weighting | The model uses label smoothing to prevent overconfident predictions, which is particularly important for pitch detection where the boundaries between notes may not be perfectly defined. Sources: [basic_pitch/models.py39-126](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L39-L126) ## Model Parameters The default model configuration has the following parameters: | Parameter | Default Value | Description | |-------------------------|---------------|--------------------------------------------------| | n_harmonics | 8 | Number of harmonics used in harmonic stacking | | n_filters_contour | 32 | Number of filters for contour convolutional layers | | n_filters_notes | 32 | Number of filters for note convolutional layers | | n_filters_onsets | 32 | Number of filters for onset convolutional layers | | no_contours | False | Whether to exclude contour output (for optimization)| Sources: [basic_pitch/models.py173-178](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L173-L178) ## Implementation Details The model implementation uses several custom techniques to improve performance: 1. **Kernel Initialization**: Uses variance scaling initialization for stable training 2. **Kernel Constraints**: Applies unit norm constraints to control filter magnitudes 3. **Early Downsampling**: For efficient CQT computation on lower frequencies 4. **Multi-resolution Output**: Different frequency resolutions for different tasks (3 bins/semitone for contours, 1 bin/semitone for notes) The model is designed to be lightweight while still achieving high-quality transcription results, with a particular focus on accurate pitch contour detection that enables capturing expressive pitch characteristics like vibrato and pitch bends. Sources: [basic_pitch/models.py129-136](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/models.py#L129-L136) [basic_pitch/layers/nnaudio.py30-135](https://github.com/spotify/basic-pitch/blob/f423902a/basic_pitch/layers/nnaudio.py#L30-L135)