Commit
feat(clap): zero-shot sample type + instrument classification
Commit details
Commit notes
Use CLAP's shared text/audio space to classify samples instead of the heuristic spectral/envelope rules engine. The audio embedding already computed for semantic search is now also compared against a versioned bank of text prompts (taxonomy.py) and softmaxed per group to label each sample's sample type (one-shot/loop/phrase), instrument (kick, 808, vocal, ...) and tonality (tonal vs percussive). No training and no model files required.
CLAP service: - taxonomy.py: versioned prompt bank + instrument->family map - classifier.py: dependency-free cosine/softmax scoring (unit-tested) - clap_model.py: cache label-prompt embeddings, embed_and_classify() - main.py: /classify and /classify-s3 return embedding + classification from a single forward pass; warm label cache on startup - tests/test_classifier.py: 17 tests covering taxonomy + scoring
Backend: - clap-client.ts: typed classifyAudioBytes / classifyAudioFromS3Key - worker.ts: CLAP job now persists clap_embedding + sample_type, instrument, instrument_confidence, classification_source, and fills category when empty - schema + migration 0025: additive sample_type/instrument columns + indexes - clap-search.ts: filter by instrument / sample_type - backfill script: also enqueues rows missing classification
Co-Authored-By: Claude Opus 4.8 <[email redacted]> Claude-Session: [private session redacted]
- Files changed
- 14
- Lines added
- +931
- Lines removed
- −28