- Updated analyze_embedding to accept additional parameters for rhythm, tonal, loudness, spectral, and envelope features.
- Expanded the embedding to a 64-dimensional vector by combining MFCCs with other audio features for improved similarity search.
- Added detailed documentation on the structure of the embedding and its components.
- Ensured backward compatibility with the TypeScript implementation while normalizing the embedding for cosine similarity.
This change improves the audio analysis capabilities of the Essentia service, allowing for richer feature extraction.
- Cleaned up docker-compose.yml by standardizing quotes and formatting for better readability.
- Updated type hints in analyzer.py, main.py, and models.py to use Tuple and List from typing for consistency.
- Simplified Dockerfile by switching to the official mtgupf/essentia image, removing unnecessary build stages, and ensuring Python dependencies are installed correctly.
- Adjusted requirements.txt to reflect the pre-installed Essentia library and updated version constraints for other dependencies.
- Added essentia to dev:services script (starts with other services)
- Added dev:services:core for starting without Essentia (lightweight)
- Added dedicated Essentia commands: essentia:up, essentia:logs, essentia:build, essentia:health
- Updated AGENTS.md with Essentia service URL and commands
To run Essentia:
bun run dev:services # Starts all services including Essentia
bun run essentia:health # Check if Essentia is running
Co-authored-by: armin.naimi <[email redacted]>
All audio analysis is now exclusively handled by Python/Essentia.
If the service is unavailable, analysis fails with a clear error message.
Changes:
- Remove TypeScript analyzer caching and fallback logic
- analyzeAudioHybrid now throws if Essentia service unavailable
- Update worker thread comment
- Simplify code by removing unused fallback paths
This ensures:
- Single source of truth for audio analysis
- No hidden fallback behavior that could mask issues
- Cleaner debugging (analysis either works or fails clearly)
Co-authored-by: armin.naimi <[email redacted]>
- Add DrumClassification model and analysis to Python Essentia service
- Drum type detection using spectral and temporal features
- Classification categories: kick, snare, hihat, clap, tom, cymbal, rim, perc, loop
- Uses Essentia's spectral analysis combined with envelope characteristics
- Simplify hybrid analyzer to use Python for all classification
- Remove redundant TypeScript classification when Essentia is available
- Python/Essentia is the single source of truth for analysis
- TypeScript only used as fallback when Python service is unavailable
- Improve Essentia client reliability
- Add health check retry with exponential backoff (3 retries, 500ms/1s/2s)
- Adaptive cache duration based on service health
- Track consecutive failures to avoid hammering down service
Benefits:
- Single source of truth for audio analysis (easier debugging)
- Reduced tech debt (no duplicate classification logic)
- Better classification accuracy (Essentia's mature algorithms)
- More efficient (skip redundant analysis when Essentia succeeds)
Co-authored-by: armin.naimi <[email redacted]>
- Fixed port mismatch in env.example: Essentia is exposed on 3004 (not 3003)
when running the backend locally with Essentia in Docker
- Added comprehensive logging to track Essentia service availability checks
- Added startup check in worker to log Essentia service status
- Added Essentia status to worker health endpoint for monitoring
- Enhanced hybrid analyzer logging to track analysis mode (Essentia/TypeScript)
- Added Essentia section to .env.docker.example for Docker deployments
The issue was that the backend was trying to connect to localhost:3003 but
docker-compose exposes Essentia on port 3004 externally (3003 internally).
Now logs will clearly show whether Essentia is being used or falling back
to TypeScript analysis.
Co-authored-by: armin.naimi <[email redacted]>
- Remove libavresample-dev (deprecated in FFmpeg 4.0, removed in 5.x)
libswresample-dev is already included and provides the same functionality
- Replace libfftw3-3 with libfftw3-double3 (package renamed in Bookworm)
Co-authored-by: armin.naimi <[email redacted]>
Scaling improvements to Python Essentia service:
## ProcessPoolExecutor Architecture
- Each analysis runs in a separate process (bypasses Python GIL)
- Pool size = CPU cores - 1 (leave headroom for main process)
- Workers are pre-initialized with Essentia algorithms on startup
- Cross-process serialization via Pydantic model_dump()
## Parallel Batch Processing
- /analyze/batch endpoint processes files concurrently
- Semaphore-based concurrency control (configurable max_concurrent)
- Maintains original file order in results
- Graceful error handling per file
## Non-blocking Async
- run_in_executor() ensures event loop isn't blocked
- FastAPI can handle multiple concurrent requests
- Single uvicorn worker (parallelism via ProcessPoolExecutor)
## New Endpoints
- GET /stats - Pool status, worker count, CPU count
## Configuration
- ESSENTIA_WORKERS - Number of analysis workers (default: cpu_count - 1)
- max_concurrent query param for batch control
This matches the scaling pattern of the Bun worker's WorkerPool but
uses multiprocessing (true parallelism) instead of worker threads.
Co-authored-by: armin.naimi <[email redacted]>
- Add LoopBpmEstimator and LoopBpmConfidence for loop detection
- Enhanced EssentiaResult to include loop analysis:
- isLoop: whether audio is a seamless loop
- loopBpm: BPM detected specifically for loops
- loopConfidence: confidence score for loop detection
- bars: number of bars if beat-aligned
- beatAligned: whether duration aligns with beat grid
- Update hybrid-analyzer to use Essentia for all three:
- BPM: Uses RhythmExtractor2013 or LoopBpmEstimator
- Key: Uses KeyExtractor with EDMA/Krumhansl/Temperley profiles
- Loop: Uses LoopBpmConfidence for loop quality assessment
- Add loopBars, suggestedTrimMs, beatPositions to AudioFeatures interface
- Update type declarations for essentia.js with new algorithms
Co-authored-by: armin.naimi <[email redacted]>
- Install essentia.js npm package for WebAssembly-based audio analysis
- Rewrite essentia.ts to use essentia.js library correctly
- Use pre-initialized WASM module (not factory function)
- Implement RhythmExtractor2013 for BPM detection
- Implement KeyExtractor with EDMA, Krumhansl, Temperley profiles
- Implement Danceability estimation
- Use FFmpeg for reliable audio decoding to Float32Array
- Update hybrid-analyzer.ts to work with new essentia.js API
- Run Essentia.js and TypeScript analysis in parallel
- Merge results intelligently for best accuracy
- Build key consensus from multiple Essentia profiles
- Add TypeScript type declarations for essentia.js module
- Remove Python service approach in favor of JS bindings
Co-authored-by: armin.naimi <[email redacted]>
- Updated audio analysis worker to use optional chaining and nullish coalescing for key detection properties.
- Ensures that musicalKey, keyMode, keyConfidence, and camelotCode default to null if keyResult is not defined, improving robustness of audio analysis.
- Create new shared module: backend/src/audio/key-detection.ts
- Contains all key detection logic, constants, and profiles
- Exports: detectMusicalKey, extractKeyFromFilename, getCamelotCode
- Exports: KEY_NAMES, CAMELOT_MAJOR, CAMELOT_MINOR, KEY_PROFILES, etc.
- Update audio-analysis.ts to import from shared module (~1000 lines removed)
- Update audio-analysis-worker.ts to import from shared module (~250 lines removed)
- Both files now use the same key detection implementation
- All tests pass
Co-authored-by: armin.naimi <[email redacted]>
This commit significantly improves audio analysis performance by:
1. Worker Thread Pool (src/workers/pool.ts):
- Creates a pool of worker threads based on available CPU cores
- Automatic load balancing across workers
- Task queuing when all workers are busy
- Graceful shutdown with task completion
2. Dedicated Audio Analysis Worker (src/workers/audio-analysis-worker.ts):
- Runs CPU-intensive FFT, spectral analysis, BPM detection in separate threads
- Parallel execution of independent analysis tasks within each worker
- Complete audio feature extraction without blocking main event loop
3. Increased Queue Concurrency:
- Audio analysis queue: 3 -> 8 concurrent workers
- Guest sample processing: 2 -> 6 concurrent workers
- Reduced job timeouts (analysis is now faster)
4. Integration:
- analyzeAudioFromFileWorker() function for worker-based analysis
- Automatic fallback to main thread on worker failure
- Proper shutdown handling for worker pool
Expected performance improvements:
- 3-5x faster throughput on multi-core systems
- Main event loop remains responsive during analysis
- Better handling of batch uploads from multiple users
Co-authored-by: armin.naimi <[email redacted]>
Reverted unified processing queue - the separate queue architecture
is intentional because:
- Processing (preview/sequencer) is fast (~2-3s) → instant upload feel
- Analysis (BPM, key, AI tagging) is slow (~10-30s) → background work
Retained optimizations:
1. Batch AI Tagging - accumulates requests, sends up to 10 samples
per API call (80-90% reduction in AI API latency)
2. Loop Detection Early Exit - skips expensive analysis when metadata
or filename clearly indicates loop status
3. Existing metadata skip for BPM/key detection
Updated PROCESSING_OPTIMIZATIONS.md with correct architecture explanation
Co-authored-by: armin.naimi <[email redacted]>
Key optimizations implemented:
1. Unified Processing Queue
- Combined audio processing + analysis into single job
- Downloads file ONCE from S3 instead of twice
- ~50% reduction in S3 bandwidth and latency
2. Batch AI Tagging
- Accumulates tagging requests and sends in batches (up to 10)
- Single API call instead of 10 separate calls
- ~80-90% reduction in AI API latency overhead
3. Quick Metadata Skip Logic
- Loop detection now has early exit for metadata tags
- Combined with existing BPM/key filename detection
- Instant results for well-labeled samples (Splice, Loopmasters, etc.)
4. New unifiedProcessingQueue
- Concurrency: 4 workers
- Replaces parallel audio-processing + audio-analysis queues
- Simpler job flow, better error handling
Expected performance improvement for 100-sample batch:
- Before: ~10 minutes
- After: ~2 minutes (5x faster)
Documentation: See PROCESSING_OPTIMIZATIONS.md for full analysis
Co-authored-by: armin.naimi <[email redacted]>
- Added support for retrying failed uploads, including tracking retry attempts and managing upload state.
- Enhanced the upload queue to store file references and relative paths for better retry capability.
- Updated the UploadProgressPanel to include retry and clear failed uploads options.
- Introduced methods in LibraryState to handle failed uploads and mark them for retry.
Files modified:
- frontend/src/lib/api/upload.ts
- frontend/src/routes/library/+page.svelte
- frontend/src/routes/library/library-state.svelte.ts
- frontend/src/routes/library/components/UploadDropzone.svelte
- frontend/src/routes/library/components/UploadProgressPanel.svelte
- Added detailed event handling for audio processing stages, including success and failure notifications.
- Introduced a new ProcessingStatusIndicator component to visually represent processing states in the UI.
- Updated backend event emissions to include processing stages for better tracking.
- Enhanced library routes to support new error and analysis status fields.
Files modified:
- backend/src/worker.ts
- backend/src/routes/library.ts
- backend/src/routes/upload-links.ts
- backend/src/services/library-events.ts
- frontend/src/lib/components/ProcessingStatusIndicator.svelte
- frontend/src/lib/hooks/useLibrarySSE.svelte.ts
- frontend/src/routes/library/+page.svelte
- frontend/src/routes/library/components/LibrarySampleGrid.svelte
- Introduced a logo image in the header component for branding.
- Updated the import statement to include the asset function for proper asset resolution.
Files modified:
- frontend/src/lib/components/Header.svelte
- frontend/static/grainstash-black.svg (new file)
Ensure Biome extension doesn't interfere with Svelte file formatting.
The frontend uses ESLint + Prettier, while Biome is only for backend.
Co-authored-by: armin.naimi <[email redacted]>
- Remove old .eslintrc.cjs (ESLint 8 format conflicting with ESLint 9)
- Rewrite eslint.config.js with proper flat config for ESLint 9
- Configure proper Svelte 5 support with eslint-plugin-svelte
- Add VS Code/Cursor settings for format-on-save
- Add recommended extensions for Svelte, Prettier, ESLint, Tailwind
The old setup had two conflicting ESLint configs (.eslintrc.cjs and
eslint.config.js) which caused issues. This consolidates to ESLint 9's
flat config format with proper TypeScript and Svelte 5 support.
Co-authored-by: armin.naimi <[email redacted]>
- Simplified health server port configuration to prioritize WORKER_HEALTH_PORT.
- Refactored library page to improve loading, error, and empty states for samples.
- Removed LibraryFilters component to streamline the library interface.
- Adjusted filter popover position and button styles for better UI consistency.
Files modified:
- backend/src/worker.ts
- frontend/src/routes/library/+page.svelte
- frontend/src/routes/library/components/LibraryFilters.svelte (deleted)
- frontend/src/routes/library/components/LibrarySearchFilter.svelte