PronounceAI: Explainable Pronunciation Coaching
An NLP pronunciation coach that scores speech at the phoneme level and renders the learner's own voice in a target accent.
Why I Built This
PronounceAI was my May 2026 NLP class project. Most pronunciation tools return one opaque score and ask the learner to try again. I wanted feedback that identifies the specific sound, shows when it happened, and gives the learner a concrete correction.
The second idea was more personal: a target-accent example should still sound like the learner. Instead of replacing their voice with a generic speaker, PronounceAI renders new sentences in their own voice with either a General American or Received Pronunciation accent.
How the Scoring Pipeline Works
One browser recording fans out across four independent signals:
- Phoneme alignment: wav2vec2 CTC forced alignment locates each phoneme and produces timestamped GOP confidence scores
- Prosody: Parselmouth and librosa compare pitch movement, timing, and rhythm
- Phrase grounding: faster-whisper checks that the learner actually spoke the requested phrase
- Holistic assessment: an optional WavLM Large assessment head provides a bounded second opinion calibrated on human ratings
The final score remains explicit rather than hiding the fusion behind another model:
overall = 0.55 × phoneme_accuracy
+ 0.20 × intonation
+ 0.15 × stress_rhythm
+ 0.10 × vowel_quality
The response includes the four component scores, per-phoneme timestamps, a pitch overlay, phrase-match status, and an actionable pronunciation tip.
Voice Cloning with Accent Control
The learner records roughly ten seconds of enrolment audio. Kokoro 82M TTS creates a deterministic source in the selected accent, then CosyVoice 3 voice conversion transfers the learner's speaker identity onto that source. Whisper validates the generated speech and returns word timings.
The result is new text in the learner's voice, rendered in either General American or Received Pronunciation. The live sample gallery includes the original recording, both accent conversions, native reference clips, and score breakdowns.
Product and Architecture
- Practice: phrase library and free-text recording with phoneme tape, pitch overlay, and targeted feedback
- Voice Lab: enrolment management and voice rendering with accent and emotion controls
- Progress: local session history, streaks, and per-phoneme mastery without requiring an account
- Settings: target accent, theme, sound effects, and profile reset
The Next.js 16 and React 19 frontend records audio through MediaRecorder. A FastAPI backend exposes five primary routes for scoring, speech synthesis, accent conversion, voice enrolment, and TTS. The scoring pipeline runs its independent model calls asynchronously, while shared caches keep repeated model and audio work bounded.
Results
| Evidence | Result |
|---|---|
| Backend test suite | 34 tests, designed to run without network access |
| Inspectable score fixtures | 4 committed audio/JSON examples |
| Learner recording | 79.5 overall score with phrase match ok |
| “Ship or sheep?” reference | 84.5 overall score |
| “She sells seashells…” reference | 92.4 overall score |
| “The quick brown fox…” reference | 92.3 overall score |
| Target accents | General American and Received Pronunciation |
| Local deployment | Docker Compose frontend + backend reviewer path |
The repository also includes a documented API contract, a stable deployment runbook, training utilities for the WavLM assessment head, and a manifest-driven evaluation pipeline for recorded fixtures.