This is a proof-of-concept for Cantonese speech-to-text-to-jyutping. Wiring this up took… five minutes; Elixir / Nx / Livebook is a really superb combination.
The first minute of the video is colorful narration, with the technological magic starts at 1:15. (The prose is a classic internet prose, from maybe twenty years ago, called 壞過凱婷 Worse than Hoi Ting. The speaker is a Grade 7 student trying to impress on the victim that she is the most bad-ass girl in school.)
The transcript, lightly edited with corrections, is here (on mobile browsers you’ll need to download PDF to see):
The speech-to-text uses out of the box Whisper-Large-v3, with `yue` language tag. The implementation is as an Elixir / Nx Livebook app. The video was recorded on a Apple Silicon M1 (GPU is not used), the one minute audio took two minutes to transcribe; this is already a Docker app, can be readily pushed to HuggingFace for running on GPU.
The output is then copied out to Pages, displayed using the Cantonese Font v2 (2.0.21.1).
I was astonished at the combined accuracy: the editing needed are mostly for characters of the same sound (凱婷 / 海婷; 引/忍; 方溫/form one). There were only two major off-phrases: 入黑社囉, which was transcribed to 入黑寫law, and 鎅手 which was transcribed to 戒酒 (which does fit what came before). The font assignments needed to be corrected is 咁 (`gam2` / gam3) and 呀 (aa1/aa3). Vernacular, especially final particles, are general tricky but they seem to be handled reasonably well.
One of the frequently wish-listed items from learners are videos with Chinese and Jyutping subtitles. Super useful, but pain-in-the-butt to prepare. I had previously worked out that the Canto Font works just fine in Adobe Premiere Pro:
Since the transcripts are already time-stamped, we can load the transcript files in, and have a direct approach to subtitled movies / podcasts.
(Maybe I need to investigate renting GPUs, and provide a video -> closed-caption video service… 🤔)
Leave a Reply