Video to Notes
The first complete† visual piano transcription system.
It reads the keyboard and writes full MIDI from silent video alone.
Watch it transcribe, note by note.
A real recording plays on top. Below, V2N's prediction lines up against the ground truth. Pick a performance, and switch between the real audio and V2N's transcription.
Filled bars are V2N, outlines are the ground truth. Where they line up, the model got it right, across onset, offset, key hold, and velocity. Nothing runs live. These are saved predictions, so you can check them against the truth and against the sound.
Audio cannot tell when a key lifts.
Press the sustain pedal and a note keeps sounding long after the key lifts. Audio cannot tell when the key lifted, only when the sound fades. Video sees the key itself.
Physical key release
On video, you can see whether a key is down or up. So V2N catches the exact moment it lifts, even when the pedal hides that from audio.
Velocity, from sight
The first video-only system to read note velocity, how hard each key is struck, from the motion alone rather than the sound.
When audio fails
Video still works when the sound is silent, noisy, reverberant, or mixed with other instruments.
One backbone, four heads.
Onset
The frame a key goes down.
Offset
The frame it physically lifts.
Key hold
The full span it stays pressed.
Velocity
How hard it was struck.
State of the art, and the first with velocity.
Read the columns as cumulative. Onset counts a note when its pitch and onset land within the tolerance. +Off also requires the offset to match, +Vel also requires velocity within 10 percent, and +Off+Vel requires all four at once.
| Method | Onset | +Off | +Vel | +Off+Vel |
|---|---|---|---|---|
| Trained on PianoVAM, tested on PianoVAM | ||||
| S2S [1] | 60.1 | 21.8 | (39.4) | (14.3) |
| V2R [2] | 86.5 | 55.4 | (58.5) | (36.8) |
| Li et al. [3] | 94.2 | (21.0) | (63.6) | (12.8) |
| PPAN [4] | 84.9 | 45.9 | (57.0) | (29.7) |
| V2N | 94.7 | 89.5 | 82.8 | 78.3 |
| Trained on R3s+R3x, tested on R3s | ||||
| S2S | 44.0 | 17.2 | (24.9) | (9.8) |
| V2R | 38.7 | 15.8 | (21.6) | (8.8) |
| PPAN | 40.7 | 14.9 | (23.4) | (8.7) |
| V2N | 78.8 | 69.5 | 57.9 | 51.2 |
| Trained on R3s+R3x, tested on R3x | ||||
| S2S | 46.6 | 14.7 | (33.5) | (10.3) |
| V2R | 46.4 | 14.9 | (33.7) | (10.3) |
| PPAN | 47.7 | 13.9 | (33.7) | (9.3) |
| V2N | 73.3 | 68.5 | 59.7 | 56.6 |
What each part adds.
Multi-task heads and multi-frame context. We add each one at a time and measure the gain on PianoVAM, F1 at 50 ms.
Multi-task heads
Predicting every attribute at once lets the heads teach each other. Velocity climbs from a constant-value proxy to real dynamics, and dedicated offset supervision is what unlocks physical key release.
| Heads trained | Onset | +Off | +Vel |
|---|---|---|---|
| Key hold | 92.8 | 81.4 | (62.3) |
| + velocity | 92.7 | 85.7 | 80.6 |
| + onset | 94.3 | 86.7 | 82.8 |
| + offset | 94.7 | 89.5 | 82.8 |
Multi-frame context
Seeing more of the performance at once compounds. Supervising every frame, modeling the frames as a sequence, and widening the window each add accuracy, and the long context is decisive on the harder R3 repertoire.
| Model | Onset | +Off | +Vel |
|---|---|---|---|
| Center-frame loss 0.2 s | 90.5 | 78.5 | 77.4 |
| + multi-frame loss 0.2 s | 91.8 | 83.0 | 78.5 |
| + sequence model 0.2 s | 94.3 | 88.7 | 80.5 |
| + longer context 1 s | 94.7 | 89.5 | 82.8 |
Each row adds one component over the row above. Bold marks the best score in each column.
Code, checkpoint, dataset.
@inproceedings{kim2026v2n,
title={Multi-Task Multi-Frame Visual Piano Transcription},
author={Yonghyun Kim and Hoyeol Sohn and Juhan Nam and Alexander Lerch},
booktitle={Proceedings of the 27th International Society
for Music Information Retrieval Conference (ISMIR)},
year={2026},
address={Abu Dhabi, UAE}
}
†Complete refers to the full set of MIDI note attributes recovered from video alone: onset, offset, key hold, and velocity. It is not full symbolic notation, so fingering, articulation, and other expressive markings are out of scope. Prior visual systems focus on onset and leave offset accuracy far behind, and V2N is the first video-only system to report note-level velocity.
[1]S2S, A. S. Koepke et al. Sight to Sound: An End-to-End Approach for Visual Piano Transcription. ICASSP 2020.
[2]V2R, K. Su et al. Audeo: Audio Generation for a Silent Performance Video. NeurIPS 2020.
[3]Li et al., Y. Li et al. A Two-Stage Audio-Visual Fusion Piano Transcription Model Based on the Attention Mechanism. IEEE/ACM TASLP 2024.
[4]PPAN, U. Zivanović et al. Pay Attention to the Keys: Visual Piano Transcription Using Transformers. IJCAI 2025.