Multi-Task Multi-Frame Visual Piano Transcription
Summary
This paper introduces V2N, the first complete visual piano transcription system that jointly predicts onset, offset, keyhold, and velocity from video, achieving state-of-the-art results on PianoVAM and R3 benchmarks.
View Cached Full Text
Cached at: 08/05/26, 01:45 PM
Paper page - Multi-Task Multi-Frame Visual Piano Transcription
Source: https://huggingface.co/papers/2608.03419
Abstract
Audio-basedpianotranscriptionperformswellononset,pitch,andvelocity,butthesustainpedalletssoundpersistlongafterkeyrelease,soaudiosystemspredictpedal-extendedoffsetsratherthanphysicalkeyrelease.YetexistingVisualPianoTranscription(VPT)systemsfocusononsetdetectionfromshortvideowindows,offsetaccuracylagsonsetbyawidemargin,andnote-levelvelocityhasnotbeenreported.Toaddressthesegaps,wepresentV2N(VideotoNotes),thefirstcompleteVPTsystem:asharedtemporalbackbonefeedstask-specificheadsforonset,offset,keyhold,andvelocity,jointlytrainedwithper-framesupervisionratherthanonlyatthewindowcenter.Ablationsshowthatmulti-tasksupervisionenablesoffsetandvelocitypredictionwhileimprovingonsetaccuracy;longertemporalcontextyieldsfurtherimprovements.V2Nsetsnewstate-of-the-artresultsonPianoVAMandR3.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2608\.03419
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.03419 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.03419 in a dataset README.md to link it from this page.
Spaces citing this paper1
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription
DuET is a dual-encoder transformer that jointly performs MIDI-to-score transcription and beat tracking to improve timing and duration accuracy in expressive piano performance transcription, achieving large MUSTER error reductions on the Gangqinpu dataset.
PianoKontext: Expressive Performance Rendering from Deadpan Context
PianoKontext generates variable-length expressive piano performances from deadpan MIDI scores by aligning audio and MIDI in latent space using Dynamic Time Warping and flow matching with DiT blocks.
Show HN: I trained a 125M model to autocomplete piano on-device
A developer trained a 125M-parameter transformer model to autocomplete piano performances in real-time on-device using MIDI data and DPO post-training, and released the RollTab app for iOS.
MuSViT: A Foundation Vision Model for Sheet Music Representation
MuSViT is the first foundation vision model for sheet music, pre-trained on millions of pages via Masked Autoencoders, achieving superior performance in score recognition and symbol detection tasks.
SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
SheetSage2 is a unified music transcription framework combining synthetic MIDI supervision, task-specific structured decoding, and autoregressive distillation, outperforming prior systems on 12 of 15 benchmark-metric pairs with publicly available model weights and code.