Extract a Provisional Pose Study from Performance Video
Extract a Provisional Pose Study from Performance Video
You can have a moving skeleton drawn over your performance clip before lunch. The question is whether the thing you're left holding is worth showing to anyone.
What you can honestly build from an authorized adult performance clip is a provisional overlay: a set of estimated body landmarks drawn over the original frames, tied to the source's own clock, with the weak stretches marked and left visible rather than scrubbed away. Its job is to give a room a shared object to point at while talking about a gesture — "here, this part, the arm crosses." Nothing more. It is not captured acting, not measured anatomy, and not a rig an animator can pick up. Those are three different artifacts with three different standards of evidence. The whole discipline of this exercise is refusing to let one of them impersonate another.
Choose a gesture and define permission first
You probably already have the clip. Someone sent it, it's sitting on a shared drive, it's in the review thread. Having it is not the same as being allowed to run an automated analysis over someone's body and put the resulting diagram into a treatment discussion. Those are two separate agreements, and the second one is the one that matters here. Before you touch the file, know the answer to: did this performer agree to this processing and this use? If that's fuzzy, the fuzziness is the task. Not a footnote to it.
Then pick your material. Short — a few seconds, not the whole take. Nonsensitive, which means steering well clear of anything that invites a story about the person: no injury, no distress, no intimate gesture, nothing that puts health or emotion on the table even as a remote possibility. A reach, a turn, a weight shift. And it needs a change of pose. A landmark study of someone standing still has nothing to study; you've built an expensive still image.
One more property, and it's the one people skip: the passage should contain a moment where a body part is briefly hidden. Not because you're hoping something breaks, but because that span is where every interesting question about this technique lives. We'll come back to it.
Before the run, settle the route out loud. The sensible default is local — decode the video in a browser on a machine you control and process it there. Frames stay put. That changes what you're asking of the performer, the client, and the studio, and it's a sentence worth saying before you start rather than after someone asks.
And keep the study narrow. The same pass that returns body landmarks can also hand you facial points, gaze, and a pile of other signals, and the temptation is real because the data is already there. But you asked for a movement study and you were given permission for a movement study. If a later conversation wants to say the character "reads anxious," that's a directing judgment and it belongs to a person watching the performance — not to a model's output that nobody authorized.
Keep the video clock attached to the estimate
Name your tools and write the names down. In the browser route this typically means the @mediapipe/tasks-vision package, the PoseLandmarker class, running mode set to video, plus one of the shipped model bundles. Pin the exact package version, the exact model file, and the exact browser build you ran. You will want them again, and "whatever was current in September" is not an answer.
Video mode means frames are handled as a stream, in sequence, rather than as a pile of independent stills. Each call carries a frame timestamp in milliseconds. Meanwhile the browser's own media clock, HTMLMediaElement.currentTime, reports seconds. Two units for what feels like the same quantity, and the API has no way to know which one you meant.
That matters because the number you hand the API becomes the number you store beside those landmarks. Pass seconds where milliseconds are expected and your log is wrong by a factor of a thousand. Nothing necessarily crashes. Depending on how you've wrapped the call, it may raise an error or it may simply return. Either way, the times you now hold are not video times.
The second clock trap is quieter. If the video is paused and your loop keeps firing, you will keep generating fresh landmark sets from an unchanging image — and if the timestamps advance along with the loop, the result looks like the body is doing something over time when the picture hasn't moved an inch. Decide explicitly what repeated analysis of the same frame means in your setup: skip the call when the time hasn't changed, or record the identical time for it. Do not let the cadence of your own loop become the motion.
Then verify the timing against the source before you evaluate any pose. Put the overlay next to the original at a distinct beat — the top of a reach, a clear turn — and check they agree. If the overlay lags or leads, nothing downstream is readable. A bad clock doesn't make the pose bad. It makes the pose unlocatable, which is worse, because it's harder to notice.
Say what the coordinates mean
Two different representations can come back, and mixing them in one unexplained drawing is where a study quietly becomes a lie.
Normalized image landmarks are tied to the frame: x and y scaled across the image, which is why they map neatly onto a canvas with the video's aspect ratio. Their z value is a depth estimate, expressed at roughly the same scale as x, and not a distance you can use.
World landmarks are an estimated three-dimensional position in metres, reconstructed from a single camera.
Both are described in Google's Pose landmark detection guide for Web. Pick one for your diagram and label which. Then label the axes, and label any mirroring or cropping you applied to the display. If you flipped the video horizontally for a familiar self-view and didn't flip the landmark x values along with it, the skeleton is drawn on the wrong side of the body. Crop the frame without adjusting and the points land off the figure. Scale the canvas at a different ratio than the video and x stretches relative to y, so the figure looks subtly wrong in a way that has nothing to do with the model. None of these are estimation errors, and every one of them looks like an estimation error.
Visibility is worth its own warning. The output carries a visibility value per point — a likelihood that the point is visible in the frame. It is a statement about the model's own input, not a certificate that the joint is where the drawing says it is. A high-visibility point can be in the wrong place; a low-visibility point can happen to be right. Read visibility, don't trust it as ground truth.
And resist the 3D figure. A skeleton rotating in space is enormously persuasive, and it was inferred from one viewpoint. You cannot measure a performer's reach, stride, or limb lengths from it, and once you show it as though you could, the whole study inherits a rigor it never had.
What the documentation establishes is what the API returns and how it wants to be called. It does not establish accuracy on your footage. That part is yours to find out.
Inspect the failure span, not the cleanest frame
Don't review the tidiest moment. Clean frames tell you the setup runs, which you already knew the instant anything drew at all. Go to the occlusion.
In a reach across the body, the hand disappears behind the arm or the torso. For that stretch the model has no pixels — it has no observation. What it returns is whatever it returns. A returned coordinate is not an observation of the hidden body part. It is an estimate produced while the evidence was unavailable.
So look for specific signatures. Missing points: no coordinate reported for that landmark. Implausible jumps: a wrist that travels an impossible distance between consecutive frames. Left/right confusion: labels swapping when the arms cross the midline. And the most seductive one — points that continue smoothly along a plausible path behind the obstruction, so that the drawing looks most confident exactly where you have the least information.
Record the interval. Mark the source times where the estimate stops being trustworthy, and inspect the edges as carefully as the middle. If the hand's position just before it vanishes and just after it reappears are both reasonable, the middle might be a smooth interpolation with no evidence underneath it — which means the two ends are the reliable part and the centre is a guess wearing the ends' credibility.
One naming note before it costs you an afternoon. "Left" in a landmark set means the person's left, which appears on the right-hand side of an unmirrored camera image. Decide the convention once, print it on the diagram, and don't debug a swap that turns out to be a naming convention.
Mark, omit or retry instead of hiding the uncertainty
Three acceptable moves, roughly in order of usefulness.
Mark it. Render the doubtful points differently — faded, dashed, with a shaded band across the timeline. The reader sees the shape of your uncertainty instead of having to take your word for it.
Omit it. Take those points out of the drawing entirely. A gap is a true statement about what you know.
Retry with clearer input — which produces a new input, not a repaired observation of the original. If you rerecord with the hand visible, you now have two studies of two performances. Don't merge them and present the result as a study of the first.
There's a useful middle check here too. On the frames where the hand is clearly visible — before the occlusion, and again after it clears — annotate a few by hand and put your annotation next to the model's output for those same frames. You're not establishing ground truth for the hidden span; you can't. You're establishing a baseline for how well the estimate tracks this particular footage when it has something to work with, which tells you how much weight the confident stretches can carry.
What you must not do is smooth the failure into a confident trajectory. Smoothing is the default behavior of nearly every tool that touches this data, and it converts "we don't know here" into "here is where the hand was." At that point the diagram stops being an estimate of the performance and becomes an invention borrowing the performance's authority.
Know where to stop, too: before rig mapping, before performance evaluation. No retargeting these points onto a character, no claims about the performer's weight or technique. Separate jobs, separate evidence.
A proposed fixture, and two failures that are not the same failure
Everything below is hypothetical. No performer has been recorded, no model has been run, no browser test has been performed. The fixture defines what a run should capture, so that when the footage exists, the record is already shaped to receive it.
The setup: one consenting adult performer, one nonsensitive original action — a reach across the body with the left arm, so the left hand passes behind the right arm and the torso for a moment. Source clip at 24 frames per second, about two seconds long. The reach begins near 0.4 s. The hand is obscured roughly between 0.9 s and 1.3 s. The arm clears around 1.5 s. In milliseconds: 400, 900, 1300, 1500.
The study carries three synchronized tracks — the unmodified source, the raw landmark overlay with no smoothing applied, and a timeline band marking 900–1300 ms as uncertain.
Now, two failures to keep strictly apart.
The first is yours. The run passes currentTime in seconds into a call that wants milliseconds, and stores the returned value as the frame time. Nothing looks wrong on screen. The overlay sits over the correct picture, because you drew it on the frame you just decoded. The damage is entirely in the log: every recorded time is a thousandth of its true value, so 1.2 s of performance is filed at 1.2 ms, and the uncertainty band you draw from those numbers collapses into a sliver at the head of the timeline. Any statement about when something happened is now unanchored. The fix is a conversion at one boundary and a naming convention that carries the unit — sourceTimeMs, so nobody downstream has to remember. Rerun, and the band now sits under the actual occlusion at 900–1300 ms.
The second belongs to the model. Whatever the landmarks do inside that span. And here is the point worth carrying out of the whole exercise: fixing the first failure changed the estimate not at all. Not one landmark moved. The clock repair changed what you could say about where the estimate sat in the performance. It did nothing about the estimate's quality. Two problems, two fixes, and no amount of implementation care turns an estimate into an observation.
Record the second failure as it happens. Whether the hand landmarks drop out, drift, jump, or carry on behind the arm — write down what you saw, mark the span, and let the record be what it is. Going in with a predicted failure mode is how you end up describing your expectations instead of your footage. Keep a run log alongside: source times, chosen versions, the consenting fixture, what actually occurred.
What the study can and cannot support
A version of this that can leave the room looks like this:
Reach across body. Source 24 fps, 0.0–2.0 s. Landmarks from [package version] / [model bundle] / [browser build], processed locally from the original file. Coordinates shown are normalized image landmarks; no mirroring or cropping applied. Estimate follows the reach closely from 400–900 ms and from 1300 ms onward. Between 900 and 1300 ms the hand is behind the arm; points returned in that span are provisional and are not observations of the hand's position. Not motion capture. Not a measurement. Nothing here describes what the performer intended.
Every claim in that paragraph can be walked back to a frame in the source, and that traceability is the only reason the diagram is allowed into the conversation at all. When someone points at the overlay and says "so the hand goes here," you can answer precisely: yes, on this drawing, between these times — and here is the span where we would be guessing.
Then mind the naming, including in the deck, where terms tend to harden into promises. What you estimated was a set of two-dimensional points per frame plus a guess at depth. What remains to be authored is everything else: volume, contact with the ground, weight, the relationship between two performers, and every choice the performer made on purpose. Say both halves, in that order, and the technique settles into what it should be — a way of pointing at a moment in a performance, with the performance still underneath, still the thing you're actually looking at.
Frequently asked questions
What is a provisional pose study actually meant to be?
Estimated body landmarks drawn over the original frames, tied to the source's own clock, with weak stretches marked and left visible. Its job is to give a room a shared object to point at while discussing a gesture. It is not captured acting, measured anatomy, or a rig for an animator.
What has to be settled before analysis begins?
Whether the performer agreed to this processing and this use, not merely whether the clip is available. The material should be short, nonsensitive, contain a change of pose, and include a moment when a body part is briefly hidden. A local processing route is the sensible default.
Why do the two clocks matter so much?
Video mode expects frame timestamps in milliseconds, while HTMLMediaElement.currentTime reports seconds. Passing seconds where milliseconds are expected stores times wrong by a factor of a thousand, so timing becomes unanchored even if the overlay looks right. A paused video with a firing loop can also create apparent motion from an unchanging image.
How should the occlusion span be handled?
Do not review the cleanest frame; inspect the span where the body part is hidden. Mark doubtful points, omit them, or retry with clearer input, which creates a new input rather than a repaired observation. Returned coordinates during occlusion are estimates, not observations, and smoothing them into a confident path turns uncertainty into invention.
What can the finished study support?
It can support statements tied to source frames, such as where estimated two-dimensional landmarks appear, and it can mark the span where the estimate is provisional. It cannot support motion-capture claims, measurements, rig mapping, performance evaluation, or statements about what the performer intended.