Salvatore Prochilo
All projects
prototype

PenaltyVisual – Penalty Direction Prediction from Kicker Motion

Computer vision on penalty-kick footage: YOLO11 tracks ball and kicker, and a simple physics model pins down the kick frame to within one frame (40 ms), even when the leg hides the ball.

Role
Sole developer (dataset, models, evaluation)
Organization
Personal project
Period
2025 – 2026
  • computer vision
  • sports analytics
  • YOLO11
  • object detection
  • video
  • physics-based modelling
  • PyTorch
repo

Context

A personal project with a long-term goal: from the video of a penalty, predict where the ball will go before it gets there. To do that, a model has to look at the right moment: the run-up and the instant the foot meets the ball. Everything downstream depends on finding that instant.

I built a dataset of 383 penalty clips from public match footage and annotated each one by hand: shot side and height, kicking foot, run-up speed, feints, camera angle. Then I built the pipeline around it.

The problem

At 25 frames per second, one frame is 40 ms, and a struck ball travels about a metre in that time. Being “close” to the kick frame is not good enough: two frames late, the clip already shows the ball in flight.

Broadcast footage makes this hard:

  • The ball is tiny: 15–20 pixels across in a 720p frame.
  • The camera pans and zooms throughout the run-up, so a ball at rest still moves on screen.
  • The contact itself is often invisible. In many clips the kicking leg hides the ball for two to seven frames, exactly when the kick happens.

Approach

Pipeline in two stages. Timing: a penalty clip goes through a fine-tuned YOLO11 that finds the ball and the kicker in every frame; the kick frame is found by extending the ball’s flight back to the penalty spot, with the closest ball–kicker frame as a fallback; the kicker is then isolated from the background. Prediction, a prototype: a run-up window and a kick window are encoded by MViT-v2 and a small classifier predicts left, centre or right.

Detection. I fine-tuned YOLO11 (nano and medium) from COCO weights on two classes, ball and kicker. It runs on every frame. For the ball, each frame keeps the detection closest to the previous position.

Kick frame: a physical model instead of a heuristic. My first version took the frame where the ball and the kicker are closest. It is simple, and often right, but it can be wrong by more than half a second: the closest point between two bounding-box centres depends on the kicker’s posture, not on when the ball moves.

So I went back to the physics of the event:

  1. Remove the camera. Phase correlation between consecutive frames measures the pan; subtracting it gives the ball’s motion on the pitch, not on the screen. A ball at rest now really sits still.
  2. Before the kick, the ball is at rest on the spot. After it, it flies in a near-straight line at near-constant speed.
  3. Fit the flight and extend it backwards. A line through the first detections after the ball reappears, extended back to the spot, gives the moment it left the spot, even if the contact itself was hidden by the leg.

In other words, the kick is the single impulse that takes the ball from zero velocity to flight, and you can find it from what happens after it.

Animation in four-times slow motion, drawn only from YOLO11 detections with camera motion removed, no video frames. The kicker’s box runs up to the ball resting on the penalty spot. Around the kick the ball is hidden by the leg for three frames; it then reappears in flight, leaving a trail of dots. A dashed line extends the flight back to the spot and marks the kick at frame 24.6. A timeline below shows the three phases (ball at rest, hidden, flight), the kick time and the current frame.

Chart of the ball’s distance from the penalty spot, frame by frame, for one penalty at 25 fps with camera motion removed. The ball sits at the spot until frame 22, is hidden by the kicking leg for three frames, then reappears moving about 3.1 ball diameters per frame in a straight line. Extending that line back to zero puts the kick at frame 24.6; checked by eye, the kick is at frame 24. The closest ball–kicker frame is 31, seven frames (280 ms) late.

When the physical model cannot be applied (the ball is never seen at rest, or too few detections after the kick), the pipeline falls back to the closest ball–kicker frame.

Prediction (prototype). From the kick frame, the pipeline cuts two windows: 32 frames of run-up and 16 frames around the kick. It removes the background (temporal median of the clip, with only the kicker’s box pasted back) and encodes each window with MViT-v2, a video transformer pre-trained on Kinetics-400. A small two-branch classifier, fed with these embeddings, predicts left, centre or right.

My role

Everything: the dataset (download, cutting, a small web tool to annotate 383 clips), YOLO fine-tuning, the timing method and its evaluation, and the prediction prototype.

Results

Detection. On the validation split, the five YOLO11 runs reach a mAP@50 between 0.81 and 0.89 (ball and kicker together). The nano model used for the timing analysis scores 0.88.

Kick frame. I checked the kick frame by eye on 12 clips that I had not used to design the method:

Method Exact frame Within ±1 frame Worst error
Closest ball–kicker (first version) 5 / 12 8 / 12 15 frames (0.6 s)
Peak of ball acceleration 4 / 12 8 / 12 11 frames
Flight extended back to the spot 6 / 12 12 / 12 1 frame (40 ms)

The physical model can be applied to 23 of 40 clips in a sample; the others use the fallback. The reference frames come from my own visual check, at broadcast resolution, so they carry about ±1 frame of uncertainty themselves.

On real footage I can show. The training clips are TV broadcasts, which I can’t republish, so the frames below come from a free stock video: a close-up penalty filmed from behind the kicker.

Six real frames of a penalty filmed from behind the kicker, with the boxes found by a generic YOLO11 model and a cross marking where the ball rests. Frame 10: the ball on the spot is found with confidence 0.92; a false “ball” is also found on the keeper’s glove. Frames 32 and 36: run-up, the ball still on the cross. Frame 37: the foot meets the ball, which is found already shifted off the cross, the kick. Frame 39: the ball is hidden by the leg. Frame 46: the ball is found again in flight, with low confidence.

Footage: Pexels, Pexels License. Frames cropped and annotated by me.

At this distance the physics needs no extrapolation: the ball sits still on the spot until frame 36, and at frame 37 it is found displaced. That is the kick, the same frame I see by eye. Two honest caveats. The boxes come from the generic YOLO11m (COCO “sports ball”), not from my fine-tuned models: trained on TV footage, where the ball is a few pixels wide, they barely find a ball this large. And on a second free video, filmed from the stands, the fine-tuned model tracks ball and kicker up to the contact but never detects the ball in flight, a small blurred streak. Both point to the same next step: training data with more camera distances, and blurred balls in flight.

Prediction. Not there yet. The only trained version of the classifier reaches 44% accuracy on a validation set of 27 clips. Always guessing the most common side would score 45%. The model never predicts “left” correctly. With a few hundred clips, generic action embeddings don’t seem to capture where the kicker is aiming. That’s why I keep this part labelled as a prototype.

Stack

Python, PyTorch, Ultralytics YOLO11, OpenCV (phase correlation, optical flow), torchvision MViT-v2, scikit-learn, Weights & Biases, Roboflow.

What I’d do next

  • Bring the physical model fully into the pipeline and extend the manual check from 12 clips to a proper annotated test set.
  • Replace generic video embeddings with features that carry intent: body pose at the moment of contact (hip and plant-foot orientation), with the kick frame as the anchor.
  • Remove the hand-annotated inputs from the classifier: some, like whether the shot was on target, would not be available before the kick.
  • Record my own footage, so the project can show real frames, not only charts.