CLIP Exercise Recognition

A small API for exercise recognition on still images using the OpenAI CLIP model, packaged as a deployable demo with FastAPI, Streamlit, and Docker.

Why CLIP

CLIP was the first zero-shot paper that actually worked reasonably well across many datasets. It is trained on roughly 20,000 image-text pairs spanning a vast ontology, so it extends to new domains without supervision — though you can still add a classifier on top of the extracted features if you want to.

That matters for exercise recognition because labelling exercise video is slow and expensive, and every new movement means another collection round. With CLIP the labels are just text.

Goals

1. Exploration — see what prediction looks like on exercise classification, and how it behaves frame by frame across a video as a time series.
2. A simple, deployable API others can explore and build on.

Supported labels

The demo ships with four prompts:

• a person standing
• a person repeating a squat
• a person repeating a jumping jack
• a person performing a plank

Editing src/labels.txt changes the demo’s vocabulary, or the texts variable in the notebook. Prompt engineering is the whole interface.

Results

The notebook runs CLIP frame by frame over video with a mean smoothing window of 5, which is enough to stabilise predictions across a repetition.

Future work

1. Use the CLIP image encoder as an embedding for image retrieval — collect a dataset of stills and match new images against it.
2. Use the CLIP image encoder when learning new tasks, as an additional loss acting as a regulariser to push a network toward more generic features.