Say What You See: a Pandemic Project

For many, the Coronavirus lockdown was an opportunity for people to get creative, and I was one of them. Since my choir had been cancelled and I could no longer go bouldering, I had a bit more free time in my evenings.

Early on into lockdown, a few of my friends wanted to carry on the local tradition of the Monday night pub quiz. We would take it in turns each week to host a quiz, coming up with our own questions that the other households would try to answer. I wanted to give my quiz a 90’s theme. I don’t know how I came to that decision, but it was probably because most pub quizzes I’ve been to have questions from before my time. Turns out my questions were much too difficult – I don’t think anyone got more than 1 point in the science round – but I managed to redeem myself with a round of puzzles from our favourite gameshow of the 90’s, Catchphrase!

Image shows a fish with caterpillar tracks and a turret. What do you think the catchphrase is?
Players would connect their phones to BuzzIn.live. Whoever buzzed in with the correct answer first won the point.

In order to put the round together, I had to find some good puzzles from old episodes of Catchphrase that I found on YouTube. I had to extract the frames from the puzzle and turn them into a GIF. I started off by downloading a video from YouTube and opening it in some video editing software. Extracting a short clip from the episode was tedious, and while I had the patience to extract 10 clips for my quiz, I wondered how many other great puzzles there were from other episodes, and if there was a way I could extract the puzzles using a computer program instead.

Teachable Machine

I remember attending a talk at work about machine learning. One of the tools demonstrated during that talk was Google’s Teachable Machine. It allows you to quickly train a model using images from your Webcam, all without you having to code anything. The presenter used this model to pause and resume a radio show on BBC Sounds just by waving their hands around. It was a memorable presentation, and I thought perhaps I could create a model using Teachable Machine that could tell if a video frame was a Catchphrase puzzle or not.

When you set up a Teachable Machine project, you need to create a set of classes. These are the categories that your images will be sorted into. I created a class for Puzzles, Answers, and a third for “everything else” – frames that could be safely discarded. I then needed to provide a set of training data. These are example images used to help define what a “puzzle” is. I made a script to convert 4 episodes (1 from each series I had downloaded) into a set of stills, then manually sorted them into folders.

Screenshot of Google's Teachable Machine in action. Contains three classes full of sample images.
Images were a Puzzle whenever the catchphrases were visible full-screen, and an Answer whenever the text was visible in the bottom-left corner.

Training the model takes a few minutes, but it was great to know that all this processing happened on my laptop and not in the Cloud somewhere. The only bit of Cloud processing happened when converting the generated model into a file that could be used in a Python script.

Teachable Machine also provides some sample code that will use the model you generate to classify any image of your choice. I modified that code slightly to let it process an entire folder of images, and a few more hacky scripts: one that grouped the classified images into sequences and turned them into GIFs; another to convert the GIFs to MP4; and a third to run an OCR process to extract the text from the “answer” images to make each phrase searchable. It took 3 days to process the whole lot, but the results were excellent.

Seven files output per puzzle
Each puzzle was available as a GIF and MP4, along with its solution in GIF, MP4, PNG and plain text varieties.

I was surprised at how effective the model was at detecting full-frame puzzles, but I noticed that quite a few puzzles were cut short, meaning that it was difficult for a human to work out the correct answer. When searching for a selection of puzzles for my quiz, I checked that the puzzle made sense and people would be able to work out the answer. It was easy enough to do for a small set, but I wondered if I could crowdsource it to find out the best puzzles.

And here’s a Web app

So, now I have all these videos, how can I collect feedback on each puzzle? With a database and a bit of Python!

https://saywhatyousee.uk will keep giving you random Catchphrase puzzles. You have the opportunity to guess the answer, then you can reveal the correct answer and upvote/downvote it. Sometimes the clue video will be cut short, or the answer text was extracted incorrectly. Whenever that happens, the user can flag the puzzle and it will be hidden until I can come and address it.

Video demonstration of SayWhatYouSee.uk

Feel free to spend some time on the nostalgia train.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s