Telepresence Game Experiments

Montage made up of two photos separated by a zigzag. The photo on the left shows a man in a baseball cap sitting in front of a computer showing a video feed of a room and a control pad with direction arrows. On the right a small cardboard robot holds a smartphone showing the man's face

This post is about work I’ve done with The Crafty Robot but as it is more a story about design and technology experimentation and development, than a project writeup, it felt like it fitted here better.

Cardboard telepresence experiments

When the pandemic first started I was desperate to do something about the heartbreaking way it was separating people.

In designing Smartibot originally, I had tried to create a piece of electronic hardware that could easily be adapted to different uses (by users, but also myself) with only mechanical design (through hand making, hacking existing physical products, doing 3D printing or making kit expansions) and writing simple software (like web-apps and device firmware in JavaScript or with code blocks).

I realised that telepresence (allowing someone to drive a robot around over the internet in order to be more ‘present’ in a space that they are physically separated from) could be one of these uses, and with a little bit of new software I could allow owners of the few thousand Smartibots that were already out there to use them as basic telepresence robots.

Flowchart showing a laptop with a control pad and picture of a tea mug on the screen connected via the internet to another laptop which is connected via bluetooth to a Smartibot (a small cardboard robot carrying a smartphone) with a mug of tea in front of it.

 

I hacked a couple of web-apps together, using PubNub to handle the communication of the driving commands, and letting users use whatever app they wanted for the actual video call (this meant that users at the robot end of the connection needed two devices, one to put on the robot which handled the video call and one running the web-app that worked as the relay for the controls). I published a writeup of how to get it all working and a video and it proved quite popular, suggesting that there might be an appetite for a better developed version of the idea.

Looping video of a small cardboard robot holding a smartphone which is showing a video call with a man with a moustache. The man is moving from side to side within the video frame and the robot is driving from side to side

A product

I started to work on a properly integrated telepresence system that would work with a single device with Altrubots (who got in touch after seeing the writeup and had been working on a system to allow people to control rubbish cleanup robots over the internet).

A toddler sits at a table looking at a small cardboard robot which is holding a smartphone on which a woman's face is displayed

Using the original system with the A.I. Bot chassis that comes in the Smartibot kit I realised that, in order to be able to fully interact with people in a space it would be very helpful to be able to tilt the phone, allowing you to look up and down. The Smartibot kit only includes two motors so it made sense to do an expansion kit that included a third motor and a new cardboard chassis designed specifically for telepresence.

Photo of a small cardboard robot with an arm on the top which holds a smartphone cradle. The robot is shown from front and rear three quarter views.

With a new cardboard robot design and a mostly intact telepresence system we took it to Kickstarter to get enough pre-orders to finish the software and put the expansion into production.

Cardboard Wolfenstein 3D (first person game)

Looping video of a small wheeled robot with a smartphone in the centre of the chassis and a cardboard fist mounted on a motorised lever at the front driving up to and punching a cardboard cutout Nazi soldier

I realised that the experience of driving the robot around and seeing from its point of view felt a bit like playing a first person computer game. I wanted to make a bit of content that would raise the profile of the Kickstarter campaign so I built a cardboard recreation of the first, first person game I remember playing, Wolfenstein 3D. I made a robot that could hold a phone and had a pixelated fist attached to a small servo, allowing players to drive themselves around the labyrinth and punch Nazis (detailed writeup here).





I posted the link to drive the robot on Reddit and ran the game for a few days. The people of Reddit really liked it and I had people come and play from all over the world. It was a strange and magical experience, having the robot suddenly spring to life in my studio, being controlled by a random person in a random location. Sometimes they would enable their microphone and camera and we would have a brief chat. At that time, here in the UK, we still were not having many social contacts so it felt nice to connect with people.

It also turned out that the a telepresence system connected to a small and flexible robotics platform works surprisingly well as a game engine.

Remote telepresence game workshop (first person games)

Fellow designer and researcher Dries De Roek was an original Smartibot Kickstarter backer and been following the progress of the Smartipresence system. He suggested that I run a workshop as part of the annual Thingscon conference’s 2020 incarnation, the remote-only Good Things Fest, based on making telepresence games.

A few weeks before the workshop I sent each participant a Smartibot kit and a couple of small servo motors. During the workshop we used cardboard modelling techniques to built a basic cardboard chassis and then participants modified theirs and put together a game experience around them. At the end of the workshop we all played each others’ games. There is a more detailed write up over at The Crafty Robot.





Robot unicorn jousting (third person game)

In the spring of 2021 I was asked by the amazing Emma Beaman (who runs the wonderful Playful Anywhere a Leeds-based organisation who make wonderful playful things happen, some of which I’ve been fortunate enough to have got involved with in the past) to run an activity as part of Love To Play, which was a remote event that felt like a mixture of lovely play experiences and fascinating reflections on what play actually is. 

I had recently made a simple jousting game by putting paper figures with lances on top of the cardboard unicorn chassis we include in the Smartibot kit, mainly to entertain our toddler. I figured that the Smartipresence system might work just as well for third person games as first person ones, in which case the Smartipresence system combined with the jousting game could be a good fit for Emma’s event.

Photo of two brightly coloured paper knights riding cardboard robot unicorns. In the background small toy figures are cardboard robots look on

It was! I set it up with a webcam for the main event feed looking across the centre of the playing field and two smartphones on tripods behind each robot unicorn. We had a jousting tournament with participants taking in in turns to control the unicorns for each jousting bout.

Hybrid telepresence/on site game

The organisers of the V&A Museum’s Digital Design Weekend thought the jousting might work well as part of that event. We were keen to tie it into the Alice in Wonderland exhibition that was running at the V&A at the time. I remember the the second instalment of the Alice story contains a fight between two mounted knights, the white knight and the red knight. Those knights have helmets in the shape of horses heads (like the chessmen) so I designed a folded paper helmet for the little paper knights.

Hand drawing in black and red showing a checkerboard playing surface on a table with two mounted knights with lances on it. There is an iPad on a stand at each end of the table and to one side is a table with craft materials and a person holding a paper knight.

 

Unlike the previous events I had run with the Smartipresence system, this time there were going to be people physically present in the museum, but we were keen to connect those people (and the physical space they were in) with people who were joining remotely.

Composite of two images. On the left a photo of a cardboard robot unicorn with a blank white paper knight with a horse shaped helmet and lance sitting on it. On the left a template with the cut and fold lines for all the parts to make that knight out of paper.

 

We printed A3 templates so that visitors in the museum could design their own knights and then cut out all the parts, fold them and glue them together.

Photo of some children standing busily around a table covered with craft materials. One child is smiling whilst being handed a pair of scissors by a woman wearing a red paper hat in the shape of a horse head.

We then put their knights on the Smartibot unicorns which we placed on the chessboard like playing surface.

Photo of a man in his 50s and a girl of about 10 standing behind a large table holding iPads. On the table is a red and white chessboard pattern on which two robot unicorns carrying paper knights are colliding. Around the edge are other hand decorated paper knights.

We were keen that players had the same chance to win whether they were in the museum or not so we made everyone drive the unicorns using the Smartipresence pilot web app. We had some iPads we could lend to visitors so they could do this but we also put the links to drive each unicorn online so people from anywhere in the world could drop in and control one. Before the event we built a system the put uses in a queue if the robot they wanted to drive was not available.

Photo of the table with the chessboard playing surface and robot unicorns with paper knights. There is a smiling child and a man wearing a white paper hat in the shape of a horse head standing to one side and you can see at the end of the table a large screen shows what is happening on the playing surface.

We ran the activity over 3 days and it worked really well (Here’s a Twitter thread I did at the time with videos). People in the museum made some absolutely beautiful knights and we had remote participants from all over the world (Argentina, Texas, Indonesia, all over the UK and Europe). We even had a few people make knights and participate in the museum on one day and the participate remotely from home on another, which was very pleasing.

Photo of two small girls standing by the chessboard playing surface wearing the two horse shaped paper hats

I’m excited about what we can do next with the technology as it seems like both an effective way to bring people together and an interesting way to empower people to make their own games. 

Leave a Reply