It’s been a couple of years since I’ve started a project knowing exactly how to finish it. It’s part of the job description really; and it’s what I love so much about it. But every now and then you get one where you’re not really even sure how to start.
Some time around the start of last year, I found myself in a meeting room at McCann Melbourne. Alex Wadelton and Andy Jones had dug up some literature about a long dead french anatomist. They had grainy black and white photos of this man, Duchenne de Boulogne, applying electrodes to a patient. The electrodes were forcing the patient’s face into all manner of contortions. A series of gross parodies of human expression. Typical of his era, Duchenne’s endeavours were pretty pseudoscientific and steeped in some pretty out-there religious ideas.
But he had a neat party trick: he knew how to artificially stimulate facial expressions.
The client was the Melbourne International Film Festival. Their problem, in a nutshell - is that in the era of Netflix, this Season’s Sequels and the Monthly Marvel Movie; cinephiles were accustomed to purchasing tickets knowing exactly what they were getting.
But a film festival is different. A film festival is like a box of chocolates - you’re never sure what you’re going to get. Maybe it’ll be a heart warming story about a puppy on an adventure to find his kitten friend; maybe it’s a depressing family drama set in the middle of the Irish potato famine. Maybe there’ll be explosions, maybe there’ll be boobs.
The McCann insight was to review the films in the program - not in the academic language of film reviewers - but in the pure, visceral language of emotion. Their draft tagline: “Feel the film; before you see the film”.
The very nascent campaign that Andy and Alex explained to me that day touched on many different media with an insightful strategic plan that went well beyond what they wanted us for.
Airbag and I were there to talk about one thing. The mad Frenchman.
See once you had this data about the emotional arc of a film, what to do with it? How do you communicate that data? Common sense says an emoticon on the website would probably do the trick - but luckily for us common sense was at a different agency that day.
Alex and Andy wanted me to design a system that would allow us to play back the emotional arc’s of these films, using a human face as the display.
I said, “Sure thing”.
After all if Duchenne could do it in the 1860’s, I could do it in 2015.
Of course the thing I didn’t know until later; was that the man in Duchenne’s pictures had a medical condition - possibly a form of neurosyphilis. His entire face was permanently numb.
He couldn’t feel a thing.
But I didn’t know that then. When I went home that night I took a small multi-tap transformer I found in a drawer at home and wired it up in reverse to my signal generator. Then I taped two pieces of damp sponge to my inner arm and taped the wires on to that. I set the signal generator for a pure sine wave at 100Hz. Winding the current limiter right down, and dropping the amplitude down as far as I could - I switched it on.
I gently upped the voltage until I started feeling it tingle; about 45 volts. A little higher still and it felt like ants crawling on my arm. Around 60 volts and the ants started biting. At 70 my arm was on fire and my hand was twitching, but it wasn’t a sustained muscle contraction. I pushed it a little more but there was no improvement. I couldn’t take it any more. I killed the power and removed my makeshift electrode. My skin was red, blotchy and a weird combination of numb and itchy.
And that was on my arm. There was no way I was trying this on my face. I have a mouth full of old school amalgam fillings after all.
Do I need to tell you not to try this at home? Of course by now I might have given the impression I’m a completely cavalier idiot and a potential Darwin Award Nominee.
The truth is I’ve been near fatally electrocuted twice; and of course as a farm kid I’ve had plenty of exposure to the electric fence. Electrocution is a painful experience, and one totally not worth your while. Not only does it hurt in that instance; you end up with burns the take time to heal. Your whole body cramps, your muscles ache and you have bruising that seems to last for weeks. I've had some neurological weirdness too - twitches and numbness etc.
The worst part is that as little as 70 milliamps in the wrong place can kill you. For example the vagus nerve in the neck is responsible for numerous parasympathetic systems in the abdomen - including maintaining your heart rhythm. Hit that and you could end up in fibrillation. Or just drop dead.
So I have a healthy respect for electricity. Mostly though I was captivated by that sense of adventure that accompanied 1860’s type scientific endeavour. And I had (I thought) enough electrical and anatomical know how to approach it - gingerly.
That week I tried a few more ideas, but mostly it was fruitless. Fundamentally there were too many variables at play. Location of the electrodes, voltage, amps and the nature of the signal. What frequency? What period, function and duty width? Monophasic or biphasic? Was a single wave sufficient or would it need to be modulated? Etc etc.
Adventurism over, it was time to hit the books. Over the next week or so I read a lot of books. Pretty much crammed the neuroscience and anatomy modules from first year medicine. I was lucky to get the personal attention of Dr Ivanusic, Senior Lecturer of Anatomy & Neuroscience at Melbourne Uni. Our wide reaching conversations were very helpful. He was very interested in the project. I asked him if there was anyone at the University doing anything similar?
“Are you kidding? There’s no way a university ethics committee would let you try that.”
Good to know.
Meanwhile the experimentation continued. By now I’d upgraded to experimenting with TENS devices. These are medical devices you can buy off the shelf. So they’re approved for use by the general public in Australia and considered ‘safe’. Plus I had proper electrodes of numerous sizes, shapes and chemical composition.
A TENS device is most widely used for pain management - more on that in a second. But there are variants of TENS devices sold as an aid for muscle toning (typically referred to as EMS - Electro Muscular Stimulation). I'm told sports people do use them. But they’re also often encountered as 'work-out free' miracle weight loss machines. As seen on TV!
This EMS variant of TENS interested me a lot. If the device is designed to contract muscles - then that’s what I want, right?
But we should back up a moment. Let’s pretend for a moment that there are two main types of nerves. It’s a whopping over simplification; but for the moment let’s pretend. There are the nerves that carry signals from the body up to the brain (the Afferent nerves). And there are the nerves that carry signals the other way; from the brain back out to the body (the Efferent nerves).
One set of wires to make you feel, and one to make you move.
And that’s the catch. As an electronics engineer, it’s very easy to fall into the trap of thinking of the human body as a straightforward electrical system. Wires attached to sensors and motors.
But in truth, you quickly find that your typical human is less robot and more fleshy bag of salt water. The ‘wires’ you want are mostly bare and all tangled. In fact they’re not even like wires at all; they’re more analogous to rivers. Alive and prone to move and chart their own courses. If you’re an intelligent design advocate: God obviously hadn’t really set on a standard when he started assembling us. Just kind of draped the nerves where ever they fit; like an over excited six year old decorating a christmas tree.
Worst of all those two sets of nerves, the ones for movement, and the ones for pain: are generally right next to each other. When you’re trying to make yourself a meat marionette; you really only want the movement without the pain.
Unless you’ve got access to people with completely numb faces of course. Or you’re working with the recently dead I suppose.
So the next thrust of my research became about how I could activate the Efferent nerves without hitting the Afferent. Very difficult from outside the body. Even I didn’t have the gumption to ask whether me, sans any medical qualifications, would be allowed to drive needles into random members of the general public. So I had to get it to work from outside the skin.
Which brings me back to TENS and pain management. The theory behind TENS that by overloading the Afferent Neurons with electricity, you effectively blocked the pain pathways back to the brain. Again; a massive over simplification.
So maybe it's not practical to avoid hitting the nerves you don’t want . So instead you can try and shut them down. Combine the principles of both TENS and EMS. The trick becomes finding the combination of stimuli that shut down one set, whilst activating the other. Especially using devices that really aren’t supposed to be used on the face.
Through lots of experimentation I ultimately managed to isolate combinations of stimulus and electrode placements that worked for each of the important muscle groups on the face. Mostly it was actually about being strategic about the placement of those electrodes. But it remained a fickle thing and required careful tuning per person. It was very easy to get it wrong and trigger the pain receptors.
I was running out of time and I needed to begin building the machine. We knew that it wouldn’t just be me going in the machine, so it needed to be safe. My very last battery of experiments consisted of attaching the electrodes myself and driving the stimuli all the way up to their maximum power to ensure that no lasting damage could be done. It hurt like hell and for about 20 minutes that day my left eyebrow was paralyzed. Kind of a lopsided “surprise botox” effect. But it faded without trace, and I set the machine up so that it was effectively limited to about 25% of that power. In practice most people put in the chair operated at between 16% and 22% power.
The hardware was setup to be entirely powered by batteries for safety sake. Anywhere mains electricity was used, it was completely isolated from the systems attached to the subject. The control system was built up with a handful of arduinos, which I wrote software to communicate with an ipad/iphone wirelessly. All battery powered. I also designed and built a UI that was designed to be projected on a big screen - showing the changing values of the various TENS channels and the ‘expressed emotions’.
At this point I was also had to write some software to convert the data the McCann guys had gathered about the films, into a format that worked for the Emotion Simulator. This was an interesting challenge as the system they’d settled on not only involved automated biometric readings - it also revolved around the reviewers self reporting their emotions by pushing buttons on an app. This meant that the emotional sampling rate was highly inconsistent. Some films had 50 data points, some had over 500. On top of that we realised quickly that if you didn’t insert ‘rests’ between emotion poses, the user was inclined to get a bit overwhelmed and ‘tense up’.
In the end we settled on each film being represented by a sequence of facial expressions lasting less than a minute. An expression each for "Anger, Disgust, Fear, Happiness, Sadness and Surprise". Each ‘emotion’ held for no more than a few seconds. Anything longer or more densely packed tended to overwhelm the user.
To give the electronics some stage presence and gravitas we sourced a terrific old fashioned hydraulic dentist’s chair from the late 19th century. I actually found it in a shed outside Geelong. It was rusted to all hell, but the lovely Seth Aitken and the team at Illusions repainted it and reupholstered it with with red velvet. The effect we were shooting for was 'science meets weird cinema chair'. I’m very happy with how the chair looked, but I rued the decision none the less as it was entirely cast iron and weighed nearly 300kgs. It made moving it around very onerous.
We also needed something to hold all the electronics, so Seth and his gang at Illusion also built us a bespoke cowl to hold all the electrodes. I rigged it up with the emotion simulator electronics, addressable LEDs (because, why not?) and cameras.
I’m the first to admit that in the end the visual effect was pretty odd.
But I was kind of shooting for that.
Once we were ready, we put several people in the chair and made a little film about the experience. This was also how we went about generating the eponymous 'Emotional Trailers'. McCann took that footage and linked it with the relevant films on the MIFF website. Creating a unique interpretation of each film.
For the opening gala we setup a live demonstration on stage. In a moral victory for production companies everywhere, a last minute casting hiccup on launch night meant that previously mentioned McCann Creative Alex Wadelton was forced to fill in as Test Subject #1. Mr Wadelton dutifully withstood a good 10 minutes in the chair, bringing much joy to the audience, and suffering as we all do - for his ideas.
Then we transitioned into activation mode.
During the festival we ran Emotion Simulator on stage at the Forum Theatre every other night. Volunteers from the audience were given the opportunity to sit in the chair and ‘experience’ a film on behalf of the onlookers. When you think about it; it’s a pretty weird proposition. But the response was fantastic. And people always wanted to talk at length about how it all worked.
The hardest question - and I seemed to get it every time I set up shop - was “which university are you from? And what’s the actual subject of your paper?”.
“It’s just a marketing stunt” never seemed a very satisfying answer. There’s something about the whole exercise that seemed to ask questions of the participants nobody had good answers to. What is the real connection between our faces and our feelings? How does what’s on our face translate to another human? What does it mean to decouple expression from emotion? In that sense those unanswered questions made it feel almost more like an art installation than anything else.
But in truth it was very effective in it’s own way. When your goal is discussion and a little buzz, doing something a bit out-there is always a fruitful path. In that sense we owe a debt to MIFF and McCann. It takes a risk to make something great.
I also have to give a shoutout to my partners at AIRBAG. We have this little matrix we filter the jobs through - are we going to make money? Is it good for the reel? Etc. This project it was super hard to say yes to on any of the normal fronts. The intrinsic question is rarely one that get’s a showing. Is the project worth doing, simply because it’s worth doing?
That’s faith, and that’s what makes working at AIRBAG so special.
The MIFF Emotional Simulator recently won Gold, Silver and Bronze Lions at Cannes in the Creative Data categories. It's also won a Grand Prix for Interactive at ADFEST, along with two gold, two silver and four bronze across various other categories.
Steven Nicholson is an AIRBAG partner and our Chief Geek. You can see more of his work on this site.