Translator: Leyre Bastyr Reviewer: Rhonda Jacobs
People with synaesthesia experience the ordinary world
in quite extraordinary ways.
For some people, as they're listening to my words now,
each word is its own, distinct colour,
perhaps projected in their mind's eye or floating like a ticker-tape in space.
For other people, when they listen to music,
it's not just an auditory experience,
it's an animated visual spectacle of dancing shapes, colours and textures.
For some people, such as Carol Steen,
in this picture, when she has acupuncture
or feels touch or pain in her body, she experiences dynamic floating commas,
in this case bright red against a deep velvety background.
So synaesthesia seems very extraordinary to us,
but to people with synaesthesia, that is the only reality they know.
This is how they experience the world,
and for them this is a perfectly normal way of experiencing things.
It isn't just about vision.
In this example, this is James Wannerton's depiction of the London Underground.
So for him, every word that he hears, thinks about or reads
is like an ebb and flow of flavours and tastes
that are subjectively appearing on his tongue.
So some train stations might taste like apple pie or celery, and so on.
And to him, this is how he experiences things since childhood.
For other people, they may have auditory experiences.
Can we have this movie, please?
I don't know whether people can hear this?
Well, this is a silent movie, so you shouldn't be hearing it.
But for some people with synaesthesia,
this creates a whooshing, shooshing sound, for instance,
or a change in pitch, very dynamic.
So for these people, everyday life is very noisy,
whether or not they're literally hearing something or not.
More formally we can think of synaesthesia as a kind of joining of the senses,
in which they experience something extra tagged on to that which most people do.
So in this case, they're having an extra auditory experience
on top of a more typical visual experience.
Synaesthesia turns out to be quite common.
You all know somebody with synaesthesia, but you probably don't know who it is
unless you've asked them, 'What colour is this piece of music?'
Synaesthesia is a real biological condition.
It runs in families,
although the particular associations don't necessarily.
So, for instance,
somebody who experiences synaesthetic taste
may have family members who experience synaesthetic colour.
So it's a general disposition here,
and we know that there is a genetic basis to synaesthesia
through studies of the genome.
Although we don't fully understand what these genes are doing.
They're almost certainly affecting the way that the brain matures
because synaesthetes have differences in their brain,
both in terms of gray matter - they've got more gray matter density
in, say, parts of the brain to do with seeing.
So parts of the brain seeing colour, they have more of this.
And similarly with regards to connectivity -
they have more white matter connectivity between different regions of the brain.
So they might, for instance, connect the auditory parts of the brain
with the visual parts of the brain in a way that other people don't normally.
And I say 'don't normally',
but there's one claim in that we all had synaesthesia at one point of our life
and that's when we looked like this - when we were infants.
So one suggestion is that we're all born with synaesthesia
and most people lose it as part of the normal maturation process.
But synaesthetes, due to their different genetic composition,
retain these roots that link together the senses.
And we may not all fully lose synaesthesia entirely;
we're all able to link our senses together,
and synaesthesia reveals the rules
by which we can understand the links between vision and music.
And this has important implications for arts, for instance.
So nobody knows whether Kandinsky was a synaesthete or not.
Some people say he is, some people say he wasn't;
but he knew about synaesthesia.
Synaesthesia was well documented at the time he was producing his artwork.
And what Kandinsky wanted is that people should understand his artwork
not solely through the visual medium,
but they should understand it more as a multi-sensory kind of gestalt
that encompass all the different senses.
So I'm just going to do a little test now,
and I'm just going to play two different sounds,
and I want you to think which sound goes with this Kandinsky painting.
He didn't to this experiment, by the way, this is just me.
So if we could listen sound one, please.
(Discordant violin music)
And sound two.
(Harmonious violin music)
There's no right or wrong answer,
yet most of people tend to go with sound one here,
that's more discordant, whereas sound two is more harmonious.
So again, this might sound trivial, but actually, why is that?
What are these particular rules?
And it turns out that there's a whole set of rules
for linking vision and music together,
and we can analyse this by looking at the experiences of synaesthetes,
and a lot of them are present from the early stages in infants too.
So we know that they're not purely cultural.
If anything, the way that our senses interact
might be influencing our culture rather than vice versa.
So, for instance, if you're a synaesthete,
listening to a high-pitched sound relative to a low-pitched sound,
a high-pitched sound is going to be brighter,
it's going to be smaller,
it's going to be higher up in space
and it's going to be more jaggedy, for instance.
And most synaesthetes tend to have these principles,
although they might differ in exactly how it's experienced.
So we've done various experiments with these particular kinds of stimuli.
So one of the things that we've done is that we've taken these descriptions
of how synaesthetes see the world, and we've tweaked them in some way.
So, for instance, we can simply just rotate it left to right,
or flip it top to bottom.
We can change the colour -
so if a synaesthete experiences a yellow disc doing this,
we can flip that colour to, say, red, or green,
or brown, or something like this.
And what we can then do is play the pair of movies to people
and ask them which image, which movie goes best with the sound.
And what you find is that people who don't have synaesthesia,
they choose the synaesthetic original movie
as being the best representation,
or as being the most aesthetically pleasing one that goes with it.
So people who don't have synaesthesia can tune into these properties
that synaesthetes naturally experience within their mind's eye.
We've also got people who don't have synaesthesia
to try and draw from their own experiences
what it might be like to experience synaesthesia.
And, again, if we show those experiences with a true synaesthete,
people prefer the true synaesthe's experience here
relative to the one that somebody's tried to recreate.
So the synaesthete probably has a richer, more vivid kind of vocabulary
for matching this,
but other people use the same rules as well.
What does that mean for creativity and art?
Are synaesthetes more creative?
Well, we've done formal measures of creativity
and showed that there does seem to be some trends
for synaesthetes to score better on some tests.
What is perhaps more striking is that if we look at occupations of synaesthetes,
or their hobbies,
we find that they tend to gravitate more towards the arts.
So these unusual experiences are probably very beautiful for them,
and it probably inspires them to create art.
But it might also affect their brain in ways that actually is helpful.
So we know that synaesthetes are better at distinguishing
between different colours that to most people look very similar.
They can tell them apart much better.
So that would be a raw ability that would help in the world of art.
This is one of the artists I've worked with
who lives near me in Brighton.
She's a portrait artist,
but rather than painting people's faces, she paints people's voices.
So this is from a CD, but she sometimes does interviews with people
and paints their voices as she's having a conversation with them.
So this a very novel concept that's inspired by her unique experience.
So that's visual experiences. What about touch or pain?
So when I was a lecturer at UCL, I sent an email out saying:
Here's this thing called synaesthesia, does anybody think that they have this?
And I got a reply back saying,
'Well I don't know whether this is synaesthesia,
but whenever I see somebody being touched,
I feel touch on my own body.'
So what she reports when looking at this movie on the left
is that when seeing a finger going up and down somebody's face,
such as this,
she would say, 'I feel a tactile sensation on my right cheek',
as if you're looking in a mirror of that particular person.
They tend not to report tactile experiences
when looking at objects touched although there's some variability in that.
And what we find is that when we look in the brains of people
who are watching this kind of image here seeing other people being touched,
this is not just activating the visual parts of the brain.
So this is a purely visual stimulus,
but the brain does not interpret it solely in visual ways.
And this happens both
whether you have synaesthesia or whether you don't.
So you activate parts of the brain
that are involved in perceiving touch or pain.
So it's almost as if you are embodying what that other person is feeling.
But for a synaesthete, you experience that consciously,
and for other people it's more an implicit embodiment.
Synaesthetes also use some of these same mechanisms,
but they use it in a somewhat different way,
drawing on different brain regions
that probably enable them to experience this consciously.
What does this mean for art
and for other kinds of constructs such as empathy?
What it means is that mapping between the visual sense
and the bodily senses is something we do naturally,
and it may enable us, for instance in this image,
to empathise with people in pain, and so on.
But this could also be an artistic device.
So artists, when they're depicting figures and pain and so on,
they might not understand that this is how the brain works,
but they're wanting people to actually be affected viscerally,
to kind of empathise with what they see.
And this is the actual underlying mechanism of this.
In collaboration with Daria Martin, we've also looked at how these people
with what we call mirror-touch synaesthesia
respond to artworks.
This is a Giacometti sculpture -
Giacometti produced these distorted bodily images -
and this is what one of our participants says:
'I love to stand in front of the Giacometti ...
and it's a very good feeling.
So I love to stand in front of them and feel I am getting longer.'
So what this person is reporting is that their own body feels
like it's being stretched or distorted in some curious way.
This image isn't depicting touch or pain, it's just an unusual body,
but nevertheless what these people have
is a tendency for vision to override their own internal bodily senses.
In a way, they become what it is that they're seeing,
in a way that is really interesting and profound.
So what does this all mean
with regards to the relationship between art and science?
For me, I think one of the most profound things that synaesthesia tells us
is that there really are multiple ways of experiencing the world.
And people have debated this obviously for a long time -
lots of other people have drawn that conclusion -
but science can give us a real handle on that,
and say, actually, there are differences in the brain
we can chase this all the way back to genes
and do the link right from genes to unusual conscious experiences.
But also that these different ways of experiencing the world matter.
They're not just epiphenomenal.
Synaesthetes perform differently on various measures.
So seeing colours, they're good at; they're better at memory, for instance.
They gravitate towards the arts.
These are real differences that affect society.
And synaesthetes aren't vanishingly rare.
They're a few percent of the population, we would say.
I think the other thing that we get from synaesthesia
that tells us about the arts is that it tells us the rulebook
for how the different senses play out and relate to each other,
and this can be used in various artistic ways.
So, for instance,
you can figure out what the best sound is that goes with the colour red,
and you can use this, for instance, in art installations.
One of the ways we're using this is to work with people who are blind
and have auditory depictions of the visual world,
so what we'd want to do is to sonify this image that I'm seeing now
so that they can understand what colours are out there -
the different spatial relationships between objects.
And that's not a trivial problem.
How do you convert something from vision into sound?
But synaesthesia gives you a handle on how you might go about doing that
and have some real-world implications.
So for me, synaesthesia is an absolutely fascinating biological entity,
but it really tells us something about the human experience.
Thank you for your time.
(Applause)