- So, as Michael said, my name is Vinoba Vinayagamoorthy.
But the only person who calls me Vinoba is my mother
and only when she's pissed at me, so Vino will do.
I work at the BBC, as I said,
and my collaborator on this paper was Maxine Glancy.
This project is actually a collaboration
between BBC R&D and IRT, who is on the right there,
I've kind of pushed them outside.
Christoph Ziegler who is with me here is also a co-author.
For those of you in the crowd who don't know about
the BBC's research and development unit,
we essentially look at technologies to
future-proof the BBC.
And IRT does the same for the public broadcasters
in Germany.
And this problem space is essentially about putting augments
using AR technologies for television.
It is not a new problem space,
the idea has been around for quite a while.
But where we defer is I suppose the technology
has moved on to allow the video frame synchronization
between a television and a connected device
in your home network.
So, our general problem space has been
as a public broadcaster,
how do we let our audiences watch, listen and experience
everything that we make anytime, anywhere
and in any way that they want.
So we have a background of work that has been going along
where we are looking at a wide range
of platforms and and devices,
because we have to serve pretty much everyone
with whatever devices they might have.
We like to create well-designed, accessible user experiences
and we tend to select technologies
that might maximize these services in a cost-effective way.
We also like to work in partnership with others,
so IRT is a very good collaborator of ours
for a lot of projects.
And we tend to do, in addition to the sort of paper writing
and distributing of our findings
across academic conferences,
we tend to do a lot of standard specification work.
The DVB Project for instance,
is one for the digital media broadcasting projects,
and we have worked with industry partners
to ensure that there's a way for televisions
to talk to devices on a home network.
Through the companion screens and screen specification.
This means that any TV generally in the UK
that has the Freeview Play patent on it,
has the potential to have a profile of HbbTV 2.0
which will allow synchronization
between a connected TV and devices.
And so, as part of a new project called 2-Immerse,
we wanted to see how far we can push the protocols.
So, before I go on to the actual thing that we did,
here's a video that shows us having two video sources
on two separate devices.
And in a few minutes you'll have a split screen
where you'll see that the top left-hand side
shows the sort of timing information there,
and it's pretty close, it's frame accurate.
I have decided to spare you the inconvenience
of listening to the beeps that goes with
this calibration video.
But it's very exciting stuff,
we had a standing ovation in the office and everything.
It's also a video I've shown over and over again
for the last two years.
Anyway.
So, in this particular project,
our objective was to see how can we use AR technologies
to deliver something quite important
to the BBC Access Services.
Our sign language interpretation
for our television programs.
Now we are the R&D component,
so this is not going to be a service for the BBC.
That is not at all what I am saying.
But we wanted to see if this was a potential thing
that we could do.
So it's essentially a, you know,
passive non-interactive experience,
where you sit on our couch watching our television program
and you have an optical head-mounted display,
in this case a HoloLens,
not for any special kind of purpose,
the HoloLens was the first HMD we had
and IRT had a HoloLens as well,
so it seemed like a good device to use.
Our research questions were:
can AR be used to augment TV programs using DBBCSS?
What should AR sign language interpreters look like?
I mean, the idea is there
but no one has really designed a sign language interpreter
outside the television.
And what do users with hearing impairments
people who would be our target demographic,
what do they think of our AR sign language interpreters?
And figure out the limits and potential.
So, this image hopefully gives an idea
of what we were working with.
The top column shows the sign language interpretation
sort of invision,
which is essentially the traditional way
that sign language interpreters
are shown to the UK audiences.
The bottom right, on the far end, has the German version.
We conducted this project in a sort of balanced way
between the UK and Germany.
We also designed two AR interpreters,
each for each of the countries.
Our half-body interpreter,
that's the middle images on the slide,
and then the full-body interpreter.
They are both gender matched
because we had an English program in Britain
and a German program in Germany,
and we had a female and a male narrator,
so we gender matched to the narrator.
Before we go on further,
I wanted to give this talk with the sort of idea
of telling you things
that I've not actually put in the paper,
because you know, they're kind of funny
and not quite scientific, but I kind of think it.
When we were designing the AR interpreter,
we got to play a bit of dress up.
Now this is an email that I think I could send off
to my signer,
she was very instrumental on trying to figure out
how to build a AR sign interpreter.
Now there are a lot of guidelines in the UK
for how to build one for the broadcast industry,
but no one has actually thought about how to do it
if you are going to project it on a wall outside the TV.
So, essentially I'm telling her that, you know,
these are the colors that we have a chrome-keying
technology which works with the blue color.
If you have black hair, that's gonna be an issue,
please don't wear white, don't wear light blue,
don't wear navy, don't wear jewelry, don't wear earrings.
There's a lot of information that we had to get across
in a nice way, of course.
But Dominique was quite instrumental, as I was saying,
and she was happy to ask questions
and come back and say, you know,
"Would you like me to buy a couple of outfits"
"and come along?"
And so we had to have a dressing area that she sat in.
We had a little bit of a mannequin play,
dress up kind of thing to see the right sort of outfits
she could wear.
I don't think the guys in IRT had this issue.
So they had a green screen, as shown in this image,
and, you know, he just didn't have to wear green.
He also doesn't have to tie his hair up
or no jewelry problems.
So it's a little bitter thing.
So I just wanted to show a little bit
of the conditions themselves,
essentially the stimulus set up in Germany.
So this is the view of the traditional invision
through the HoloLens.
Participants were not wearing the HoloLens
while they were looking at the traditional invision,
but this is not an image I can put on a paper,
so I thought I'll put it up here.
This is the half-body interpreter in close-up.
As you can see, she's sort of near life-size
and kind of fills in from the top of the TV frame
to the bottom of the TV frame.
And if you're looking at it through the HoloLens,
then she's sitting on the TV stand.
Doesn't look like it on this image,
but take my word for it.
And then this is the full-body interpreter.
She is grounded on the floor
and she goes all the way to the top of the TV frame.
So this is a video that we have.
As you can see it's very similar to traditional signing.
There's a resting phase
where she's looking at the television
as if she's watching it with you.
And then when she starts interpreting,
she starts talking to the viewer.
And I love her,
I mean this is my favorite condition I think.
She looks very much as if she's there.
I mean I don't use sign language on a regular basis,
but the minute I saw this I got to thinking
surely this is going to be the best condition
out of all three.
Oh, sorry.
That video doesn't play automatically because, you know.
Anyway, so this is the half-body interpreter,
and she's also quite larger than life
and she's quite there, but cut off.
The full design of the interpreters are available
on a work in progress paper from TBX last year.
But this is a summary of the guidelines that we used.
The heads of the AR interpreters are aligned
to the top of the TV.
The bottom edge of the half-body AR interpreter
is aligned with the bottom of the TV frame,
almost sitting on the TV stand.
The feet of the full-body interpreter
are grounded to the floor of the physical room.
The direction of the gaze of the AR interpreter
has to be horizontally facing towards the TV
while in resting phase
and has to be directly facing towards the viewer
during the interpreting phase.
And positioned to the right, just outside the TV frame
with a very slight overlap over the TV
in order to minimize distance between
the content on the TV and the AR interpreter.
And these are all guidelines that we took
after talking with our pilot candidates
and also the sign language interpreters themselves.
And then we designed the study.
The study itself again was mirrored
in both the UK and Germany,
so we had a training phase initially,
we don't have the weather program that we used,
but we had a weather program where participants
could sit with us and talk about the AR technology.
They could whinge about how heavy the HoloLens device was,
they could talk about how they had to keep their body
quite straight,
essentially get their frustrations about the technology
out of the way.
We really wanted to focus on the design of the interpreters
and not on the limitations of the technology.
So we had a training phase where they could
kind of get used to the whole aspect of it.
And it took a little bit of time,
longer than we would with people
who are more used to the technology.
But I think this was an important phase.
The programs matched the languages of the country,
so you know, we had English and British sign language
in the UK, BSL.
And German with Deutsche DGS in German,
I cannot pronounce it
but Christoph over there will be able to do it.
I'm just going to call it the German sign language.
The gender of the interpreter was matched to the narrator,
and each participant who came to us saw three clips.
The clips were themselves on a continuous storyline.
So that was the first clip,
that was always the second clip
and that was always the third clip.
But the sign language interpreters themselves
were randomized.
So people could, for instance, see the traditional,
followed by the half-body, followed by the full-body,
or traditional followed by full-body and half-body,
and all the other permutations.
But the storyline was kept intact.
There we go.
Our participants all had hearing impairments
and they were all regular TV watchers,
about more than an hour a day.
They had no biases against factual documentaries
and they didn't hate public service broadcasters.
I'm not sure about Germany,
but in the UK there are folks who hate the BBC.
I don't know who these folks are and I don't know why,
but they're there.
So we asked for people who did not
particularly have that feeling against us.
The folk who we invited in the end
stated that they had a preference for subtitles
because it was perceived as more accurate.
And they used it to improve
their written language skills.
But they did think the sign content,
the signing made it a more comprehensive thing
because you had a lot of emotions and things like that,
which were not captured in subtitles.
And this was especially the case if the participant
only had a basic command of the written language.
You have to understand that sign language
is a language in itself
and it's the first language for many of the folks
in the community.
All the participants understood the sign language
and unsurprisingly participants mentioned
that they could only lip read on TV
if the subject was facing outward, facing them.
All participants also reported they were expert-levels
of familiarity with computers,
but only one of them in each country
called themselves an AR expert.
This is essentially the demographic information
that we have.
It's nearly sort of gender balanced.
We have quite an older sort of range of folks
who came to us.
And the folks also had different ways
of describing themselves.
So the profoundly deaf, hearing impairments,
hard of hearing, all those are key terms
that they used to essentially tell us
what their particular impairments were.
We also have collected information
about how proficient or fluent or basic,
whether it was a first language or the main language,
and all those sort of situations.
Extra details all in the paper, so please have a look.
We then had some post condition questionnaires.
After each of the condition in the three conditions,
we had questionnaires to ask them what they thought
of that particular condition.
And we also had a post study questionnaire at the very end
to compare in between conditions.
And we had an interview.
But the interviews was more conducted in parallel
to the interviews, the post study questionnaire interviews.
So whereas we would have essentially got
folks to sit down and do a questionnaire by themself
under normal conditions,
for this particular study we chose to help them along.
So we would sit down there and conduct the interview
and get their responses in person and type it all out.
We had some findings of mild interest.
Again, it's in the paper.
But in the UK, the responses on whether they liked it
was split three ways.
For the questionnaire data.
And in Germany, it was sort of split nearly halfway
with preferences for the traditional winning out,
with then the half-body,
and one person said they liked the full-body AR.
The more interesting findings came
in the qualitative information that we gleaned.
So novelty factors.
We always have factors that they wanted to improve on.
So for instance, participants always had opinions
or expectations of what they call a hologram would be.
So "I've always wanted to watch a hologram"
"and there has to be something"
"in which a hologram just happens."
"Something like a projection."
"Rather than something you have to put on."
So there's an already an idea
of what the technology should be capable of.
They would like to get things
like view closer to the videos.
There was also a question of cost and barriers
to adopting the technology if you had cochlear aids,
that was an issue.
So these are all factors that we hadn't taken into account
and we should in future studies.
Physical discomfort.
"Having glasses on your head for two hours? No way!"
"And I can imagine you put them on and everyone"
"will look at you and they will recognize you are deaf."
"I don't like showing the outside world that I am deaf."
"I feel too vulnerable."
So there are also these personal feelings,
very human feelings of problems that you have to consider
when working with this particular community.
"For a longer film holograms are unsuitable,"
"since that our eyes got tired fast."
"In this case subtitles are better."
"Maybe in the future holograms could be displayed"
"without HoloLens or other glasses?"
"Might be difficult for women with wonderful hairstyles?"
"I am also wondering about people"
"who wear distance glasses,"
"can those glasses (fashionably bigger ones)"
"fit into the HoloLens. Like Dame Edna for instance."
We did not have Dame Edna, but you know,
these were all very human problems that came across.
Position of the interpreter.
They liked the fact that in the traditional method
the picture in picture was closer to the actual content.
Even though they were impressed with the HoloLens,
so with things like you know,
with traditional method you can see everything
within your eye-line.
You have to move attention from one to the other
in the HoloLens, so the distance was a bit of a problem.
Conditions like a game of tennis.
"I wanted the signer on the picture"
"even if it obscured the picture itself."
"I think the full-body interpreter is better."
"However, it would be worth trying to place the interpreter"
'half-body' over the TV"
"Because you will not be so left or right heavy."
Same to do with the clarity of the interpreter.
"Position and opacity of the signer needs to be controlled"
"by the user to make a huge difference."
"If I could move the signer"
"and if I can change the luminosity of the signer,"
"make her darker?"
"Same with subtitles - different size, different colors."
So it went on with essentially people telling us
that they wanted control over the way they had signers.
In terms with traditional and AR, for instance,
folks wanted to be sure that they could figure out
which signer to use, where to place them,
whether they could add subtitles in AR as well.
So, final discussions.
"Having control over where to place the signer"
"would be good."
"People have different preferences and it would be good"
"to adjust to type of program, people's visual capability"
"and wallpaper."
"I don't have time to make adjustments to such a system"
"and fine tune things, there should be something universal."
So what came out very clearly was that personalization
is key, because the preferences are quite individual.
Some folks like the emotional type of interpreters,
the other folks they want more information.
So, again, all of these are in the paper
and I think I'm running out of time quite quickly there,
so I'm gonna stop and show you this first.
We do have more pieces of work
that will be published in due course,
including production guidelines and things like that.
Thank you.
I don't think we have time for any questions, right?
(clapping)
- (offscreen male speaker) So unfortunately
in this case I would invite you
to talk to Vino after, you know, after the session.
So thanks Vino.
- Thank you.