Hi, I’m Haijun Xia, an Assistant Professor from University of California, San Diego,
Crosscast is a joint work with Jennifer Jacobs from UC Santa Barbara, and Maneesh Agrawala
from Stanford University. Just within the United states, There are more
than 20 million people listening to travel podcasts to learn about new locations and
get inspiration for future trips. However, the primary appeal of travel is the visual
experience of seeing famous landmarks or breathtaking scenery. While other forms of travel media
focus on recording and delivering engaging visual experience, audio travel podcasts lack
a visual component. Here is a short clip of a travel podcast about Tokyo.
The Tokyo of our sort of visual imagination, which is, the blinking signs, this sort of
like.. blinged out ... Giant intersections, yeah. Yeah, giant intersections, very busy
kind of like sides of buildings, very tall buildings.
If you have never been to Tokyo, it’ is very hard to imagine what Tokyo looks like
just from audio. The visual nature of travel suggests an opportunity of enhancing travel
podcasts with relevant, high-quality visual content.
Crosscast is a system that automatically adds relevant images and maps to travel podcasts.
Shinjuku is among the more famous. It's the one that people ... Well, there are multiple,
but I think, it's one of the ones that people think of most strongly when they imagine the
Tokyo of our sort of visual imagination, which is the blinking signs, giant intersections,
very busy kind of sides of buildings, very tall buildings. Shinjuku is tall, for the
most part. There are short parts of it. But, Shinjuku is also the place where Yakitori
Alley exists. And that sort of like, on the other side of the train tracks.
Taking the audio and transcript from a podcast as input, Crosscast fist identifies relevant
locations and visually significant entities being discussed at any moment of the podscat,
combine them as search queries to obtain relevant visuals, and aligns the visuals to the audio
to create an audio-visual experience.
The core of crosscast is the heuristics-based algorithms that score the relevance of locations
and visually significatn entities, based on four factors, freshness, specificity, subject
relevance, and detail relevance.
Let’s take a look at some examples.
A newly mentioned location is likely to be of interests for the listeners.
I'm fascinated by Akihabara. Yeah. The gaming neighborhood, which to me is just peak Tokyo
culture that there's neighborhood dedicated to it.
Here, Crosscast shows the map and images of Akihabara when it’s mentioned. However,
it’s challenging to identify the most relevant location, when multiple locations are mentioned
within a short period of time.
After describing the details of gaming culture of Akihabara, the hosts made the following
comments:
Which I just think is sort of one of those only in Tokyo. Nowhere else in the world.
Only in Tokyo. Only in Tokyo. All those themed cafes. They got hedgehog cafes. The maid cafes.
Despite Tokyo being mentioned 3 times, Crosscast is able to maintain Akihabara as the most
relevant location based on two heuristics:
First the more geographically specific locations are often more relevant,
Crosscast utilizes the hierarchical relationship among locations, and assigns higher scores
to more specific locations.
Second, locations described in notable details are usually more relevant. Here, details of
Akihabara, such as the gaming culture and theme cafes, were mentioned, Crosscast is
able to capture these details by matching the content in the transcript with the wikipedia
description of Akihabara.
Crosscast also identifies the relevant locations based on how they are mentioned. In this clip,
But there's this neighborhood called Koenji which is not far from what we're talking about
Shinjuku and all of that.
Here, Koenji is more relevant because it’s the subject of the sentence, whereas Shinjuku
is the object for comparison.
Crosscast infers the relevance of an entity to the subject using the grammatical structure
of a sentence, where the root of tree, corresponds to the main verb in the sentence. The closer
a word is to the root, the more directly the word is related to the main verb, and the
more relevant to the subject of the sentence.
Utilizing these design heuristics, Crosscast generates audio-visual travel content in a fully automatic
manner, and enables podcast creators and listeners to quickly produce, iterate, and enjoy the
desired visual experience.
It also opens future directions of leveraging domain knowledge and linguistic signals to
automatically collect visual content to accompany audio and text for other domains such as science,
news, history as well as with more diverse visual layouts and effects.
Thank you.