Welcome, I'm Abel Valente,
as Simone, thank you, said.
I'm from IBM.
I'm going to speak about the work we did last year
about career recommendations and
in the context, in particular, of a chatbot.
So I'm gonna go through a few examples
of the techniques we focused on,
and then details on how we implemented our recommender,
and finally, how we measured the performance
of the recommender and the system all in all.
So consider a network of jobs and transitions.
We want to discover common patterns
as people flow through this network.
For example, here we have two patterns.
The first one on jobs one, three, and four.
And two, three, and five will be the second pattern.
Besides the actual list of jobs,
there is information that we gather from the jobs
and from the people traveling through these paths
that we want to use to make the recommendations.
So I'm gonna go through a few examples
of how we want to use this information.
Consider, first, if first user which is in Job1,
then we likely will use Pattern1
because the job is part of the pattern
and we should be recommending
either Job3 or Job4 for this user,
mostly based on the flow defined by that path,
but also additionally some information about similarity
of the profile of the user and the profiles of the jobs.
Consider a second example
which may not define a clear trajectory.
Through our switch,
we want to get data the user.
In this case, we will use some other information
such as past jobs.
In this case, it's clear that the user
is matching the second pattern.
And thus, we should be recommending Job5.
A second example will be one
that doesn't have information about the past job.
And in this case,
we will run a similarity analysis
of the profile of the user and the profiles of the patterns.
And in this case,
if we consider the user to have a skill d,
we should be, again, matching Pattern2
and that's recommending Job5.
So that's the basic idea.
And we put this in the context of an interactive chatbot.
That means that the user will be running this process
of recommendation several times.
And as it goes forward,
it should collect feedback from the user
and fine tune the recommendations
and eventually converge
to the job that the user is looking for.
So I'm going to go through the details
of how we made this work.
First, the training of the system.
The first thing we do is collect those patterns.
For this, we used an existing algorithm
from the literature,
which is maximal sequential patterns algorithm.
After we collect the pattern,
we collect also additional information
from the jobs and the users taking part of these patterns.
And that's basically the center of the model
that we used to produce the recommendations.
So let's go to actually the process of recommendation.
First of all, we collect all the relevant patterns.
And by this I mean all the patterns
that intersect with the user.
And this would be the ones that are mostly used
to produce the recommendations.
Then we collect the candidate jobs
from all the jobs that we have.
And this is coming from mostly the features of the patterns,
which is for those relevant patterns,
the jobs that lay ahead of the user
from the left intersection point with the pattern.
And additionally, we also collect jobs that
feature some intersection
between the profile of the user
and the profile of the patterns.
So after we collect all the candidates,
we run them through a scoring function.
The scoring function is composed of mainly three aspects.
The first one is the pattern scorer
which is coming from how well the user matched the pattern.
In this case,
it's composed of mostly asset similarity,
function, which is comparing the profiles
of the pattern and the user,
and those who are backward compatible with the score.
And this is reflecting how old is the information
that was used from your profile to match the pattern.
So if we match the pattern only
because you have a very old job
that was in the pattern,
this may not be relevant for you.
A second angle that is present in the scoring function
is the proficiency score,
which is basically how well we estimate
that you will do the job.
This third and important angle
to the scoring function is the preference score,
which is how well this job represents your preferences.
And this is related deeply
with the experience of the dialog,
because this is where we will put
all the feedback collected through the dialog.
This is mostly coming from three areas,
opinions that you gave on the jobs,
opinions that you gave on the skills
and the features of the jobs,
and also an opinion of whether you are looking
for a change in the short term
or in the long term in your career,
which we call the seniority or stage.
Finally, we combine all the scores.
And in particular,
I want to point out that the preference score is important
for us because it's deeply linked
with the chatbot experience.
And thus, we want to react and give the user
a feeling of control when they are on the chat,
and make sure that the recommendations reflect
some change when they give an opinion.
And that's why the preference score
determines the whole score.
If the preference score is zero,
we completely discard the recommendation.
After this, we determine the top N recommendations
and return them to the user.
So that's basically the recommendation process.
And in addition to that we,
for improving the experience in the chatbot,
we also produce a list of features
that are good splitters
of that top N recommendations.
We make this because it could happen
that the user gives an opinion
of an aspect of a job that doesn't really impact
the top N recommendations just given to him.
And in the iterative nature of this interaction,
we hope every feedback to really count
and reflect on the new recommendations.
And if the user gives an opinion on a feature
that doesn't really relate to this top N,
it might not change,
which is kind of a bad experience
and a bad perceived control
for the user within the chatbot.
So that's the central points and I'm going to go Through
how we measure the performance of the system.
And this is mostly done in two areas.
First is the actual recommendation
as in one call for the recommender.
And the other one is trying
to measure the interactive nature
and how well these feedback channels
allow the user to get to better recommendations.
I'll go through the first one now.
So if we consider a test set within our data,
we split the careers of the job and tried to predict,
as ground truth,
the second halves of the careers of the user.
What we found is that when done in two datasets,
one public and one from a company that allow us
to publish the statistics,
that the results are performing pretty well
compared to classic solution from the literature.
In this case,
I'm comparing three variations of our system
against two classical methods,
which is a popularity-based recommendation
and the collaborative filtering recommendation.
You may see that the first row from our solution,
which is ICGR,
is performing slightly better on the first dataset
and somewhat even better on the second dataset.
And what I want to point out also is
that there are two more variations of our system
in which we deactivated some components
of the scoring function.
And the first one,
which is entirely based on patterns,
it performing much better actually than the combination.
And the second one,
which is only the skills,
is not performing very well.
We seen this in the two datasets
and we attribute this to poor information
because it was hard for us
to actually find good descriptors for those positions,
to have a good skill list for those jobs,
and other limitations.
The second part of this evaluation is long-term
and is trying to predict jobs that are
a little more ahead in the careers of the user.
In this case,
we skip the first jobs in the ground truth
and we run the experiment again.
What we found is that
the classical methods couldn't do this, actually.
And we were not able to predict the jobs in the long-term.
Whereas in our solution,
we were getting some results
even if the skill configuration,
which is still showing the same signs with less performance,
but still producing some results.
So for the second part of the evaluation,
we tried to make sure those channels of feedback.
In this case,
we made artificial users that would give the chatbot
some aspect of their desired job
and express those feedbacks through different strategies.
So after this we see
if eventually we get to the ground truth
and how many turns in the dialog it takes.
So we found that considering a baseline
of an artificial user that doesn't give any feedback,
we see that this user which essentially
calls the recommender every turn of the dialog
and only says, "I didn't want these jobs.
"I don't like these recommendations, give me more."
So it bans those jobs from the recommendation process
and just calls again.
That takes an average six dialog turns to get there,
to get to the ground truth.
Then we made three actually interesting artificial users,
which provided some skill information
and some seniority information.
And what we saw is somewhat expected.
The ones that give skill information,
which we actually already have saw
that this was somewhat poorer in predictive power,
do get better.
And it's reducing the number of steps
that the user has to take to get to the ground truth.
But it's not so much of an improvement.
So in this case,
it's five turns and 4.4 turns of the dialog
to get to the ground truth on average.
Finally, the interesting thing was that
by giving the expectant seniority of the job,
which is basically...
So if you imagine those patterns
that I showed at the beginning,
you will see those as a line
and the seniority will be an index on that place, right?
How far or deep those path you wanted to go.
In this case,
it was a good predictor
and we will get 2.2, on average,
dialog turns to get to the ground truth
or even less if you consider
the actual calls to the recommender
because, well in this case,
we assume that the first call
already contained this information.
Finally, the solution combined is somewhat in the middle.
It's an arbitrary combination of those averages.
All right, I'm a bit early.
So as a take away,
the recommendations were better
than the classical baselines.
The elicitation process,
where the user gives feedback through the dialog,
makes the recommendation better
and it improves all the amount of information,
the amount of effort that you have to put in the dialog
to get to the ground truth.
Also, I wanted to point out
that the technique is simple and very explainable.
You can track down the origin
of the recommendation very easily.
And finally, we do have a pending user study
which we expect to see whether the recommendations
actually are perceived well by the users.
There could be a difference between the actual ground truth,
which is the actual jobs the user have,
and what's perceived as a good recommendation for a job.
That's my talk, thank you.
(audience applauds) - All right, let's thank Abel.
(audience applauds louder)
And we have some time for questions.
- [Male Audience Member] Sorry, I might have missed this,
but you said that this is all delivered through a chatbot.
So does it matter
that everything is delivered through conversation?
- Good point,
so actually the measures that we took are independent.
These strategies that we made are completely analogous
to something that you could have done through another means.
And we expect to see impacts
when we actually try with humans
because of the experience
of chatting through natural language
could make different experience overall.
So we expect to see more results on that area
once we have the user study.
Thank you.
- [Moderator] Okay, more questions?
Well, in that case,
I have a question.
So it actually does relate to the user study.
I think one of the assumptions
that you had going through the artificial user
is the quicker interaction,
in a way, the fewer turns the better.
- Yep.
- [Moderator] Are you able to say anything more about that
and what your sort of finding already in the user study?
Do you think that that is actually going to be better?
- Well I may have something on there, actually.
I commented on something that I didn't say earlier,
which is this.
So there are two strategies
for private feedback related to skill.
The first one was based on the discriminator list
that I commented earlier,
which was aimed at providing a better perceived control
for the user.
And actually, that doesn't have
the objective of conversion earlier
to the best recommendation.
So it could be that you are interesting me
in a better experience in the chatbot
rather than conversion faster.
So it's gonna be an interesting thing to see in the study
how those two approaches balance out.
- [Moderator] Do we have any more questions?
Okay, let's thank Abel again.