- Hi, everyone.
My name is Zang from University of Illinois
and Urbana-Champaign.
So, as you may hear from today's key note,
I'm here to present a normal use of conversational agent
that aim to help team effort.
My collaborators are Dr. Zhou from Juji and professor Fu
from University of Illinois.
So, as an old saying goes,
"Sticks in a bundle are unbreakable."
Teamwork: one's ability to collaborate with others
and effectively work in a team was identified as one
of the most important and desirable skills
by many organizations and companies.
Therefore, in education environment, it is crucial
to engage students in team projects for them to practice
and thereby improve their team work skills.
In other words, we should equip students with adequate
teamwork skills and experience before they get
to the real world basis.
Bad team experience may prevent students from future work.
But, forming effective teamwork is not easy.
To form effective teams, instructor need to know student
well before forming teams, which makes is especially hard
for the time frame in the educational environment.
That is, a semester is usually only 16 weeks
and a quarter is only 10 weeks.
Also in educational environments, students may came
from diverse backgrounds.
Therefore, no one team formation formula can fit for all.
Also instructors who are not organizational
psychologists, although they know their classes the best,
without knowing the the right formula
it's very hard for them to without (mumbles)
it is very hard for them to actually come up with a formula
for forming good teams.
So, meanwhile, understanding team composition is hard
for researchers as well.
First of all, team goal varies on a different context.
Best team formation formula in one situation may not
generalize to others.
Second, oh sorry.
Yeah.
So, second from team personality composition
to individual team members characteristics.
There is so many factors that effect team dynamics.
Therefore, an effective team is required to aid team effort.
So, in this paper, we'll be introducing INDIGO:
Individual Differences for Group Optimization.
This is our novel use of consular agent that aim
to activity engage with student.
Get information such as student's team preference
and their team expectation.
Infer key factors such as a student personality trait
that may have effect team dynamics.
Also, INDIGO is intended to design a longterm team
companion that follows students throughout the semester.
So, I'll first introduce the system behind INDIGO.
So, INDIGO is a consular agent that developed
on Juji's chatbot platform.
There three reasons of using such platform.
First of all, chatbot can be easy to configure through GUI,
which lowers the barrier for instructors to customize
INDIGO and for researchers to adapt INDIGO
in the future usage in different scenarios,
and replicate our study results.
Second, the chatbot should be able to function robustly
and support long conversations,
since understanding the student deeply.
A normal human interviewer may take more than 30 minutes.
So, we want our conversational system to have
the same ability to conduct long conversations.
So, third, Juji's chatbot can automatically infer
student big personality traits during the interaction
without directly asking any personality related questions.
Which avoids such as our ability issues in a paper based
traditional personality test.
So, to validate the take of INDIGO,
we asked three research questions.
First: how well students interact with INDIGO?
What's their perceptions or impression of INDIGO?
Second: how effective can INDIGO get information
from student?
Third: how well can inferred personality provide insights
into team dynamics and team performance?
So, to answer our research questions,
we did a field deployment of INDIGO in a classroom.
It is an undergrad level user interface class where
a semester on group project is emphasized.
So, the goal for the group project is to create an
innovative interface that aim to solve real world problems.
Student will start from brainstorming and end with a
hyphenated prototype.
The group project constituted 40%
of the student final grade.
There were 184 students in 40 groups consented
to join the study, and INDIGO interact
with this student twice.
Once at the beginning of the semester
and once end of semester.
While the course is offered
in the computer science department,
there are students from non-engineering majors.
For example, there are came from industrial design,
information science, business, and psychology.
To investigate whether INDIGO could offer a use for insight
for team function, we wanted to learn the existing team
formation cues.
In our study, half was grouped or formed randomly,
and half of the group were formed by a widend use
team aiding to academy.
So, in here I gonna have a brief introduction
of INDIGO's interview.
So, the INDIGO's project interview has five stages.
The first stage is a warm up where the student and INDIGO
introduce themselves to each other.
And the INDIGO will explain briefly the agenda
of the interview.
Second, the student will ask to fill out an impression
management scale since the IM scale is highly related
in the self-control and social adaptation in work place.
We hypothesize that it might effect the team dynamic.
Third, student will ask about their team preference
and the past experience.
Then the next section is about individual's characteristics
such as their strengths and weakness.
Lastly, student will ask to some questions about their
impression of the INDIGO.
So, the last section is solicit student feedback
about INDIGO.
The student will ask to express their impression of INDIGO
and rate INDIGO on a number of scales.
Such as, its helpfulness and likability.
So, here I want to highlight some results from our study.
So, first, we show student interaction with INDIGO.
On average students spent 60 minutes and typed 620
words in a pre-project interview, and 20 minutes
and 289 words in a post-project interview.
This project may demonstrate that the student were willing
to spend a considerable amount of time with INDIGO,
and offer information steering their interaction
with INDIGO.
And here, we show the student perceptions with INDIGO.
Students were asked to read INDIGO's
on five point Likert scale.
Including the likable, helpful, enjoyable and trust.
And also asked to describe INDIGO in three key words.
As we can see here, the students pursued INDIGO
as a likable , helpful, trustworthy, but not very enjoyable.
Which aligns with the robotic impression
we can see over there.
So, one student also provided feedback saying that
INDIGO is not a huge conversationalist.
Not too much in comparison or talking out of a script.
So this we believe may reduce enjoyment in the conversation.
Since we are with INDIGO always follow the script.
So, additionally, student perceived INDIGO as a more
as counselor than a friend.
So, further, we show the student who feel INDIGO has
a similar personality to their own tends to enjoy
the conversation and trust INDIGO more.
The correlation here is statistically significant.
So, moving to the next research question.
Yeah.
So, we compare some questions that ask
by those academy and INDIGO.
When analysis opened these results we gathered from INDIGO,
we performed LDA and essential analysis to extract the
key topics students mentioned, and we found that students
provided rich information that won't be able to copy
by choice based questions.
Which were often using existing teaming to academy.
Second from the conversations, students expand more
about their input.
Next I'm gonna introduce an example.
So, this question is asked by those academy survey
choice based questions from later to follower
and a open ended question in INDIGO's conversation.
As we can see, student responses in INDIGO not only
cover the later and follower,
but also one point did mention that it wants
to take turns during the semester.
Which won't be able to capture the predefined
choice based survey.
So, one might argue that open-ended question could be
inserted into a regular survey
to capture needed information.
However, extensive survey that testing shows that people
are waiting only waiting to spend a few seconds per
question if the conversation, if the surveys lasts more
than five minutes.
So, our use of INDIGO indicate that the students are willing
to spend time interacting with the agent
and offer useful information.
Which suggests an effect way to carry information
from student.
So, next we're gonna show the insight, present the
insight from the INDIGO on the team result.
We first did an analysis of 35 personality facets,
and the result give us 7% in makeup
that best fit student profile.
Including the emotional, collaborative, social,
open-minded, responsible, sensitive, and self-disciplined.
So, then we finished several regression models
on the team outcome.
We found that groups who had a higher variance on their
emotional makeups tends to perform better in the end.
One potential explanation here is a student who are
low on the emotional factor are often more anxious
about the deadlines, which helps the team to always
be on time.
In the meanwhile, students who are in the group
have a higher in the lower anxious about the deadlines
they may keep always keep the group in good shape.
Such composition eventually will benefit the team.
So, another model that shows that the group who are
less sensitive in general has a lower conflict according
to the previous research.
Students who are less sensitive in emotions may able
better cope with in difference in intentions,
which reduce the team conflict in the process.
So, in here we, the study finding demonstrate INDIGO's
ability to proactively engage with students,
gather useful information, and provide insights
on the team outcomes through the lens
of the team personality composition.
Such findings can benefit the design of the intelligence
system team in choosing general.
First, from the interaction between INDIGO and student
in the class, we learned that the student who trusted INDIGO
and perceived INDIGO as a counselor.
In the future design of a search of an intelligence
agent for teaming, we could leverage such perception
and frame the agent as team coach.
So, similar to the counselor in real life,
who could follow up with their kind,
team coach can follow up with the individual team
and team members
throughout their team efforts, and provide help
to reduce the potential team conflicts.
So, second, our results show that students prefer INDIGO
more if they perceive INDIGO having a personality
similar to theirs.
Which is personality influence the capability
of the intelligence such as INDIGO can learn a user's
personality on the fly, and then adapt this behavior
to that of the similar personality.
For example, they can mirroring the user's the wording
of the user's are using.
So, third since our finding reveals that team personality
composition predict team outcomes, such as team performance,
and the team dynamic.
We can use such findings to argument INDIGO.
In particular, we can extend INDIGO to automatically
recommend team formation based on the inferred personality
trait of the potential team members.
For example, it can select each team by matching the
variance of the emotional characteristic
of the team members.
Also, we want to recognize the limitations of the study.
First, the measure of the team performance was submitted
on a one project score.
From the example we know that the project score does not
fully reflect the team work skills,
and what student have learned in the process.
Second, our field from INDIGO was situated in a CS class
in a large US public university.
Which may not be representative of student teammate
situations in other cultures.
Third, the team perception was collected once at the end
of the semester.
We want to collect team perceptions throughout the teaming
efforts that detects the changes over time.
The best access of the team dynamics.
So, as a conclusion to summarize our result,
we examined a novel use of conversational agent
in aiding team effort.
INDIGO can proactively engage with student and
collect valuable information.
And, INDIGO also can provide insight on team outcomes
throughout the lens of the team personality composition.
So, at the end, we want to thanks all for your listening,
and we would like to thank all reviewers
for their valuable feedback,
and the Air Force Office of Scientific Research
for its support.
(applause)
- [Moderator] Thank you, are there any questions?
- [Speaker] So, it's great that personality is a factor
in the overall performance of the team,
but there are lots of other factors
such as skill, for example.
And, I'm sure things vary by what the task is.
So, how important is personality compared to other factors?
Did you test that?
And, did you
and, is there even one thing,
which like, okay,
teaming is forming a team that, do you need the same mix
of personalities for every task?
Or is it just, or is it varied by task, for example?
- So, I think this is a great question.
Since there is a lot of factors that effect the team
performance and team dynamics.
So in our study, we do control how the team performs.
So, in the academy, the instructor forms a team
based on the, like the team members, their skills,
and their preference in the leadership, and also their
thinking styles.
And we didn't find any significant effect
from how the team were formed.
But we do see that the personality play a role here.
But, like, as we said that the some research in
the personality also demonstrate that the team work was in,
kind of, because of the varies of the team goal,
the team personality composition may varies.
So that's why we want to apply INDIGO in different
scenarios to see if there is a personality composition
that best fits for certain scenario, or if there is
a general personality composition that works
for all the teams.
So, in the future study, we want to, like, just deploy the
INDIGO and also let the other researchers in different
classes use an instructor use INDIGO to give us more
insight on that.
- [Moderator] Thank you, any more questions?
- [Male Speaker] Oh, I have one.
You know because and it may be a question more for
Michelle and I don't see outright, but you know,
as I understand it, you really use this for students
to team up, right?
And you know, I mean the results look good,
but would you do it again?
(laughs)
You know, in the next class, next year, do it again?
(woman's voice)
(laughs)
- [Male Speaker] Okay, yeah, so it's good to hear.
So, this work we did in, I think is one year ago.
And this class still going on, and we still use this
conversational agent to kind of interview students,
get information, and we hope to get more data to tell
a more robust model that can do the team recommendations.
- [Male Speaker] Excellent
- [Moderator] Any more questions?
- [Male Speaker] (mumbles) Would this style work
if you were trying to form basketball teams?
- I think that maybe, since they haven't tried
this scenario, and we know that personality can
really predict some outcomes, and we can see, like,
the personality, for example, if the team player already,
kind of, agreeable to each other,
the team may be more united,
and they may perform better.
But, it's a very interesting scenario to test our agent.
Yeah.
(laughs)
- (woman's voice drowned out by background noise)
you're asking about the task we are dealing, right?
So, the task we tested in this particular scenario
it's kind of a short term task, right?
Because its only one semester.
So, we think that why a reason why the emotional composition
showed up very strongly you can see the
(voice becomes mumbled)
so, then we said, it may not work as well for this
particular composition for the long term
are team engagements, like the basketball.
Let's say basketball team they have to last
for several years, right?
That's why in the scenario we're very kind of interested
in trying it out.
How the personality composition, as Zang mentioned that
maybe the compositions will be different
for different teaming situations.
Structurally, they tested I think, (mumbles)
so, the academy, they tested about 29 surveys.
So they offered 29 other attributes from students
they collected from academy tool.
And, none of them is predictable.
That's how amazing. (laughs)
None of it was predictable!
Another thing is including the student's previous
academic performance, right?
So, that's wasn't even predictable either.
But in terms of a team outcome and the team dynamics.
That's why it made us very excited about this.
- [Male Speaker] Thank you.
Let's thank the speaker