[MUSIC]
I just heard a lot of theory, let's call it, about what makes partnerships work,
but sometimes it's helpful to hear bad stories too.
When it goes wrong, what goes wrong?
What is the big driver?
And hopefully, and this is going to be a question for you, Josh.
>> Mm-hm.
>> I'm sorry.
[LAUGH] But I think you made one point that I wanted to highlight if you can
try to fit in, which is the difference between kind of generalized knowledge and
just kind of a story.
And so is there any lesson that permeates through a sequence of failures
that you think is an important lesson of what not to do with the story of,
I told you this question in advance.
[LAUGH] >> It's okay.
So I think that, I guess,
the aspect of the portfolio approach I think is very,
that Susan was talking about, is something important.
A good thing not to do is to be very narrow in the problem you're going to work
on or the people you're going to potentially bring to the table.
Because then you're just unlikely to get a match up
where both the academic is happy and the partner's happy.
So I think that is definitely a possiblity.
The other nice thing about the portfolio approach is that there's going to be
failure.
We talk about failure in different ways.
But one of the big failures is that your champion within
the institution of the partner goes away.
And that champion is so important to the success of the project.
And you talk about financial services firms, the big ones,
they are intentionally moving these executives around.
And so you start with something some executive working on something and
they get transferred to another area.
A new executive comes in and says, well, if I take on this thing and
it's successful it'll be viewed as that last guy's success and if it fails,
I'll be stuck with the baggage.
So they generally kind of walk away.
So the more you can have a number of irons in the fire and a number of things
you're doing with one partner, the more likely you are to be successful.
And you get the benefit of sort of spreading that
relationship you developed over a number of different projects.
>> Do you want to answer?
>> Go ahead. >> Okay, so,
one other tension which I think you guys kind of both talked a bit about, but
I want to focus on is the tension of measuring impact.
And the basic problem that, so I'm going to just pose the problem, and
then the question will be like, what do you think of that?
Which is, at least, as I see the problem is that it's easy to make poor claims.
And so even well-intentioned people might be running a for-profit firm that
needs to make money, but is yet attracting money from an impact investing space,
maybe it's below market rate or maybe not.
But they are telling a social impact story.
And so there's often a tension there with the question, the claim they're making is
a long-term claim about long-term impact, it's about habit formation.
It's something that, by its very nature, is long run, and
that's the interest of the academic.
But the problem is there's there's kind of short run and
maybe bad metrics that can be used that satisfy too may people.
And so they can kind of raise the money they need off of bad metrics.
And so there's an incentive problem.
And that's my non-question question and so the question is what do you think of that?
>> [LAUGH] >> [LAUGH]
>> Yeah, I guess that was a,
in my for-profits sector work, that's a huge problem.
Because you don't want your colleagues to not get their bonuses
because the short-term metrics don't look good
because you were investing in long-term impact.
And so that was a constant tension, when you would sort of say, all right,
wouldn't it be better to do things this way.
But it's going to to look bad for three or six months and
actually that was a problem for Yahoo kind of on its way out that all they
cared about was the next month and trying not to get acquired.
And so you could see a lot of things went wrong, putting up more and more ads, and
other stuff that just were not consistent with the long-term impact.
So one of the things that I've tried to do out of that is really to,
I think at that point you have to solve the whole problem.
You have to get some of the key decision makers, whether it's higher-up executives,
or its funders, in my board positions really making sure everybody agrees
that this problem will not be solved by short-term metrics.
So let's make sure everybody understands that the way you're being evaluated for
this particular issue is a different metric.
And if you didn't solve that,
it was just very hard to get people to buy into doing the right thing.
But on the plus side is that people really want to do the right thing.
It's not fun to work for any kind of company,
when you know what you're doing is gaming this month's numbers.
Maybe it's exciting for a month to see them pop, but it's not a satisfying job.
And so actually people will be very motivated to go to some
effort to actually do the right thing.
I think most people understand when they're not doing the right thing.
>> I would just add I think the problem's even compounded when you're talking about
social issues.
So you take something like college completion, right?
We don't really know if you're successful for, if it's a community college two or
three years, if it's a four year institution, four to six years.
And you're looking at things like, well, did we get students to fill out
their financial aid form in a timely manner and get more financial aid?
because we know small amounts of money can cause people to drop out or
do you see completion from one year to the next?
So you're looking at these proximate measures, but it's hard to know.
And then if you think about well you want to have,
you learn from the tech world, you want to have a fastly iterating feedback loop and
problem, that you try something, see if it works.
And then you can take a school year, right, so
you're only being able to iterate maybe once a semester to once a year.
It's a significant challenge.
I'm very interested in something that Susan mentioned, is as we gather more data
along the spectrum of the outcomes, are there different statistical models,
or ways to figure out what's really predictive and what's not?
>> Yeah, and that's actually something that also comes up in education and
training, as well.
You might have lots and lots of data watching people engage in an application.
And if you can just really make sure you understand what that means in terms of
ultimate learning outcome, then you can iterate really quickly.
But until you do that, you're not going to be able to.
And that's one of the things in all of our collaborations we talk about.
That's a long-term research project.
It takes a lot of thought.
So it's a great thing for the academics to go off and think about, and can work on
over a course of years as you get more data and so on to keep doing better and
better at making the short-term measure you're using the optimize represent
the long-term impact >> For what it's worth on that, I've
always found there's a, I mean, there's obviously win cases that overcome this.
But young organizations, young firms, are so young that they can't wait that long.
And large organizations have the problem, you mentioned, where your champion moves.
>> [LAUGH] >> That's not to say there's not success
cases, lots of them.
But that seems to be a- >> Well, do you have any tactics?
>> No just try to get incentives aligned.
That's the key is you need the champion and keep incentives aligned.
It's gotta make sense for the for-profit firm, at the end of the day.
Your question to me now just completely sucked away the question I was about
to ask you.
I know what it was.
So I think you both touched an issue of intellectual freedom which is,
in some cases, at least in my experience, more of a problem and
sometimes everything is perfectly aligned.
My favorite line I've ever heard from a company that I refuse to work with,
they told me that they were totally happy to give me intellectual freedom, but
they were just going to decide that later.
>> [LAUGH] >> Needless to say,
that did not go very far.
Any magic insights on how to deal with this?
And then in particular, I think one of the things that we don't face, so part two of
the question is how do we think about this about the individual researcher in terms
of individual researchers sometimes doing private science and
sometimes doing public science.
And is that a viable model to do both?
Or does that taint the entire portfolio for an individual?
>> Well, this is something we've had a lot of panel discussions among economics,
especially those of us who are using data from Uber or from Microsoft or whatever.
How do you think about the conflict issues and so on?
I guess, my personal view is that we can't stop doing the research.
We have to go where the data is, where the action is.
We don't want to not study something really large like, say,
Uber because we think those obstacles are insurmountable.
So I think a lot of it has to do with being very clear about what you're doing,
defining projects that are credible.
And making sure that things can be, in some sense, replicated in some way.
Whether, inside the firm they say the dataset and
some cost people can get in to look at it.
Making sure that some level, there's different types of credibility problems
that you can come up with and thinking through what they are in their particular
setting and making sure that they're addressed.
And then I think in terms of the partnerships that we're forming,
we really are looking for people who are motivated by improving the space.
In the end, a great partner is someone who just wants to ed tech to improve,
that wants workforce training to improve.
We look out and actually so many of these places and there's a lot of room.
And it's not like if this firm's market share goes up,
this other one's going to go down in the same kind of way.
Everybody's solving slightly different problems.
So there's a lot of opportunity to find social impact partners who
are very excited to share their research.
And putting aside the social impact, those are the great kind of partners.
Now, I think there's one bit of conflict of interest about just if what
they want is an impact evaluation that says, does this work or not?
Obviously, there's an issue there along that line.
And you have to treat those kinds of impact evaluations a little
bit differently.
But if you want to show, does personalization work,
in some sense, the organization wants to know whether it works or not.
And if you can show that personalization works, that's a, I think,
a credible thing there.
They didn't have a vested interest in whether it worked or didn't,
they want their product to work.
But at some level getting these insights about mechanisms and
what types of things work is much easier and has less conflict.
So I guess, it's basically managing those conflicts in a variety of ways.
And one other slight point on that is that even before we had word about companies
with this, if you wanted to go work with a school district and measure something.
Well, it's going to be the high performing school districts
that agree to be part of the study.
You're going to have selection of governments in terms of who agrees to work
with you and you have to keep those partners happy.
So in some sense, anytime you work with the world there are conflicts.
And so I guess my view is that what we need to do is just have a lot of integrity
and transparency as researchers, but we can't just stop working on the world.
That's our job is to work on the world.
>> Good note to end on, too.
Thank you both.
Thank you, Susan, thank you, Josh.
>> [APPLAUSE]
[MUSIC]