JON SKRIP: It's good.
It's a nice-- I feel like I'm "Between Two Ferns," but not--
JEN PERSON: We're hoping to get on that one.
JON SKRIP: That would be nice.
SPEAKER: Team 1, take 1, calling mark.
JEN PERSON: Hey, all you Firebase developers.
Welcome to another episode of AskFirebase, the show
where we answer your burning Firebase questions.
I'm Jen Person, your host.
And today my co-host is Jon Skrip.
Thanks so much for coming on the show.
JON SKRIP: Not a problem.
JEN PERSON: So are you ready to answer some questions about A/B
testing?
JON SKRIP: I am so ready.
JEN PERSON: Then let's dive right in.
Can you believe that I get paid to do this?
JON SKRIP: I know, right?
JEN PERSON: It's ridiculous.
Our next question is from yarism, who says,
"Is there any way to run multiple tests
at the same time?
Or am I going about that the right way?"
JON SKRIP: You can definitely run more experiments or more
than one experiment at once.
One thing that you kind of want to watch out for is you
don't want to target--
if it's an RC experiment, you don't
want to target the same RC parameters
between the experiments.
The way that things come through gets a little weird
in doing that.
And plus, it kind of messes up your whole flow
of testing in general because you're doing multiple tests
on the same value.
So it doesn't really provide as much use.
But say you wanted to test, simultaneously,
some welcome screen and some store within your app,
or something like that.
That's 100% doable.
You just target those things.
And it should work out of the box
JEN PERSON: Yeah, I'm almost wondering,
if you have some use case where you're
saying you want to target the same Remote Config values,
for instance, then go ahead and let me know.
Because I'm interested in what that scenario is.
And maybe we could point you to a better
strategy for doing that.
JON SKRIP: Yup, for sure.
JEN PERSON: Here we go.
The next question comes--
see, I don't even have to try to pronounce this one.
There are no vowels in that.
And I appreciate that because then I
don't have to butcher it like I usually do.
JON SKRIP: (WHISPERING) "Kuh-tar."
JEN PERSON: Yeah, K-Q-T-R--
JON SKRIP: Butchered.
JEN PERSON: Yeah, maybe.
Quarter-- K-Q-T-R. We're going with that.
JON SKRIP: Got it.
JEN PERSON: "When A/B testing with Remote Config,
if a new app version uses a new Remote Config parameter,
will older versions automatically
be excluded from the results of the experiment
because they do not use that parameter in the code?"
JON SKRIP: So short answer is no.
Unless you are specifically targeting app version
within your targeting parameters,
the older versions that are not using it
will still pull the value and get
enrolled, depending on if you have activation events and all
that stuff.
But they'll pull it.
They can potentially get enrolled.
They just won't have any use for that value.
So if you want this thing, you will
want to target app version specifically.
JEN PERSON: Yeah, that makes sense.
So otherwise, it seems like your results could be really skewed
because you have all these other people using an older version,
and it's not doing the thing that you're expecting it to do.
JON SKRIP: For sure, yup.
JEN PERSON: Looks like we got some more.
JON SKRIP: Great.
JEN PERSON: Good?
JON SKRIP: Yeah.
JEN PERSON: Iamronak-- see, this is what I'm talking about.
Iamronak.
JON SKRIP: Hi, Ronak.
JEN PERSON: "I have an A/B testing experiment running with
three variants-- control, variant A, and variant B--
at 33.3% distribution for each.
And I want to stop that experiment
and run it again with just two variants-- control and variant
B. That way, they'll have 50% distribution for each.
My question is, will the 33.3% of users
who got variant B during the first run
of the same experiment get a different variant when
I run this experiment again?"
JON SKRIP: So yeah, good question.
In short, no.
When sampling the populations, you want random sampling.
This means that between experiments, there's
zero linkage.
There is a good chance that a majority of the people in that
33% end up in the 50% afterwards,
just because they qualify for whatever you're targeting.
But there is no guarantee that it will be the exact same group
or that all 33.3% will end up in that 50% afterwards.
JEN PERSON: Yeah, that totally makes sense,
because let's say that one third happened to just
really get along with variant B specifically for some reason.
If you put the same people in there,
you're going to get a very similar result, which may not
reflect the other 2/3 of your population
that's using your application.
JON SKRIP: Yeah.
So Jen, this question is for you.
JEN PERSON: All right.
JON SKRIP: This comes from restonweb.
"Do you know how to fix this?
'No clients were able to be added to your Firebase project
for the following reasons--
an app with this package name and SHA1 is already connected
to a Google project.'"
JEN PERSON: So they deleted the app and project
in another account, but it's still giving them this error.
And so what I'll do in answering that is actually ask you
a question, which is, at what point
did you delete that previous app?
Because normally, it can take you even a week or two for that
app to be 100% actually deleted--
that project to be completely deleted--
which is really to your benefit, because quite frequently,
people contact support saying that they didn't actually
mean to do that.
And we're able to restore that information.
So I would say if it's been within--
usually a week, but give it a couple weeks just to be sure--
this is possible that it would give you that error.
If it's been longer than that, and you're
certain that you did delete the previous one,
then I recommend that you reach out
to support because they may have more insight into it.
It's always possible that in the interim,
someone else made a project with that.
Or maybe you were just removed from permissions
from the previous project, potentially.
So they can really dig down into what's happening there.
But if this is something that you're experiencing,
just make sure you wait a couple weeks to be sure
that the previous project has actually
been completely deleted.
[MUSIC PLAYING]
John, thanks so much for coming on and answering
all these burning questions about A/B testing.
JON SKRIP: Sure.
JEN PERSON: I really appreciate it.
And thank all of you who have asked questions and submitted
questions.
And if you have your own burning question,
be sure to put it on social media
with the hashtag #AskFirebase.
And who knows?
You might see it on a future episode.
Also, if you like what you see here,
be sure to subscribe to the Firebase YouTube channel,
where you'll see all sorts of other interesting content.
Thanks so much for watching, and we'll
see you on a future episode.
JON SKRIP: I just drank too much coffee.
JEN PERSON: That's the right mood
to be in, though-- too much coffee.
JON SKRIP: It is, yeah.
But it dries my mouth out.
JEN PERSON: Yes.
All right, here we go.