FELIPE HOFFA: Hello, everyone.
I am Felipe Hoffa, Google Cloud Developer Advocate,
and today, I'm in Johannesburg, South Africa,
to meet the CTO of Nomanini, Dale Humby.
This company enables micro-transactions
in various countries around Africa.
With these devices, they create.
Dale is going to tell us more about their story
and why they chose to build their backbone with Google
Cloud.
Also, we'll show you how you can use open data and BigQuery
to visualize the state of the mobile networks that
connect the world.
So join me.
It's my first time in South Africa.
And I think there's so much we can learn about Africa just
by looking at what you're doing.
Not only South Africa, but Africa.
DALE HUMBY: That's correct.
So what Nomanini does is we build point-of-sale systems
to help remote villagers get access
to prepaid airtime, prepaid electricity, and basic banking
services like being able to withdraw or deposit cash.
FELIPE HOFFA: Nice.
And Nomanini got into this business just to help people?
DALE HUMBY: Yes.
Back in 2011 when we started the company,
we focused a lot more on air time.
We wanted to get scratchcards out into rural villages,
but it's very difficult and expensive
to send boxes of scratchcards, which are worth
money, into far-flung villages and then collect cash and bring
it back to the cities to pay for that air time.
If you wirelessly deliver air time, these vouchers,
to machines that are in the village,
it's a lot cheaper and more cost effective.
So people in the villages who don't have a lot of money
to start with don't have to travel far to go and get
the essential services like air time or electricity,
and now even banking services.
FELIPE HOFFA: So as the CTO of Nomanini,
you created all of these devices back in 2011?
DALE HUMBY: Yes.
We started off with a terminal.
It was very simple to use-- no screen, just some buttons
where you press the mobile network like Vodafone,
and you press the 10 rand denomination or [INAUDIBLE]
Meti Cash, wherever you are on the continent.
And it prints out a slip, and you got a PIN number
that you type into your phone.
FELIPE HOFFA: And then you started with the cloud
back in 2011.
DALE HUMBY: Yeah.
Before I joined Nomanini, I had my own company
and we were doing the early days of internet of things.
And I absolutely fell in love with App Engine
because I'm more of an embedded system guy,
so what I knew back then about setting up high availability
servers was a bit scary.
So App Engine solved my problem.
And then when I joined Nomanini, I took that knowledge with me
and we built up our systems backend,
that does all the transaction processing
and the maintenance and monitoring of all
these devices, on App Engine.
FELIPE HOFFA: Now, to be able to pull this off,
you need to have network connectivity.
DALE HUMBY: Yes, that's correct.
FELIPE HOFFA: How's that working in Africa?
DALE HUMBY: So each of our devices
has a SIM card in them that links up to the network
and of course a REST API on our system.
But the networks, while they're there,
they can be a little bit flaky.
So knowing what the signal strength thing is
is very important for us.
FELIPE HOFFA: How can I learn more about it?
DALE HUMBY: Well, there's a project called the OpenCellID
Project.
And people with GPSes and data monitors--
typically tracking companies--
take their data and anonymize it and upload it
to this open-source project called OpenCellID.
And this project then takes that data and figures out
where all the base stations are around the world.
FELIPE HOFFA: Can you show me that?
DALE HUMBY: Absolutely.
FELIPE HOFFA: This sounds fascinating.
DALE HUMBY: Yeah.
So I've got this page open here.
This is the OpenCellID page, and you
can see all those base stations around the world
that they've collected data for.
I've registered on the API, so if I log in then
I can go to Download and can download the latest cell tower
data from this link.
And what I did--
I didn't want to download it onto my laptop
and then re-upload it into Google Cloud
Storage or BigQuery.
So I spun up an f1-micro, which is a free instance,
and I used CURL to download the file which
is about 800 megs or so.
And then from there, I used gsutil to upload it
from that f1-micro instance into Google Cloud Storage.
So here you can see the cell phone towers--
886 megs of gzip data.
FELIPE HOFFA: And if you want to analyze this--
DALE HUMBY: Yeah.
Well, it's no use just like sitting in Google Cloud
Storage, so I used BigQuery.
So one of my favorite tools with BigQuery
for importing or creating a table
is you can point it straight at a Google Cloud Storage bucket,
and you can point it to a gzip file.
You don't even need to worry about unzipping it first.
I told it it's a CSV format and I know all the columns.
FELIPE HOFFA: Radio, MCC, net--
DALE HUMBY: Radio, so it's like GSM or LTE.
MCC-- Mobile Country Code, so that's
the country that it's in.
The network is like Vodacom, MTN, whatever.
Area is like the suburb, and cell is the unique cell ID.
And from there, I imported it, and so I've got those data here
so you can see all the columns it imported.
And if I look at the details, it's
4 gigs, so it went from 880 megs to 4 gigabytes decompressed.
And it's got just shy of 40 million unique base stations
from around the world.
I can then have a look at, for instance, this one--
is finding out the latitude and the longitude and the radio
type.
So having a look at, basically, where's GSM?
Where is the LTE networks?
How is the world moving from first gen, second gen,
third gen, fourth gen cell phone towers?
So running this query, if I run it again--
FELIPE HOFFA: So you're grouping here by--
DALE HUMBY: Grouping it by the--
FELIPE HOFFA: --the latitude and longitude?
DALE HUMBY: --the latitude and the longitude.
So that's just doing some rounding with that division
there just so we don't get too many data points,
and then looking at the case when it's GSM or LTE.
And then importing that, or at least
visualizing that, inside of Data Studio.
So I took those four network types that they've recorded--
CDMA, which is primarily, you can see,
used in the US and some in the east.
The world has moved on to GSM.
So throughout the world, we've got GSM everywhere.
And you can see, particularly in Africa,
is huge amount of coverage with GSM networks there.
Then we move over to third gen, so this is like your 3G data.
Obviously also widely rolled out, but less so in Africa,
but there's still quite a lot of 3G available.
And then we move over to the fourth-gen networks, so LTE.
Also some in Africa, mostly in the big cities.
And huge in the east, so LTE is big there.
Africa's got some catching up to do.
And that took me about five minutes to do.
FELIPE HOFFA: Exactly.
You downloaded all of this data, you load it into BigQuery,
and suddenly we're looking at the whole world--
how antennas are looking.
DALE HUMBY: That's right.
FELIPE HOFFA: Can we focus on Africa?
What else can you see about Africa?
DALE HUMBY: Sure.
So what I had a look at was, given the mobile country
codes--
FELIPE HOFFA: That's the MCC?
DALE HUMBY: That's the MCC--
the mobile country code.
So each country has a unique code,
like South Africa's is 655.
And we did a join with the mobile country codes with
country names-- so there's a database that has all
the country names and the country codes--
and then doing a count to have a look at how many antennas,
unique base stations, for every country in the world.
And taking that, and I wanted to explore that a little bit more,
so putting in the ISO code, which
is like ZA for South Africa, or GB for Great Britain, or Puerto
Rico--
PR.
And then going even one step further,
joining that with one of Google's country info
databases that's available on the internet.
So having a look at population data versus how many antennas,
and seeing, is there a correlation between population
size and how much a mobile network has rolled out.
FELIPE HOFFA: How many antennas do we have per people?
DALE HUMBY: Exactly, exactly.
And that's this division here, the number
of antennas per capita.
And this is actually per 1,000 people,
so like Australia has got 17 antennas for 1,000 people,
right?
So if we once again go into Data Studio,
we can have a look at what that looks like across Africa.
And if we plot that on a log-log scale
of the total antennas versus the total population,
here you can a look.
See with the big countries, it correlates
with having lots of mobile base stations,
and the smaller countries with less population
have fewer bases, which sort of makes sense.
So there's this linear trend.
FELIPE HOFFA: But you still see a big disparity
where some countries have a lot of antennas.
DALE HUMBY: That's right.
If you look, this is a log scale.
So have a look at country like the Seychelles.
It's got very few people, at 88,000,
but they've got just under 1,000 antennas there.
And other countries like Chad, which
is right in the middle of Africa here--
FELIPE HOFFA: That's the light spot.
DALE HUMBY: The light spot--
they've got quite a high population but not widely
rolled-out mobile network.
And the countries that have--
South Africa, Nigeria, Egypt--
and you look up at the top here.
There's Nigeria, Egypt, South Africa.
So for South Africa's population of just under 50
million people, we've got the most widely rolled-out
mobile network in Africa.
FELIPE HOFFA: And in which countries is Nomanini?
DALE HUMBY: Nomanini is operating a little bit
in South Africa, but mostly in Mozambique, and Zambia,
and Kenya, and a little bit in Ghana, as well.
FELIPE HOFFA: And here you can get
a look of where you are getting real coverage reported
by people.
DALE HUMBY: That's right.
Yeah.
FELIPE HOFFA: Nice.
And then you can take all of this data--
this is public data.
DALE HUMBY: That's right.
FELIPE HOFFA: And you can combine it with your own data.
DALE HUMBY: Yeah.
So we have a bunch of diagnostic data
that's uploaded to our systems.
So if I have a look at one of ours--
this is a particular user in Mozambique.
Every minute we're just uploading
diagnostic information.
Everything from user interfaces to here
we've got the latitude and longitude of the GPS,
and importantly, the cell phone diagnostic information-- signal
strength.
And then we do a survey of the seven
surrounding base stations, looking
at what the signal strength is of the surrounding network
as well.
FELIPE HOFFA: All of these devices that you have deployed
are reporting to you what's happening.
DALE HUMBY: That's right.
Right where they are.
So if a client phones up and says my phone isn't connecting
or my device isn't connecting, I can't use your system,
then we can do an investigation and understand,
is it a signal problem or is there some other problem?
FELIPE HOFFA: And you are getting this data now
into Kubernetes, or how are you--
DALE HUMBY: Yeah, so all the data
is uploaded from the devices to our service.
Some of it is in App Engine and some of it is in Kubernetes.
And from there, it's streamed live into BigQuery.
That's what we use for some of our analyses.
So if I look at selecting the data from one
of our clients for today, we've got
a ton of data that's been uploaded already.
And I can go and look specifically
for something like this GSM environment.
So I go [INAUDIBLE]--
FELIPE HOFFA: That's the GSM environment.
DALE HUMBY: GSM environment for one
of our clients in Mozambique.
Now I can have a look at all of today's data.
And what we've got is this raw diagnostics
that's been uploaded.
But unless you're a mobile network engineer,
this doesn't mean a lot to you, right?
Yeah.
This is in BigQuery.
And then I can use BiqQuery's just as in JSON.
I can use JSON methods within BigQuery.
I can split on commas.
I can pull out very specific pieces of this information,
like this is the country code and the network code
and the base station IDs, as well as all the signal strength
for all these base stations.
FELIPE HOFFA: So what I see, you have
like a big JSON blob of whatever the devices are sending?
DALE HUMBY: That's right.
FELIPE HOFFA: Yeah.
That's good modeling?
DALE HUMBY: Yeah, so a lot of our data
is JSON, especially for this particular table
which is just our event log.
It just takes raw data.
There's no ETL or anything on it.
Just whatever the device or the engineers who
build these devices or developers that
make the app just want to send up, we send up,
and we put it into BigQuery.
And then later on, once we've got all the data,
we can decide what sort of questions we want answered.
And because it's in JSON and BigQuery can operate on JSON so
nicely, we can extract just the pieces of information
that we want and build new data tables over that.
FELIPE HOFFA: Nice.
So you can parse all of that in real time
with whatever you want to focus.
DALE HUMBY: That's right.
Then from there, what I've done is I've pulled out--
each row here has got seven diagnostic readings in it.
FELIPE HOFFA: So it's whatever the modem is telling you.
DALE HUMBY: Yes.
It's telling me all these signal strengths and the base
stations.
So what I did is I ran one of the queries
to extract from JSON and got the time, the device that
uploaded it, the area--
so that's like the suburb, I guess-- the suburb ID, and then
the cell tower that it's connected to, as well
as the signal strength.
The closer it is to zero is the better the signal strength is.
If it's at minus 100, it's pretty bad.
You're not going to be able to do anything.
FELIPE HOFFA: So the device knows to what cell
tower it's connected--
DALE HUMBY: That's exactly it.
FELIPE HOFFA: But it has no idea where
the cell tower is located.
DALE HUMBY: That's right.
So if a client phones us up and says,
hey, I don't have good signal.
Can you see what the problem is?
It allows us to have a look at the device
and have a look at the signal strength that it's receiving,
as well as, then, join that to the OpenCellID tower.
So given this, I'm doing a join on the cell
ID between the towers databases and our devices database,
and being able to join.
And what we get is, for each cell,
we get the latitude and longitude
of where that base station is as well as the maximum,
the minimum, and the average signal strength
that our clients are experiencing on that cell
tower.
And from that, we can put it into a dashboard.
And this is all the cell towers that we've
connected to in Maputo.
That's the capital of Mozambique.
FELIPE HOFFA: Is this Redash?
DALE HUMBY: Yes, it is Redash.
FELIPE HOFFA: Oh, good.
I love Redash.
DALE HUMBY: It's super for doing this sort of geo-visualization.
So you see largely this is green, which is good.
It means that the cell towers have got good signal strength.
FELIPE HOFFA: In the capital of Mozambique.
DALE HUMBY: In the capital of Mozambique, yeah.
But this one device that I showed you earlier--
FELIPE HOFFA: And these are your devices.
DALE HUMBY: These are our devices, yeah.
If we look at this guy--
he's in Maputo, but he is sort of a little bit
to the north of the city.
So we have a look in Redash here, and the north
of the city where he is--
it's a bit of yellow and red.
So we can explore down and see that there
is a cell phone tower that has not got a great signal
strength.
And because we've got the cell tower,
we can then phone up the mobile network
and say that, in this particular area,
this base station has got very poor signal
strength for our users.
Could they fix it?
FELIPE HOFFA: Wow.
So you can use your own data to tell the networks
where they have problems?
DALE HUMBY: Yes, because they're not going to report it to you.
But if we can give them data, then they
have fixed stuff before.
FELIPE HOFFA: Well, I would love to learn more about how you're
managing micropayments all over Africa,
but let's save that for another episode.
DALE HUMBY: Absolutely.
FELIPE HOFFA: Excellent.
Thank you, everyone, for joining us.
Thank you, Dale.
DALE HUMBY: Thank you for having me.
FELIPE HOFFA: And everyone, stay curious.
I'll see you soon.
[MUSIC PLAYING]