Hi and welcome to 'This is My Architecture' in New Zealand.
I am Nick and I'm joined by Carlos today.
For those of you people that don't know who Halter is.
Can you tell us a little bit about Halter and the device you're holding?
Sure thing, so Halter is a New Zealand based startup
that allows farmers to remotely manage their farms.
And our solution is based on this collar.
So this collar is a solar-powered and GPS enabled
that uses mainly sound and vibration to give cues to the cows.
And this collar goes on each cow on the farm,
and we also have a mobile app that allows farmers to create the virtual fences,
physically shift the cows within the farm,
and also monitor the health and well-being of the animals 24/7.
Well, I'd love to know more,
and one thing I know about farms is that,
they're often very remote.
How have you solved their connectivity challenges?
Yes, collars can talk to our back-end in two ways.
The first one is over Wi-Fi.
So when connected on Wi-Fi, the collar talks to IoT Core directly over MQTT.
So Wi-Fi is great for throughput.
The problem is that the Wi-Fi range is not great and also it's super power hungry.
So we use another radio technology called LoRa.
So this is a long range and low power technology.
So the collar talks to a base station on the farm
and then this base station is connected on the Internet and talks to EC2.
And we just host a third-party software here on EC2.
And that's pretty much how the data comes from the collar to the back-end system.
That's great. And how do you ingest and process all this data from your devices?
We need to combine the data coming from the two sources.
So the first thing we do is putting that on the Kinesis Stream.
So this is the raw stream of data.
So the data here is in a binary format and can be duplicated
because it comes from two different sources.
Then we have a Lambda function that consumes the data from here,
dedupes the data and also decodes it
and makes it a lot more digestible to the upstream services.
So we ship it as JSON to the next Kinesis Stream.
So this is the processed Kinesis Stream data.
And this one is the main stream in the system.
So we have lots of different parts of the back-end consuming data from here.
One of them is an Apache Flink application that runs on top of Kinesis Analytics.
So what this application does is aggregating this cow location data
into buckets of ten minutes.
And we also enriched the data with a geo index
and we use this geo index later on to do aggregation.
So once the data is ready,
we just ship as a file to S3
and we use farm ID and data as the partition keys
in order to avoid hard partitions.
Awesome. There must be a lot of data.
How do farmers actually make use of this data?
Sure, we have a feature in our app which is a cow location heat map.
So this shows where cows have been like in the last six or twelve months.
So when the farmer wants to request that report,
the app sends a request to our back-end which is hosted on ECS Fargate,
and the ECS Fargate will do the orchestration on querying the data.
The data is stored on S3, as I mentioned,
so we use Athena to do the actual query.
So this is a classic MapReduce problem
where you have to go through a lot of data points and aggregate and count it.
So Athena does a great job at it.
We can process roughly 25 million data points in roughly 10 seconds.
And then when the result of the query is done,
the ECS Fargate application will go to S3, gather data
and then send it back to the app and the app can render a nice heat map.
Wow. That's actually a lot more data points than I imagined.
And what do farmers do with this data?
Sure, this can give a great insight to the farmers
to know where cows haven't spent too much time on the farm
so they know where to put fertilizers
or even change their grazing patterns
in order to put the cows in those areas
in order to have more nutrients there naturally.
So this will help farmers to comply with New Zealand regulations
around how much nitrogen you can apply per hectare on the farm
and also possibly make them save tens of thousands of dollars per year.
Awesome. Thank you for sharing this interesting architecture.
It's been really interesting learning about
how Halter is using location data to optimize pasture management for farms around the globe,
and thank you for watching 'This is My Architecture'.