Hello, and welcome to 'This is My Architecture.'
I'm Joe. I'm joined here today with Steve,
Steve from INT.
How are you doing?
Great! Glad to be here.
That's great! So, we appreciate you coming in.
Can you talk to us a little bit about INT? And what you guys do there.
Sure, INT is a software company, that we build vizualization
for Geoscience data.
And, we'll be talking today about our main flagship,
which is IVAAP.
Great, definitely, seismic data is always something that is very important in,
you know, geographic surveys and oil and gas.
So, you know, can you walk us through,
kind of, the first entry point into your IVAAP solution?
Sure, so IVAAP is a client server application.
So, we've got code running it in the browser and also in the server.
So the very first step is to get logged into the system,
because this is all sensitive data.
So all of the access is, you know, definitely, locked down, controlled.
So the first thing you're going to do, is do a login,
which is going to go -
the request is going to go through our IVAAP reverse proxy,
and then, go down into user management,
which is backed by Amazon Cognito, in this case.
OK, great, so Cognito,
I'm assuming you have to set up these accounts,
the users, the roles.
How do you manage that by customer basis?
That's very simple, we don't.
We let the customer or our partner, like Amazon, deal with that.
OK, and then from a service endpoint,
are you leveraging Cognito for service authorization, as well?
No, we have a component that's running here, on the EC2 instance,
that is a Node.js instance.
OK? That then talks to a postgres instance.
In this case, it's running on Amazon RDS.
And we use that to store project configuration.
So when a user gets started, they'll configure a project
with particular data sources,
and also set up access control rights.
And, this also keeps track of what users have access to what.
Almost like groups and policies around the groups.
Exactly, it's a role-based access control.
Great, great. OK. Walk us through this OSDU,
and how you leverage OSDU, and what is OSDU?
So OSDU is a consortium that builds out a set of APIs,
to let you remotely access your geoscience data.
- OK. - So, from IVAAP's perspective,
OSDU is a data source.
It's one of many data sources that we support.
So, the way we do that, is we have a series of adapters.
And so, we'll have a particular adapter that will talk to OSDU.
So our architecture in the backend, is a microservice architecture.
See, each one of these adapters, is a microservice.
And we have, of course, very many other ones,
that talk to a whole lot of other different data sources.
But, in this case for today, we'll just be talking about OSDU.
OK, great. Are these custom-built adapters?
Or are these AWS services? How did you build those?
These adapters are definitely custom-built by INT.
And so, the cool thing is that the client doesn't have to know anything
about which adapters are running.
The microservices just tell the client, you know, what data is available,
and the client doesn't have to care, even, which adapter is which or anything.
Great, great. I'm assuming OSDU is going to have a lot of data.
- So-- - That's right.
Just, assuming that, talk to me how you guys are managing
the performance and the throughput data.
Well, I think it's best to talk about that, in the context of the seismic data.
- OK. - So, the seismic data is when
you have, like a-- this is active reflection seismic data.
So this is where you have an active source.
Like in water, you would have an air gun,
that just goes, sort of, burp-burp, burp-burp,
sending out sound and seismic, out into the ground.
It reflects off of the rock layers, under the ground,
back up into sensors, and the sensors measure.
And so, each sensor is measuring, what we call a trace,
and that's about 15 hundred samples for each trace.
The datasets for seismic are just huge,
like a large dataset, is maybe, say a terabyte.
A very large one could be a petabyte,
and that's for a single dataset.
- Wow. - So the way that we manage this,
is, to begin with, is the client--
well, to begin with, they first need to do a search,
to find what data they want to look for.
So that goes through the EC2, up through our adapter,
and over to OSDU for their search.
And that's backed by Elasticsearch.
And it handles lots of files.
Once they get the results back and look at them,
they'll make a display.
And, so then, the first step, to building the display,
is to know which traces, it is, that you want to look at.
So, to build the display,
you're probably only going to need, maybe two or three hundred traces.
That's maybe 10 or 15k worth of data.
So that's a lot smaller than a terabyte.
So that's the first thing that's done,
is the selection of which data you need, to paint the pixels.
- Right. - And, that's super critical.
So we'll make the request up here, which goes to the adapter.
Then the adapter is going to need to access the data in S3.
But, to do that, we need to, in the adapter we need to
compute the byte-offsets into the dataset.
Because the traces are not necessarily in order.
They might be out of order, and the dataset might be irregular.
So we have indexes, an index for each of the keys,
and we combine that,
and combining that gives us the unique offset for each trace.
Wow... it sounds like a very complex orchestration,
complex technology you've leveraged?
Yes it is.
And it sounds like a really good solution.
It is, and, the part that's really great,
is the way that it matches up with S3.
Because these datasets are just huge,
and so, when it's uploaded,
S3 is not like, just this single thing.
It's this whole distributed network.
And, so when we're asking for the traces,
we actually do the request.
Most of the time we have to do it in parallel.
Because the traces are skipping around in the dataset.
So, parallel query and parallel searches through S3,
providing you performance.
- That's right. - Outstanding, I love that.
And S3 really stands up to that kind of level of just hammering on it.
- Great, great. - It's really great.
So then as the request comes back, we compress the data.
We have some native code that runs there,
that's using AVX extension instructions,
that are vector instructions.
So we get about a 6x speedup on the compression code.
And we're able to compress the data that's coming back,
about, like, anywhere between 4 to 13 times.
And finally it gets back to the client.
We decompress it, using WebAssembly.
And then, we build up this display.
And, the display is built up on top of an HTML5 canvas element.
Great, sounds like a great solution for your customers.
You know, I do appreciate you showing this to us.
It's a great option, with leveraging AWS,
as well as custom-built solutions that you put into this, as well.
I do appreciate you coming in, talking about it.
- All right. Thank you. - Thank you, Steve.
- Thank you. - And thank you for watching
'This is My Architecture.'