Welcome to another episode of 'This is My Architecture.'
Well, typically we love to have customers talk about their solutions,
but today I have a fellow solution architect with me.
Provanshu, how are you? Good, how are you, Adrian?
I'm well. Now, you work in the Media & Entertainment team
and you work with a lot of customers that are looking to ship around
vast volumes of files, audio, video, large data sets.
What are some of the challenges that you see that they have?
Typically, it's the amount of time and money that they are spending
by maintaining those infrastructures for moving those files.
As an example, if somebody has to move hundreds of gigabytes or terabytes of data over the internet,
it takes considerable amount of time.
And also, they have to pay for the licenses that are associated with
that software that is doing it for them, etcetera.
Also, security is paramount.
In our industry, all the transports have to be secure,
because this is really, extremely valuable content.
And you've built a really elegant solution, so let's dive in.
Yeah, so, in this solution we are putting S3 as the backbone for those transfers.
The reason we are looking at S3 is because a lot of our customers
are already using S3 as part of their media processing workflows.
So what we have done is that we have taken an S3 bucket
and put it in the middle for whoever is producing the content
and whoever is here to receive the content.
But the thing about this particular S3 bucket is that it is secured
and access-controlled for the specific transactions
that they're having to do with themselves.
Now, what is happening here is that whoever is producing content,
they are simply copying the files into this bucket, the S3 bucket that is in the middle,
and the other side is coming to go and pick it up.
So they have write permissions, the other side has read permissions too.
So you're using a publisher/subscriber model. Exactly.
How do you maintain control of access between all the different buckets?
That's a great question.
The access is being controlled by AWS Identity and Access Management.
There are no credentials to manage.
So, what this bucket is configured for is that the publisher account
has access to this bucket, to put assets in, and the subscriber account
has access into this bucket to read those files in.
It's really simple.
It's as if you are putting the files in a staging area and then you are--
the other side is coming and picking it up from there.
Great, so let's walk through the whole orchestration of how it works.
Yeah, so, what is happening here is that every time the publishers
put a file in between, that generates an event which gets sent to an EventBridge
and these notifications, the object creation notifications, get passed around to both parties.
That way, what happens is that either the publisher or the subscriber
can integrate their existing workflows to it.
As an example, if somebody is receiving an asset, they can look at those events
and start transporting them immediately.
They don't have to wait for the whole thing to complete, etcetera.
They do not have to come back and check when the assets are transferred.
So, EventBridge, which is our serverless event bus, makes it really scalable,
no matter how many publishers and subscribers that you have.
Tell us a little bit about how we now manage the security,
because we've talked about the robustness, but how do customers feel
that their data is secure?
Yeah, absolutely.
One key point here is that there are no changes to be made here,
in this side of the board, in the publisher account
or on the subscriber side of things.
All of this is being deployed in its own account in a siloed area,
where it is completely locked out, and the assets as they're moving in.
What is happening from the publishers' point of view is that they are writing into a bucket
that happens to be in a different account, and they have the required permissions.
Similarly, on the subscriber side, they have the required permissions to read from there.
And on top of that, everything is encrypted with AES-256 and AWS KMS in the mix.
Which is here.
What we are doing in this case is that the KMS is generating the keys that are being used to
encrypt each and every object, and the parties, publisher and subscriber,
have access for encryption and decryption.
That way, it's not only secure from S3's point of view, but it is also secure
with a shared encryption key that is only available to these 2 parties.
Great, and those access logs are sitting in that S3 bucket for querying later on.
That's the reporting, and reporting page in the sense that every time
objects are being written to or read from, this particular bucket that produces access logs,
and the publishers and subscribers have access to it, that way they can build
reporting in all of it, who is and when the files are being picked up, etcetera.
Very elegant. How do customers get access to this?
So, this is available in the AWS Solutions page, as well as it's available in the GitHub,
our customers can go and download it today and start using it.
Fantastic, love all the serverless use and the simplicity of it.
Thanks, Provanshu, for coming to share your architecture with us today.
Thank you.
And thank you for joining us on 'This is My Architecture.'