in this video I show you how to use Azure automation and event grid to
automatically copy blob files from Azure storage to a local file system
hello everyone my name is Travis and this is ciraltos I posted a video a while ago
on keeping local files in sync to Azure blob storage with AZ copy I'll post a
link to that video up here someplace in response Asgar M asked about using
Azure functions or web jobs to do the opposite copy files from blob storage to
the local file system thanks for the question I can't promise to help
everyone in these videos but it seemed like an interesting challenge so I
decided to give it a try before we get started don't forget to subscribe and
for those who have thank you here's the problem I'm trying to solve I have
an azure storage account when a new blob file is placed into the account I want
to copy that file to my local file server first thought is to use web jobs
or Azure function the problem I ran into is there is no easy way to move or
copy that data into a file system behind a firewall I did do some quick research
on this it could be possible with a web job and a relay a hybrid connection or
with vnet connectivity on a web app or Azure function but that would require
coding beyond PowerShell and that's just not something I'm able to do instead I'm
going to stick with what I know for this challenge that's Azure automation and
Azure automation is well suited for this at a high level here's how it works
a new file arrives in the storage account that triggers event grid
Event grid sends a web hook to start and as your automation runbook passing details
along the way as JSON payload Auzre automation starts the runbook on the
hybrid worker inside the network the run book downloads the file to the local
file share you will need to have Azure automation configured with a hybrid
worker if you want to follow along I have a playlist that covers that the
link is above you will also need an insurer storage account I have an
overview of Azure storage accounts published here of course I'm going to
start the at the most logical place the middle I
need to first create a run book run book code isn't important at this point but
what is important is that the run book is published so I can get the webhook
I'll come back and finish the run book in a later step once the run book is in
place I create the web hook the web hook has a special token that authenticates
it triggers the run book once I have the web hook I can configure a vent grid to
trigger the run book the web hook has a JSON payload with details such as file
name and container I'll use this in the run book for the download command the
run book is configured to run on a hybrid worker in my network this gives
me the ability to copy the file to a local file system
lastly I'll put it all together now when a new file is added to blob storage the
run book will download that file to a local filesystem
let's get started by creating the run book okay buckle up everyone it's demo
time Here I am at the azure portal inside my Azure automation account and
the first thing I'm going to do is add a new run book I'll give it a name
Cir blob demo of course you can give it whatever name you would like if you're
following along I'm gonna select PowerShell and I'll create ok now that
it's created I'm going to edit it I'm just gonna add some code as I mentioned
before the goal right now is just to get the webhook but I'm gonna pay some code
in that will help me see what's getting passed in this code simply grabs the
input that's coming in from the web hook and then writes it to the output stream
I'm gonna save and publish and we'll come back and edit us in a few steps but
for now all I'm looking for is to be able to do the add web hook first I'm
gonna create the web hook and I'll give this a name I'll just call it I'll just
call it blob web hook I'm gonna make sure it's enabled I'm gonna set the date
for a year out keep in mind that if you're doing this in production you want
to make note of that expiration date because it will quit working once that
expires I'm gonna copy the URL I'm gonna use this in a later step and you
can't come back and get this copy and paste it someplace safe the URL does
have a token in it with that token anything can trigger this run book so
it's a security object and you can't retrieve it later I'm gonna click OK
I'm gonna configure parameters so I'm not gonna pass any parameters in but I'm
gonna select hybrid worker as I said in the slides you need to run this on the
hybrid worker that way you have access to the file system behind the firewall
and again if you're not sure what a hybrid worker is go back to my is your
automation playlist and watch it I have plenty of details there so next I'm
going to click OK and create next I'm going to go into my storage account and
I'm gonna find the storage account that I set up for this demo there it is I go
into blobs you can see I have a container here called cirdemo that's
where I'm gonna upload my files although the way this works it doesn't really
matter any blob that's uploaded to this storage account will get downloaded
locally next I'm gonna go into events this is where event grid is found I'm
gonna go into more options and webhook so this is where I'm gonna set up event
grid to fire a web hook every time a new files uploaded so I'll just call this
blob event I can leave the schema I can leave the topic details I do need to
change the event type I only want this to fire when a new blob is created so
I'm gonna uncheck the delete under endpoint details I'll go to webhook and
it's gonna ask me to select an endpoint here's where I'm gonna paste that
webhook and then confirm selection and create so that's gonna create my weblog
in event grid and next I'm gonna go to access keys
these are blurred out for now you're actually probably going to be able to
see them in a little bit but that's ok because this will be long gone by the
time you're watching the video but I'm just gonna copy the key to the first key
and paste that someplace as well I'll come back to that later but you will
need this key for a future step so next I'm going to go into a storage Explorer
preview I'm going to go to my blob container I'm gonna go into the
container I set up for this test and I'm going to upload a file I've got a test
one on my desktop and I'll upload okay so that's been
uploaded now I'm going to go back to as your automation go into my automation
account and under jobs let's see if we have a job running yep there it is so
what did I do I uploaded a new file to that container
using Storage Explorer and as soon as that happened that webhook fired and
that's running a job in Azure automation and we're gonna see the
results of that job in just a second okay now it's completed and we're gonna
look at output and there it is there's the output so it did pass along the data
I need I'm gonna go into input though and I'm gonna just select this this is
the data that came in and if I copy that this is all JSON data so I'm gonna go to
vs code create a new file and we'll just call this data dot JSON I'm gonna paste
that in and I'm gonna format it there so now we can see the data that was passed
in this is handy if you actually want to see what's being passed between the
webhook and the actual run book if we scroll over we can see some information
that's kind of helpful so under subject you can see we have
data which is blob services default containers and that's the container name
also if we go to uri i think it is let's see here
URL here it is we can see that the very last is the file name so I'm gonna use
this data to build the final run book so with that let's go in and build the run
book Here I am at the finished Run book I wasn't gonna make you actually watch
me type all this out that would have been pretty horrible
maybe someday I'll set up a twitch stream but for now I'm just gonna give
you the final product and I will make this available so you don't have to try
to copy all this down it'll be on my github page and you can find a link to
that in the video description so first I have the parameters just like before so
it's capturing that data as it's passed into the run book I have some error
control in here I'm just setting there air action default to whatever it is
when it starts next I'm going to identify the local path
so this is the path that the file will get downloaded to you can change that if
you need to but this is gonna copy everything to the C:\blob directory on
the local server storage account name that's the name of the account we're
targeting you'll need to create a run book for every storage account you want
to use this against next is a storage access key so this is the security token
or the key that we copied previously we need that to actually interact with the
storage account in this run book I don't want to put the key in my actual code so
I've got get automation variable and blob copy key so let's set up that
variable and as your automation Here I am in the storage accounts I'm gonna go
to variables and we got a handful in there already but I'm gonna add a
variable I'm gonna leave it as string I'm gonna set the value as the actual
key the value is the key that we copied before and again I'm not gonna blur this
out it's gonna be gone by the time I post this important click yes to encrypt
it and create here we can see it's been created with an unknown type because
it's encrypted so let's get back to vs code okay so now we have this storage
account key set the great thing about doing it this way is you can call that
same key for other run books and if that key ever needs to be rotated you only
have to rotate it or change it in the one encrypted variable you don't have to
go into each run book next I'm going to set the storage of context and you'll
see I'm using the try catch and then sometimes finally block for error
control what happens is it's going to try a command and if it hits the error
action stop it's going to write the air to the outputs the error output stream and
then stop the script so if you need more information on that there's a great book
on PowerShell.org it's a E-publication on error control but for now just know that's
what that's about here's the command though that I'm running is it's creating
the storage context so that's gonna give me the access permission to log in and
pull that file down next I'm taking the webhook data that was input and I'm
going to the request body portion of that and I'm converting it from JSON and
that's gonna be in a variable called request body and again more error control
so here what I'm doing is I'm getting the request body.subject
and we looked at that before let's find that here's subject so here's what I'm
dealing with right now in this little section of code and what I did is I use
the split command to split that based on the forward slash so you can hear you
can see here it's gonna split all these items up and on 1 2 3 4 the 4 is the
container name and I think that's gonna be pretty consistent no matter what's
passed in it will always be the container name will be the 4th object in
that string so I'm telling that the container is gonna be the subject array
number 4 I guess technically that may be the 5th because the array may start with
a 0 anyway it works the next command finds the actual file
name so I'm grabbing data again from the request body and I'm looking for that
URL and the storage account name is a variable as well as the container and
what I'm doing is just stripping all this off because I'm doing a replace
command and I'm replacing it with nothing so that's gonna leave me with
the file name if the file name is in a path it's gonna leave the path of the
file name and here we can go down and the next command is I'm doing a test to
see if that local file path exists so again this is the local file to C:\ what
did I call it let's grow up and take a look C:\blob so it's gonna test to see
if that path exists and if not it's gonna create it ok here I've got a big
block of write outputs that I was using for testing I'm gonna leave them in
there I think you can uncomment it if you're curious on what everything is
coming out as it's helpful if you're trying to identify you know what is
actually being passed back and forth but for production I don't really need it
because I know it works the final step is gonna actually run the copy command
it uses the get AZ storage blob content command and passes in all the variables
that we've set up so far along with the context that was created to give us
access to that storage account I have a couple notes they don't want to point
out for this one I'm using the AZ module if you're using the azure RM the default
that comes with azure automation you'll want to change the command
get azure storage blob content instead of AZ storage blob content also if you are
using the AZ command I had a problem where I was getting a illegal character
in path error and it took me a while to figure out what was going on but I've
got two links here that reference the issue and what happens is I had to add a
file called Orchestrator dot sandbox dot exe config and it's all outlined in
these two articles it's possible that you won't get that by the time you watch
this it may have been fixed but it's a bug within the AZ module also one other
thing that I kind of skipped over but I want to go up and make sure we don't
overlook it here under the storage context I'm using the AZ command so
these are two locations if you're using the default modules that come with
azure automation you'll need to change them from AZ
to the older command so with that let's copy it in and paste it into that run
book and I'll just paste that in there are a hundred different ways to manage
this or at least three but I'm doing the old-school copy and paste so I'm going
to save it and publish it so that run book is done let's trigger it by adding
another file to the storage account so I'm going to go back into the storage
account and I'm going into storage Explorer this is just a way to emulate a
file appearing into that storage account and I'm going to upload a file let's see
if the job ran so I'm going to go back to azure automation go back into my
automation account jobs and it did run it ran on the domain group and we can
see the input-output errors there's no errors so that's a good sign
next I'm going to go into the hybrid worker that I ran on ok here it is here
you can see I'm at the C:\ Drive I'm gonna go into blob and there it is so let's
upload another one and just see that if that works again back to storage
accounts go into my storage account go into storage Explorer I'm going to
upload again this time I'm gonna select test 3 and
I'm gonna override the file if it exists and upload there you can see it just
appeared that indicates it works so we've got a working run book I want to
test one more thing I'm going to create a new folder we'll click OK and I'm
gonna upload test 3 to that let's see what happens back on my hybrid worker
look at that it created the folder and there's the file so not only does it
copy new files that occur in that storage account but it will also create
the directory structure if you add folders and add files so that's a
success every time a file is uploaded to the storage account it's downloaded onto
the local machine there's a lot of things you can do to configure this like
specify maybe a container as a parent folder instead of having everything just
drop into one local file path defined in the variable but hopefully this gets you
started and understanding how storage account event grid run books and hybrid
workers can all work together to automate some pretty common tasks that's
it for the demo thanks everyone I hope you found this helpful don't
forget to subscribe and thanks for watching