[MUSIC PLAYING]
VIKRAM PAROLKAR: Hey, guys.
Vikram Parolkar from UiPath here.
And today we're going to talk about UiPath UI automation
capabilities.
To begin, I want to do a quick recap
of what is an application.
An application has two components.
The first is the user interface.
A user has the ability to interact
with things like text boxes, buttons, labels, and other UI
elements visible to them.
And applications also have a backend.
This is where various construction logic or business
logic resides or rules on where a button navigates to
or how data is actually stored in the application.
Now, when developers are building
both of these components, they use something called selectors.
For web applications, that's things like HTML code.
For desktop applications, that's something called Win32 tags.
But UiPath's bread and butter is the ability
to interact with the user interface
and reliably interact with all of those UI elements.
And so let's take a deep dive into the multiple ways
that UiPath will actually interact
with any type of application.
The first method is what we call Multi-Anchor Unified Targeting.
This is our default way of interacting
with an application.
And we actually use four concurrent methods
in order to actually reliably interact with the UI element.
You, of course, have the selector
that I mentioned before.
You have a fuzzy version of it, which eliminates
dynamic things like IDs.
You have the text.
And you have image as a fallback.
Now, if UiPath isn't actually installed
on the same device as an application,
you might need to use things like Citrix Automation
or Native RDP Automation.
The way this works is the robot would sit on the actual client
machine.
But there's a virtual channel client
that would interact with a remote runtime that's
installed on the remote application server.
And the third option, if neither Multi-Anchor Unified Targeting
and neither Citrix or Native RDP Automation is available,
is what we call AI Computer Vision.
AI Computer Vision leverages machine learning models
to basically allow the robot to interact with the screen
exactly like how a human does.
Let's do a technical deep dive into our Multi-Anchor Unified
Targeting method.
You'll notice I have UiPath Studio installed
on the same machine as my application
that I want to automate.
We have a series of UI automation activities.
And these activities are separated into two categories.
The first is what we call our parent container activities
where I would define a specific application or browser that I'm
interacting with.
And once I've defined that, I can then
combine it with a child action, such as a click type
into activity and many more, and interact with a specific UI
element.
If I scroll down, I've already indicated a parent application
here called Allscripts PM.
And if I want to interact with a specific UI element,
all I need to do is indicate that specific target on screen.
You'll notice that when I enter what we call our selection
screen mode, I'm able to hover over specific elements
and indicate them.
I'm going to go ahead and indicate this appointment
scheduling navigation tab.
As you can see, we're going to return four different aspects
of that specific UI element, the selector, a fuzzy selector,
the text, and an image.
If for any reason I feel that this specific selector isn't
reliable enough, I can add an anchor to it.
This is kind of allowing the robot to interact with that UI
element the same way humans do.
If the appointment scheduling label
were to move around on the screen,
humans would still be able to identify it because it's
under the scheduling section.
And so similarly, we can define anchor points for our robots.
And these anchor elements will have the same
for selector, fuzzy selector text, and image elements
to ensure reliable interaction with.
Another cool thing is that if, for example,
we have things like indexes or names
that are dynamically changing from one page to the next,
I can replace these names with variables, regular expressions,
and fuzzy matching.
And so this allows me to ensure that as a developer,
I'm defining the exact selector that I want to interact with.
Now, if I want to dive into the specific details,
I have something called UI Explorer.
UI Explorer gives me a full view of the visual tree
of an application, the various properties behind a specific UI
element, and allows me to select the specific items
of a selector that I want to include
whenever a robot tries to interact with the specific UI
element.
Now, our AI driver algorithm automatically
selects what it thinks is the minimum required
in order to reliably interact with a specific element.
But developers can go into the details
and change them as required.
So what happens in scenarios where
I'm not able to install UiPath on my remote desktop
or my Citrix server for any reason?
In those scenarios, we're able to still leverage our UiPath
Studio on an external machine and interact
with the application reliably.
In order to do this, we have to establish
a virtual channel between our machine and the application
machine.
And in order to do this on the application machine,
you have to install the UiPath Remote Runtime.
Once this is installed, when you want to just indicate
a specific element on a remote desktop,
you will have the ability to do that.
And it will look exactly like it does as if the UiPath
Studio was installed natively.
So next step, I want to introduce AI computer vision.
And so if I hit Enter my selection screen mode again,
you'll notice if I hover over, let's say, a specific tab,
this application doesn't expose them to our UiPath driver.
However, that doesn't mean that we still can't reliably
interact with them.
This is where we have a series of computer vision activities.
Let me go ahead and bring him up on the screen.
You'll see them here.
These activities are split again into two.
We have a parent scope where I'm actually
able to indicate a specific screen.
And you can see I'm going to indicate the whole application
or I could indicate the subset here.
Once I do that, it's going to pass the image of a screen
through our UiPath screen OCR engine.
And once it's done that, I am able to drag a simple UI
element activity, such as a click activity,
and indicate on scope.
You'll notice I have an image of the specific application.
And if I hold the S key, all of the actual elements
that are identified by our AI computer vision model
are going to be exposed to me.
And you'll notice all of these tabs
now have their individual elements that I can indicate
and also add anchors to
Thanks for joining me in this quick tutorial of how UiPath
interacts with any type of application
in three different ways.
Till next time, bye.