SPEAKER: Hello, and welcome to the UiPath AI Computer Vision.
Today, we'll take a look at the computer vision
activities, which are now available in UiPath Studio.
First, we'll compare the selector based activities
with computer vision in order to understand the benefits
that this new feature brings.
We'll then go into detail about how to use the computer vision
activities, and finally, we'll take a quick look
at some of the best practices in building solid automation
workflows that incorporate AI.
Let's begin.
Here is a workflow that contains a simple click, which you're
already familiar with followed by a set of computer vision
activities.
As you can see, the activities are
included in a CV screen scope.
But we'll get to that later.
Now, let's see how this normal click
works on our test applications.
If we select indicate on-screen, we
can see that in this desktop application,
all elements are easily recognized when
we hover the cursor over them.
However, if we move on to the VDI environment here,
we cannot interact with any individual elements
on the screen.
Only the buttons at the top of the window
are recognized because they are part of the desktop app.
In the web browser, the same thing happens.
We can interact with some elements.
But the fields in this image are obviously not
recognized because it's just an image.
So that was the normal behavior.
Now, let's see how computer vision
handles these applications.
We'll select indicates on-screen again.
But this time, let's try using a computer vision type
into activity.
All the fields in the application are identified.
So there's no difference in that regard.
the very same application is open in the VDI environment.
With a simple click activity, we could not
perform any action here.
But with computer vision, we can interact
with every element, text, input fields, buttons,
and checkboxes.
And because computer vision works based on how things look,
it also works on the image in the web browser.
We can detect input fields texts, and also
the Submit button.
Pretty cool.
All right.
Now, that we've gone through the capabilities of the computer
vision activities and how they differ
from the normal approach, let's see them in action.
We'll start with an empty workflow,
and add a CV screen scope.
We'll agree with the terms and conditions here.
We'll indicate the application, the RDP window, in this case.
By doing that, we generated a selector
that's used at runtime to get the screenshots
of the application and analyze it.
If we double click the informative screenshot,
we can see a preview of that image
and the results of the analysis with the detected elements
highlighted in blue.
Next, we'll add a type into activity.
Select indicates on screen and then click on the field
where the text should go.
When we do that, computer vision automatically selects an anchor
and generates the appropriate activities.
You want to find a label and the other to actually type in it.
We'll just add some text to be typed here and done.
Now, two blocks were generated in the type into activity.
As we just mentioned, the one on the left
contains the anchor text, while the one on the right
contains the input field itself.
We can easily modify our field selection
by clicking indicate on screen again
and then choosing something different.
Say this one.
The activity was updated and the anger
was detected automatically.
So they on-screen option enabled us to change our selection
altogether.
Let's see what happens if we select edit on the screen.
The anchor and the field are displayed
and we can edit or remove any one of them.
Let's say that we want to select a different anchor.
We need to remove the current one first.
The color of the fields has changed,
the red color in this field indicates
that we don't have an anchor, and a yellow color
shows us the fields that are similar
to our current selection.
Those fields are also called false matches.
On the upper left corner, we are notified that we
need to indicate the anchor.
But we've seen that it works even automatically.
So let's try something else.
We'll pick a different field, such as this one, as well
as a new anchor.
The color of the fields changed again to orange.
The explanation is once more mentioned
in the wizard in the upper left corner of the screen.
There are duplicate matches.
So we need to choose another anchor.
So the wizard provides details about the selection
and notifies us if anything is wrong.
If there is no problem, the fields are colored in green.
So to recap this final part, yellow
indicates similar matches to the currently selected one,
and red is displayed if there is no anchor,
or if the current anchor is not valid for the target.
Orange signifies that there are duplicates or false matches.
So we need to inspect our selections
to see what's wrong. a
Now, in this particular case, we could solve the issue
by making use of the image automation feature.
To do that, we simply need to drag click and draw a rectangle
around a unique anchor.
And it worked.
The selection is colored in green, which
means that everything is fine.
If we look at the wizard on the left side,
we can see that the target and the anchor are OK.
We can hit Enter now.
The activity was updated once again,
and our selection is displayed in the type into activity.
Let's click edit on screen again and remove this anchor.
Another feature that might be very useful
sometimes is the ability of selecting a part of the text
or a couple of words.
The fields are orange, which means
there are duplicate matches.
For now, we can stick with the selection
that solved our problem earlier and hit Enter.
Now, let's see a very important aspect
when it comes to computer vision, refreshing the scope.
Let's say that we perform some actions
in a certain screen of an application,
and then we're led to a different screen.
In that case, because the screen changed,
we need to add one more CV screen scope
activity for the second screen.
Why is that?
Well, because the screen analysis
is cached in each screen scope individually.
As a, result all the activities inside a screen scope
actually work on the analysis performed
by that specific screen scope only.
To demonstrate this, let's use a click activity
to navigate to a different screen in our application.
First, we select indicates on screen,
and then we choose withdrawal to navigate away
from the screen to a different one.
Let's run this workflow.
As you've just seen, it types the value in the cache in field
and then clicked withdrawal and the screen changed.
Now, we need another CV screen scope
for the activities that are performed on the new screen.
We'll add a new screen scope and indicate the application again.
Studio is running computer vision and OCR
to detect everything in the application.
And now, it's ready.
Let's use the get text activity to retrieve
some data from the screen.
For example, the transaction number.
We need to store the results in a variable, which
will call trans number.
Now, we can output the number to a message box.
So let's enter the variable we've just created here.
Before we run this, let's go back to deposit.
So the workflow consists of writing
this volume in the cash in field going to withdrawal
and retrieving the transaction number.
Let's go back to studio and run the workflow.
And here is the transaction number.
We can click OK.
One more useful option when it comes to computer vision
is error reporting.
Now, it's a very valuable action.
Because based on it, we can improve the machine learning
model to ensure that every single element will be
recognized in future versions.
To see how errors can be reported,
let's select edit on screen.
On a side note, we could do this in the indicate mode, too
without a problem.
Now, let's imagine that a certain region on the screen
is not clearly visible.
If any such region exists, we can simply
click R, which is the reporting shortcut
and select that specific area.
Every element in the application was recognized.
But if there was an unidentified one in here,
this is how we'd report it.
To validate this SELECTION, we can hit Enter.
Reports successfully sent.
So please report any unrecognized elements
as that will contribute to improving
the accuracy of the computer vision activities.
OK.
We can close this for now.
That's about it.
But before we end the video, here
are a few best practices to use when automating with computer
vision activities.
In some situations, we might need
to isolate a portion of the application
and work only on that.
Now, this approach can avoid some false matches issues,
and also speed things up a bit.
So first, instead of clicking the entire application,
you can drag select a smaller region.
In our case, we don't need anything in the lower region
here.
So we can surround the area that we're interested in
and ignore everything else.
If we look at the screenshots, we
can see that it only includes the selection we have just
made.
Another recommendation is to store selectors in variables.
If there are a lot of screen scopes, in order
to avoid indicating it every time,
and update everything more easily if needed.
To do that, copy the selector create, a new variable.
We named it selector here, and paste its value in the default
column.
Now, before we end our demo, let's go over one more
useful tip, the default ocr engine
used in CV screen scope activity is Microsoft OCR.
But it's a good idea to try all of them
out if you run into any problems and see which one works best--
ABBYY, Google Cloud Vision, Microsoft Project Oxford
Online, or tesseract.
At the moment, the Google Cloud Vision or Abby OCR engines
might return the most accurate results.
ALL right.
That was all for today.
In this demo, we've seen what the UI Computer Vision
activities look like, how they work,
and when they can be used.
Thank you for watching.