Hello and welcome to UiPath Essential training ‐ Image-based automation.
In a previous training you learned about recording desktop applications, and how to manually
add actions to the recorded workflow.
You saw examples for 3 out of the 4 available types of recording: Basic, desktop, and Web.
Today you’ll get to know the 4th one, Citrix.
The tools and methods it uses are made for automating virtual machines,
but they are useful in other types of automation too.
VMs usually run on a server, and only the image of the interface is streamed to the
user.
Therefore, UiPath cannot address the interface through the operating system, as you learned,
with Basic recording and selectors.
We’ll need some special tools and techniques
for that, and fortunately UiPath has methods of dealing with these constraints.
But there is one important limitation: you cannot automatically record a set of actions,
and have them played back.
You have to hold the robot’s hand, and show it what to do; at least for now.
So, how do you do that?
You try to teach it how a human does it: by finding elements on the screen – like buttons
or text fields – and doing something, in relation to those elements: clicking buttons
and controls, or filling in a text box.
Let’s see a practical example.
We have this Remote desktop connection, and on the remote desktop we have a dummy banking
app.
Let’s say we want to click on some radio buttons and fill in a few text fields.
What we’d normally do is read the name of an element, then click the corresponding
control, and maybe type something.
A robot does the same.
Using clickImage, you can click on mostly anything: buttons, menus, text, or somewhere
relative to such an element.
As long as the image you choose is unique.
Let’s select this area for the Deposit radio button.
There is no other identical element on screen so we should be good.
We can choose where to perform the click, inside or outside the selected area.
We’ll hit ok for a click in the center of the rectangle.
Right; next, let’s fill out these fields using the same method.
Since we cannot select something as generic as a textbox, we’ll choose something
unique that’s close to it, and doesn’t move in relation to our textbox.
It’s name is usually a good choice.
So, select the Cash-in area, and this time choose to indicate where, relative to the
yellow rectangle, to actually make the click.
We’ll pick the text field.
And type something to help us see the focus.
And we’ll do the same for the next one: ClickImage action, click&drag to select, indicate
position... in the textfield next to it.
And type.
You could use this method for the 3rd editable field too, but let’s see a complementary
action, clickTxt.
The difference from its sister method, is that clickText uses OCR to scan the screen
of the virtual machine for text.
Hence, it works even if things like windows-theme or text size are different.
ClickImage, on the other hand, is faster and very reliable, but more sensitive to such
graphical variations, and can fail if colors or background details change.
So it’s more prone to errors in fluctuating environments.
So let’s try clickText on the last input field.
You’ll recognise this window from the ScreenScraping training, but this one has a few more options
on the bottom here.
The Text-to-be-found field, is the text that you want to click on, or click relative to.
We want to click on the 3rd textfield, so we’ll just enter it’s name.
Now UiPath will click on it.
We want to click next to it, so we’ll click Set mouse position, and just like before,
we’ll indicate where to click; in this case, on the editable field next to it.
After we insert one more type action, we’ll do another clickText to press the Accept button.
That’s it, exit and run the generated workflow.
Great.
It correctly identified the radio button and the text fields, clicked in the designated
spots, and typed in the text fields.
Inside the generated workflow, there are a
few important parameters you should know about.
For the clickText and clickImage actions, options for single or double click, and left
or right mouse buttons can be changed here, and modifier keys can be added here.
Click Image also has an Accuracy parameter you can play around with, and at it’s maximum
value, 1, the images must match 100% to register as found.
0.8 is a good balance between accuracy and reliability, but feel free to try other values.
You should know that ClickImg and clickTxt have some drawbacks which can make them, in
some situations, not 100% accurate.
On virtual machines, ClickTxt relies on OCR to find the desired text, so even one misidentified
letter can throw it off track.
And clickImage is very reliable, unless the theme or other graphical elements have changed.
So here’s an easy alternative that can be faster, safer, and works in desktop & web
apps too.
We’ll go in further detail about it in a future training, but it’s pretty straightforward,
so here’s how it works.
The trick is simply to avoid using mouse interactions, and replace them with keyboard actions such
as navigating with the Tab key, and using keyboard shortcuts to activate different functions
of the application.
Basically, just add Tab and other navigation keys using the type tool.
For example, let’s type in this field, and after the value, we’ll add a Tab key to
move to the next field.
And so on for the rest.
Or, you can combine multiple keyboard actions into a single one.
Type the first value, then add a Tab, type the value for the next field, then add a tab,
and so on.
After the last field, the focus moves to the Accept button, so we’ll add a Space to press
it; and we’re done.
That’s all there is to it.
As you can see, it’s a very fast and powerful method to interact with applications, virtual
or not.
So now you know how to navigate a virtual machine interface and activate various menus,
buttons, and text boxes, with 3 actions: clickText ,clickImage, and TypeInto.
To get information out of virtual machines, there are again, 2 methods: Select&Copy, and
ScrapeRelative.
Select&Copy is the easiest, but works only for selectable text, like these textboxes
here.
Remember that all commands are sent to the whole virtual machine window, so the action
is performed on the active textfield.
That’s all.
The two resulting actions are, you guessed it, select & copy.
The copied text is available as the output variable of the second action, Copy selected
text.
Of course, you should combine it with any of the 3 input actions to bring the focus
to the intended text field.
For example, we could simply drag in a Type action, and simulate pressing the Tab key
to move to the next input.
And drag in a writeline for testing.
Let’s run it.
That’s the easiest method of getting text out of virtual machines.
But what do you do when some text that you want is not in a selectable field?
Easy.
Remember screen scraping, and how that gave you the contents of the whole window or container?
You aply the same output technique, but relative to a fixed element.
The action is called Scrape Relative, and it’s here under Screen Scraping.
It allows you to scrape just a portion of an image, relative to an anchor.
Let’s get the result of this deposit and
also the transaction iD. We first select the anchor, Total deposit,
and then indicate the area to scrape.
In the scraping wizard, if we play around with the setting we can see we get a better
result with google OCR and a scale of 3.
And continue, then repeat the process to get the transaction id.
Screen scraping > scrape relative > select the anchor > and indicate the area to scrape.
Let’s see what we got.
You learned about most of these actions in the ScreenScraping training, but here’s
the newcomer.
The ScrapeRelative action is actually a series of 3 actions:
first finding the anchor image.
when an image is found, a region identical to the anchor selection you made, is the actual
result; then, UiPath calculates the region you indicated,
based on the anchor image; and gets the content of that region, using
GetOCRText.
There’s also a 4th action that resets the clipping region, because it’s a shared resource,
and this avoids interference with other operations.
It’s not mandatory to understand what each of these actions does, but you should know
that the scrape relative action generates these 4 activities.
Now all we have to do is add in a couple of writelines and we’re done.
The output of the ScrapeRealtive action is in the GetOCR actions.
Looks like our results are pretty good, except a couple of spaces we can clean up later.
That’s all you need to know to start automating
virtual machines.
Let’s do a quick recap to see what we learned.
Today we focused on image automation, and its special limitations.
The first tool you learned about was ClickImage, and you saw how to use it to correctly identify
and click on certain interface elements, like radio buttons and text fields.
You also got a chance to try out the ClickText tool, which scans the whole virtual machine
for text location, and lets you to click on any identified text, or relative to it.
There was also a very useful technique of using the keyboard exclusively, to move around
an app’s interface using special keys like Tab, arrows, and other shortcuts.
In order to get information OUT of apps, the first and simplest method was Select and Copy.
As its name implies, it simply selects editable text, and copies it to the UiPath environment,
through the clipboard.
For more difficult, non-selectable text, you learned about the Scrape Relative action.
It locates fixed elements on the screen, and then, using OCR, extracts your relevant information.
That’s all for image automation, but if you’d like to know more, have a look at
the advanced training.
We’ll be covering a few more techniques related to virtual environments.
Goodbye!