Sometimes, we need to develop an RPA automation to operate processes on different VMs (Virtual Machines). In this article, we will better understand what an RDP connection is and how to use it to your advantage when accessing systems on virtual machines so that your process is automated.
What is an RDP connection?
RDP (Remote Desktop Protocol) is a protocol developed by Microsoft to control a computer remotely using a graphical interface, providing remote access through the Windows operating system safely and efficiently.
How to connect to a VM via an RDP connection?
Before you can connect to a remote computer using the RDP protocol, you need to know the IP address or name of this computer, the username, and the password. If the process were manual, you would need to obey the following steps:
- On your local computer, search for “Remote Desktop Connection” in the Windows Start menu and click on the suggested application;
- Enter the name of the computer or its IP address in the “Computer” field;
- Then click on “Connect” and, when prompted, enter the credentials with username and password so the authentication happens successfully;
- After authentication proceeds, you can access the remote computer as usual.
But, as we will build RPA automation to access one computer from another, we will use the power of computer vision to automate the processes.
What to know before developing your RPA automation
For your automation to execute the necessary processes, it is crucial to ensure that your local computer, or the VM running the robot to access the other VM where the systems are, has the necessary permissions to execute PowerShell commands, scripts, etc.
Identifying which tasks must be executed and validating all the credentials needed to access the VMs and systems involved is essential. Additionally, consider storing these credentials securely. The step-by-step guide will explain how to do this using BotCity Orchestrator.
Another important consideration will be the issue of VM resolutions. This is because we will use computer vision to look for screen cutouts, as all steps will be on the desktop. So, during the development process, we will consider fixed resolution configuration.
Why use the RDP connection for your automation?
This approach can be advantageous in cases where we cannot execute scripts and commands directly on the computer.
Furthermore, it can also be an excellent support for your automation if it is only possible to access the software through a remote connection.
Development example
Our automation must access a virtual machine through a Windows RDP connection to register products from a .json file in the Fakturama, a free desktop software for managing products and invoices.
Below is an example of a file with products called payload.json that we will use to register using our automation:
{ "load": { "customer": "BOTTEST", "products": [ { "item_number": "0000000000001", "name": "PRODUCT DEMO 1", "category": "ELECTRONICS", "gtin": "0000000000123", "supplier_code": "SUPPLIER1", "description": "PRODUCT DEMO 1", "price": "100.00", "cost": "50.00", "allowance": "0.00", "vat": "Free of Tax", "stock": "100" }, { "item_number": "0000000000002", "name": "PRODUCT DEMO 2", "category": "ELECTRONICS", "gtin": "0000000000124", "supplier_code": "SUPPLIER1", "description": "PRODUCT DEMO 2", "price": "94.90", "cost": "47.50", "allowance": "1.90", "vat": "Free of Tax", "stock": "143" } ] } }
Additionally, here is an architectural model for our example case with BotCity Orchestrator:
How to access a remote computer through automation
To develop this automation, we will use the Python desktop framework. You can follow the first steps through this tutorial in our documentation and install the BotCity Studio plugin for Visual Studio Codeif that is your IDE.
Following the step-by-step, you will have a project with the following structure (we are calling the robot BotRDP):
BotRDP
├── bot.py: this is where we will develop our robot.
├── resources: folder containing the auxiliary files for the bot.
├── build.bat: batch script to generate the package.
├── build.ps1: PowerShell script to generate the package.
├── build.sh: shell script to generate the package.
├── requirements.txt: file describing all external dependencies for your bot.
└── BotRDP.botproj: file used to load the project into BotCity Studio if not used by the Visual Studio Code plugin.
Within this structure, we will add the payload.json file to the resources folder containing the products to be registered. Or, you can send the file contents as a parameter for your automation.
Find out more:: Documentation about parameters.
Reading the payload.json file
# open .json file with open(r'resources\payload.json') as json_file: json_data = json.load(json_file) # defining a name to be able to search for credentials according to the client's name customer_label = json_data["load"]["customer"] # defining a name for the list of products contained in the payload.json file products = json_data["load"]["products"]
How to add credentials in BotCity Orchestrator
To secure your automation, you can use the Credentials feature available in BotCity Orchestrator.
In the menu on the left side of the Orchestrator, identify the “Credentials” option under “Operation Tools.” Click on “New Credential” and add the requested information:
- Label: add the credential’s name;
- Repository: enter the repository that should have access to these credentials;
- Secrets: click “Add” to add information:
- Key: name of the credential key;
- Value: the value of the credential.
You can add as many keys/values as needed.
In our example, we will do it as follows:
- Label: BOTTEST;
- Repository: DEFAULT;
- Secrets:
- Key: remote_ip;
- Key: username;
- Key: password;
- For each one of the keys, add their respective values.
In our code, you can access credentials as follows:
remote_ip = maestro.get_credential(customer_label, "remote_ip") username = maestro.get_credential(customer_label, "username") password = maestro.get_credential(customer_label, "password")
Note: if you want to run the bot locally so that you can connect to the Orchestrator and validate that you can access the credential, you must first add this code to log in to the tool:
maestro.login( server="add the server URL here", login="add the login here", key="add the key here" )
To find out what the login and key are for this login to be successful, you must access the “Dev Environment” area through the menu on the left side of your Orchestrator. Find out more: Dev Environment documentation.
⚠️Remember that you will use this code snippet only while running the bot on your computer. You must remove this code before deploying your bot in Orchestrator to run automatically.
How to create a function to connect to the VM via RDP
We will use the Python subprocess library to connect to the remote computer, which will help us run Powershell, making it easier to open the connection. See how we developed the code, creating a new commands.py file and adding the following code:
import subprocess def run_start_rdp(remote_ip: str, user: str, password: str): # command to start session: mstsc start_rdp_command = f''' cmdkey /generic:"{remote_ip}" /user:"{user}" /pass:"{password}" mstsc /v:{remote_ip} /w:1600 /h:900 cmdkey /delete:{remote_ip}''' # running powershell with the command created above subprocess.run(["powershell", start_rdp_command])
In the command, we use the /w and /h items to set the screen resolution of the remote computer you will access. Fixing the resolution and positioning (the position part we will do later in the code) will help ensure the automation runs successfully.
⚠️Note that in the first runs, dialog boxes may appear for some certificate confirmations or even login and password for the RDP connection. You can use computer vision to interact with these dialog boxes or leave a check on the “do not ask again” option.
After that, import this file into your main code:
from commands import run_start_rdp
Then call the function:
run_start_rdp(remote_ip, username, password)
How to set VM screen in focus for automation execution
Let’s add the following code snippet:
bot.connect_to_app(best_match="Remote Desktop Connection", timeout=30000) main_rdp_window = bot.app.top_window() main_rdp_window.set_focus() bot.wait(3000)
It may be a good practice to use the wait() function from the BotCity framework in some steps of the code. Remember that you will execute something in a virtual machine that is typically slower than a local computer.
Fakturama
To open Fakturama, search on the virtual machine’s Windows Start menu bar. But before, let’s limit where it should occur so that the computer vision does not confuse the cut made with your local computer or the incorrect VM.
So before opening, let’s ensure the position of the window:
main_rdp_window = bot.app.top_window() window_rect = main_rdp_window.client_rect() x, y, width, height = window_rect.left, window_rect.top, window_rect.width() + 50, window_rect.height() + 50
In this code, x and y are the screen’s position, and width and height will guarantee the screen resolution, which, at the beginning, to connect to the remote computer, we fixed as 1600 x 900. In other words, the query will only consider within this screen rather than outside it, avoiding problems of confusion on which screen you should find what will be requested.
Mapping VM start menu search bar by computer vision
We can follow the next steps using the BotStudio plugin for Visual Studio Code. If you want to do it this way, follow the step-by-step installation and configuration instructions in the Studio plugin documentation.
If you want to use Fakturama to follow the example, install it following the instructions in the Fakturama documentation.
In Visual Studio Code, leave the cursor where the new code should be automatically inserted. Click on the plugin in the left-side menu, and you will have a window to take screenshots of the clippings you want to map for your automation.
Now, map the search icon to type “Faktumara.exe” in the virtual machine’s Windows Start menu.
Leave the BotCity Studio screen on top of the screen we will take the screenshot of. Click on the icon in the menu on the left side identified as “Screenshot” with a camera symbol. This action will create a screenshot from the screen.
After that, click once on the search icon in the start menu and then click and drag the mouse, selecting the corresponding icon. If you make a mistake, press the “ESC” key on your keyboard and click where you want to select again.
When selecting the search icon, add a name for it and the “Click” action, leaving the “Image” mode selected and clicking “Submit.”
When returning to the code, you will notice that the computer vision added a code similar to the quote below to your project:
if not bot.find("search_icon", matching=0.97, waiting_time=10000): not_found("search_icon") bot.click()
Your project’s resources folder will store all the clippings you map. Without these cutouts, the code cannot find the items on the screen you need for the automation to run correctly.
In this code, in the find() function, we will add the previously configured position and resolution as parameters. Then the code will look like this:
if not bot.find("search_icon", matching=0.97, waiting_time=10000, x=x, y=y, width=width, height=height): not_found("search_icon") bot.click()
After that, the robot must write “fakturama.exe” in the search bar and press “enter” to open the system. And to ensure time for the software to open before the following steps, we will add time.
bot.type_keys("fakturama.exe") bot.enter() bot.wait(4000)
Another way to add the name to the search is to use another function, according to the paste() documentation guidelines.
Product registration on Fakturama
To ensure that we have the Fakturama screen highlighted, we can map the system name and consider a longer waiting time so it will only continue if it finds Fakturama highlighted on the screen.
Following the same procedure as mapping the search icon cutout, we will map the name of Fakturama. The code may look similar to the one below:
if not bot.find("label_fakturama", matching=0.97, waiting_time=60000): not_found("label_fakturama")
As we have a list of products inside the payload.json file in the resources folder, we will use “for” to iterate through this list and register all products. The code may look similar to the one below:
for product in products: # we first need to click on “New Product” if not bot.find("button_new_product", matching=0.97, waiting_time=10000): not_found("button_new_product") bot.click() # the next step is to add the values in the fields if not bot.find("label_item_number", matching=0.97, waiting_time=10000): not_found("label_item_number") bot.click_relative(84, 4) bot.type_keys(product["item_number"]) print(product["item_number"]) # you can use “tab()” or map each field using computer vision bot.tab() bot.type_keys(product["name"]) bot.tab() bot.type_keys(product["category"]) bot.tab() bot.type_keys(product["gtin"]) bot.tab() bot.type_keys(product["supplier_code"]) bot.tab() bot.type_keys(product["description"]) bot.tab() # as there is already a number 0 in some fields, we can use control_a() to select all the content and type the correct value above bot.control_a() bot.type_keys(product["price"]) bot.tab() bot.control_a() bot.type_keys(product["cost"]) bot.tab() bot.control_a() bot.type_keys(product["allowance"]) # the vat field won't need to be changed, so we'll just use tab() to go to the next bot.tab() bot.tab() bot.control_a() bot.type_keys(product["stock"])
After filling in the product data, we can save what we registered and close the product tab. To do this, we map the save button, still inside the “for” code:
if not bot.find("button_save", matching=0.97, waiting_time=10000): not_found("button_save") bot.click()
Also, inside the “for,” we map the x of the product tab so we can click on it to close it:
if not bot.find("button_close_product", matching=0.97, waiting_time=10000): not_found("button_close_product") bot.click()
How to close the Fakturama and RDP connection
As a good practice, we should always close the systems and connections used in process automation. One of the ways to do this is with the following code (already outside the “for” so that they only close after product registration has finished):
# as the Fakturama screen is highlighted, we can just close it by command bot.alt_f4() # closing RDP connection main_rdp_window = bot.app.top_window() main_rdp_window.close()
We must add task completion since we will run our automation through the Orchestrator. Orchestrator. Here, you can also map other situations, such as errors and different statuses, depending on what happens with the automation. See the example below and explore other opportunities in our documentation about tasks.
maestro.finish_task( task_id=execution.task_id, status=AutomationTaskFinishStatus.SUCCESS, message="Task Finished OK." )
Full code
See how is the complete main code:
from commands import run_start_rdp from botcity.core import DesktopBot from botcity.maestro import * # Disable errors if we are not connected to Maestro BotMaestroSDK.RAISE_NOT_CONNECTED = False def main(): maestro = BotMaestroSDK.from_sys_args() execution = maestro.get_execution() print(f"Task ID is: {execution.task_id}") print(f"Task Parameters are: {execution.parameters}") bot = DesktopBot() with open(r'botPython\resources\payload.json') as json_file: json_data = json.load(json_file) customer_label = json_data["load"]["customer"] products = json_data["load"]["products"] remote_ip = maestro.get_credential(customer_label, "remote_ip") username = maestro.get_credential(customer_label, "username") password = maestro.get_credential(customer_label, "password") run_start_rdp(remote_ip, username, password) bot.connect_to_app(best_match="Remote Desktop Connection", timeout=30000) main_rdp_window = bot.app.top_window() main_rdp_window.set_focus() bot.wait(3000) main_rdp_window = bot.app.top_window() window_rect = main_rdp_window.client_rect() x, y, width, height = window_rect.left, window_rect.top, window_rect.width() + 50, window_rect.height() + 50 if not bot.find("search_icon", matching=0.97, waiting_time=10000, x=x, y=y, width=width, height=height): not_found("search_icon") bot.click() bot.type_keys("fakturama.exe") bot.enter() bot.wait(4000) if not bot.find("label_fakturama", matching=0.97, waiting_time=60000): not_found("label_fakturama") for product in products: if not bot.find("button_new_product", matching=0.97, waiting_time=10000): not_found("button_new_product") bot.click() if not bot.find("label_item_number", matching=0.97, waiting_time=10000): not_found("label_item_number") bot.click_relative(84, 4) bot.type_keys(product["item_number"]) bot.tab() bot.type_keys(product["name"]) bot.tab() bot.type_keys(product["category"]) bot.tab() bot.type_keys(product["gtin"]) bot.tab() bot.type_keys(product["supplier_code"]) bot.tab() bot.type_keys(product["description"]) bot.tab() bot.control_a() bot.type_keys(product["price"]) bot.tab() bot.control_a() bot.type_keys(product["cost"]) bot.tab() bot.control_a() bot.type_keys(product["allowance"]) bot.tab() bot.tab() bot.control_a() bot.type_keys(product["stock"]) if not bot.find("button_save", matching=0.97, waiting_time=10000): not_found("button_save") bot.click() if not bot.find("button_close_product", matching=0.97, waiting_time=10000): not_found("button_close_product") bot.click() bot.alt_f4() main_rdp_window = bot.app.top_window() main_rdp_window.close() maestro.finish_task( task_id=execution.task_id, status=AutomationTaskFinishStatus.SUCCESS, message="Task Finished OK." ) def not_found(label): print(f"Element not found: {label}") if __name__ == '__main__': main()
And here is the commands.py file code:
import subprocess def run_start_rdp(remote_ip: str, user: str, password: str): start_rdp_command = f''' cmdkey /generic:"{remote_ip}" /user:"{user}" /pass:"{password}" mstsc /v:{remote_ip} /w:1600 /h:900 cmdkey /delete:{remote_ip}''' subprocess.run(["powershell", start_rdp_command])
Deploy and execute the task
Before proceeding to this step, it is always worth testing what you have developed locally. And remember, before deploying to the orchestrator, delete the login code snippet.
To deploy, follow this article about “how to create and execute your automation tasks in BotCity Orchestrator” or follow the step-by-step instructions in our deployment documentation.
To perform your task, you must have the BotCity SDK installed and open, through it, the BotCity Runner on the computer that will execute the automation and that needs to open the VM to run the system contained in it.
What did you think of the possibility of automating processes in VMs using an RDP connection?
We invite you to join our global RPA community to go deeper into this topic. Just join our Slack or get in touch via our forum .. If you have any questions, just call us! And don’t forget create your free account to test the features we are exploring in our content. When creating your automation to learn how to use the tool, remember to share with the community via Bot Repository.
