micropsi-industries MIRAI UX1 AI robot control system
- June 4, 2024
- micropsi-industries
Table of Contents
micropsi-industries MIRAI UX1 AI robot
Setting up MIRAI
- Physical setup: First mount the camera fixture (with the camera and lens) and then the force torque sensor (F/T sensor) on the robot flange (Figure 1). Be sure the cables do not apply any force to the F/T sensor. If forces cannot be avoided, fix the cables to the robot tightly so that the forces are constant and do not change in different robot poses.1
- Cabling and setup: Connect the camera to the MIRAI controller using the included USB cable. Connect the robot and the OnRobot F/T sensor ethernet cable to the MIRAI controller’s robot network port using an ethernet switch (Figure 2). For the UR e-Series, make sure the robot is put into remote mode. Connect MIRAI’s remaining network port to WAN (internet), which provides a DHCP service. Turn on the MIRAI controller and wait for the second beep before using the MIRAI Training App.2
- Training app setup: Join the MIRAI-#### Wi-Fi network on your tablet (#### is the ID number of the MIRAI controller). The ID number and the password appear on the product sticker on the side of the MIRAI controller. Once connected, your tablet will notify you that there is no internet connection available through the Wi-Fi connection. You can ignore this warning. Use the tablet’s browser to go to http://mirai:6543/mint/apk to download and install the MIRAI Training App.3
- Network configuration: On your tablet, in the MIRAI Training App, go to the Network Configuration section and enter the IP settings of your robot. Update the last IP field for the F/T sensor(s) and the MIRAI controller. Please make sure that all the devices (F/T sensor, robot, and the MIRAI controller) are connected in the same subnet. (Alternatively, you can configure the robot and the F/T sensor to use the same subnet as the MIRAI controller. In that case, configure the robot to use 192.168.100.100, the OnRobot F/T sensor to use 192.168.100.15, and the ATI F/T sensor to use 192.168.100.20.)4
- URCaps setup: Install the MIRAI URCap extension using the included USB drive. Make sure not to install the URCap for the OnRobot F/T sensor.5
Before proceeding: (1) recheck that all devices (the MIRAI controller, the robot, the F/T sensor, and the camera) are connected, (2) verify the IP addresses of the devices, and (3) ensure that MIRAI has working access to the internet. For the UR e-series, be sure that the minimum software version 5.9.0 is installed. For the UR CB3 series, the minimum software version 3.9.0 should be installed.
Application Design
Think about what you want the robot to do. Visualize the entire movement that the robot will make; it can be broken down into smaller sections. For each section that requires movements between fixed positions, use the UR programming environment to program those movements. Sections that require camera guided movements to reach the target position should be implemented using MIRAI. For a good MIRAI skill, find the shortest, most consistent path that needs to be guided by camera control. Design the skills to be as small as your problem permits. When in doubt, split into multiple skills. You can intermittently use both MIRAI skills and fixed movements in the UR programming environment.6
Positioning Skill
A positioning skill allows you to precisely position the end effector relative
to a visible object using the optimal (that is, shortest) path. Positioning
skills are useful for a variety of applications and use cases, for instance
when:
- Placing a gripper in a pre-grip pose over a variable object;
- Positioning a work piece precisely for insertion;
- Placing a sensor tool relative to a work piece for quality inspection.
The following pre-requisites must be met for using positioning skills:
- The direct path between the robot’s pose at the start of MIRAI’s execution phase and the goal pose must be free of obstructions.
- The involved objects must not change their location during training for the duration of the MIRAI-controller motion. You can move them after training an episode.
Motion Skill
Create a motion skill for more complex or dynamic motions, where the
positioning of a tool is not sufficient but you want to have full control over
the robot’s trajectory (the exact path, velocity, and acceleration). Examples
of such applications include:
- Different types of insertion tasks (e.g., cable plugging);
- Path or contour following tasks (e.g., gluing);
- Picking or placing objects from and to a moving conveyor;
- Quality inspection;
- Tool positioning where there are obstacles and the direct path from the starting point to the goal is not attainable.
Creating a MIRAI Skill
On your tablet, in the MIRAI Training App, create a new skill. For skill type,
please select positioning skills or motion control depending on the problem
you want to solve. (Figure 3).
Use the live video feed from the camera to verify that the camera captures all
the parts that play a role in your task, such as the end effector and the
target object(s) (Figure 4). To fine-tune the image, adjust the focus ring and
aperture on the camera itself. Aperture ranges from 1.4 to 16; a setting of 8
is recommended.
These physical settings on the camera are global; they affect all the skills
on your MIRAI controller. If altered, all the other skills that were trained
using these settings will break. Try to look for a combination of focus and
aperture that will work for all skills that your application needs. If this is
not possible, consider adding a second camera.
In the MIRAI training app on your tablet, tweak the exposure and gain to
minimize over- or under-exposure. Guide the robot through the intended path(s)
to verify that the lighting is always sufficient. Take your time with this
step; play around.7
Skill Training for Positioning Skills
After completing the physical setup, proceed with recording episodes.8
- Tap Add episode on your newly created skill to enter the recording loop and follow the on-screen instructions (Figure 5). For each recording, the robot can be freely guided. Guide the robot such that the tool is placed precisely at the final target position at which the robot should stop the MIRAI execution and should perform the next task (e.g., insertion or picking). Validate that the gripper and the target position/objects are visible in the camera.
- Tap Start recording and guide the robot for about one minute in a criss-cross, spiral motion over the target and around it, covering all positions that would be encountered during execution. If rotations are enabled, also rotate the tool around the relevant axis throughout the episode. For precision, spend most of the recording time in the space close to the target. (Figure 6)
- Once your scan is complete and you let go of the robot, it will return to the target position. Tap Stop recording.
- Tap Use for training. Discard the recorded episode if you see that the target object was moved accidentally or if the gripper and the object were not in the camera view, even if for a brief moment.
- Continue with Step A to add another episode. To make your skill robust, vary the task — move the target object, vary the tool position, change the background or the lighting condition, or change the object (different color, shape, or size).
After recording a minimum of five episodes, exit to the home screen of the training app and tap Start cloud training for this skill.9
Skill Training for Motion Skills
After completing the physical setup, proceed with data recording. During the
recording sessions, you will demonstrate the motion you want the robot to
perform in many varied repetitions of your task.10
- Tap Add episode on your newly created skill to enter the recording loop and follow the on-screen instructions. In each recording, the robot can be freely guided. For each episode, vary the task (move the target object, vary the tool position, change background, change the object) and guide the robot to a start position that you choose within the working space around your target (validate that the tool and objects are visible in the camera feed).
- Tap Start recording and guide the robot in a smooth motion to the target position motion. This can be a curved trajectory to move around obstacles, but it should be the shortest, most direct path. Once the trajectory is complete, hit Stop recording. (Figure 7)
- Tap Use for training if the recorded episode is a good representation of the task. (Figure 8)
- Continue with Step A to add another episode.
After recording around 50 episodes, please record additional recovery episodes (approximately 50). Recovery episodes should start in a position that is slightly off the target; the goal is to bring the robot back in case it goes off-track. Recovery episodes are crucial for a robust and precise skill.
Exit to the home screen and tap Start cloud training for this skill. For simple tasks with few varying aspects, a first competent skill will emerge when trained on about 50 recorded episodes. A robust behavior is usually available after 200 episodes.11
Testing and Configuring a MIRAI Skill
- Skill testing: When MIRAI receives a skill back from cloud training, the notification “Updated skill available” appears on the home screen. Start testing it by selecting Test and configure skill.12
- Configuration: For each skill, configure the condition (end-state) when the MIRAI skill should hand back the control to the UR host program. This end-state is typically reached when (Figure 9):13
- A certain force is applied, for instance touching or bumping (force-based end-state);
- The robot stops moving because it has reached the desired target position (position-based end-state);
- The TCP slows down, for instance when the target position is reached (speed-based end-state);
- The visual end-state setting automatically detects when a skill ends by comparing the live camera feed to the recorded images during training and can be configured to a level of confidence (default is 0.7).
- You can also increase the speed of the skill execution or define speed profiles for skills to further optimize the execution time (Figure 10).14
- Application setup: After you have a robust MIRAI skill, you can embed the skill in a Polyscope program using the MIRAI URCaps.15
Best Practices and Guidelines for Skill Training
This section contains valuable advice on how to train a MIRAI skill in a way
that ensures a robust skill. We strongly recommend that you read through this
section before you proceed with training a skill.
Golden Training Rules
Golden rule 1: When you look at the video feed at any point during the robot’s
intended movement, can you determine how to proceed based on what you see? If
the answer is yes, MIRAI will be able to learn the skill. If no, MIRAI will
not learn how to move either.
Golden rule 2: Include all the kinds of variance that you require the skill to
be robust against (e.g., varied positions of objects and the tool, varied
appearances, or lighting conditions).
Variance
A key strength of MIRAI is its ability to deal with variance. This means that
one can teach a MIRAI-powered robot to move differently depending on the
constellation of things seen on camera. Similarly, because MIRAI can also
ignore irrelevant variance, one can teach it to robustly move the same way
even if something related to the object changes — such as variations in
lighting. Through demonstrating the desired motion by directly guiding the
robot, showing it a variety of possible paths, one achieves both reactiveness
and robustness. Examples of sources of variance that are often relevant in
automation tasks are:
- Variations in the starting and end points of trajectories; variations in the position of target objects in a task or during the manipulation of the working part.
- Changing background or moving objects in the background. This is only relevant if the background is visible in the recorded episodes.
- Differences in the color and/or shapes of the objects/working parts.
- Changes in lighting conditions (e.g., changing intensity of daylight). Direct exposure to sunlight can negatively impact the robustness of the skill and should be avoided if possible.
- Imprecisions and variations in gripping positions at the TCP.
- Training Positioning Skills
Before recording:
- The tool should be placed precisely at the final target position at which the robot should stop and perform the next task (e.g., insertion, picking, testing).
- Make sure that there is no obstruction in the direct path from the intended starting point to the target.
- Introduce all sources of variance within their possible range and include extremes that are relevant to make the skill robust. For example, if your target object varies a lot, it is better to spend 20 seconds on 50 different objects for a positioning task rather than a minute each on only five varied objects.
During recording:
- Do not move the target or introduce any variance during recording.
- When guiding, hold the robot below the F/T sensor, ideally keeping any appendages out of the camera frame.
- Please make sure that the gripper and the target position/objects are visible at all times in at least one of the two possible camera feeds. You can review the recorded episodes afterwards as well.
- Starting from the target position, guide the robot in an overlapping spiral motion away from the target. Your goal is to visit all tool poses near the target that the robot may pass by during the execution of the skill.
- Continue your random motions at an increasing distance away from the target all the way back to the handover point where the skill should take over from the previous program step.
- Spend most of the recording time near the target.
- For rotations-enabled skills, repeat the spiral motion at various orientations of the tool.
- Occasionally, let go of the robot. It will then move towards the target pose on the shortest path. This provides a preview of the final trajectory and direction the trained skill will perform from a given starting pose.
Training Motion Skills
- When guiding, hold the robot below the F/T sensor, ideally keeping any appendages out of the camera frame.
- Pick a consistent strategy and stick to it: Never show two different routes to reach the same goal. After all, the system cannot decide which of two is best. It will try and fail to create a middle way. The easiest strategy is to always go the shortest path from start to goal, wherever possible.
- Move with consistent speed: MIRAI will pick up speed changes (e.g., first moving quickly, then slowly). Make sure there are no pauses.
- During the training, the tool center point (TCP) and the object should be always visible in the recordings.
- Be ready to tap stop recording the mo ment the task is done; do not wait.
- Some tasks are difficult to solve by robot guiding. You may want to practice a few times before you start recording the data. MIRAI will do what you do, not what you want. Show it clear, decisive examples.
Refining an Existing Skill
It is common that after the first skill iteration there are some weaknesses in
the new skill. For instance, the motion might be imprecise in certain task
configurations, or it may overly depend on a particular lighting situation. In
these cases, the reason is that the situations where the skill is still weak
are not yet sufficiently represented in the recorded demonstrations. Hence, in
areas where you notice weaknesses, record more episodes, focusing on the
situations in which the skill needs help. Repeat the whole process until the
skill becomes robust. If the skill works well in one lighting condition but
not another, record additional episodes with varied lighting. If a skill moves
too far beyond a target object, record some episodes in which you pick a start
position that is already beyond the target, and then just demonstrate the way
back (recovery episodes). Repeat the whole process until the skill tests
competently in all the desired situations.
Include Recovery Episodes
To make your skills more robust, be sure to include episodes in which you show
MIRAI how to recover from states it should never reach. This will ensure that
it knows how to get back on track if it ever makes a mistake. Include episodes
that start slightly off course. Show MIRAI how to move towards the target
position if it finds itself in those undesirable states.
References
Read User Manual Online (PDF format)
Read User Manual Online (PDF format) >>