Hardware

Franka Emika Panda Arm

Minimum system requirements

For this part we refer to the minimum system and network requirements written by Franka Emika.

Setting up a real-time kernel

For this part we refer to setting up a real-time kernel written by Franka Emika.

Boot VUB desktop with correct kernel

Note

This section describes how to boot the desktop that is already set up to work with the Panda 2 robot in the R&MM lab at Vrije Universiteit Brussel.

When you start up the desktop, you will get the following screen. Navigate with the arrow keys to the student boot and press F2 for more boot options.

In order to control the Panda robot later on using libfranka, the controller program on the desktop must run with real-time priority under a PREEMPT_RT kernel. Since a real-time kernel is already set up on the desktop, we can now navigate with the arrow keys to the rt9 version as shown in the figure below and press Enter.

Afterwards, you can choose the Main User account and enter the password MainUser.

Set up the robot network

Hardware connections

In the figure below you can see the official hardware connection documentation from Franka Emika which you can find in the Franka Emika Panda manual.

In our setup we have the following connections:

• desktop to Panda Arm with ethernet cable

• desktop to Panda Control with ethernet cable

• Panda Arm to Panda Control with connection cable

• external activation device to Panda Arm

• Panda Control to Panda Arm with earth cable

• Panda Control to power supply

• dekstop to an ethernet port with an ethernet cable (if there is no Wi-Fi)

Set up the network connections

Turn on the Panda Control via the On-Off switch at the back of the Panda Control.

After the Control has been switched on, or supplied with energy, Panda boots up. In this process, the safety locking system is activated, movements are thus locked mechanically. The display lights on the base and the pilot flash yellow.

As soon as Panda has booted up, the display lights are lit yellow continuously.

Note

The part below is specifically written for the use of robot Panda 2 in the R&MM lab at Vrije Universiteit Brussel with the desktop that is already set up to work with this robot. This robot has shop floor IP address 192.168.2.106 and Netmask 255.255.255.0.

For the general version of this documentation we refer to setting up the network for the Franka Emika Panda robot.

In case there is no Wi-Fi connection, you should see three ethernet connections:

• a connection to FrankaControl

  

• a connection to FrankaArm

  

• a connection to Internet connection

  Note that this is the personal static IP address that is assigned to Kelly Merckaert at Vrije Universiteit Brussel. At VUB, they work with static IP addresses, so if you need internet via an ethernet connection, you have to ask for your own IP address.

  

Depending on which ethernet ports you have connected, you have to select the right ethernet connections. In the example below we have connected port enp10s0f0 to FrankaControl, port enp10s0f1 to FrankaArm, and port enp12s0 to Internet connection.

Start and shut down the robot via Franka Desk

Note

To follow this section for another robot than the Panda 2 in the R&MM lab at Vrije Universiteit Brussel, you will have to replace 192.168.2.106 with the shop floor IP address of your own Panda robot and change the username and password required to enter Franka Desk.

Start Franka Desk

Once the Panda has booted up, you can start Franka Desk. Open your web browser and type 192.168.2.106 in the search bar. You will see a window where you will have to give your username and password to enter Franka Desk. The username is admin and the password is franka123.

When the robot is well connected and you have signed in, you should obtain the following window:

Settings

You can go to the settings window in Franka Desk by clicking on the two lines in the right upper corner. Like that you can go to the settings menu, but you can also download the User Manual and the Cobot Pump Manual.

In the settings menu, you can see in the Dashboard tab the system, robot, and network settings. In the Network tab, you can change the robot and the shop floor network. In the System tab, you can install features like the Franka Control Interface, do a system update, download log files, etc.

In the End-Effector tab, you can select the standard Franka Hand or select Other.

• If you are using the standard Franka Hand end-effector, then select the Franka Hand. In case the Franka Hand is uninitialized in the Dashboard tab, you can re-initialize the hand by clicking on HOMING. Afterwards the hand should be initialized in the Dashboard tab.

  

• If you detach the Franka Hand end-effector and want to attach another end-effector, then you have to select Other. Like that, you will be able to give the properties of your new end-effector, as in the example below.

  

To go back to the Desk, click again on the lines in the upper right corner and click on Desk.

Unlock the joints

At this point, the display lights of the Panda Arm should be lit yellow continuously. This means that the Panda has booted up and that the safety locking system is activated, whereby movements are locked mechanically.

Make sure that the external activation device is in the closed state (pressed down).

The safety locking system can now be opened with the button click to unlock joints in the sidebar of Franka Desk.

After you have clicked on open, you should hear seven clicks of the seven joints that are being unlocked.

The display lights should now be lit white continuously. Desks’s sidebar shows now joints unlocked. Panda is now in the monitored stop state.

Shut down the robot

When you are done using the robot, don’t forget to put the external activation device in the closed state (pressed down) and lock the joints before shutting down the Franka Control and the desktop.

The safety locking system can be closed with the button click to lock joints in the sidebar of Franka Desk. You will here one click which means that all the seven robot joints are locked and afterwards the display lights should be lit yellow continuously.

Robot Operating States

There are status lights on both sides of the base which take on the corresponding color. These status lights will only flash during boot up, during other processes the lights will glow continuously in the corresponding color of the status Panda is in.

On this page we will explain the states you can get in during this work. For the explanation about the other states, we refer to the Franka Emika Panda manual.

Locked

After the Panda has booted up, the display lights are lit yellow continuously, which means that the safety locking system is activated.

After you have unlocked the joints, the display lights should be lit white in case the external activation device is in a closed state (pressed down), the robot is now in the interactive state.

Interactive

When the display lights are lit white, Panda is in the monitored stop state, which means that the Panda Arm is ready for interaction.

By simultaneous pressing of the enabling button and the guiding button at the Arm’s grip, it is possible to guide the Arm manually. In this state, the Arm can also record the positions it passed when it was guided manually by a human.

When the two buttons are released, Panda goes back to the monitored stop condition.

Activated

When the display lights are lit blue, Panda is in the activated state, which means that an automatic program can be started. In other words, the robot can now be controlled with commands send by your own program.

In what follows, we will go in detail on how to run a program on the Panda robot.

Conflict

A conflict can occur when the robot is in the activated (blue) state and you try to manually guide the arm. In that case, the display lights will be lit pink.

To solve this conflict, you have to press down the external activation device such that it is again in the interactive (white) state.

Error

When an error occurs, the display lights will be lit red. This can occur when the robot moved close to one of its hardware limits, e.g. when it moved too fast, too brisk, or close to one of the joint angle limits. This can happen when you are manually guiding the robot in interactive (white) state or because of the commands send by your program in the activated (blue) state.

To solve this kind of problem you can try the following things. We have listed the possible solutions in chronological order, so if the first solution doesn’t work, then you have to try the next solution, and so on.

• If the external activation device is in the open state (pulled up), put it in the closed state (pressed down), and back in the open state (pulled up).

  Vice versa, if the external activation device is in the closed state (pressed down), put it in the open state (pulled up), and back in the closed state (pressed down).

• Lock and unlock the joints via Franka Desk.

• Shut down Franka Control with the On-Off switch at the back of the Panda Control, wait for a minute, and restart it.

Vicon Motion Capture

Here we explain how to use the Vicon motion capture system in general, how to make subjects with Vicon Nexus, how to make objects with Vicon Tracker, and how to use the Vicon information for reference object and obstacle recognition with the Panda robot.

Reading material

In this internship report you can find a more detailed explanation of the Vicon motion capture room and Vicon Nexus. It explains how to define objects in Vicon Nexus for three different robots and for multiple scenarios.

In this internship report you can find some general info about the Vicon motion capture and also how to define objects in Vicon Nexus for some specific scenarios with the Panda robot.

In this report the use of the Panda robot in the Vicon room is explained. It explains how to establish the ethernet connection and how to receive data from the Vicon motion capture system. It also explains how to make objects in Vicon Tracker (but currently we have Vicon Nexus as software in the lab).

Possible problems

Last year there were some problems with Vicon Nexus.

Firewall

If the cameras and the Lock Sync Box don’t light up blue when you start up Nexus, then it’s highly probable that there is a firewall problem.

To solve this issue, you have to enter the Vicon desktop.

• Go to Windows –> control panel –> all control panel items –> Windows Defender Firewall –> Allowed Apps

  

  and check all Nexus applications

  

• Go to Windows –> Windows Firewall –> Inbound rules –> enable and allow connection for all Nexus applications

  

Update the Vicon Firmware

Problems can occur when the Vicon Firmware is not updated. You can do this via the Vicon Firmware Update Utility.

Vicon Datastreak SDK

Last year there was also a problem with the Vicon Datastream SDK.

When the vicon_bridge is giving occlusions when you clearly see the subject in Nexus, do the following checks.

• Navigate in Windows to

  cd C:\Program Files\Vicon\DataStream SDK\Win64\CPP
  

  (you don’t have to do this via the terminal) and double click ViconDataStreamSDK_CPPTest

• When you get the standard non-overwritten values with zeros and ones (zero position, unity matrix for orientation) in the Global and Local Translation/Rotation as in the Static Translation/Rotation,

  

  then probably the Processing Output Level is not Kinematic Fit. Change it to Kinematic Fit as shown in the figure below.

  

• Try the ViconDataStreamSDK_CPPTest again, when you now see the correct values of the subject in the Global and Local Translation/Rotation.

• Run the vicon_bridge: you shouldn’t see the occlusion warning again in the terminal, which means you can call the subject position and orientation

Extra things that can be done if it still doesn’t work:

• install the newest version of the Datastream SDK

  

• install python

  

• run ViconDatastreanSDK_CPP test and check again if the processing output level is kinematic fit

Stereolabs ZED2

Read the general information about the ZED 2 stereo camera from Stereolabs to know for which problems this camera can be a solution and what its technical specification are.

The ZED 2 camera comes with a Software Development Kit (SDK). Read the ZED SDK documentation to know with which third-party libraries and environments it can be interfaced and on which platforms you can run it.

After reading the general information about the camera and the ZED SDK, we recommend to read the introduction before continuing.

Getting started with ZED 2

First go through the quick start guide.

As you can read, the ZED SDK is available for Windows, Linux, and Nvidia Jetson platforms. However, you need an NVIDIA graphics card to run CUDA. In what follows, we will use the Nvidia Jetson Xavier NX platform to run the ZED SDK. We explain below why we have chosen this embedded platform.

• We don’t have any laptop or desktop with an NVIDIA graphics card, so an embedded platform as the Jetson is the cheapest solution.

• The controller programs on the desktop that send commands to the Panda robot must run with real-time priority under a PREEMPT_RT kernel. Since NVIDIA binary drivers are not supported on PREEMPT_RT kernels, buying an NVIDIA graphics card is not a solution.

• We tried first with an NVIDIA Jetson Nano, but this platform was not powerful enough. The obtained point cloud and depth map were not as detailed as shown in the examples on the Stereolabs website and the skeleton display was lagging a lot with respect to the real human body dysplay in the body tracking example of the SDK tutorials. Stereolabs told us that the camera resolution could be raised and the depth mode could be changed to ULTRA to improve the point cloud and depth map quality, but at the cost of a consequently lower frame rate when using the Jetson Nano. According to Stereolabs, the Jetson Nano can run the ZED SDK in real-time, but with big compromises in terms of performance. If good performances are a requirement for the project, Stereolabs suggested to switch to a Jetson Xavier NX that is on another level.

• Check also the recommended specifications for ZED SDK and the differences between the Jetson Modules.

Side note. We advise you to put back the ZED 2 camera back in the box when you are not using it. To do this, be careful to put first the built-in USB 3.0 cable in it before putting in the ZED 2 camera. Then you don’t have to pull the USB 3.0 cable to get the camera out of the box.

ZED SDK overview

It is really important to read through the SDK overview and find specific information about the camera, sensors, depth sensing, positional tracking, spatial mapping, and body tracking.

Install the ZED SDK on NVIDIA Jetson Xavier NX

Follow the Stereolabs guide to install ZED SDK on NVIDIA Jetson.

Download and install JetPack

Note

VUB researchers who want to use this for their own research, should have their own microSD card, such that they don’t mess us my installations and such that they are forced to test the install procedure and give comments on it to improve this tutorial.

To download and install JetPack, you are redirected from the download and install JetPack section to NVIDIA’s JetPack SDK. There the SD Card Image method is explained to install the JetPack on a Jetson Xavier NX Developer Kit. Follow the getting started with Jetson Xavier NX developer kit tutorial, which explains the required hardware, the steps to write an image to the microSD card (I have followed the instructions for Windows), and how to setup and boot the Jetson.

Below you can find some extra info on top of the NVIDIA tutorial Write Image to the microSD Card.

• NVIDIA proposes to use a microSD card with minmimum 16GB UHS-1. We are using a SanDisk Ultra 64GB microSD card class 10 UHS-1 U1.

  Warning

  It is very important to use an microSD card with the right specifications. We were using a premium high speed microSD card (check correct specs, add link), but we had the problem that the Jetson crashed a lot of times. Afterwards the Jetson always restarted automatically, but then everything on the microSD card was erased. According to the contributors on the NVIDIA forum, this could be because the file system over the SD was broken or because the SD card itself was broken. We could solve this problem by using the SanDisk Ultra 64GB microSD card class 10 UHS-1 U1.

• In case the microSD card you want to use for this project is not empty (and you don’t want to use it for another project), then you will have to format it first. When your microSD card has multiple partitions, then you first have to repartition the microSD card before formatting. Therefore, you can use DiskPart which is included on Windows. See Repartition-SD-Card-in-Windows for more information. After you have repartitioned the microSD card, you have to format it before usage.

• When you select the microSD card drive on which you want to write the image, you can get the message that the microSD card is locked. Usually that means that the microSD card is physically locked and that you have to physically unlock the SD card.

Below you can find some extra info on top of the NVIDIA tutorial Setup and First Boot.

• Since the Jetson Xavier NX reference carrier board includes 802.11 plug-in WLAN & BT module preinstalled with antenna, it can automatically connect to a Wireless Network, which is not the case for the Jetson Nano. For the Jetson Nano a WiFi USB adapter is required to connect to WiFi. If that doesn’t work, check this problem and its solution.

• The name, computer’s name, username, and password we have used to log in at VUB are:

  • name: JetsonXavier1 / JetsonXavier2

  • computer’s name: xavier1-desktop / xavier2-desktop

  • username: xavier1 / xavier2

  • password: JetsonXavier

• When you are asked to choose the APP Partition Size, it is recommended to choose the maximum accepted size.

• When you select the Nvpmodel Mode, keep at the beginning the default settings, which is MODE_10W_DESKTOP - (Default). We refer to the NVIDIA Jetson Linux Developer Guide for further information.

• When you log in onto the Jetson Xavier NX, be careful that you sign in on the Ubuntu version.

  

Note

Register to the NVIDIA forum to ask questions in case you have problems with the NVIDIA Jetson Xavier NX.

Download and install ZED SDK

For this part, we followed the instructions from the download and install the ZED SDK section on the Stereolabs website.

We have downloaded the ZED SDK for Jetpack 4.6 and installed it in silent mode. Silent mode enables you to perform an installation in a non-interactive mode, so in this mode you don’t have to answer questions on dependenies, tools, and samples installation in the terminal during the installation procedure.

Check also which python3 version and pip3 version is installed on the Jetson. We have executed the described experiments with python 3.6.9 and pip 9.0.1.

python3 --version
pip3 --version

ZED SDK tools

After downloading and installing the ZED SDK, the next step in the quick start guide is to run the ZED Explorer, which is a good way to check that the ZED SDK is properly installed.

• Connect the ZED 2 camera to a USB port of the Jetson Xavier NX.

• Go to the terminal on the Jetson Xavier and navigate to the ZED tools folder where you can run the ZED Explorer:

  cd /usr/local/zed/tools/
  ./ZED_Explorer
  

• If the ZED SDK is properly installed and the ZED 2 camera is recognized by the Jetson, you should see a 3D video from the left and right camera.

  

In the ZED tools folder you can also find ZED_Depth_Viewer, ZED_Diagnostic, ZEDfu, ZED_Sensor_viewer, and ZED_SVO_Editor that you can run in a similar way as the ZED Explorer.

Warning

Although it is possible to recalibrate the camera manually using the ZED Calibration tool, i.e. ZED_Calibration, it is totally not recommended for the ZED 2 cameras as is mentioned here!

ZED SDK tutorials and samples

To play around with the ZED tutorials and samples, you first have to download the example code in the directory of your choice.

git clone https://github.com/stereolabs/zed-examples.git

There are C++ and python examples, we will use the C++ examples. Before you can run a C++ example, you will have to read how to build a C++ application on Jetson which is explained for the first tutorial that publishes the serial number of the ZED 2 camera.

Try out all the tutorials to familiarise with the ZED SDK. All the tutorials give a text output printed in the terminal.

Try also all the samples. Building the C++ samples is done similarly as for the tutorials. Below you can find some results.

Note

If you get the Gtk-Message: Failed to load module “canberra-gtk-module”, then you have to install this module by running the following in the terminal.

sudo apt install libcanberra-gtk-module libcanberra-gtk3-module

Camera control

To change the camera settings, click on the camera control window such that your cursor is not in the terminal. Push s and you will see in the terminal which setting you can change. Push s again if you want to adapt another camera setting. Whereas my numlock was turned off, I couldn’t use the + and - keys that are on the right side of my keyboard, but I could use the + and - keys of my main keyboard.

Depth sensing

This sample shows how to retreive the current point cloud.

Body tracking

This sample display a skeleton over the live image and shows hot to detect and track 3D human bodies in space.

Object detection (birds eye viewer)

In the object detection (birds eye viewer) sample, detected objects are displayed in a 3D view with the current point cloud. You have to press c to clear filters such that the program can detect objects from all the available object classes. Without clearing the filters, the program can only detect people and vehicles.

Avoid crashing and latency issues

You can try the following things in case you have crashing and latency issues with the NVIDIA Jetson Xavier NX. If the following things don’t solve your issues, then it is best to contact Stereolabs or Nvidia.

Change power mode

You can switch the power mode of the NVIDIA Jetson to MODE 15W 6CORE, as depicted in the figure below.

Turn on the fan manually

In order to reduce lags on the Jetson turn on the fan to 100%, by doing the following steps.

• Install jetson-stats:

  git clone https://github.com/rbonghi/jetson_stats
  sudo apt-get update
  sudo apt-get install python3-pip  # install pip3 if you don't have this yet installed
  cd jetson_stats
  sudo -H pip3 install -U jetson-stats
  

• Restart your Jetson and run :

  jtop  # start jtop
  

• The following window will open with all the information of the Jetson

  

• Navigate with the arrow keys to tab 5CTRL. You can set the fan method to manual by clicking with your mouse on manual. In the manual fan mode, you can increase the fan speed by pressing the p key and decrease the fan speed by pressing the m key.

  

If you don’t have any crashing or latency issues, you can put the fan mode on default as we did to execute our programs.