top of page
Purpose

This manipulator is designed to sit on a corner of a desk and help people perform trivial tasks within the range of the robot arm. It obtains environmental information through the camera, and then controls the movement of the robot arm to complete the corresponding task according to the command of the human. These tasks include cleaning desktops, cleaning up trash and passing things around.

Structure and Hardware

  • Depth Camera For visual sensors, we chose the depth camera Realsense D435i

  • Platform For the platform and controller, we choose Jetson Nano, Arduino and TIC500.

3D modeling and URDF

We already have the integral 3D modeling of the robot manipulator in SolidWorks, after doing some modifications and defining some joints, we exported the modeling Assembly to URDF from the SolidWorks to URDF Exporter.

  • An element within each element must be properly specified and configured.

  • Add an element for every joint and link

  • Set proper damping dynamics

  • Add actuator control plugins

  • Add an element for the element

  • Add a link if the robot should be rigidly attached to the world/base_link

  • Verifying the Gazebo Model Works

  • Viewing the URDF In Gazebo

We have done the path planning and able to subscribe the Joint_states topic by the Arduino node. Also, the base control part in arduino is finished. However, for some reason, the arduino can’t execute the message in the joint_states topic which it has subscribed. One reason for this is that the joint_state is the sensor massage in the Rviz simulation environment, which is using a fake controller.

Object Detection

To use depth camera and implement computer vision, we need to update and install some dependences:

 

  • Update Cmake Version  

  • Install QT5.9.9, VTK7.1.1 and PCL 1.8.1

  • Install OpenCV

 

Some resources for this step:

 

  • Cmake:   Download Tutorial 

  • QT5.9.9, VTK7.1.1 and PCL 1.8.1 : QT Install    VTK and PCL Install    PCL Tutorial  

  • OpenCV 3 :  Tutorial 

 

The next step is to install depth camera function package. We use Realsense 435i in this project, so we need to install Realsense SDK. The Install guidance is as following:

 https://github.com/IntelRealSense/realsense-ros

 

Note: Many ROS packages depends on Realsense SDK, so we  need to use Method 2 to install.

Using the command:

 

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud

 

Subscribing 3 topics (

  • /camera/color/image_raw

  • /camera/depth/image_rect_raw

  • /camera/depth/pointclound.              

         ) in Rviz, we can get 3 outcomes (RGB Image, Depth Image and Point cloud Image). In this program, we will subscribe RGB Image for object detection and subscribe Depth Image for publishing object’s position information, because in depth image, each pixel bring the depth information of that point.

Model: Yolo4 with DarkNet

Problems

  • Depth Camera could not be assembled to the manipulator

  • The parts of the manipulator were 3D printed by ourselves, mechanical mistakes caused the failure of the path planning in practice

  • Other sim to real problems need to be solved 

bottom of page