Image Collection for the Mapper Robot II

A project of the Robotics 2020 class of the School of Information Science and Technology (SIST) of ShanghaiTech University. Course Instructor: Prof. Sören Schwertfeger.

Wenqing Jiang, Ziqi Gao

Abstract

In order to avoid the limitation of the ROS transmission bandwidths on the second version of the MARS Mapper Robot dataset collection, this project is to develop a new image compression method which can produce fast and highquality image compression and saving. Combinations of different method in raw image acquisition, compression and saving have been tested and the best combination is using Spinnaker’s API for image acquisition and TurboJPEG for image compression and saving. Different values of JPEG quality are used for testing the best value with respect to quality-speed trade-off, and the best one can achieve a frame rate of 60 Hz with two cameras on a NUC with high image quality.

Introduction

Localization and mapping are essential robotic tasks and are often solved together in a Simultaneous Localization and Mapping (SLAM) system, which highly depends on data feeding from sensors, including Lidars and cameras. Consequently, an advanced mapping system with high-resolution sensors is needed to collect data for benchmarking datasets - the Mapper Robot in MARS lab. By upgrading it with higher frame rate and higher-quality image production, we can improve the quality of the dataset significantly. In this project, we use a Intel® NUC mini computers and FLIR cameras to collect images and test the performance of images collection. We use a variety of image acquisition, compression, and saving algorithms including Spinnaker’s API, libjpeg-turbo, and ROS’s original methods. We evaluate different methods with different number of cameras based on average frame size, Frame Per Second (FPS), CPU usage, frame dropped, Signals To Noise (SNR).

System Description

Hardware

  • NUC 

NUC (Next Unit of Computing) is a small-size barebone series computer designed by Intel. We use an Intel® NUC Kit NUC8i7BEH in this project for image processing.

  • FLIR camera

Grasshopper3 USB3 (GS3-U3-51S5C-C: 5.0 MP, 75 FPS, Sony IMX250, Color) provides highperformance, high-quality imaging. We use 2 cameras for image acquisition.

Software

  • ROS

ROS (The Robot Operating System) is a set of software libraries and tools that help us build robot application and our software is based on that.

  • Spinnaker

The Spinnaker application programming interface (API) is used to interface with FLIR’s USB3 Vision and GigE Vision cameras. We use that for image acquisition.

  • TurboJPEG

TurboJPEG API provides an easy-to-use interface for compressing and decompressing JPEG images in memory. We use that for image compression and saving.

Methods & Evaluation

Methods

We test combinations of different image acquisition and compression and saving techniques and choose the best one to test on different number of cameras and different JPEG quality.

  • Raw Image Acquistion
    • subscribing ROS topic /cam num/image raw
    • Spinnaker’s API 
       
  • JPEG Compression 
    • ROS’s image compression based on OpenCV 
    • Spinnaker’s API 
    • TurboJPEG
       
  • Compressed Image Saving
    • rosbag record /cam_num/image raw/compressed (ROS’s compression)
    • Spinnaker’s API 
    • TurboJPEG

Evaluation

The performance of image collection is based on average frame size, average FPS, CPU usage, frame dropped rate, SNR in flat, and the algorithm is tested using different number of cameras and different JPEG quality.

Result

Best Combination Found

By keeping other variables regarding to images (JPEG quality=80, image\_form=GREY) and camera's parameters the same, we found out that the best algorithm is the combination of using Spinnaker's API for raw image acquisition and using TurboJPEG for image compression and saving. Three typical combinations are shown in Table 1. Compared with orginal ROS implementation and using ROS for image acquisition and Spinnaker for compression and saving (and all of the other combination), the best method has the highest FPS and the lowest frame dropped (zero), and we use that combination for further testing.

Performance with Different JPEG Quality

Table 2 and Figure 7 show the result of the evaluation metrics by testing with one camera on the NUC. We calculated FPS by recording the time for receiving, processing and saving 2000 images. CPU usage here is the sum of two processes: one is SavePics, the other is Nodelet. The CPU usage of Nodelet is almost the same (~32\%) all the time, while the SavePics process uses more CPU's computation resources with the improvement of quality. 

By shooting at the same location and calculating the average among all collected images (shown in Figure 8), Signals to Noise (SNR) does not change dramatically with image quality, which has two possible reasons. One is that the camera we use has not been calibrated, so the image is quite blurred. The other reason is that different JPEG quality does not affect the quality of images badly, convincing us that a relative small JPEG quality (e.g. ~90) is a good choice since the FPS is decreased rapidly for a slight increase in quality after 90. 

Table 3 and Figure 9 demonstrate the performance of 2 cameras on an NUC, which reaches our expectation of a FPS ~60 image collection for the Mapper Robot II at JPEG quality 90.

 

Extra Information Saving

For each image collection, we save a corresponding text file which contains its frame_id, sequence number, time stamp, resolution and each image’s path for the convenience of further dataset collection.

Video