SK-TDA4VM: Visual Localization with Arducam Stereo Setup

Hello Aditya,

1. These 3 files are the outputs of mapping. Please note that 3D Lidar data and image are used for mapping and Calra simulator was used for the demo.

2. Not sure if I understand this question correctly. DKAZE network outputs 2D key points along with descriptors. The 3D position of each key point is from the corresponding 3D Lidar point. Camera and 3D Lidar should be calibrated. 

3. I do not think DKAZE trained using Carla simulator works for real warehouse. The DKAZE network and the demo was mainly developed for the proof of concept. Since we haven't tested it in real world, we do not recommend to use it for real-world use cases.

4. I'd liked to recommend to use public network, e.g., SuperPoint. The trained network should be compiled for C7x/MMA. Please refer to GitHub - TexasInstruments/edgeai-tidl-tools: Edgeai TIDL Tools and Examples - This repository contains Tools and example developed for Deep learning runtime (DLRT) offering provided by TI’s edge AI solutions.

Best regards,

Do-Kyoung

Hi!

Thanks for the response. Could you please clarify few more questions

1. Could you please also confirm if it is a 3D descriptor (from point cloud) to 2D descriptor matching (from DKAZE on images) or in other words,  if the match between the 3D keypoint descriptors from point cloud of corresponding 3D keypoints of 3D sparse map and the 2D keypoint descriptors of the image from the DKAZE network is used for localization? 

2. Could you please clarify few things related to 3d sparse map files especially 3D keypoints binary file, 3D descriptor binary file? From my understanding, it seems to be that point cloud from LIDAR was obtained and a separate network was used for obtaining 3D keypoints and their corresponding 3D descriptors of 64 dimension from the LIDAR data. Could you please let me know if it was a deep learning network (if yes, which model) or a classical CV keypoint and descriptor algorithm that was used for this 3D keypoint and descriptor for 3D sparse map files? 

Hi Aditya,

Let me summarize the mapping and localization process in this demon. Please note that two processes are quite different in terms of sensors and algorithms used.

1. Mapping

• Sensors: IMU (for ego vehicle's pose estimation), 3D LIDAR, camera. 
• These sensors should be calibrated. After calibration, we are able to find the corresponding 3D point from LIDAR for the feature point in image. (It is not always true though since 3D LIDAR is sparse. There could be feature points in image, for which we cannot find exactly matching 3D points. Such points won't be added to the map or interpolation technique can be used)
• DKAZE network will provide feature points and their descriptors 
  
  • Other DL networks or CV based methods can be used instead of DKAZE
• So the map will have a bunch of points in it. Each point has the following information: X, Y, Z (3D coordinate) and 64-D descriptor  

2. Localization

• Sensors: IMU, camera
  • 3D LIDAR is not used 
  • In localization, IMU is optional. But it should help improve localization accuracy.
• DKAZE network will provide feature points and their descriptor. Each point has x, y (image coordinate) and 64-D descriptor
  • The same feature detection method should be used in mapping and localization. Otherwise, for each point we cannot find the corresponding points from the map.
• Localization
  • For the detected feature points in the image, we find the corresponding points in the map by comparing 64-D descriptors 
  • Using the matched feature points, localize the camera (i.e. ego vehicle)
    • We have multiple pairs of (x, y) - (X, Y, Z)
    • We can localize by solving PnP (Perspective-n-Point) 

Best regards,

Do-Kyoung