Items with no label
3335 Discussions

Faster D435 distance reading

RTin0
Beginner
1,738 Views

Hi,

 

I was wondering if there was any faster algorithm to get distance after providing an x and y coordinate. Right now getting the distance is way too slow and very inaccurate if my analogue is moving. The readings I'm getting are always showing either 0 or a distance that is too far. What I'm doing is detecting the contours from a background subtracted image and feeding the centroids of the contour into the get distance formula on python. Any help in this is appreciated.

 

I'm using a D435 and have set the depth stream (along with the colour stream) at 848 x 480.

# After getting the centroids for items in matches: x, y, w, h = items[0] mid_x, mid_y = items[1] depth_z = aligned_depth_frame.get_distance(mid_x, mid_y)

 

I'm trying to detect the blue do in the midddle, ignore the shadow at the back depthmap.png

0 Kudos
1 Solution
MartyG
Honored Contributor III
770 Views

I had an idea for making the insect more detectable on the image. A trait of the D415 camera model is that when fast-moving objects are observed, motion artifacts may appear on the image. This is because the D415 has a slower 'Rolling' type shutter than the D435's fast 'Global' shutter, which is suited to fast motion tracking.

 

My thought then is that if you slow the shutter speed on your camera then the insect may leave an impression on the image as an artifact that is easier for the camera to analyze than a tiny insect. Although RealSense cameras cannot have their physical shutter speed adjusted, you can do it in an indirect way by altering the speed of the color exposure value. This is because shutter speed is also known as 'exposure time'.

View solution in original post

0 Kudos
6 Replies
MartyG
Honored Contributor III
770 Views

It sounds as though there are a lot of calculations going on - the background subtraction, and then working out the centroids.

 

I wonder if an alternative approach that might work for you is to use Deep Neural Network (DNN) object detection and MobileNet SSD. The link below goes to a page with a list of 20 DNN programs for Python.

 

https://github.com/topics/mobilenet-ssd?l=python

 

I have seen good reports about a program on this list called MobileNet-SSD-RealSense,

 

https://github.com/PINTO0309/MobileNet-SSD-RealSense

 

The animated images on its program page show that it can not only detect an object but also provide a real-time depth value for that detected object.

 

Also, a table on the page that compares MobileNet-SSD to other forms of object recognotion rates the program's speed performance as "Medium" in a scoring range of low, medium and high.. The images on the page are a very good visual indicator of real-world performance.

0 Kudos
RTin0
Beginner
770 Views

Thanks for the suggestion, but the eventual detection is used for insect in flight which is hard to train a DNN on due to the different forms and shapes insects have in flight. While there are a lot of calculations, the program has been running I am able to draw a centroid on the object consistently, the only thing that is not consistent was the depth measurement because of how fast the object might be moving. I would still love any other options to make it faster, I've been looking at cython and possibly implementing a kalman filter.

0 Kudos
MartyG
Honored Contributor III
770 Views

Are you recording the data whilst it is being observed, please? Recording can cause a bottleneck in the performance speed if the hard disk drive cannot keep up (a solid-state SSD drive is preferable), or the computing hardware's processor is of a type that has problems with the compression algorithm in a recorder (in these circumstances, disabling compression can improve performance).

0 Kudos
RTin0
Beginner
770 Views

I've attached my code below for easier referencing, but no there isn't any disk I/O everything is running through the RAM. It seems like I would have to implement it in cython

 

import pyrealsense2 as rs import numpy as np import cv2 import matplotlib.pyplot as plt   class KalmanFilter:   kf = cv2.KalmanFilter(4, 2) kf.measurementMatrix = np.array([[1, 0, 0, 0], [0, 1, 0, 0]], np.float32) kf.transitionMatrix = np.array([[1, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 0], [0, 0, 0, 1]], np.float32)   def Estimate(self, coordX, coordY): ''' This function estimates the position of the object''' measured = np.array([[np.float32(coordX)], [np.float32(coordY)]]) self.kf.correct(measured) predicted = self.kf.predict() return predicted   def init_intel_pipeline(): pipeline = rs.pipeline()   # configure stream, get the depth stream and colour stream # colour stream needed for background subtraction config = rs.config() config.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 90) config.enable_stream(rs.stream.color, 848, 480, rs.format.bgr8, 60)   # start streaming profile = pipeline.start(config)   # Getting the depth sensor's depth scale (see rs-align example for explanation) depth_sensor = profile.get_device().first_depth_sensor()   # Create an align object # rs.align allows us to perform alignment of depth frames to others frames # The "align_to" is the stream type to which we plan to align depth frames. align_to = rs.stream.color align = rs.align(align_to) return align, pipeline     #Dilation to remove noise and objects that are too far away that appears inconsequential def morph_mask(fg_mask):   # Create ellipse kernel for morph ops elipse_kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2,2))   # Fill holes close = cv2.morphologyEx(fg_mask, cv2.MORPH_CLOSE, elipse_kernel)   # Remove noise opening = cv2.morphologyEx(close, cv2.MORPH_OPEN, elipse_kernel)   # Dilate to merge and increase detected in mask dilation = cv2.dilate(opening, elipse_kernel, iterations=2)   return opening   def get_midpoint(x,y,w,h): mid_x = x+ (w/2) mid_y = y+ (h/2) return (mid_x, mid_y)   def detect(fg_mask, min_contour_width = 10, min_contour_height = 10): matches = []   # finding contours im, contours, hierarchy = cv2.findContours(fg_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_TC89_L1) contours = filter(lambda cont: cv2.contourArea(cont) > 10 , contours)   if len(contours) > 5: pass else: for c in contours: (x, y, w, h) = cv2.boundingRect(c) contour_area = w * h if contour_area > 2000: continue   valid_contours = (w >= min_contour_width) and ( h >= min_contour_height )   if not valid_contours: continue midpoint = get_midpoint(x,y,w,h)   matches.append( ( (x,y,w,h), midpoint ) ) return matches   def output(current_points, current_z, frame): cv2.putText(frame,current_points,(10,20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2) cv2.putText(frame, current_z, (10,40), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)   def main(): # Create bgs with 500 frames in cache and no shadow detection init_bgs = False frameNo = 0 fg_mask = None kf = KalmanFilter() predictedXY = np.zeros((2,1), np.float32) mog2 = cv2.createBackgroundSubtractorMOG2(history = 500, detectShadows = False)   align, pipeline = init_intel_pipeline()   xy, output_z = (" ",) * 2 try: while True: matches = [] frames = pipeline.wait_for_frames()   # Align the depth frame to color frame aligned_frames = align.process(frames)   # Get aligned frames aligned_depth_frame = aligned_frames.get_depth_frame().as_depth_frame() # aligned_depth_frame is a 848*480 depth image color_frame = aligned_frames.get_color_frame() if not aligned_depth_frame or not color_frame: continue depth_image = np.asanyarray(aligned_depth_frame.get_data()) color_image = np.asanyarray(color_frame.get_data()) color_copy = color_image.copy() if frameNo != 500: # Init BGS for the first 500 frames to have a clear template of the background mog2.apply(color_image, None, 0.001) frameNo = frameNo + 1 else: #set a higher learning rate in deployment fg_mask = mog2.apply(color_image, None, 0.005) dilated_mask = morph_mask(fg_mask) matches = detect(dilated_mask) for items in matches: x, y, w, h = items[0] mid_x, mid_y = items[1] xy = "X: %d, Y:%d" % (mid_x, mid_y)   depth_z = aligned_depth_frame.get_distance(mid_x, mid_y) if depth_z != 0 and depth_z < 2.0: output_z = "Depth: %f" % depth_z   future_x_y = kf.Estimate(mid_x, mid_y) cv2.putText(color_copy,'Object detected',(x+w+10,y+h),0,0.3,(0,255,0)) cv2.circle(color_copy, (mid_x, mid_y), 3, (0,255,0), -2) cv2.circle(color_copy, (future_x_y[0], future_x_y[1]), 3, (0,0,255), -2) output(xy, output_z, color_copy)   # Show everything depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET) cv2.imshow("color ", color_copy) cv2.imshow("depth", depth_colormap) if fg_mask is not None: cv2.imshow("mask", dilated_mask) key = cv2.waitKey(1) if key & 0xFF == ord('q') or key == 27: cv2.destroyAllWindows() break finally: pipeline.stop()   if __name__ == "__main__": main()            

 

0 Kudos
MartyG
Honored Contributor III
770 Views

Programming is not one of my specialist areas unfortunately so I do not have suggestions for how your code could be enhanced, though readers of this forum who understand Python may be able to offer useful input.

 

Researching the processing speeds that are achievable through Cython, it does seem to be a reasonable choice for increasing your program's performance and using memory more efficiently. The article below studies benchmarks of Cython's speed against other forms of Python processing such as NumPy.

 

http://arogozhnikov.github.io/2015/01/06/benchmarks-of-speed-numpy-vs-all.html

0 Kudos
MartyG
Honored Contributor III
771 Views

I had an idea for making the insect more detectable on the image. A trait of the D415 camera model is that when fast-moving objects are observed, motion artifacts may appear on the image. This is because the D415 has a slower 'Rolling' type shutter than the D435's fast 'Global' shutter, which is suited to fast motion tracking.

 

My thought then is that if you slow the shutter speed on your camera then the insect may leave an impression on the image as an artifact that is easier for the camera to analyze than a tiny insect. Although RealSense cameras cannot have their physical shutter speed adjusted, you can do it in an indirect way by altering the speed of the color exposure value. This is because shutter speed is also known as 'exposure time'.

0 Kudos
Reply