Stick timeout, only full power cycle recovers

[idata]

Hi, we are on our way to deploying a four to five figure number of Movidius NCS. We run two sticks for our product, connected via usb hub, either with Raspberry Pi or intel NUC (Ubuntu 16.04) base systems. We have encountered the following error with a number of sticks (at least five different sticks), seemingly at random. The sticks only recover after a full power cycle and work fine afterwards. Environment temperature is usually around 23° with AC. Here is the error report: Scenarios: 1) 2 Movidius Sticks on powered hub (4A, more than enough) on Intel Nuc (Ubuntu 16.04) 2) 2 Movidius Sticks directly plugged into Intel Nuc 3) 2 Movidius Sticks on powered hub on Raspberry Pi 3 Model B, Raspian Stretch 4) 2 Movidius Sticks on powered hub on Raspberry Pi 3+ Model B, Raspian Stretch We tried USB 3.0 Hubs from Raidsonic, Amazon Basics and other providers. We also tried USB 2.0 Hubs. Other conntected devices: USB Camera (One or two), sometimes LTE Modems. Task: Running two models on in background threads, one of them is on quite constant load (inference time around 80ms), one only working occasionally (inference time around 300ms). Problem: After some varying time, one of the two movidius sticks throws the Timeout Error (Exception: mvncStatus.TIMEOUT). Closing or opening it then does not work (as seen in the logs below). Then not even a reboot solves the problem, just a complete power cycle. Intel Nuc Syslog: Sep 6 18:46:50 nuc1 systemd[1]: Started cartwatch entrance. Sep 6 18:46:52 nuc1 start_entrance[13445]: Stream thread started. Sep 6 18:47:07 nuc1 start_entrance[13445]: Traceback (most recent call last): Sep 6 18:47:07 nuc1 start_entrance[13445]: File "demos/detect_movi.py", line 116, in <module> Sep 6 18:47:07 nuc1 start_entrance[13445]: sys.exit(main(parse_args())) Sep 6 18:47:07 nuc1 start_entrance[13445]: File "demos/detect_movi.py", line 17, in main Sep 6 18:47:07 nuc1 start_entrance[13445]: checkout = Checkout(FLAGS) Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/checkout.py", line 96, in __init__ Sep 6 18:47:07 nuc1 start_entrance[13445]: self.detector_process = Detector(self.frames_queue, self.detections_queue, FLAGS.det_graph) Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 46, in __init__ Sep 6 18:47:07 nuc1 start_entrance[13445]: self.device.OpenDevice() Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 147, in OpenDevice Sep 6 18:47:07 nuc1 start_entrance[13445]: raise Exception(Status(status)) Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception: mvncStatus.ERROR Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception ignored in: <bound method MovidiusDetector.__del__ of <MovidiusDetector(Detector Thread 1, initial)>> Sep 6 18:47:07 nuc1 start_entrance[13445]: Traceback (most recent call last): Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 144, in __del__ Sep 6 18:47:07 nuc1 start_entrance[13445]: self.device.CloseDevice() Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 153, in CloseDevice Sep 6 18:47:07 nuc1 start_entrance[13445]: raise Exception(Status(status)) Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception: mvncStatus.INVALID_PARAMETERS Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Main process exited, code=exited, status=1/FAILURE Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Unit entered failed state. Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Failed with result 'exit-code'. Sep 6 18:47:18 nuc1 systemd[1]: entrance.service: Service hold-off time over, scheduling restart. Sep 6 18:47:18 nuc1 systemd[1]: Stopped cartwatch entrance. Sep 6 18:47:18 nuc1 systemd[1]: Started cartwatch entrance. Sep 6 18:47:20 nuc1 start_entrance[13671]: Stream thread started. Sep 6 18:47:35 nuc1 start_entrance[13671]: Traceback (most recent call last): Sep 6 18:47:35 nuc1 start_entrance[13671]: File "demos/detect_movi.py", line 116, in <module> Sep 6 18:47:35 nuc1 start_entrance[13671]: sys.exit(main(parse_args())) Sep 6 18:47:35 nuc1 start_entrance[13671]: File "demos/detect_movi.py", line 17, in main Sep 6 18:47:35 nuc1 start_entrance[13671]: checkout = Checkout(FLAGS) Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/checkout.py", line 96, in __init__ Sep 6 18:47:35 nuc1 start_entrance[13671]: self.detector_process = Detector(self.frames_queue, self.detections_queue, FLAGS.det_graph) Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 46, in __init__ Sep 6 18:47:35 nuc1 start_entrance[13671]: self.device.OpenDevice() Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 147, in OpenDevice Sep 6 18:47:35 nuc1 start_entrance[13671]: raise Exception(Status(status)) Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception: mvncStatus.ERROR Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception ignored in: <bound method MovidiusDetector.__del__ of <MovidiusDetector(Detector Thread 1, initial)>> Sep 6 18:47:35 nuc1 start_entrance[13671]: Traceback (most recent call last): Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 144, in __del__ Sep 6 18:47:35 nuc1 start_entrance[13671]: self.device.CloseDevice() Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 153, in CloseDevice Sep 6 18:47:35 nuc1 start_entrance[13671]: raise Exception(Status(status)) Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception: mvncStatus.INVALID_PARAMETERS Sep 6 18:47:36 nuc1 systemd[1]: entrance.service: Main process exited, code=exited, status=1/FAILURE Sep 6 18:47:36 nuc1 systemd[1]: entrance.service: Unit entered failed state. And so fourth.. Rebooting.. then running the command lsusb shows: Bus 002 Device 004: ID 05e3:0612 Genesys Logic, Inc. Bus 002 Device 002: ID 05e3:0612 Genesys Logic, Inc. Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 004: ID 8087:0a2b Intel Corp. Bus 001 Device 007: ID 05a3:9422 ARC International Bus 001 Device 009: ID 12d1:1506 Huawei Technologies Co., Ltd. Modem/Networkcard Bus 001 Device 008: ID 05a3:9230 ARC International Bus 001 Device 006: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 001 Device 005: ID 05a3:9230 ARC International Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 001 Device 018: ID 03e7:2150 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub -> Showing only one movidius stick. Doing a full power cycle then restores to the initial situation.

[idata]

@cs_cw Looking into this issue.

[idata]

Hi @cs_cw and @Tome_at_Intel

I'm experiencing a similar issue! Did you manage to identify the problem or find a workaround?