Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

multistick neural network example

idata
Employee
905 Views

@Tome_at_Intel : I looked at the multistick example that runs two different networks on two different sticks. For my research, I'd like to partition one neural network across two neural sticks. To this end, I'd like to have the same network on two devices and gate the appropriate layers to achieve the partition effect. However, I'm not sure how to get started with this idea. Can you provide some pointers on this?

0 Kudos
7 Replies
idata
Employee
631 Views

@Tome_at_Intel : Did you get a chance to look at my message? Is my explanation about my problem clear?

0 Kudos
idata
Employee
631 Views

@kodukulav You want to have each NCS device process different layers of a network? I assume by passing data from each layer/NCS device to the next layer/NCS device. Just wondering what you are trying to do. This may turn out slower than processing the entire network on one NCS device.

0 Kudos
idata
Employee
631 Views

@Tome_at_Intel : For my research project, I'd like to partition a neural network into two halves and place each of them on one of the neural compute sticks. This idea is similar to a recent academic work called NeuroSurgeon, link provided below, from ASPLOS'17, where they partition the neural network between the client and server for better energy-efficiency and latency. Also, similar to Neurosurgeon, I'd like to adapt the partition dynamically based on some input parameters.

 

https://dl.acm.org/citation.cfm?id=3037698

 

From reading your message, I got a feeling that this idea can be implemented on the NCS. A couple of questions in this regard.

 

(1) It looks like the partition is possible with the NCS and SDK available. I'm wondering how the flow works. Do we need to construct and compile, using mvncompiler, two different partitions separately and place them on the NCS devices?

 

(2) If I were to change the split dynamically, do I need to rebuild and recompile?

 

(3) Yes, the performance of the split network is going to be slower than the original one because the data needs to cross the slow interfaces. However, I need to do this to balance some trade-offs for my project.

0 Kudos
idata
Employee
631 Views

@Tome_at_Intel : Are my explanations clear?

0 Kudos
idata
Employee
631 Views

@kodukulav Yes I believe I understand. You want to split one network into two and run them on separate devices. Just off the top of my head, you would have to rename the input layer of the second half to data. You should be able to compile two graph files for each half of the network. Theoretically this may be possible, but I don't think anyone has tried it yet.

0 Kudos
idata
Employee
631 Views

I am using SSD mobilenet (caffe) and I get 8 FPS…. but i still need more speed ….. can I use two NCSs to gain better speed with this same network ? I use raspberry pi camera module for live video…

0 Kudos
idata
Employee
631 Views

@amare You can use two NCS to boost performance of your application, however this has to be done on the application side because there is no information sharing/communication between NCS devices. How this works is multiple NCS devices can be used to process inference on more camera frames at a time. Think of each NCS device as an "inference machine" that you can send some work to be done and after the work is completed, you receive an output result. With two NCS, you can send one camera frame to the first NCS device and the next camera frame to the second NCS device. This effectively lets you process more camera frames at a time.

0 Kudos
Reply