Tensorflow Finetuned models GPU device error

[idata]

Hi There,

I am trying out custom tensorflow models with NCS. Started with finetuning Inception V3 model on a new dataset using the slim models as described in: https://github.com/tensorflow/models/blob/master/research/slim/README.md. Getting error for GPU device on running NC Compile on the generated models:

mvNCCompile -s 12 model.ckpt-1000.meta -in input -on=InceptionV3/Predictions/Reshape_1

mvNCCompile v02.00, Copyright @ Movidius Ltd 2016

/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead

if d.decorator_argspec is not None), _inspect.getargspec(target))Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1297, in _run_fn self._extend_graph() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1358, in _extend_graph self._session, graph_def.SerializeToString(), status) File "/usr/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status))tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device. [[Node: gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/softmax_cross_entropy_loss/Mul_grad/Shape, gradients/softmax_cross_entropy_loss/Mul_grad/Shape_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/bin/mvNCCompile", line 118, in create_graph(args.network, args.inputnode, args.outputnode, args.outfile, args.nshaves, args.inputsize, args.weights) File "/usr/local/bin/mvNCCompile", line 104, in create_graph net = parse_tensor(args, myriad_config) File "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py", line 192, in parse_tensor saver.restore(sess, path[:path.rfind('.')]) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1560, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message)tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device. [[Node: gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/softmax_cross_entropy_loss/Mul_grad/Shape, gradients/softmax_cross_entropy_loss/Mul_grad/Shape_1)]]

Caused by op 'gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs', defined at:

File "/usr/local/bin/mvNCCompile", line 118, in create_graph(args.network, args.inputnode, args.outputnode, args.outfile, args.nshaves, args.inputsize, args.weights) File "/usr/local/bin/mvNCCompile", line 104, in create_graph net = parse_tensor(args, myriad_config) File "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py", line 190, in parse_tensor saver = tf.train.import_meta_graph(path) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1698, in import_meta_graph **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/meta_graph.py", line 656, in import_scoped_meta_graph producer_op_list=producer_op_list) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 313, in import_graph_def op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.

[[Node: gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/softmax_cross_entropy_loss/Mul_grad/Shape, gradients/softmax_cross_entropy_loss/Mul_grad/Shape_1)]]

Please describe specs models that are compatible with NCS. Do we have to compile model on a CPU to be able to run on NCS?

Thanks!

[idata]

Hi There,

I am using the Tensorflow Slim to finetune Inception model on new dataset for running on NCS. On running compile, getting error for GPU device (see below). Is there any special requirements for NCS on how to train models. Can we use GPU for model training?

mvNCCompile -s 12 flowers/model.ckpt-1000.meta -in input -on=InceptionV3/Predictions/Reshape_1

mvNCCompile v02.00, Copyright @ Movidius Ltd 2016

/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead

if d.decorator_argspec is not None), _inspect.getargspec(target))Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1297, in _run_fn self._extend_graph() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1358, in _extend_graph self._session, graph_def.SerializeToString(), status) File "/usr/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status))tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device. [[Node: gradients/softmax_cross_entropy_loss/Mul_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/softmax_cross_entropy_loss/Mul_grad/Shape, gradients/softmax_cross_entropy_loss/Mul_grad/Shape_1)]]

[idata]

@gchauhan For training, the training method probably isn't an issue whether it is CPU or GPU based. However, we have not tested the SDK with the GPU flavor of Tensorflow and I think this could be the problem here. Please try using the the CPU flavor of Tensorflow and see if this resolves the issue. Thanks!