[Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm

[idata]

When I try to run any of the examples in the ncappzoo for TensorFlow I get the following error:

[Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm

As one example, I get the following output when I run make all in ncappzoo/tensorflow/inception_v1:

test -f output/inception-v1.meta || ((wget http://download.tensorflow.org/models/inception_v1_2016_08_28.tar.gz && tar zxf inception_v1_2016_08_28.tar.gz && rm inception_v1_2016_08_28.tar.gz) && ./inception-v1.py) mvNCProfile -s 12 output/inception-v1.meta -in=input -on=InceptionV1/Logits/Predictions/Reshape_1 mvNCProfile v02.00, Copyright @ Movidius Ltd 2016 /home/tsteiner/fla-gtsc/fla_root/usr/python/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py:766: DeprecationWarning: builtin type EagerTensor has no __module__ attribute EagerTensor = c_api.TFE_Py_InitEagerTensor(_EagerTensorBase) /home/tsteiner/fla-gtsc/fla_root/usr/python/env3/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) [Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm mvNCCheck -s 12 output/inception-v1.meta -in=input -on=InceptionV1/Logits/Predictions/Reshape_1 -i ../../data/images/cat.jpg -id 829 -S 2 -M 128 -cs 0,1,2 mvNCCheck v02.00, Copyright @ Movidius Ltd 2016 /home/tsteiner/fla-gtsc/fla_root/usr/python/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py:766: DeprecationWarning: builtin type EagerTensor has no __module__ attribute EagerTensor = c_api.TFE_Py_InitEagerTensor(_EagerTensorBase) /home/tsteiner/fla-gtsc/fla_root/usr/python/env3/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) /home/tsteiner/fla-gtsc/fla_root/usr/python/env3/lib/python3.5/site-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15. warn("The default mode, 'constant', will be changed to 'reflect' in " [Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm test -f graph || mvNCCompile -s 12 output/inception-v1.meta -in=input -on=InceptionV1/Logits/Predictions/Reshape_1

I am able to successfully run hello_ncs_py and hello_ncs_cpp, my NCS is plugged in to my computer, and TensorFlow seems to be configured correctly. I get the same error when I try inception_v3 (any inceptions) or the mobilenet example, as well. Is this a known issue? I'm not sure what to do with this error message.

[idata]

Hi @tsteiner2 we have seen this error with the new Tensorflow v1.4 and are looking for a fix. You should not see this with the earlier v1.3. Please let us know if that fixes you issue for now.

[idata]

Hi Ramana,

Thanks for providing such a quick response. I dropped the version of my TensorFlow to v1.3 and now the examples for Inception v1, v2, and v3 all work. However, the examples for Inception v4 and MobileNet still give the same "FusedBatchNorm" error. Does that match up with your testing, as well?

[idata]

@tsteiner2 I other place we saw this was when the model itself was changed to include fusedbatchnorm. The included Makefile always pulls the latest design file and hence the fails. If you modify the Makefile to use an older model file, that would work too. I’ll try to do that and check it in until the fix is available in the NCSDK.

[idata]

Hi

@tsteiner2 @ramana.rachakonda,In my linux system i am having tensorflow 1.3 version . and my bazel version is 0.7.0 which is installing 1.4 tensorflow version because when i am printing tensorflow version in retrain.py,i am getting 1.4.0 version of the tensorflow. So how can i downgrade bazel tensorflow version and make it 1.3 . please help me i am strugling lot to use this this device.

[idata]

Thanks @ramana.rachakonda. That makes sense to me. I've done a lot of searching and am unable to find an older version of this model. The only model files I can find are of the form mobilenet_v1_*_2017_06_14.tar.gz. I also tried using the older version of the mobilenet_v1.py file but got the same error message. If you could point me toward where to get the older model file that would be extremely helpful.

Also, if you have any rough estimate on when the NCSDK will support FusedBatchNorm it would be really helpful. My team is trying to include MobileNet on the NCS on a prototype we're working on and need to demo soon so any scheduling guidance there is very useful for planning purposes.

@imran I'm sorry but I don't use Bazel so I'm not sure how to downgrade TensorFlow with that tool. I am still prototyping with the pip-installed version, but that made downgrading to 1.3 temporarily pretty easy by calling pip install -I tensorflow==1.3 in my virtualenv. My [still weak] understanding of the Movidius chip is that using a custom optimized build of TensorFlow will not provide any speedups because the model is being evaluated on the NCS rather than on your CPU, so there is likely no benefit to maintaining a custom Bazel build to use with the NCS. There may be value in a custom build for speedups when training, but you could do that separately.

[idata]

Hi, @tsteiner2

Thanks for your effort

please see this, i think you can understand my problem in better way

https://ncsforum.movidius.com/discussion/402/error-5-toolkit-error-stage-details-not-supported-batchnormwithglobalnormalization#latest

[idata]

Hi,

@tsteiner2 I had the same issue lately with tensorflow 1.7 and 1.8.Following your advices I finally did a pip uninstall tensorflow and the issues remained the same,Then I removed the tensorflow build folder (apparently the unique way to uninstall a source install)Now I have tensorflow 1.7 installed in a python3 virtualenv and I still issue the same problem (full stacktrace below if it could help):/home/user/workspace/tensorflow-venv-3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py:875: DeprecationWarning: builtin type EagerTensor has no __module__ attribute EagerTensor = c_api.TFE_Py_InitEagerTensor(_EagerTensorBase) /home/user/workspace/tensorflow-venv-3/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) /usr/local/lib/python3.5/dist-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. from ._conv import register_converters as _register_converters /home/user/workspace/tensorflow-venv-3/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) [Error 5] Toolkit Error: Stage Details Not Supported: BatchNormWithGlobalNormalization

Regards