OpenVINO EP not able to use CPU device

[CristianCerasuoloo]

Describe the issue

Hi, I am trying to integrate in my application the use of onnxruntime with OpenVINO EP using the C++ API.
To do this, I followed the directions in the documentation so:

• Installed OpenVINO version 2024.4.0 from here
• Compiled from source the onnxruntime v1.20.0 library with the following command

./build.sh --config RelWithDebInfo --use_openvino CPU --parallel 4 --build_shared_lib --update --build

I compiled my application by integrating the libraries produced in build/Linux/RelWithDebInfo and the headers in onnxruntime/core/session and produced the application code that would load the provider:

std::unordered_map<std::string, std::string> options;
options["device_type"] = "CPU";
options["precision"] = "FP32";
# Desperate attempt, not sure this option works in this way (documentation not clear)
std::string config = R"({
		"CPU": {
		"precisions": "FP32"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

When starting the application, the available providers appear to be (correctly) OpenVINOExecutionProvider and CPUExecutionProvider. However, when loading OpenVINOExecutionProvider, I get the following error:

2025-03-26 09:57:31.781753197 [E:onnxruntime:, inference_session.cc:2117 operator()] Exception during initialization: Exception from src/inference/src/cpp/core.cpp:261:
Exception from src/inference/src/dev/core_impl.cpp:566:
Device with "NPU" name is not registered in the OpenVINO Runtime

OpenVINO is trying to use the NPU device even though it has not been specified!

To probe all the options I recompiled the library specifying NPU as a device as well, then with the following command

./build.sh --config RelWithDebInfo --use_openvino HETERO:CPU,NPU --parallel 4 --build_shared_lib --update --build

However, specifying NPU as the device I got the device unavailability error (as it should be)

std::unordered_map<std::string, std::string> options;
options["device_type"] = "NPU";
options["precision"] = "FP16";
# Desperate attempt, not sure this option works in this way (documentation not clear)
std::string config = R"({
		"NPU": {
		"precisions": "FP16"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

onnxruntime::OpenVINOExecutionProvider::OpenVINOExecutionProvider(const onnxruntime::OpenVINOExecutionProviderInfo&) [ERROR] [OpenVINO] Specified device - NPU is not available

Instead, specifying as device HETERO:CPU,NPU I get the same initial error

std::unordered_map<std::string, std::string> options;
options["device_type"] = "HETERO:CPU,NPU";
options["precision"] = "FP16";
std::string config = R"({
	"NPU": {
		"precision": "FP16"
	},
	"CPU": {
		"precision": "FP32"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

2025-03-26 10:13:29.006646437 [E:onnxruntime:, inference_session.cc:2117 operator()] Exception during initialization: Exception from src/inference/src/cpp/core.cpp:261:
Exception from src/inference/src/dev/core_impl.cpp:566:
Device with "NPU" name is not registered in the OpenVINO Runtime

On the OpenVINO library side, running the HelloQueryDevice sample I get the following output:

hello_query_device

[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] Available devices: 
[ INFO ] CPU
[ INFO ]        SUPPORTED_PROPERTIES: 
[ INFO ]                Immutable: AVAILABLE_DEVICES : ""
[ INFO ]                Immutable: RANGE_FOR_ASYNC_INFER_REQUESTS : 1 1 1
[ INFO ]                Immutable: RANGE_FOR_STREAMS : 1 4
[ INFO ]                Immutable: EXECUTION_DEVICES : CPU
[ INFO ]                Immutable: FULL_DEVICE_NAME : Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
[ INFO ]                Immutable: OPTIMIZATION_CAPABILITIES : WINOGRAD FP32 INT8 BIN EXPORT_IMPORT
[ INFO ]                Immutable: DEVICE_TYPE : integrated
[ INFO ]                Immutable: DEVICE_ARCHITECTURE : intel64
[ INFO ]                Mutable: NUM_STREAMS : 1
[ INFO ]                Mutable: INFERENCE_NUM_THREADS : 0
[ INFO ]                Mutable: PERF_COUNT : NO
[ INFO ]                Mutable: INFERENCE_PRECISION_HINT : f32
[ INFO ]                Mutable: PERFORMANCE_HINT : LATENCY
[ INFO ]                Mutable: EXECUTION_MODE_HINT : PERFORMANCE
[ INFO ]                Mutable: PERFORMANCE_HINT_NUM_REQUESTS : 0
[ INFO ]                Mutable: ENABLE_CPU_PINNING : YES
[ INFO ]                Mutable: SCHEDULING_CORE_TYPE : ANY_CORE
[ INFO ]                Mutable: MODEL_DISTRIBUTION_POLICY : ""
[ INFO ]                Mutable: ENABLE_HYPER_THREADING : YES
[ INFO ]                Mutable: DEVICE_ID : ""
[ INFO ]                Mutable: CPU_DENORMALS_OPTIMIZATION : NO
[ INFO ]                Mutable: LOG_LEVEL : LOG_NONE
[ INFO ]                Mutable: CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE : 1
[ INFO ]                Mutable: DYNAMIC_QUANTIZATION_GROUP_SIZE : 32
[ INFO ]                Mutable: KV_CACHE_PRECISION : f16
[ INFO ]                Mutable: AFFINITY : CORE
[ INFO ] 

therefore the device is correctly recognized and identified as CPU. The sample benchmark_app also correctly performs inference on the provided onnx model.

[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration

[... Other not important steps ...]
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               1092 iterations
[ INFO ] Duration:            60338.34 ms
[ INFO ] Latency:
[ INFO ]    Median:           216.96 ms
[ INFO ]    Average:          220.76 ms
[ INFO ]    Min:              209.33 ms
[ INFO ]    Max:              358.57 ms
[ INFO ] Throughput:          18.10 FPS

I want to to clarify that the very same problem happens with different append methods. Since the documentation of AppendExecutionProvider says to not be available for OpenVINO but only QNN, SNPE and XNNPACK I also tried:

Appending OV with the v1 API (as specified in inference sample)

OrtOpenVINOProviderOptions options{};
options.device_type = "CPU_FP32";
state->session_options->AppendExecutionProvider_OpenVINO(options);			

Appending OV with the v2 API:

std::unordered_map<std::string, std::string> options;
options["device_type"] = "CPU";
options["precision"] = "FP32";
state->session_options->AppendExecutionProvider_OpenVINO_V2(options);

Also, the previous without specifying any options
state->session_options->AppendExecutionProvider_OpenVINO(OrtOpenVINOProviderOptions ());
or

std::unordered_map<std::string, std::string> options;
state->session_options->AppendExecutionProvider_OpenVINO_V2(options);

Moreover, since I didn't specify it before, the error seems to born in the Session creation and so, after the provider creation:

state->env = new Ort::Env(ORT_LOGGING_LEVEL_WARNING, "onnx_environment");
OrtOpenVINOProviderOptions options{};
options.device_type = "CPU_FP32";
state->session_options->AppendExecutionProvider_OpenVINO(options);
std::cout << "Provider loaded" << std::endl;
state->session = new Ort::Session(*(state->env), file, *(state->session_options)); // HERE

Lastly, here the informations about my machine

Ubuntu20.04 x86_64
CPU: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
GPU: Not available
NPU: Not available
Onnxruntime v1.20.0
OpenVINO 2024.4.0
CMake 3.26.6
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

I've also asked the very same question into the intel/onnxruntime fork but still no answer.

To reproduce

1. Compile onnxruntime v1.20.0 with OpenVINO EP using the provided building commands on the same machine/architecture and the very same versions
2. Try to use OpenVINO EP in a c++ application by providing the same options as shown above
3. Check the stdout to read the logs.

Urgency

Solving this issue is very important in order to put this new feature in the new release of my application (coming in few days).
So, the urgency is very high. Please help me as soon as possible.

Platform

Linux

OS Version

Ubuntu20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

v1.20.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

OpenVINO 2024.4.0

[preetha-intel]

@CristianCerasuoloo There was a fix introduced for this error in Release 1.21.
I have build the onnxruntime with 1.21 from source and using openvino 2025.0 which is the latest releases and using the command shared above
./build.sh --config RelWithDebInfo --use_openvino CPU --parallel 4 --build_shared_lib --update --build

The issue is not reproducible.

onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo>.\onnxruntime_perf_test.exe -m times -r 1 -e openvino -i "device_type|CPU precision|FP32" ".\onnx_models_public_ovep_por_20240405\opset16\vgg19_bn_Opset16\vgg19_bn_Opset16.onnx"

Session creation time cost: 4.38354 s
First inference time cost: 114 ms
Total inference time cost: 0.106671 s
Total inference requests: 1
Average inference time cost: 106.671 ms
Total inference run time: 0.106723 s
Number of inferences per second: 9.37007
Avg CPU usage: 50 %
Peak working set size: 3823554560 bytes
Avg CPU usage:50
Peak working set size:3823554560
Runs:1
Min Latency: 0.106671 s
Max Latency: 0.106671 s
P50 Latency: 0.106671 s
P90 Latency: 0.106671 s
P95 Latency: 0.106671 s
P99 Latency: 0.106671 s
P999 Latency: 0.106671 s

[CristianCerasuoloo]

Thanks for the help @preetha-intel.
Since I'm still having troubles compiling v1.21.0 with OpenVINO 2025 could you share more information on your configuration (gcc, g++, python or whatever else can be useful)?

I got this error with g++/gcc 9

onnxruntime::openvino_ep::SharedContext::SharedWeights::Metadata::Key&) const’ cannot be defaulted
   30 |         bool operator==(const Key&) const = default;

This using g++/gcc 10

/usr/include/c++/10/type_traits:2554:11: error: no type named ‘type’ in ‘struct std::enable_if<false, std::from_chars_result>’
 2554 |     using enable_if_t = typename enable_if<_Cond, _Tp>::type;

This error with g++/gcc 11

[ 11%] Building CXX object _deps/protobuf-build/CMakeFiles/libprotobuf.dir/src/google/protobuf/message.cc.o
/tmp/cchOcF9B.s: Assembler messages:
/tmp/cchOcF9B.s:53525: Error: unsupported instruction `vpdpbusds'
/tmp/cchOcF9B.s:53616: Error: unsupported instruction `vpdpbusds'

Again, thanks for all.