OpenVINO EP not able to use CPU device

[CristianCerasuoloo]

Describe the issue

Hi, I am trying to integrate in my application the use of onnxruntime with OpenVINO EP using the C++ API.
To do this, I followed the directions in the documentation so:

• Installed OpenVINO version 2024.4.0 from here
• Compiled from source the onnxruntime v1.20.0 library with the following command

./build.sh --config RelWithDebInfo --use_openvino CPU --parallel 4 --build_shared_lib --update --build

I compiled my application by integrating the libraries produced in build/Linux/RelWithDebInfo and the headers in onnxruntime/core/session and produced the application code that would load the provider:

std::unordered_map<std::string, std::string> options;
options["device_type"] = "CPU";
options["precision"] = "FP32";
# Desperate attempt, not sure this option works in this way (documentation not clear)
std::string config = R"({
		"CPU": {
		"precisions": "FP32"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

When starting the application, the available providers appear to be (correctly) OpenVINOExecutionProvider and CPUExecutionProvider. However, when loading OpenVINOExecutionProvider, I get the following error:

2025-03-26 09:57:31.781753197 [E:onnxruntime:, inference_session.cc:2117 operator()] Exception during initialization: Exception from src/inference/src/cpp/core.cpp:261:
Exception from src/inference/src/dev/core_impl.cpp:566:
Device with "NPU" name is not registered in the OpenVINO Runtime

OpenVINO is trying to use the NPU device even though it has not been specified!

To probe all the options I recompiled the library specifying NPU as a device as well, then with the following command

./build.sh --config RelWithDebInfo --use_openvino HETERO:CPU,NPU --parallel 4 --build_shared_lib --update --build

However, specifying NPU as the device I got the device unavailability error (as it should be)

std::unordered_map<std::string, std::string> options;
options["device_type"] = "NPU";
options["precision"] = "FP16";
# Desperate attempt, not sure this option works in this way (documentation not clear)
std::string config = R"({
		"NPU": {
		"precisions": "FP16"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

onnxruntime::OpenVINOExecutionProvider::OpenVINOExecutionProvider(const onnxruntime::OpenVINOExecutionProviderInfo&) [ERROR] [OpenVINO] Specified device - NPU is not available

Instead, specifying as device HETERO:CPU,NPU I get the same initial error

std::unordered_map<std::string, std::string> options;
options["device_type"] = "HETERO:CPU,NPU";
options["precision"] = "FP16";
std::string config = R"({
	"NPU": {
		"precision": "FP16"
	},
	"CPU": {
		"precision": "FP32"
		}
	})";
options["load_config"] = config;
state->session_options->AppendExecutionProvider("OpenVINO", options);

2025-03-26 10:13:29.006646437 [E:onnxruntime:, inference_session.cc:2117 operator()] Exception during initialization: Exception from src/inference/src/cpp/core.cpp:261:
Exception from src/inference/src/dev/core_impl.cpp:566:
Device with "NPU" name is not registered in the OpenVINO Runtime

On the OpenVINO library side, running the HelloQueryDevice sample I get the following output:

hello_query_device

[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] Available devices: 
[ INFO ] CPU
[ INFO ]        SUPPORTED_PROPERTIES: 
[ INFO ]                Immutable: AVAILABLE_DEVICES : ""
[ INFO ]                Immutable: RANGE_FOR_ASYNC_INFER_REQUESTS : 1 1 1
[ INFO ]                Immutable: RANGE_FOR_STREAMS : 1 4
[ INFO ]                Immutable: EXECUTION_DEVICES : CPU
[ INFO ]                Immutable: FULL_DEVICE_NAME : Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
[ INFO ]                Immutable: OPTIMIZATION_CAPABILITIES : WINOGRAD FP32 INT8 BIN EXPORT_IMPORT
[ INFO ]                Immutable: DEVICE_TYPE : integrated
[ INFO ]                Immutable: DEVICE_ARCHITECTURE : intel64
[ INFO ]                Mutable: NUM_STREAMS : 1
[ INFO ]                Mutable: INFERENCE_NUM_THREADS : 0
[ INFO ]                Mutable: PERF_COUNT : NO
[ INFO ]                Mutable: INFERENCE_PRECISION_HINT : f32
[ INFO ]                Mutable: PERFORMANCE_HINT : LATENCY
[ INFO ]                Mutable: EXECUTION_MODE_HINT : PERFORMANCE
[ INFO ]                Mutable: PERFORMANCE_HINT_NUM_REQUESTS : 0
[ INFO ]                Mutable: ENABLE_CPU_PINNING : YES
[ INFO ]                Mutable: SCHEDULING_CORE_TYPE : ANY_CORE
[ INFO ]                Mutable: MODEL_DISTRIBUTION_POLICY : ""
[ INFO ]                Mutable: ENABLE_HYPER_THREADING : YES
[ INFO ]                Mutable: DEVICE_ID : ""
[ INFO ]                Mutable: CPU_DENORMALS_OPTIMIZATION : NO
[ INFO ]                Mutable: LOG_LEVEL : LOG_NONE
[ INFO ]                Mutable: CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE : 1
[ INFO ]                Mutable: DYNAMIC_QUANTIZATION_GROUP_SIZE : 32
[ INFO ]                Mutable: KV_CACHE_PRECISION : f16
[ INFO ]                Mutable: AFFINITY : CORE
[ INFO ] 

therefore the device is correctly recognized and identified as CPU. The sample benchmark_app also correctly performs inference on the provided onnx model.

[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration

[... Other not important steps ...]
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               1092 iterations
[ INFO ] Duration:            60338.34 ms
[ INFO ] Latency:
[ INFO ]    Median:           216.96 ms
[ INFO ]    Average:          220.76 ms
[ INFO ]    Min:              209.33 ms
[ INFO ]    Max:              358.57 ms
[ INFO ] Throughput:          18.10 FPS

Lastly, here the informations about my machine

Ubuntu20.04 x86_64
CPU: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
GPU: Not available
NPU: Not available
Onnxruntime v1.20.0
OpenVINO 2024.4.0
CMake 3.26.6
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

To reproduce

1. Compile onnxruntime v1.20.0 with OpenVINO EP using the provided building commands on the same machine/architecture and the very same versions
2. Try to use OpenVINO EP in a c++ application by providing the same options as shown above
3. Check the stdout to read the logs.

Urgency

Solving this issue is very important in order to put this new feature in the new release of my application (coming in few days).
So, the urgency is very high. Please help me as soon as possible.

Platform

Linux

OS Version

Ubuntu20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

v1.20.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

OpenVINO 2024.4.0

[CristianCerasuoloo]

I want to to clarify that the very same problem happens with different append methods. Since the documentation of AppendExecutionProvider says to not be available for OpenVINO but only QNN, SNPE and XNNPACK I also tried:

Appending OV with the v1 API (as specified in inference sample)

OrtOpenVINOProviderOptions options{};
options.device_type = "CPU_FP32";
state->session_options->AppendExecutionProvider_OpenVINO(options);			

Appending OV with the v2 API:

std::unordered_map<std::string, std::string> options;
options["device_type"] = "CPU";
options["precision"] = "FP32";
state->session_options->AppendExecutionProvider_OpenVINO_V2(options);

Also, the previous without specifying any options
state->session_options->AppendExecutionProvider_OpenVINO(OrtOpenVINOProviderOptions ());
or

std::unordered_map<std::string, std::string> options;
state->session_options->AppendExecutionProvider_OpenVINO_V2(options);

Moreover, since I didn't specify it before, the error seems to born in the Session creation and so, after the provider creation:

state->env = new Ort::Env(ORT_LOGGING_LEVEL_WARNING, "onnx_environment");
OrtOpenVINOProviderOptions options{};
options.device_type = "CPU_FP32";
state->session_options->AppendExecutionProvider_OpenVINO(options);
std::cout << "Provider loaded" << std::endl;
state->session = new Ort::Session(*(state->env), file, *(state->session_options)); // HERE

[preetha-intel]

Issue not reproducible with ORT 1.21 and OV > 2024.3

microsoft#24208 (comment)