前言

最近有个项目，线上环境需要使用 C++ 来进行识别。为了方便调试，模型使用 python 代码进行训练，模型保存为 Tensorflow pb 格式。

环境

Tensorflow Share Library

直接安装

一般的包管理工具的 libtensorflow 仅包含 libtensorflow.so，只能调用 C Api。

# MacOS
brew install libtensorflow
 
# Ubuntu
sudo apt-get install libtensorflow

编译

此方法可以根据需求自行编译 libtensorflow.so 或 libtensorflow_cc.so，自由度更大。

~/Downloads/tensorflow-2.10.0 % ./configure
You have bazel 5.1.1 installed.
Please specify the location of python. [Default is /opt/homebrew/opt/python@3.10/bin/python3.10]:
 
Found possible Python library paths:
  /opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Please input the desired Python library path to use.  Default is [/opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages]
 
# 是否启用 ROCm 计算平台支持
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
 
# 是否启用 CUDA 支持
Do you wish to build TensorFlow with CUDA support? [y/N]: N
No CUDA support will be enabled for TensorFlow.
 
# 是否使用新版本 Clang 编译器
Do you wish to download a fresh release of clang? (Experimental) [y/N]: N
Clang will not be downloaded.
 
# 其他编译选项
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:
 
# 是否启用 Android 支持
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N
Not configuring the WORKSPACE for Android builds.
 
# 是否启用 iOS 支持
Do you wish to build TensorFlow with iOS support? [y/N]: N
No iOS support will be enabled for TensorFlow.
 
# 其余支持编译时传入的参数
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
    --config=mkl            # Build with MKL support.
    --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
    --config=monolithic     # Config for mostly static monolithic build.
    --config=numa           # Build with NUMA support.
    --config=dynamic_kernels    # (Experimental) Build kernels into separate shared objects.
    --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
    --config=nogcp          # Disable GCP support.
    --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
 
# 编译 C Api libtensorflow.so
~/Downloads/tensorflow-2.10.0 % bazel build --config=monolithic -c opt //tensorflow:libtensorflow.so
 
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=209
...
INFO: Analyzed target //tensorflow:libtensorflow.so (269 packages loaded, 21402 targets configured).
INFO: Found 1 target...
Target //tensorflow:libtensorflow.so up-to-date:
  bazel-bin/tensorflow/libtensorflow.so
INFO: Elapsed time: 29.067s, Critical Path: 6.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
 
# 编译 C++ Api libtensorflow_cc.so
~/Downloads/tensorflow-2.10.0 % bazel build --config=monolithic -c opt //tensorflow:libtensorflow_cc.so
 
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=209
...
INFO: Analyzed target //tensorflow:libtensorflow_cc.so (1 packages loaded, 53 targets configured).
INFO: Found 1 target...
Target //tensorflow:libtensorflow_cc.so up-to-date:
  bazel-bin/tensorflow/libtensorflow_cc.so
INFO: Elapsed time: 6.587s, Critical Path: 5.69s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

项目构建

项目构建需要将 Bazel 编译后的头文件和动态链接库加载到项目构建中，相关文件路径可以参考 Bazel 编译完成后输出的 Target 路径。C++ API 编译同时需要拷贝其余第三方依赖库的 Header 文件。包括但不限于 Eigen, absl, protobuf。

查看模型详情

# guesslang DNN 模型
# https://github.com/yoeo/guesslang/tree/master/guesslang/data/model
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model
The given SavedModel contains the following tag-sets:
'serve'
 
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model --tag_set serve
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "classification"
SignatureDef key: "predict"
SignatureDef key: "serving_default"
 
(tensorflow) guesslang-2.2.1 % saved_model_cli show --dir guesslang/data/model --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['inputs'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: Placeholder:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['classes'] tensor_info:
      dtype: DT_STRING
      shape: (-1, 54)
      name: head/Tile:0
  outputs['scores'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 54)
      name: head/predictions/probabilities:0
Method name is: tensorflow/serving/classify
 
 
# ResNet 模型
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain
The given SavedModel contains the following tag-sets:
'serve'
 
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain/ --tag_set serve
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default"
 
(tensorflow) test-resnet % saved_model_cli show --dir ./resnet_retrain/ --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['resnet50_input'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 3)
      name: serving_default_resnet50_input:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['dense_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 5)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

C / C++ 模型加载和预测

上述流程中获取到了模型相关的详情信息，下面将以 guesslang 模型为例，看看 C / C++ 中如何对 Tensorflow 模型进行加载和预测。

C 版本

从 Tensorflow C API 的定位来说，它更偏向底层，因此代码看起来会更加复杂。

#include <stdio.h>
#include <tensorflow/c/c_api.h>
 
void NoOpDeallocator(void* data, size_t a, void* b) {}
 
int main() {
    const char* model_dir = "/path/to/model/dir"
    TF_Status* status = TF_NewStatus();
    TF_Graph* tfGraph = TF_NewGraph();
    TF_Buffer* run_opts = NULL;
    TF_Buffer* meta_g = NULL;
    TF_SessionOptions* opts = TF_NewSessionOptions();
    TF_Output *inputs = (TF_Output *) malloc(sizeof(TF_Output) * ninputs);
    TF_Output *outputs = (TF_Output *) malloc(sizeof(TF_Output) * noutputs);
    TF_Tensor **input_values = (TF_Tensor **) malloc(sizeof(TF_Tensor *) * ninputs);
    TF_Tensor **output_values = (TF_Tensor **) malloc(sizeof(TF_Tensor *) * noutputs);
     
    // 此处 tag 对应模型信息中的 tag-sets
    int ntag = 1;
    const char* tags = "serve";
     
    // 加载训练好的模型
    TF_Session* sess = TF_LoadSessionFromSavedModel(
        opts, run_opts,
        model_dir, &tags, ntag,
        tfGraph, meta_g,
        status
    );
     
    if (TF_GetCode(status) != TF_OK) {
        printf("%s\n", TF_Message(status));
        goto out;
    }
    TF_DeleteStatus(status);
     
    int ninputs = 1;
    int noutputs = 2;
 
    // 构建输入输出
    // 输入输出的 Operation Name 对应 SignatureDef 中输入输出 Tensor 描述中的名称
    TF_Output input = {TF_GraphOperationByName(tfGraph, "Placeholder"), 0};
    if(NULL == input.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
    TF_Output classes = {TF_GraphOperationByName(tfGraph, "head/Tile"), 0};
    if(NULL == classes.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
    TF_Output score = {TF_GraphOperationByName(tfGraph, "head/predictions/probabilities"), 0};
    if(NULL == score.oper) {
        printf("undefined graph operation\n");
        goto out;
    }
 
    const char* data = "package main\nimport (\n\"fmt\"\n)\nfunc main() {\nfmt.Println(\"Hello Go!\")}";
    size_t size = strlen(data);
 
    // 模型 Input 为 DT_STRING
    // 实现构建 TF_TSTRING 类型变量作为构建 Tensor 入参
    TF_TString tstr[1];
    TF_TString_Init(&tstr[0]);
    TF_TString_Copy(&tstr[0], data, size);
 
    // 此处为 Tensor 的 Shape 参数
    // 对应 Tensor 的维度信息
    // dim - 0 常量
    // dim - 1 向量
    // dim - 2 矩阵
    // ... (重要)
    int ndims = 1;
    int64_t dims[] = {1};
 
    // 构建输入 Tensor，必须根据模型入参传入对应类型的 Tensor 和参数
    TF_Tensor* int_tensor = TF_NewTensor(TF_STRING, dims, ndims, &tstr[0], sizeof(tstr), &NoOpDeallocator, NULL);
    if (NULL == int_tensor) {
        printf("construction of input tensor failed\n");
        goto out;
    }
 
    // 出入参写入对应指针指向的内存
    inputs[0] = input;
    outputs[0] = classes;
    outputs[1] = score;
    input_values[0] = int_tensor;
 
    TF_Status* status = TF_NewStatus();
     
    // 执行预测
    TF_SessionRun(
        sess,
        NULL,
        inputs, input_values, ninputs,
        outputs, output_values, noutputs,
        NULL, 0,
        NULL,
        status
    );
    if (TF_GetCode(status) != TF_OK) {
        printf("%s\n", TF_Message(status));
        goto out;
    }
    TF_DeleteStatus(status);
 
    // 输出结果数量也是根据模型 output shape 得出
    size_t num_class = 54;
     
    // 结果解析
    void* out_classes = TF_TensorData(output_values[0]);
    TF_TString* res_classes = (TF_TString *)out_classes;
    void* out_scores = TF_TensorData(output_values[1]);
    float* res_scores = (float *)out_scores;
 
    const char * tag;
    float max_score = 0;
    for (int i = 0; i < num_class; i++) {
        if(res_scores[i]>max_score){
            max_score = res_scores[i];
            tag = TF_TString_GetDataPointer(&res_classes[i]);
        }
    }
 
    //释放资源
out:
    if (NULL != int_tensor) {
        TF_DeleteTensor(int_tensor);
    }
     
    if (NULL != tstr[0]) {
        TF_TString_Dealloc(&tstr[0]);
    }
     
    if (NULL != tfGraph) {
        TF_DeleteGraph(tfGraph);
    }
     
    if (NULL != sess) {
        TF_Status* status = TF_NewStatus();
        TF_DeleteSession(sess, status);
        TF_DeleteStatus(status);
    }
 
    free(inputs);
    free(outputs);
    free(input_values);
    free(output_values);
}

C++ 版本

include <iostream>
 
#include "tensorflow/cc/client/client_session.h"
#include "tensorflow/cc/ops/standard_ops.h"
#include "tensorflow/core/framework/tensor.h"
#include "tensorflow/cc/saved_model/loader.h"
#include "tensorflow/cc/saved_model/tag_constants.h"
#include "tensorflow/core/framework/logging.h"
 
using namespace tensorflow;
using namespace tensorflow::ops;
 
int main() {
    const char* model_dir = "/path/to/model/dir"
     
    // 模型加载
    auto* loadModel = new SavedModelBundleLite();
    SessionOptions sessionOptions;
    RunOptions runOptions;
 
    Status loadStatus = LoadSavedModel(
        sessionOptions,
        runOptions,
        model_dir,
        { kSavedModelTagServe },
        loadModel
    );
 
    TF_CHECK_OK(loadStatus);
     
    const char* data = "package main\nimport (\n\"fmt\"\n)\nfunc main() {\nfmt.Println(\"Hello Go!\")}";
     
    // 构建 Tensor
    Tensor input_tensor(data);
 
    // 构建 Input Vector，绑定 Tensor
    std::vector<std::pair<std::string, tensorflow::Tensor>> feedInputs = { {"Placeholder:0", input_tensor} };
     
    // 创建 Output Vector
    std::vector<std::string> fetches = { "head/Tile:0", "head/predictions/probabilities:0" };
    std::vector<tensorflow::Tensor> outputs;
 
    // 执行预测
    auto status = loadModel->GetSession()->Run(feedInputs, fetches, {}, &outputs);
    TF_CHECK_OK(status);
 
    for (const auto& record : outputs) {
        LOG(INFO) << record.DebugString();
    }
     
    delete(loadModel)
}