-
Notifications
You must be signed in to change notification settings - Fork 690
[Model] Support Insightface model inferenceing on RKNPU #1113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 21 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
6cfb59f
更新交叉编译
Zheng-Bicheng 478f975
更新交叉编译
Zheng-Bicheng 11081c4
更新交叉编译
Zheng-Bicheng 8337768
更新交叉编译
Zheng-Bicheng f7d2ea9
更新交叉编译
Zheng-Bicheng d2eaaec
更新交叉编译
Zheng-Bicheng 77e71c7
更新交叉编译
Zheng-Bicheng 1d91da6
更新交叉编译
Zheng-Bicheng 6575184
Merge branch 'develop' into test
Zheng-Bicheng be5efbc
Merge remote-tracking branch 'upstream/develop' into test
Zheng-Bicheng 983837b
Merge remote-tracking branch 'origin/test' into test
Zheng-Bicheng 00eab71
更新交叉编译
Zheng-Bicheng 3e02c37
Update issues.md
jiangjiajun 9e9687e
Update fastdeploy_init.sh
jiangjiajun 64dc564
更新交叉编译
Zheng-Bicheng 355a6e0
更新insightface系列模型的rknpu2支持
Zheng-Bicheng c3f899c
Merge branch 'develop' into test
Zheng-Bicheng c35787d
更新insightface系列模型的rknpu2支持
Zheng-Bicheng b637ba1
Merge remote-tracking branch 'origin/test' into test
Zheng-Bicheng 0693c59
更新说明文档
Zheng-Bicheng da14acb
Merge branch 'develop' into test
jiangjiajun aa7aee4
更新insightface
Zheng-Bicheng 027b762
Merge remote-tracking branch 'origin/test' into test
Zheng-Bicheng dcc6ac2
尝试解决pybind问题
Zheng-Bicheng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| [English](README.md) | 简体中文 | ||
| # InsightFace RKNPU准备部署模型 | ||
|
|
||
| 本教程提供InsightFace模型在RKNPU2环境下的部署，模型的详细介绍已经ONNX模型的下载请查看[模型介绍文档](../README.md)。 | ||
|
|
||
| ## 支持模型列表 | ||
| 目前FastDeploy支持如下模型的部署 | ||
| - ArcFace | ||
| - CosFace | ||
| - PartialFC | ||
| - VPL | ||
|
|
||
| ## 下载预训练ONNX模型 | ||
|
|
||
| 为了方便开发者的测试，下面提供了InsightFace导出的各系列模型，开发者可直接下载使用。（下表中模型的精度来源于源官方库）其中精度指标来源于InsightFace中对各模型的介绍，详情各参考InsightFace中的说明 | ||
|
|
||
| | 模型 | 大小 | 精度 (AgeDB_30) | | ||
| |:-------------------------------------------------------------------------------------------|:------|:--------------| | ||
| | [CosFace-r18](https://bj.bcebos.com/paddlehub/fastdeploy/glint360k_cosface_r18.onnx) | 92MB | 97.7 | | ||
| | [CosFace-r34](https://bj.bcebos.com/paddlehub/fastdeploy/glint360k_cosface_r34.onnx) | 131MB | 98.3 | | ||
| | [CosFace-r50](https://bj.bcebos.com/paddlehub/fastdeploy/glint360k_cosface_r50.onnx) | 167MB | 98.3 | | ||
| | [CosFace-r100](https://bj.bcebos.com/paddlehub/fastdeploy/glint360k_cosface_r100.onnx) | 249MB | 98.4 | | ||
| | [ArcFace-r18](https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r18.onnx) | 92MB | 97.7 | | ||
| | [ArcFace-r34](https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r34.onnx) | 131MB | 98.1 | | ||
| | [ArcFace-r50](https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r50.onnx) | 167MB | - | | ||
| | [ArcFace-r100](https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r100.onnx) | 249MB | 98.4 | | ||
| | [ArcFace-r100_lr0.1](https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_r100_lr01.onnx) | 249MB | 98.4 | | ||
| | [PartialFC-r34](https://bj.bcebos.com/paddlehub/fastdeploy/partial_fc_glint360k_r50.onnx) | 167MB | - | | ||
| | [PartialFC-r50](https://bj.bcebos.com/paddlehub/fastdeploy/partial_fc_glint360k_r100.onnx) | 249MB | - | | ||
|
|
||
|
|
||
| ## 转换为RKNPU模型 | ||
|
|
||
| ```bash | ||
| wget https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r18.onnx | ||
|
|
||
| python -m paddle2onnx.optimize --input_model ./ms1mv3_arcface_r18/ms1mv3_arcface_r18.onnx \ | ||
| --output_model ./ms1mv3_arcface_r18/ms1mv3_arcface_r18.onnx \ | ||
| --input_shape_dict "{'data':[1,3,112,112]}" | ||
|
|
||
| python /Path/To/FastDeploy/tools/rknpu2/export.py \ | ||
| --config_path tools/rknpu2/config/arcface_unquantized.yaml \ | ||
| --target_platform rk3588 | ||
| ``` | ||
|
|
||
| ## 详细部署文档 | ||
|
|
||
| - [Python部署](python) | ||
| - [C++部署](cpp) | ||
|
|
||
|
|
||
| ## 版本说明 | ||
|
|
||
| - 本版本文档和代码基于[InsightFace CommitID:babb9a5](https://github.com/deepinsight/insightface/commit/babb9a5) 编写 |
13 changes: 13 additions & 0 deletions
13
examples/vision/faceid/insightface/rknpu2/cpp/CMakeLists.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| PROJECT(infer_demo C CXX) | ||
| CMAKE_MINIMUM_REQUIRED (VERSION 3.10) | ||
|
|
||
| # 指定下载解压后的fastdeploy库路径 | ||
| option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.") | ||
|
|
||
| include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake) | ||
|
|
||
| # 添加FastDeploy依赖头文件 | ||
| include_directories(${FASTDEPLOY_INCS}) | ||
|
|
||
| add_executable(infer_arcface_demo ${PROJECT_SOURCE_DIR}/infer_arcface.cc) | ||
| target_link_libraries(infer_arcface_demo ${FASTDEPLOY_LIBS}) | ||
135 changes: 135 additions & 0 deletions
135
examples/vision/faceid/insightface/rknpu2/cpp/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| [English](README.md) | 简体中文 | ||
| # InsightFace C++部署示例 | ||
|
|
||
| FastDeploy支持在RKNPU上部署包括ArcFace\CosFace\VPL\Partial_FC在内的InsightFace系列模型。 | ||
|
|
||
| 本目录下提供`infer_arcface.cc`快速完成InsighFace模型包括ArcFace在CPU/RKNPU加速部署的示例。 | ||
|
|
||
|
|
||
| 在部署前，需确认以下两个步骤: | ||
|
|
||
| 1. 软硬件环境满足要求 | ||
| 2. 根据开发环境，下载预编译部署库或者从头编译FastDeploy仓库 | ||
|
|
||
| 以上步骤请参考[RK2代NPU部署库编译](../../../../../../docs/cn/build_and_install/rknpu2.md)实现 | ||
|
|
||
| 在本目录执行如下命令即可完成编译测试 | ||
|
|
||
| ```bash | ||
| mkdir build | ||
| cd build | ||
| wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz | ||
Zheng-Bicheng marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| tar xvf fastdeploy-linux-x64-x.x.x.tgz | ||
| cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x | ||
| make -j | ||
|
|
||
| # 下载官方转换好的ArcFace模型文件和测试图片 | ||
| wget https://bj.bcebos.com/paddlehub/fastdeploy/ms1mv3_arcface_r18.onnx | ||
| wget https://bj.bcebos.com/paddlehub/fastdeploy/rknpu2/face_demo.zip | ||
| unzip face_demo.zip | ||
|
|
||
| # CPU推理 | ||
| ./infer_arcface_demo ms1mv3_arcface_r100.onnx face_0.jpg face_1.jpg face_2.jpg 0 | ||
| # RKNPU推理 | ||
| ./infer_arcface_demo ms1mv3_arcface_r100.onnx face_0.jpg face_1.jpg face_2.jpg 1 | ||
| ``` | ||
|
|
||
| 运行完成可视化结果如下图所示 | ||
|
|
||
| <div width="700"> | ||
| <img width="220" float="left" src="https://user-images.githubusercontent.com/67993288/184321537-860bf857-0101-4e92-a74c-48e8658d838c.JPG"> | ||
| <img width="220" float="left" src="https://user-images.githubusercontent.com/67993288/184322004-a551e6e4-6f47-454e-95d6-f8ba2f47b516.JPG"> | ||
| <img width="220" float="left" src="https://user-images.githubusercontent.com/67993288/184321622-d9a494c3-72f3-47f1-97c5-8a2372de491f.JPG"> | ||
| </div> | ||
|
|
||
| 以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考: | ||
| - [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md) | ||
|
|
||
| ## InsightFace C++接口 | ||
|
|
||
| ### ArcFace类 | ||
|
|
||
| ```c++ | ||
| fastdeploy::vision::faceid::ArcFace( | ||
| const string& model_file, | ||
| const string& params_file = "", | ||
| const RuntimeOption& runtime_option = RuntimeOption(), | ||
| const ModelFormat& model_format = ModelFormat::ONNX) | ||
| ``` | ||
|
|
||
| ArcFace模型加载和初始化，其中model_file为导出的ONNX模型格式。 | ||
|
|
||
| ### CosFace类 | ||
|
|
||
| ```c++ | ||
| fastdeploy::vision::faceid::CosFace( | ||
| const string& model_file, | ||
| const string& params_file = "", | ||
| const RuntimeOption& runtime_option = RuntimeOption(), | ||
| const ModelFormat& model_format = ModelFormat::ONNX) | ||
| ``` | ||
|
|
||
| CosFace模型加载和初始化，其中model_file为导出的ONNX模型格式。 | ||
|
|
||
| ### PartialFC类 | ||
|
|
||
| ```c++ | ||
| fastdeploy::vision::faceid::PartialFC( | ||
| const string& model_file, | ||
| const string& params_file = "", | ||
| const RuntimeOption& runtime_option = RuntimeOption(), | ||
| const ModelFormat& model_format = ModelFormat::ONNX) | ||
| ``` | ||
|
|
||
| PartialFC模型加载和初始化，其中model_file为导出的ONNX模型格式。 | ||
|
|
||
| ### VPL类 | ||
|
|
||
| ```c++ | ||
| fastdeploy::vision::faceid::VPL( | ||
| const string& model_file, | ||
| const string& params_file = "", | ||
| const RuntimeOption& runtime_option = RuntimeOption(), | ||
| const ModelFormat& model_format = ModelFormat::ONNX) | ||
| ``` | ||
|
|
||
| VPL模型加载和初始化，其中model_file为导出的ONNX模型格式。 | ||
| **参数** | ||
|
|
||
| > * **model_file**(str): 模型文件路径 | ||
| > * **params_file**(str): 参数文件路径，当模型格式为ONNX时，此参数传入空字符串即可 | ||
| > * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置 | ||
| > * **model_format**(ModelFormat): 模型格式，默认为ONNX格式 | ||
|
|
||
| #### Predict函数 | ||
|
|
||
| > ```c++ | ||
| > ArcFace::Predict(cv::Mat* im, FaceRecognitionResult* result) | ||
Zheng-Bicheng marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| > ``` | ||
| > | ||
| > 模型预测接口，输入图像直接输出检测结果。 | ||
| > | ||
| > **参数** | ||
| > | ||
| > > * **im**: 输入图像，注意需为HWC，BGR格式 | ||
| > > * **result**: 检测结果，包括检测框，各个框的置信度, FaceRecognitionResult说明参考[视觉模型预测结果](../../../../../docs/api/vision_results/) | ||
|
|
||
| ### 修改预处理以及后处理的参数 | ||
| 预处理和后处理的参数的需要通过修改InsightFaceRecognitionPostprocessor，InsightFaceRecognitionPreprocessor的成员变量来进行修改。 | ||
|
|
||
| #### InsightFaceRecognitionPreprocessor成员变量(预处理参数) | ||
| > > * **size**(vector<int>): 通过此参数修改预处理过程中resize的大小，包含两个整型元素，表示[width, height], 默认值为[112, 112], | ||
| 通过InsightFaceRecognitionPreprocessor::SetSize(std::vector<int>& size)来进行修改 | ||
| > > * **alpha**(vector<float>): 预处理归一化的alpha值，计算公式为`x'=x*alpha+beta`，alpha默认为[1. / 127.5, 1.f / 127.5, 1. / 127.5], | ||
| 通过InsightFaceRecognitionPreprocessor::SetAlpha(std::vector<float>& alpha)来进行修改 | ||
| > > * **beta**(vector<float>): 预处理归一化的beta值，计算公式为`x'=x*alpha+beta`，beta默认为[-1.f, -1.f, -1.f], | ||
| 通过InsightFaceRecognitionPreprocessor::SetBeta(std::vector<float>& beta)来进行修改 | ||
|
|
||
| #### InsightFaceRecognitionPostprocessor成员变量(后处理参数) | ||
| > > * **l2_normalize**(bool): 输出人脸向量之前是否执行l2归一化，默认false, | ||
| InsightFaceRecognitionPostprocessor::SetL2Normalize(bool& l2_normalize)来进行修改 | ||
|
|
||
| - [模型介绍](../../../) | ||
| - [Python部署](../python) | ||
| - [视觉模型预测结果](../../../../../../docs/api/vision_results/README.md) | ||
| - [如何切换模型推理后端引擎](../../../../../../docs/cn/faq/how_to_change_backend.md) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.