Skip to content

Get rid of pickle for fluid inference #7339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Get rid of pickle for fluid inference #7339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
85ded8a
initial commit
kexinzhao Jan 8, 2018
edf8eae
modify load inference model function
kexinzhao Jan 9, 2018
c456c9c
clean up code
kexinzhao Jan 9, 2018
9da3d72
fix io.py
kexinzhao Jan 9, 2018
875e22b
clean code
kexinzhao Jan 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion paddle/framework/framework.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,4 +143,8 @@ message BlockDesc {
// Please refer to
// https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/program.md
// for more details.
message ProgramDesc { repeated BlockDesc blocks = 1; }
message ProgramDesc {
repeated BlockDesc blocks = 1;
repeated string feed_var_names = 2;
Copy link
Collaborator

@wangkuiyi wangkuiyi Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to add feed_var_names and fetch_var_names here? The current ProgramDesc, which supports training, should be able to read training data — maybe from some data feeding operators I vaguely remember? If so, it seems that we can reuse these feeding operators, or invent some new operators for data feeding specifically at inference time?

Copy link
Collaborator

@wangkuiyi wangkuiyi Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, given the fact that we are moving to use a WhileOp to represent the iteration loop at training time, I don’t expect a ProgramDesc that describes a training program can be shared and reused without any modification at inference time. As we’d need a new ProgramDesc specifically for inference, we can have inference data feeding operators in this new ProgramDesc.

Copy link

@kavyasrinet kavyasrinet Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wangkuiyi This is the approach we have been discussing too here: #7328 . Defining a ProgramDesc for Inference separately that has all Inference specific requirements addressed separately, and that can reuse our already existing ProgramDesc.

Copy link
Contributor Author

@kexinzhao kexinzhao Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wangkuiyi for the clarification on ProgramDesc. We are exploring different paths of obtaining a Desc for inference and discussing which one to use. Based on your comment, I think we will follow things the proposal in #7328. @abhinavarora is working on a PR based on that.

Copy link
Contributor

@Xreki Xreki Jan 10, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no feed_op or fetch_op in current train and inference ProgramDesc. Each time we run a train ProgramDesc, the Executor will clone a copy of the train ProgramDesc and insert feed_ops and fetch_ops to the copy of the train ProgramDesc.
@wangkuiyi do you mean to insert feed_op and fetch_op to inference ProgramDesc when saving? It is a way to solve this problem.
There is a proposal of mine, with no need to modify the current ProgramDesc or define another ProgramDesc, but I am not sure if it works. @kexinzhao @abhinavarora @kavyasrinet

  • We can traverse all the non-persistable variables in the inference ProgramDesc. If the variable is not an output of any operators, we think it is a feed variable. If the variable is not an input of any operators, we think it is a fetch variable.

Update:
I find a disadvantage of my proposal. User may specify the output of some middle operator (some operator in the middle position of the network and has output) to the fetch list.

repeated string fetch_var_names = 3;
}
16 changes: 15 additions & 1 deletion paddle/framework/program_desc.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ ProgramDesc::ProgramDesc() {

ProgramDesc::ProgramDesc(const ProgramDesc &o) {
desc_ = o.desc_;

for (int i = 0; i < desc_.blocks_size(); ++i) {
auto *block = desc_.mutable_blocks(i);
blocks_.emplace_back(new BlockDesc(*o.blocks_[i], block, this));
Expand All @@ -59,10 +58,25 @@ ProgramDesc::ProgramDesc(const proto::ProgramDesc &desc) {
ProgramDesc::ProgramDesc(const std::string &binary_str) {
PADDLE_ENFORCE(desc_.ParseFromString(binary_str),
"Fail to parse program_desc from binary string.");

for (auto &block_desc : *desc_.mutable_blocks()) {
blocks_.emplace_back(new BlockDesc(this, &block_desc));
}
}

void ProgramDesc::GetFeedVarNames(std::vector<std::string> &var_names) {
var_names.clear();
for (int i = 0; i < desc_.feed_var_names_size(); i++) {
var_names.push_back(desc_.feed_var_names(i));
}
}

void ProgramDesc::GetFetchVarNames(std::vector<std::string> &var_names) {
var_names.clear();
for (int i = 0; i < desc_.fetch_var_names_size(); i++) {
var_names.push_back(desc_.fetch_var_names(i));
}
}

} // namespace framework
} // namespace paddle
4 changes: 4 additions & 0 deletions paddle/framework/program_desc.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@ class ProgramDesc {

proto::ProgramDesc *Proto();

void GetFeedVarNames(std::vector<std::string> &var_names);

void GetFetchVarNames(std::vector<std::string> &var_names);

private:
proto::ProgramDesc desc_;

Expand Down
21 changes: 0 additions & 21 deletions paddle/inference/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,6 @@ cc_library(paddle_fluid_api
# Merge all modules into a simgle static library
cc_library(paddle_fluid DEPS paddle_fluid_api ${FLUID_CORE_MODULES})

# ptools
# just for testing, we may need to change the storing format for inference_model
# and move the dependent of pickle.
# download from http://www.picklingtools.com/
# build in the C++ sub-directory, using command
# make -f Makefile.Linux libptools.so
set(PTOOLS_LIB)
set(PTOOLS_ROOT $ENV{PTOOLS_ROOT} CACHE PATH "Folder contains PicklingTools")
find_path(PTOOLS_INC_DIR chooseser.h PATHS ${PTOOLS_ROOT}/C++)
find_library(PTOOLS_SHARED_LIB NAMES ptools PATHS ${PTOOLS_ROOT}/C++)
if(PTOOLS_INC_DIR AND PTOOLS_SHARED_LIB)
add_definitions(-DPADDLE_USE_PTOOLS)
set(PTOOLS_LIB ptools)
message(STATUS "Found PicklingTools: ${PTOOLS_SHARED_LIB}")
add_library(${PTOOLS_LIB} SHARED IMPORTED GLOBAL)
set_property(TARGET ${PTOOLS_LIB} PROPERTY IMPORTED_LOCATION ${PTOOLS_SHARED_LIB})
include_directories(${PTOOLS_ROOT}/C++)
include_directories(${PTOOLS_ROOT}/C++/opencontainers_1_8_5/include)
add_definitions(-DOC_NEW_STYLE_INCLUDES) # used in ptools
endif()

add_executable(example example.cc)
if(APPLE)
set(OPTIONAL_LINK_FLAGS)
Expand Down
18 changes: 4 additions & 14 deletions paddle/inference/example.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,33 +18,23 @@ limitations under the License. */
#include "paddle/inference/inference.h"

DEFINE_string(dirname, "", "Directory of the inference model.");
DEFINE_string(feed_var_names, "", "Names of feeding variables");
DEFINE_string(fetch_var_names, "", "Names of fetching variables");

int main(int argc, char** argv) {
google::ParseCommandLineFlags(&argc, &argv, true);
if (FLAGS_dirname.empty() || FLAGS_feed_var_names.empty() ||
FLAGS_fetch_var_names.empty()) {

if (FLAGS_dirname.empty()) {
// Example:
// ./example --dirname=recognize_digits_mlp.inference.model
// --feed_var_names="x"
// --fetch_var_names="fc_2.tmp_2"
std::cout << "Usage: ./example --dirname=path/to/your/model "
"--feed_var_names=x --fetch_var_names=y"
<< std::endl;
std::cout << "Usage: ./example --dirname=path/to/your/model" << std::endl;
exit(1);
}

std::cout << "FLAGS_dirname: " << FLAGS_dirname << std::endl;
std::cout << "FLAGS_feed_var_names: " << FLAGS_feed_var_names << std::endl;
std::cout << "FLAGS_fetch_var_names: " << FLAGS_fetch_var_names << std::endl;

std::string dirname = FLAGS_dirname;
std::vector<std::string> feed_var_names = {FLAGS_feed_var_names};
std::vector<std::string> fetch_var_names = {FLAGS_fetch_var_names};

paddle::InferenceEngine* engine = new paddle::InferenceEngine();
engine->LoadInferenceModel(dirname, feed_var_names, fetch_var_names);
engine->LoadInferenceModel(dirname);

paddle::framework::LoDTensor input;
srand(time(0));
Expand Down
26 changes: 8 additions & 18 deletions paddle/inference/inference.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,20 +25,8 @@ limitations under the License. */

namespace paddle {

void InferenceEngine::LoadInferenceModel(
const std::string& dirname,
const std::vector<std::string>& feed_var_names,
const std::vector<std::string>& fetch_var_names) {
#ifdef PADDLE_USE_PTOOLS
void InferenceEngine::LoadInferenceModel(const std::string& dirname) {
std::string model_filename = dirname + "/__model__";
LOG(INFO) << "Using PicklingTools, loading model from " << model_filename;
Val v;
LoadValFromFile(model_filename.c_str(), v, SERIALIZE_P0);
std::string program_desc_str = v["program_desc_str"];
LOG(INFO) << "program_desc_str's size: " << program_desc_str.size();
// PicklingTools cannot parse the vector of strings correctly.
#else
std::string model_filename = dirname + "/__model__.dat";
LOG(INFO) << "loading model from " << model_filename;
std::ifstream inputfs(model_filename, std::ios::in | std::ios::binary);
std::string program_desc_str;
Expand All @@ -48,15 +36,17 @@ void InferenceEngine::LoadInferenceModel(
LOG(INFO) << "program_desc_str's size: " << program_desc_str.size();
inputfs.read(&program_desc_str[0], program_desc_str.size());
inputfs.close();
#endif

program_ = new framework::ProgramDesc(program_desc_str);
program_->GetFeedVarNames(feed_var_names_);
program_->GetFetchVarNames(fetch_var_names_);

GenerateLoadProgram(dirname);

if (feed_var_names.empty() || fetch_var_names.empty()) {
LOG(FATAL) << "Please specify the feed_var_names and fetch_var_names.";
if (feed_var_names_.empty() || fetch_var_names_.empty()) {
LOG(FATAL) << "Please specify the feed_var_names and fetch_var_names when "
"saving inference models.";
}
feed_var_names_ = feed_var_names;
fetch_var_names_ = fetch_var_names;
PrependFeedOp();
AppendFetchOp();
}
Expand Down
4 changes: 1 addition & 3 deletions paddle/inference/inference.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ class InferenceEngine {
delete load_program_;
}

void LoadInferenceModel(const std::string& dirname,
const std::vector<std::string>& feed_var_names,
const std::vector<std::string>& fetch_var_names);
void LoadInferenceModel(const std::string& dirname);
void Execute(const std::vector<framework::LoDTensor>& feeds,
std::vector<framework::LoDTensor>& fetchs);

Expand Down
37 changes: 36 additions & 1 deletion paddle/pybind/protobuf.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,42 @@ void BindProgramDesc(py::module &m) {
PADDLE_ENFORCE(desc->ParseFromString(data),
"Fail to parse ProgramDesc from string. This could "
"be a bug of Paddle.");
});
})
.def("assign_feed_var_names",
[](ProgramDesc &program_desc,
const std::vector<std::string> &var_names) {
proto::ProgramDesc *desc = program_desc.Proto();
desc->clear_feed_var_names();
for (auto var_name : var_names) {
desc->add_feed_var_names(var_name);
}
})
.def("assign_fetch_var_names",
[](ProgramDesc &program_desc,
const std::vector<std::string> &var_names) {
proto::ProgramDesc *desc = program_desc.Proto();
desc->clear_fetch_var_names();
for (auto var_name : var_names) {
desc->add_fetch_var_names(var_name);
}
})
.def("get_feed_var_names",
[](ProgramDesc &program_desc) {
proto::ProgramDesc *desc = program_desc.Proto();
std::vector<std::string> retv;
for (int i = 0; i < desc->feed_var_names_size(); ++i) {
retv.push_back(desc->feed_var_names(i));
}
return retv;
})
.def("get_fetch_var_names", [](ProgramDesc &program_desc) {
proto::ProgramDesc *desc = program_desc.Proto();
std::vector<std::string> retv;
for (int i = 0; i < desc->fetch_var_names_size(); ++i) {
retv.push_back(desc->fetch_var_names(i));
}
return retv;
});
}

void BindBlockDesc(py::module &m) {
Expand Down
26 changes: 11 additions & 15 deletions python/paddle/v2/fluid/io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,18 +204,12 @@ def save_inference_model(dirname,
inference_program = pruned_program.inference_optimize()
fetch_var_names = [v.name for v in target_vars]

inference_program.desc.assign_feed_var_names(feeded_var_names)
inference_program.desc.assign_fetch_var_names(fetch_var_names)

model_file_name = dirname + "/__model__"
with open(model_file_name, "w") as f:
pickle.dump({
"program_desc_str": inference_program.desc.serialize_to_string(),
"feed_var_names": feeded_var_names,
"fetch_var_names": fetch_var_names
}, f, -1)

# Save only programDesc of inference_program in binary format
# in another file: __model__.dat
with open(model_file_name + ".dat", "wb") as fp:
fp.write(inference_program.desc.serialize_to_string())
with open(model_file_name, "wb") as f:
f.write(inference_program.desc.serialize_to_string())

save_params(executor, dirname, main_program)

Expand Down Expand Up @@ -254,11 +248,13 @@ def load_inference_model(dirname, executor):
raise ValueError("There is no directory named '%s'", dirname)

model_file_name = dirname + "/__model__"
model = pickle.load(open(model_file_name, "r"))
program_desc_str = model["program_desc_str"]
feed_var_names = model["feed_var_names"]
fetch_var_names = model["fetch_var_names"]
with open(model_file_name, "rb") as f:
program_desc_str = f.read()

program = Program.parse_from_string(program_desc_str)
feed_var_names = program.desc.get_feed_var_names()
fetch_var_names = program.desc.get_fetch_var_names()

load_persistables_if_exist(executor, dirname, program)
fetch_vars = [program.global_block().var(name) for name in fetch_var_names]

Expand Down