Adding metadata to TFLite models

[chainyo]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I looked a lot of issues and the @zldrobit repo for TFLite export and inference.

But, I'm still lacking on answers in order to add metadatas to my tflite model. I know how to use it in a python script or even with tf.js, but I need to use my converted model in MLKit (a solution by Google) and this platform requires metadatas, especially outputs metadatas to run.

I tried also to add manually metadatas to my model with tflite_support package. Unfortunatly the default object detection template doesn't work because YoloV5 output is only 1 tensor and not 4 tensors as expected by the template. I tried to customize the metadatas add via this package, but nothing seems to work.

I'm still looking for a solution to add metadatas to my TFLite custom model.

I looked this issues' answer @zldrobit any suggestions ? #5030 (comment)

[glenn-jocher]

@chainyo I'm not sure, I'm not a TF specialist myself, but on the surface this appears to be a TFLite issue rather than a YOLOv5 issue. Perhaps raise this on the TF repo or the TF forums, or even stackoverflow.

Naturally this would be nice to have here too, as TFLite inference with detect.py lacks class names currently, so from our side at least adding the class names as model metadata would be useful.

[chainyo]

This is the actual custom script I have to add metadatas.
This is what works and what is not working:

• Models infos
• Model Inputs
• Model Outputs

Outputs don't match the metadatas because yolov5 has 1 tensor as output and the metadatas are expecting 4 tensors.

ValueError: The number of output tensors (1) should match the number of output tensor metadata (4)

The goal would be to understand how to make it fit the yolov5 output.

- subgraph.outputTensorGroups = [group]
+ subgraph.outputTensorGroups = [[group]]

import os 

from tflite_support import flatbuffers
from tflite_support import metadata as _metadata
from tflite_support import metadata_schema_py_generated as _metadata_fb


model_file = "[path_to_model].tflite"


# Creates model info.
model_meta = _metadata_fb.ModelMetadataT()
model_meta.name = "Model Name"
model_meta.description = ("Model Description")
model_meta.version = "v1"
model_meta.author = "Model Author"


# Creates input info.
input_meta = _metadata_fb.TensorMetadataT()
input_meta.name = "image"
input_meta.description = ("Input Description")
input_meta.content = _metadata_fb.ContentT()
input_meta.content.contentProperties = _metadata_fb.ImagePropertiesT()
input_meta.content.contentProperties.colorSpace = (
    _metadata_fb.ColorSpaceType.RGB)
input_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.ImageProperties)
input_normalization = _metadata_fb.ProcessUnitT()
input_normalization.optionsType = (
    _metadata_fb.ProcessUnitOptions.NormalizationOptions)
input_normalization.options = _metadata_fb.NormalizationOptionsT()
input_normalization.options.mean = [127.5]
input_normalization.options.std = [127.5]
input_meta.processUnits = [input_normalization]
input_stats = _metadata_fb.StatsT()
input_stats.max = [255]
input_stats.min = [0]
input_meta.stats = input_stats


# Creates output info.
output_location_meta = _metadata_fb.TensorMetadataT()
output_location_meta.name = "location"
output_location_meta.description = "The locations of the detected boxes."
output_location_meta.content = _metadata_fb.ContentT()
output_location_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.BoundingBoxProperties)
output_location_meta.content.contentProperties = (
    _metadata_fb.BoundingBoxPropertiesT())
output_location_meta.content.contentProperties.index = [1, 0, 3, 2]
output_location_meta.content.contentProperties.type = (
    _metadata_fb.BoundingBoxType.BOUNDARIES)
output_location_meta.content.contentProperties.coordinateType = (
    _metadata_fb.CoordinateType.RATIO)
output_location_meta.content.range = _metadata_fb.ValueRangeT()
output_location_meta.content.range.min = 2
output_location_meta.content.range.max = 2

output_class_meta = _metadata_fb.TensorMetadataT()
output_class_meta.name = "category"
output_class_meta.description = "The categories of the detected boxes."
output_class_meta.content = _metadata_fb.ContentT()
output_class_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_class_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())
output_class_meta.content.range = _metadata_fb.ValueRangeT()
output_class_meta.content.range.min = 2
output_class_meta.content.range.max = 2
label_file = _metadata_fb.AssociatedFileT()
label_file.name = os.path.basename("label.txt")
label_file.description = "Label of objects that this model can recognize."
label_file.type = _metadata_fb.AssociatedFileType.TENSOR_VALUE_LABELS
output_class_meta.associatedFiles = [label_file]

output_score_meta = _metadata_fb.TensorMetadataT()
output_score_meta.name = "score"
output_score_meta.description = "The scores of the detected boxes."
output_score_meta.content = _metadata_fb.ContentT()
output_score_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_score_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())
output_score_meta.content.range = _metadata_fb.ValueRangeT()
output_score_meta.content.range.min = 2
output_score_meta.content.range.max = 2

output_number_meta = _metadata_fb.TensorMetadataT()
output_number_meta.name = "number of detections"
output_number_meta.description = "The number of the detected boxes."
output_number_meta.content = _metadata_fb.ContentT()
output_number_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_number_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())


# Creates subgraph info.
group = _metadata_fb.TensorGroupT()
group.name = "detection result"
group.tensorNames = [
    output_location_meta.name, output_class_meta.name,
    output_score_meta.name
]
subgraph = _metadata_fb.SubGraphMetadataT()
subgraph.inputTensorMetadata = [input_meta]
subgraph.outputTensorMetadata = [
    output_location_meta, output_class_meta, output_score_meta,
    output_number_meta
]
subgraph.outputTensorGroups = [group]
model_meta.subgraphMetadata = [subgraph]

b = flatbuffers.Builder(0)
b.Finish(
    model_meta.Pack(b),
    _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
metadata_buf = b.Output()

populator = _metadata.MetadataPopulator.with_model_file(model_file)
populator.load_metadata_buffer(metadata_buf)
populator.load_associated_files(["[path_to_labels].txt"])
populator.populate()

@chainyo I'm not sure, I'm not a TF specialist myself, but on the surface this appears to be a TFLite issue rather than a YOLOv5 issue. Perhaps raise this on the TF repo or the TF forums, or even stackoverflow.

Yes I know but I was expecting any hints from @zldrobit because he worked a lot on tflite conversions 🤗

[chainyo]

By the way, it could be a really nice add to tflite conversions. And If I find the way to do it, I would PR it 😄

[glenn-jocher]

@chainyo the 4 output format is a special TF format that they apply to older SSD models. We talked to Google (Sachin Joglekar, https://github.com/srjoglekar246) about applying it to YOLOv5 but the talks stagnated. You might want to contact him and see if he'd be interested in pointing you in the right direction or providing more official support for YOLOv5.

References:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md#step-1-export-tflite-inference-graph
https://github.com/tensorflow/models/blob/master/research/object_detection/export_tflite_graph_lib_tf2.py

[chainyo]

Thanks for links and stuff! 🤝

[zldrobit]

@chainyo Adding tf.image.non_max_suppression_padded is the most viable way to support GPU delegate, but it still has dynamic-sized tensors (#2095 (comment)), which cannot be run by GPU delegate. Actually, if you only care usage on CPU and would like to integrate an NMS op, you could replace agnostic_nms() with https://github.com/zldrobit/yolov5/blob/c9702e62d9e8fbcd225dda86626abba9f57c3f68/models/tf.py#L391-L420 in model/tf.py and the agnostic_nms_layer class with https://github.com/zldrobit/yolov5/blob/c9702e62d9e8fbcd225dda86626abba9f57c3f68/models/tf.py#L376-L388. Or just use the https://github.com/zldrobit/yolov5/tree/tflite-nms-padded branch with tf.image.non_max_suppression_padded. Be careful that the output tensor order changes after TFLite conversion, so the Android Java code needs to be updated as well. I think this might be a possible way to use metadata in TFLite models.

[chainyo]

Actually, if you only care usage on CPU

Because it will run on android mobile, yes GPU support is not needed.

I will give a look and try your hints, thanks. I keep the thread up to date. I will also try to contact Sachin Joglekar as Glenn suggested.

[chainyo]

I think this might be a possible way to use metadata in TFLite models.

That works really well with detect.py file and netron.app recognise there is 4 output tensors, but MLKit doesn't know how to use it. I think it's not implemented yet...

It seems that MLKit only works with standard models from 3 years ago...

[glenn-jocher]

@chainyo the four outputs you are showing have nothing in common with the 4 google outputs:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md#step-1-export-tflite-inference-graph
https://github.com/tensorflow/models/blob/master/research/object_detection/export_tflite_graph_lib_tf2.py

[chainyo]

@glenn-jocher It is probably because I used the YoloV5 export.py script.

I will try to use the tf converter to see if it changes anything 🤗

EDIT: With tf_converter it produces 1 output tensor model, that won't be recognize by MLKit anyways. So I think it cannot help... sadly 😞

[chainyo]

After some discussion with Google employees, in order to make custom models to fit tflite_support needs for adding metadatas, custom model outputs need to fit these outputs, with the same IO signature.

They use fused operators as described here on the documentation.

[dcboy]

@chainyo did you have success to convert tflite model for ml-kit？

[zldrobit]

@chainyo @dcboy NMS support for TFLite models has been added in #5938. You could refer to the colab notebook as an example, though detection for TFLite models with NMS is not supported yet by the master branch.