Add support for object detection on Rubik Pi 3

.styleguide

@@ -19,6 +19,7 @@ modifiableFileExclude {
   \.webp$
   \.ico$
   \.rknn$
+  \.tflite$
   \.mp4$
   \.ttf$
   \.woff2$

build.gradle

@@ -40,6 +40,7 @@ ext {
     javalinVersion = "5.6.2"
     libcameraDriverVersion = "v2025.0.3"
     rknnVersion = "dev-v2025.0.0-1-g33b6263"
+    rubikVersion = "v2025.1.0"
     frcYear = "2025"
     mrcalVersion = "v2025.0.0";
 

docs/.styleguide

@@ -11,6 +11,7 @@ modifiableFileExclude {
   \.webp$
   \.ico$
   \.rknn$
+  \.tflite$
   \.svg$
   \.woff2$
   gradlew

docs/source/docs/objectDetection/about-object-detection.md

@@ -2,7 +2,7 @@
 
 ## How does it work?
 
-PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. Please note that the Orange Pi 5/5+ are the only coprocessors that are currently supported. The Neural Processing Unit, or NPU, is [used by PhotonVision](https://github.com/PhotonVision/rknn_jni/tree/main) to massively accelerate certain math operations like those needed for running ML-based object detection.
+PhotonVision supports object detection using neural network accelerator hardware, commonly known as an NPU. The two coprocessors currently supported are the {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>` and the {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`.
 
 PhotonVision currently ships with a model trained on the [COCO dataset](https://cocodataset.org/) by [Ultralytics](https://github.com/ultralytics/ultralytics) (this model is licensed under [AGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html)). This model is meant to be used for testing and other miscellaneous purposes. It is not meant to be used in competition. For the 2025 post-season, PhotonVision also ships with a pretrained ALGAE model. A model to detect coral is available in the PhotonVision discord, but will not be distributed with PhotonVision.
 
@@ -18,7 +18,7 @@ This model output means that while its fairly easy to say that "this rectangle p
 
 ## Tuning and Filtering
 
-Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). The Non-Maximum Suppresion (NMS) Threshold slider is used to filter out overlapping detections. Lower values mean more detections are allowed through, but may result in false positives. It's generally recommended that teams leave this set at the default, unless they find they're unable to get usable results with solely the Confidence slider.
+Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). The Non-Maximum Suppresion (NMS) Threshold slider is used to filter out overlapping detections. Higher values mean more detections are allowed through, but may result in false positives. It's generally recommended that teams leave this set at the default, unless they find they're unable to get usable results with solely the Confidence slider.
 
 ```{raw} html
 <video width="85%" controls>
@@ -33,22 +33,19 @@ The same area, aspect ratio, and target orientation/sort parameters from {ref}`r
 
 Photonvision will letterbox your camera frame to 640x640. This means that if you select a resolution that is larger than 640 it will be scaled down to fit inside a 640x640 frame with black bars if needed. Smaller frames will be scaled up with black bars if needed.
 
-## Training Custom Models
+It is recommended that you select a resolution that results in the smaller dimension being just greater than, or equal to, 640. Anything above this will not see any increased performance.
 
-:::{warning}
-Power users only. This requires some setup, such as obtaining your own dataset and installing various tools. It's additionally advised to have a general knowledge of ML before attempting to train your own model. Additionally, this is not officially supported by Photonvision, and any problems that may arise are not attributable to Photonvision.
-:::
+## Custom Models
 
-Before beginning, it is necessary to install the [rknn-toolkit2](https://github.com/airockchip/rknn-toolkit2). Then, install the relevant [Ultralytics repository](https://github.com/airockchip?tab=repositories&q=yolo&type=&language=&sort=) from this list. After training your model, export it to `rknn`. This will give you an `onnx` file, formatted for conversion. Copy this file to the relevant folder in [rknn_model_zoo](https://github.com/airockchip/rknn_model_zoo), and use the conversion script located there to convert it. If necessary, modify the script to provide the path to your training database for quantization.
+For information regarding converting custom models and supported models for each platform, refer to the page detailing information about your specific coprocessor.
 
-## Managing Custom Models
+- {ref}`Orange Pi 5 <docs/objectDetection/opi:Orange Pi 5 (and variants) Object Detection>`
+- {ref}`Rubik Pi 3 <docs/objectDetection/rubik:Rubik Pi 3 Object Detection>`
 
-:::{warning}
-PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care.
-:::
+### Training Custom Models
 
-:::{warning}
-Non-quantized models are not supported! If you have the option, make sure quantization is enabled when exporting to .rknn format. This will represent the weights and activations of the model as 8-bit integers, instead of 32-bit floats which PhotonVision doesn't support. Quantized models are also much faster for a negligible loss in accuracy.
-:::
+PhotonVision does not offer any support for training custom models, only conversion. For information on which models are supported for a given coprocessor, use the links above.
+
+### Managing Custom Models
 
-Custom models can now be managed from the Object Detection tab in settings. You can upload a custom model by clicking the "Upload Model" button, selecting your `.rknn` file, and filling out the property fields. Models can also be exported, both individually and in bulk. Models exported in bulk can be imported using the `import bulk` button. Models exported individually must be re-imported as an individual model, and all the relevant metadata is stored in the filename of the model.
+Custom models can now be managed from the Object Detection tab in settings. You can upload a custom model by clicking the "Upload Model" button, selecting your model file, and filling out the property fields. Models can also be exported, both individually and in bulk. Models exported in bulk can be imported using the `import bulk` button. Models exported individually must be re-imported as an individual model, and all the relevant metadata is stored in the filename of the model.

docs/source/docs/objectDetection/index.md

@@ -1,8 +1,8 @@
 # Object Detection
 
 ```{toctree}
-:maxdepth: 0
-:titlesonly: true
 
 about-object-detection
+opi
+rubik
 ```

docs/source/docs/objectDetection/opi.md

@@ -0,0 +1,19 @@
+# Orange Pi 5 (and variants) Object Detection
+
+## How it works
+
+PhotonVision runs object detection on the Orange Pi 5 by use of the RKNN model architecture, and [this JNI code](https://github.com/PhotonVision/rknn_jni).
+
+## Supported models
+
+PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 SOCs! Other models require different post-processing code and will NOT work.
+
+## Converting Custom Models
+
+:::{warning}
+Only quantized models are supported, so take care when exporting to select the option for quantization.
+:::
+
+PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rknn-convert-tool/rknn_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rknn_conversion.ipynb` notebook without needing to manually download anything.
+
+Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/opi:Supported Models>` and using the PyTorch format.

docs/source/docs/objectDetection/rubik.md

@@ -0,0 +1,25 @@
+# Rubik Pi 3 Object Detection
+
+## How it works
+
+PhotonVision runs object detection on the Rubik Pi 3 by use of [TensorflowLite](https://github.com/tensorflow/tensorflow), and [this JNI code](https://github.com/PhotonVision/rubik_jni).
+
+## Supported models
+
+PhotonVision currently ONLY supports 640x640 Ultralytics YOLOv8 and YOLOv11 models trained and converted to `.tflite` format for QCS6490 SOCs! Other models require different post-processing code and will NOT work.
+
+## Converting Custom Models
+
+:::{warning}
+Only quantized models are supported, so take care when exporting to select the option for quantization.
+:::
+
+PhotonVision now ships with a [Python Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/rubik_conversion.ipynb) that you can use in [Google Colab](https://colab.research.google.com) or in a local environment. In Google Colab, you can simply paste the PhotonVision GitHub URL into the "GitHub" tab and select the `rubik_conversion.ipynb` notebook without needing to manually download anything.
+
+Please ensure that the model you are attempting to convert is among the {ref}`supported models <docs/objectDetection/rubik:Supported Models>` and using the PyTorch format.
+
+## Benchmarking
+
+Before you can perform benchmarking, it's necessary to install `tensorflow-lite-qcom-apps` with apt.
+
+By SSHing into your Rubik Pi and running this command, replacing `PATH/TO/MODEL` with the path to your model, `benchmark_model --graph=src/test/resources/yolov8nCoco.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options=backend_type:htp --external_delegate_options=htp_use_conv_hmx:1 --external_delegate_options=htp_performance_mode:2` you can determine how long it takes for inference to be performed with your model.

photon-client/src/components/dashboard/tabs/ObjectDetectionTab.vue

@@ -90,7 +90,7 @@ const selectedModel = computed({
       class="pt-2"
       :slider-cols="interactiveCols"
       label="NMS Threshold"
-      tooltip="The Non-Maximum Suppression threshold used to filter out overlapping detections. Lower values mean more detections are allowed through, but may result in false positives."
+      tooltip="The Non-Maximum Suppression threshold used to filter out overlapping detections. Higher values mean more detections are allowed through, but may result in false positives."
       :min="0"
       :max="1"
       :step="0.01"

photon-client/src/components/settings/ObjectDetectionCard.vue

@@ -8,7 +8,6 @@ import pvInput from "@/components/common/pv-input.vue";
 import { useTheme } from "vuetify";
 
 const theme = useTheme();
-
 const showImportDialog = ref(false);
 const showInfo = ref({ show: false, model: {} as ObjectDetectionModelProperties });
 const confirmDeleteDialog = ref({ show: false, model: {} as ObjectDetectionModelProperties });
@@ -303,11 +302,33 @@ const handleBulkImport = () => {
             <v-card color="surface" dark>
               <v-card-title class="pb-0">Import New Object Detection Model</v-card-title>
               <v-card-text>
-                Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
-                640x640 YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 CPUs are
-                currently supported!
+                <span v-if="useSettingsStore().general.supportedBackends?.includes('RKNN')"
+                  >Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
+                  640x640 YOLOv5, YOLOv8, and YOLOv11 models trained and converted to `.rknn` format for RK3588 SOCs are
+                  currently supporter!</span
+                >
+                <span v-else-if="useSettingsStore().general.supportedBackends?.includes('RUBIK')"
+                  >Upload a new object detection model to this device that can be used in a pipeline. Note that ONLY
+                  640x640 YOLOv8 and YOLOv11 models trained and converted to `.tflite` format for QCS6490 compatible
+                  backends are currently supported!
+                </span>
+                <span v-else>
+                  If you're seeing this, something broke; please file a ticket and tell us the details of your
+                  situation.</span
+                >
                 <div class="pa-5 pb-0">
-                  <v-file-input v-model="importModelFile" variant="underlined" label="Model File" accept=".rknn" />
+                  <v-file-input
+                    v-model="importModelFile"
+                    variant="underlined"
+                    label="Model File"
+                    :accept="
+                      useSettingsStore().general.supportedBackends?.includes('RKNN')
+                        ? '.rknn'
+                        : useSettingsStore().general.supportedBackends?.includes('RUBIK')
+                          ? '.tflite'
+                          : ''
+                    "
+                  />
                   <v-text-field
                     v-model="importLabels"
                     label="Labels"
@@ -321,7 +342,11 @@ const handleBulkImport = () => {
                     v-model="importVersion"
                     variant="underlined"
                     label="Model Version"
-                    :items="['YOLOv5', 'YOLOv8', 'YOLO11']"
+                    :items="
+                      useSettingsStore().general.supportedBackends?.includes('RKNN')
+                        ? ['YOLOv5', 'YOLOv8', 'YOLO11']
+                        : ['YOLOv8', 'YOLO11']
+                    "
                   />
                   <v-btn
                     color="buttonActive"

photon-client/src/types/SettingTypes.ts

@@ -20,7 +20,7 @@ export interface ObjectDetectionModelProperties {
   labels: string[];
   resolutionWidth: number;
   resolutionHeight: number;
-  family: "RKNN";
+  family: "RKNN" | "RUBIK";
   version: "YOLOV5" | "YOLOV8" | "YOLOV11";
 }
 

photon-core/build.gradle

@@ -36,6 +36,12 @@ dependencies {
     implementation("org.photonvision:rknn_jni-java:$rknnVersion") {
         transitive = false
     }
+    implementation("org.photonvision:rubik_jni-jni:$rubikVersion:linuxarm64") {
+        transitive = false
+    }
+    implementation("org.photonvision:rubik_jni-java:$rubikVersion") {
+        transitive = false
+    }
     implementation("org.photonvision:photon-libcamera-gl-driver-jni:$libcameraDriverVersion:linuxarm64") {
         transitive = false
     }