YOLOBench: Evaluating Object Detection Models on Embedded Systems for Optimal Performance
Published on Sat Aug 05 2023
2842228-40 Raspberry pi | Wutthichai Charoenburi on FlickrResearchers have developed YOLOBench, a benchmarking tool that evaluates the performance of object detection models on different embedded hardware platforms. The tool compares over 550 YOLO-based models on four different datasets and four hardware platforms, including x86 CPU, ARM CPU, Nvidia GPU, and NPU. By collecting accuracy and latency data, the researchers found that various YOLO architectures, including older models like YOLOv3 and YOLOv4, can achieve a good trade-off between accuracy and latency when modern detection heads and training techniques are used.
The study also evaluated training-free accuracy estimators commonly used in neural architecture search. While most of these estimators were outperformed by a simple baseline measure called MAC count, the researchers found that the NWOT estimator could effectively identify potential Pareto-optimal YOLO detectors without the need for training. As a result, researchers were able to identify a YOLO-like model with an FBNetV3 backbone that outperformed the state-of-the-art YOLOv8 model on a Raspberry Pi 4 ARM CPU.
The findings of this research are valuable for developers and researchers working on object detection models for embedded systems. They provide valuable insights into the effectiveness of different YOLO architectures on various hardware platforms, as well as the potential to identify optimal detectors using training-free accuracy estimators. With the increasing demand for efficient object detection models in applications like autonomous vehicles, surveillance, robotics, and augmented reality, tools like YOLOBench can help optimize the performance of these models on low-power devices. The code and data of YOLOBench are publicly available for further exploration and use.
