How the RK3588 NPU software stack fits together
Rockchip's RKNN software stack separates model preparation from execution on the device. RKNN-Toolkit2 provides conversion and evaluation tools. RKNN-Toolkit-Lite2 supplies a Python interface on supported target platforms, while RKNN Runtime supplies C/C++ interfaces. The kernel driver connects the runtime to the NPU. Check the compatibility of these components with the board support package supplied for your module.
A useful deployment record identifies the board and SoM revision, Linux image, driver, runtime, conversion tool version, original model and converted model. Save the build script and preprocessing settings beside the model artifact. A filename such as detector.rknn alone cannot explain why one deployment behaves differently from another.
Step 1: establish a reference before conversion
Run the original model on a representative validation set. Save raw outputs and the application metric that matters: classification accuracy, detection quality or another task-specific measure. For a detector, preserve label ordering, confidence thresholds and postprocessing settings. This becomes the comparison point for the converted model.
Write down the entire input contract. Include dimensions, batch size, RGB or BGR ordering, layout, numerical range, normalisation, resizing and padding. A correct model can produce poor results if the application letterboxes an image differently from training or subtracts an offset twice. Keep postprocessing outside the network documented just as carefully.
Use Rockchip's RKNN Model Zoo as a starting point when it includes a matching model and target. Its conversion and inference examples are more useful than assuming that any exported network will work unchanged. Select a known repository revision and follow the instructions for that specific example.
Step 2: convert the model and check operator support
- Match the target. Select the RK3588 target and a toolchain compatible with the intended board image.
- Inspect the graph. Confirm shapes, preprocessing, outputs and support for the actual operations in the exported model.
- Build a reference conversion. Use the precision options supported by that graph and software version.
- Read the complete build log. Resolve failures and investigate warnings before adding camera acquisition or a user interface.
- Compare outputs. Run the same inputs through the original and converted models, with the same postprocessing.
An unsupported operation may require a different export, graph rewrite or an explicit CPU processing stage. Do not assume that the runtime will transparently execute every unsupported layer. If part of the workload moves to the CPU, include transfers and synchronisation in the performance measurement.
Step 3: evaluate INT8 against your accuracy requirement
Quantisation changes numerical representation. When the selected conversion flow uses calibration data, choose inputs representative of the deployment: lighting, object size, background and operating conditions. Keep a separate validation set to evaluate the result. A conversion that completes successfully still needs an accuracy check.
Compare the application output, not only the difference between two tensors. For example, a small numerical change can move a detection across a confidence threshold. Inspect difficult cases, classes with few examples and conditions where the original model was already marginal. If accuracy falls below the acceptance limit, revisit calibration coverage and preprocessing before changing the hardware.
The ONNX Runtime quantisation documentation is useful background on numerical trade-offs and accuracy investigation. RKNN conversion options and supported precision must still be checked in Rockchip's own documentation for the selected release.
Step 4: measure more than inference time
Measure the deployed workload after warm-up and again during a sustained run. Record temperature, power mode, concurrent processes and the enclosure or cooling arrangement. The useful result is a reproducible performance envelope, including accuracy and delay, under the conditions in which the product will operate.
| Measurement | What it answers | What to record |
|---|---|---|
| Model execution duration | How long does this runtime call take? | Input shape, precision, timing boundaries and runtime version |
| End-to-end latency | How old is the observation when the result is delivered? | Capture time, preprocessing, queueing, inference and output time |
| Sustained throughput | How many completed results are delivered over time? | Run length, completed requests and dropped or skipped inputs |
| Application accuracy | Does the deployed model meet the task requirement? | Validation set, task metric and decision thresholds |
| Thermal behaviour | Is the result stable after the system warms up? | Ambient conditions, cooling, temperatures and clocks where available |
As an illustrative budget, an application receiving a new frame every 40 ms must keep up with 25 frames per second if it processes every frame. A 12 ms inference measurement would leave the rest of that budget for acquisition, preparation and delivery only in a suitably scheduled design. It does not prove a 12 ms camera-to-result delay. These numbers are a planning example, not measured ForgeEdge performance.
Use a monotonic clock for software durations. Report median and tail latency rather than a single best observation, and preserve the raw measurements. For several cameras, account for the workload of all active streams, including decoding, image conversion, memory movement and recording. A queue that grows throughout the test indicates that average capacity is below the incoming workload.
Common deployment symptoms
| Symptom | First investigation |
|---|---|
| Correct model, incorrect labels or boxes | Colour order, normalisation, resize/padding, output layout and postprocessing |
| Converted model cannot initialise | Target selection, runtime/driver compatibility and model artifact integrity |
| Good short benchmark, poor sustained performance | Temperature, power mode, memory pressure and competing work |
| Acceptable inference time, visibly delayed results | Capture buffering, queue age, copies and output scheduling |
| INT8 output loses useful detections | Calibration coverage, preprocessing equivalence and task-level validation |
Turn the prototype into a repeatable product
Package the model, runtime and application configuration as a controlled release. Define startup checks, failure reporting, recovery and a rollback path. Keep a small reference-input test that can identify an incorrect deployment before a camera is connected. Separate that software check from the physical sensor test, which also exercises cabling, drivers and exposure settings.
When the board needs different connectors, power conditioning or sensor interfaces, review the SoM versus SBC decision and the carrier-board design process. Those decisions determine how a successful model demonstration becomes a maintainable system.
RK3588 NPU deployment questions
Can every ONNX model run on the RK3588 NPU?
Compatibility depends on the actual graph, shapes, operations and selected toolchain. Establish support through conversion and on-device validation of the model you intend to ship.
Does an advertised accelerator rating predict application FPS?
No. The network, precision, memory behaviour, preprocessing and runtime determine the application's measured throughput. Use the target workload and its complete data path when comparing platforms.
Should I optimise the neural network first?
Profile first. If acquisition, image conversion or queueing dominates, reducing model execution time may have little effect on the result age experienced by the application.
Benchmark the complete computer vision task
For an object-detection application, measure camera capture, resize and normalization, NPU inference, output decoding and delivery of the decision. Report the model version, input resolution, precision, confidence threshold and evaluation dataset with the timing result. Otherwise, two apparently similar edge AI benchmarks may describe different tasks.
Test the exported graph and required operators through the selected RKNN toolchain before committing to a model architecture. Check accuracy after quantization and behaviour during sustained execution, not only whether conversion succeeds. Object detection, segmentation and classification have different output-processing costs; validate the actual workload chosen for the project.
Compare FPGA, NPU and MPU design roles
Technical references
Scope your edge AI workload.
Share the model, input resolution, camera interfaces, required result rate and power envelope. We can assess the integration work around an RK3588-based edge platform.