Breakthrough in Automotive AI: Running BEVFormer on SiFive’s Early Access RISC-V Intelligence XM Platform

Pavel Chupin, Senior Director of AI Software, SiFive

We’re excited to share the successful deployment of the BEVFormer model on SiFive’s Intelligence XM IP, an early access RISC-V-based AI acceleration solution. This milestone highlights the potential of open-standard hardware to handle complex perception workloads for autonomous driving.

BEVFormer: A Proven Model, A New Challenge

BEVFormer, introduced in 2022, is a well-established transformer-based Bird’s Eye View (BEV) perception model critical for autonomous driving. It transforms six 900x1600 camera inputs into a 200x200 BEV representation, enabling robust 3D object detection, spatial reasoning, and temporal fusion.

While BEVFormer itself is not new, the real accomplishment lies in bringing this sophisticated model to life on the SiFive Intelligence XM Platform, an emerging, early access RISC-V architecture. This deployment demonstrates the platform’s ability to tackle demanding AI workloads, setting a foundation for future automotive innovations.

The Power of RISC-V and SiFive’s Intelligence XM IP Automotive AI requires compute platforms that offer high performance, customization, and power efficiency for long-lifecycle systems. Traditional proprietary processors often fall short in flexibility. RISC-V’s open-standard Instruction Set Architecture (ISA) addresses this by allowing tailored processor designs for specific workloads, such as neural inference and real-time control.

The SiFive Intelligence XM Platform, still in its early access phase, builds on this with a multi-core design featuring four X-series cores (each with a 1024-bit RISC-V vector unit) and a Matrix Engine for efficient matrix operations. Its tightly integrated architecture minimizes latency and eliminates fallback penalties common in heterogeneous systems. This enabled seamless execution of BEVFormer’s diverse modules: convolutional layers, deformable attention, and scalar operations, within a unified binary, simplifying development and boosting performance.

Conquering Deployment Hurdles

Running BEVFormer on early access IP was a complex endeavor, requiring innovative solutions to technical challenges. Using the SiFive AI/ML reference software stack, powered by IREE, our team enabled and implemented a few optimizations in the model for the XM Platform. Key steps included: Model Conversion: Addressing PyTorch 1.9.1 and OpenMMLab compatibility issues to convert BEVFormer to ONNX and then to MLIR using IREE’s iree-import-onnx tool.

• Custom Operator Support: Implementing critical operators like Modulated_Deform_Conv2D and Multi_Scale_Deformable_Attention as optimized microkernels, leveraging RISC-V Vector (RVV) intrinsics for performance.

• CUDA-to-RISC-V Porting: Translating CUDA kernels to portable C/C++ and RVV code, with Large Language Models assisting in prototyping to save engineering effort.

• Matrix Engine Optimization: Mapping matrix-heavy operations (e.g., matmul) to the Matrix Engine via IREE’s mmt4d framework, ensuring hardware acceleration.

The pipeline was validated on the nuScenes dataset using QEMU emulation of the XM Platform, achieving identical accuracy to the PyTorch baseline. Visualized outputs confirmed the model’s precision in generating BEV representations, proving the platform’s capability despite its early stage.

A Foundation for the Future

This achievement is not about BEVFormer’s novelty but about showcasing the SiFive Intelligence XM Platform’s ability to handle complex AI workloads through SiFive AI/ML reference software stack. By successfully deploying a mature model like BEVFormer, we’ve demonstrated RISC-V’s potential to rival traditional automotive compute solutions with greater flexibility and customization. SiFive AI/ML reference software stack, powered by IREE opens a potential for experimentation with more AI/ML models. Components, based on existing open source technologies are welcome to explore and reuse.

Our team is actively refining performance through kernel tuning and enhanced code generation. As RISC-V gains momentum in automotive AI, this milestone positions the SiFive platform as a game-changer for scalable, open-standard solutions.

Dive Deeper and Join Us

For a detailed look at this deployment, read the full technical paper: Deploying BEV Perception Models on RISC-V: A Practical Look into Next-Gen Automotive AI on the SiFive Intelligence XM Platform. You can also explore the BEVFormer codebase on GitHub.

Interested in evaluating the SiFive Intelligence XM Platform or collaborating on automotive AI? SiFive has the industry’s broadest set of Automotive RISC-V solutions.

Contact us Let’s shape the future of autonomous driving together!