SiFive - December 20, 2017

RISC-V QEMU Part 1: Privileged ISA v1.10, HiFive1 and VirtIO

This post covers recent development in RISC-V QEMU, the open source machine emulator and virtualizer. We’ve been playing a game of catch-up with the hardware folks so that we can match the capabilities of the Freedom U500 SDK. We’re not quite there yet, but we’ve made some important improvements that will allow for a more usable emulator.

First, some background on software emulation of Instruction Set Architectures. There are several forms of emulators and they fall broadly into these categories, from fastest and least accurate to slowest and most accurate:

Emulation type

Type	Example	Performance
Functional	QEMU	100 million to >1 billion instructions per second
Trace-accurate	Spike	10 to 100 million instructions per second
Cycle-accurate	Verilator/rocket-chip	10 to 100 thousand instructions per second

Functional

QEMU is a binary translating emulator. QEMU translates RISC-V instructions to the host CPU instruction set on the fly and is therefore much faster than an interpreted simulator. However, it doesn’t provide an instruction-by-instruction trace as it spends much of its time executing native code. The focus of a binary translator is typically simulation performance and may be used for self-hosted builds.

Trace-accurate

Spike aka riscv-isa-sim is an interpreting simulator that provides an instruction-by-instruction trace accurate simulation of a RISC-V processor. Spike is the “golden reference” simulator for the RISC-V ISA, and its behavior is the reference for hardware and software. The focus of an interpreter is typically behavioral accuracy for verification.

Cycle-accurate

Cycle-accurate simulators are generally RTL (Register Transfer Level) implementations, which provide the exact cycle-by-cycle behavior of one particular RISC-V implementation. For example, Rocket’s Chisel can be compiled to Verilog, and then compiled to C++ by Verilator to create a cycle-accurate simulator. There are various levels of cycle accuracy depending on how detailed the model is, like whether it includes a detailed model of the full cache hierarchy, and DRAM using a DRAM simulator such as DRAMSimm2.

Emulation depth

There are also orthogonal simulation categories that cover the boundary and extent of the simulation:

User Mode Simulation
Full System Emulation

User Mode Simulation

The simulator emulates the instruction set architecture to run a binary from the target architecture but maps system calls to the host kernel. In this scenario, only the user-mode code of the application binary is emulated. In a binary translator, loads and stores can be translated directly and leverage the host MMU. QEMU supports user-mode simulation on Linux hosts, which allows running RISC-V Linux binaries on different host architectures.

Full System Emulation

The simulator emulates a full system, including MMU and emulated devices, such as UARTs, GPIOs, Storage and Network adapters. Full System Emulation is usually slower than User Mode Simulation due to software emulation of the MMU. In a binary translator, loads and stores need to emulate the MMU of the target system which may differ from the host. QEMU has a comprehensive device model and supports full system emulation with a variety of different devices.

Hardware emulation

The ultimate in simulation fidelity and speed besides an actual chip usually involves running a simulation on an FPGA (Field Programmable Gate Array). However, this can be relatively costly in time and synthesis; place and route are time-consuming activities often taking hours to test a single RTL change. There are, however, cases where small changes do not warrant testing with detailed hardware simulation but instead one can relay on fast functional simulation.

RISC-V Full System Emulation in QEMU

QEMU is a fast binary translator and offers both Linux User Mode Simulation and Full System Emulation for RISC-V. Given QEMU is the fastest available RISC-V simulator, it makes a lot of sense to use QEMU for tasks that would otherwise be too costly to run on simulated hardware, such as testing every commit to the RISC-V tool-chain components such as GCC, binutils or to the Linux kernel or GLIBC library. Indeed, QEMU is fast enough to simulate a Linux environment containing a build environment for self-hosted builds.

SiFive has chosen to use QEMU as its primary platform for full system emulation, and based on this, a number of new features have been added to allow modeling of a variety of different RISC-V hardware configurations including a new ‘virt’ board that supports VirtIO.

What’s New in RISC-V QEMU

Here is a brief summary of the recent changes in RISC-V QEMU:

New Support for privileged ISA v1.10 (spike_v1.10 board)
Backwards compatibility for privileged ISA v1.9.1 (spike_v1.9 board)
Parameterizable CLINT (Core Local Interruptor)
Parameterizable PLIC (Platform Level Interrupt Controller)
sifive_e300 board that can run binaries targeting the SiFive E300 SDK
sifive_u500 board that can run binaries targeting the SiFive U500 SDK
virt board with VirtIO MMIO (virtio-net, virtio-block, etc)
Device-tree config for the spike_v1.10, virt and sifive_u500 boards

RISC-V QEMU Boards

There are now 5 RISC-V boards accessible via QEMU's -machine command line option:

spike_v1.9 - priv v1.9.1 (HTIF and config-string)
spike_v1.10 - priv v1.10 (HTIF and device-tree)
sifive_e300 - priv v1.10 (SiFiveUART, HiFive1 compatible)
sifive_u500 - priv v1.10 (SiFiveUART and device-tree)
virt - priv v1.10 (16550A UART, virtio-net, virtio-block and device-tree)

Multiple specification version support

It has been decided that given there will be hardware targeting multiple revisions of the RISC-V Base ISA and Privileged ISA, the full system emulator should support multiple versions. In the latest version of QEMU, there is backwards compatibility support for Privileged ISA v1.9.1 and the addition of Privileged ISA v1.10, which adds support for the new VM mode selection mechanism (SATP), SUM (permit Supervisor User Memory access) and PMP (Physical Memory Protection).

Device tree

Privilege ISA v1.10 specifies that a RISC-V machine configuration is described using Flattened Device Tree format. QEMU already contains support for flattened device-tree, however the previous version of QEMU was using the config string specified in Privilege ISA v1.9.1. In addition to describing the machine configuration, device-tree can also contain configuration information from non-volatile memory such as console output device and boot arguments. QEMU’s -append command line option is able to pass boot command line via device-tree e.g. root device and console.

It's now possible to run the same Linux kernel binary in the latest version of QEMU and the latest version Spike. The same kernel can be used both on the QEMU spike_v1.10 board and the virt board by passing root and console options with -append.

SiFive CLINT (Core Local Interruptor)

The SiFive CLINT block holds memory-mapped control and status registers associated with software and timer interrupts. In prior versions of QEMU, there was a separate RTC and interruptor. In the latest version of QEMU, these have been combined to more closely match actual hardware, and IPIs (Inter-Processor Interrupts) have been implemented.

SiFive PLIC (Platform Level Interrupt Controller)

The SiFive platform-level interrupt controller (PLIC) prioritizes and distributes global interrupts in a RISC-V system. The QEMU PLIC implementation supports a parameterizable number of interrupt sources and priorities. A priority can be set per each interrupt source and for each target context (hart and mode). Each context has a read-write enable bitmask and a read-only pending bitmask. The new QEMU PLIC implementation is compatible with hardware on the FE310G000 SOC, the SiFive Freedom-E-SDK and the SiFive Freedom-U-SDK.

Spike Boards

QEMU supports two different Spike boards. The default board is Spike v1.9 (-machine spike_v1.9) which implements Privileged ISA Version v1.9.1 and uses config-string to pass device configuration to bbl and its linux-kernel payload. This board was kept so that existing QEMU users with existing binaries are still supported. In addition, we have added a Spike v1.10 board (-machine spike_v1.10), which implements Privileged ISA Version v1.10 and uses device-tree to pass device configuration to bbl and its linux-kernel payload. Both of the Spike boards use the RISC-V HTIF (Host Target Interface) to provide console access.

VirtIO board

A new VirtIO board (-machine virt) has been added that implements the VirtIO MMIO (Memory Mapped IO) transport and supports VirtIO Block devices, VirtIO Network devices and a virtual 16550a compatible UART for console access. The addition of the VirtIO board allows simulation of a full RISC-V system with storage volumes and networking, and is intended to provide a convenient target for Linux distributions and other Operating Systems to bootstrap self-hosted build environments. With the addition of VirtIO support, all of QEMU's various block device (raw files, qcow files, devices, etc) and network implementations (tap, user, socket, etc) become available for use by RISC-V operating systems that implement the VirtIO standard. The VirtIO board uses the PLIC to route interrupts to the virtual devices.

We have created a simple demo Linux system that uses VirtIO Block device for its root filesystem and VirtIO Networking to provide ssh access:

https://github.com/michaeljclark/busybear-linux

If you don't have a Linux system to build the busybear-linux demo root filesystem, there are pre-built binaries available:

https://github.com/michaeljclark/busybear-linux/releases

Here is an extract from the console output:

$ sudo qemu-system-riscv64 \
  -nographic -machine virt -kernel bbl \
  -append "root=/dev/vda ro console=ttyS0" \
  -drive file=busybear.bin,format=raw,id=hd0 \
  -device virtio-blk-device,drive=hd0 \
  -netdev type=tap,script=./ifup,downscript=./ifdown,id=net0 \
  -device virtio-net-device,netdev=net0

$ ssh root@192.168.100.2
The authenticity of host '192.168.100.2 (192.168.100.2)' can't be established.
ECDSA key fingerprint is 3f:4b:69:59:01:c8:b2:9c:fb:52:a5:d4:21:c9:3c:1b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.2' (ECDSA) to the list of known hosts.
root@192.168.100.2's password:
    ____                   ____                     __    _
   / __ )__  _________  __/ __ )___  ____ ______   / /   (_)___  __  ___  __
  / __  / / / / ___/ / / / __  / _ \/ __ `/ ___/  / /   / / __ \/ / / / |/_/
 / /_/ / /_/ (__  ) /_/ / /_/ /  __/ /_/ / /     / /___/ / / / / /_/ />  <
/_____/\__,_/____/\__, /_____/\___/\__,_/_/     /_____/_/_/ /_/\__,_/_/|_|
                 /____/
root@ucbvax:~# uname -a
Linux ucbvax 4.14.0-00030-gc2d852cb2f3d #56 Thu Dec 14 10:12:10 NZDT 2017 riscv64 GNU/Linux
root@ucbvax:~# cat /proc/interrupts
           CPU0
  1:        107  riscv,plic0,c000000  10  ttyS0
  7:        115  riscv,plic0,c000000   7  virtio1
  8:        135  riscv,plic0,c000000   8  virtio0
root@ucbvax:~#

SiFive Freedom E300 SDK board

An initial version of a SiFive Freedom E300 compatible board has been added to model the SiFive FE310G000 SOC. Register address space has been added to mock the AON (Always-On) block, PRCI (Power, Reset, Clock, Interrupt), PWM (Pulse Width Modulation), QSPI (Quad Serial Peripheral Interface) and a SiFive UART for console. The support is currently limited but sufficient to allow binaries from the Freedom-E-SDK to run unmodified in QEMU.

This presently involves mocking the MMIO (Memory Mapped IO) registers that are accessed by the BSP (Board Support Package) for clock and PWM programming.

HiFive1 hello example:

$ qemu-system-riscv32 -nographic -machine sifive_e300 \
 -kernel freedom-e-sdk/software/hello/hello
core freq at 16486 Hz
hello world!

Program has exited with code:0x00000000

HiFive1 led_fade example:

$ qemu-system-riscv32 -nographic -machine sifive_e300 \
 -kernel freedom-e-sdk/software/led_fade/led_fade


                SIFIVE, INC.

         5555555555555555555555555
        5555                   5555
       5555                     5555
      5555                       5555
     5555       5555555555555555555555
    5555       555555555555555555555555
   5555                             5555
  5555                               5555
 5555                                 5555
5555555555555555555555555555          55555
 55555           555555555           55555
   55555           55555           55555
     55555           5           55555
       55555                   55555
         55555               55555
           55555           55555
             55555       55555
               55555   55555
                 555555555
                   55555
                     5

               'led_fade' Demo



55555555555555555555555555555555555555555555555
5555555 Are the LEDs Changing? [y/n]  555555555
55555555555555555555555555555555555555555555555
y
PASS

SiFive Freedom U500 SDK board

The SiFive Freedom U500 board has been extended to implement device-tree support and now supports emulation of the PLIC. The SiFive Freedom U500 board is a work in progress and will be extended in future revisions to support the topology of the U54-MC SOC.

Device Tree

The spike_v1.10, virt and sifive_u500 boards all use device-tree for configuration. If you would like to inspect the device-tree, you can use the QEMU dumpdts option to output a board's device-tree binary.

Here is an example for the virt board (notice that we execute the regular QEMU command but modify the machine e.g. -machine virt,dumpdtb=virt.out):

$ sudo qemu-system-riscv64 \
  -nographic -machine virt,dumpdtb=virt.out -kernel bbl \
  -append "root=/dev/vda ro console=ttyS0" \
  -drive file=busybear.bin,format=raw,id=hd0 \
  -device virtio-blk-device,drive=hd0 \
  -netdev type=tap,script=./ifup,downscript=./ifdown,id=net0 \
  -device virtio-net-device,netdev=net0
$ fdtdump virt.out

QEMU RISC-V virt board device tree

This section shows the device-tree for the RISC-V QEMU virt board:

/dts-v1/;
// magic:        0xd00dfeed
// totalsize:        0x10000 (65536)
// off_dt_struct:    0x40
// off_dt_strings:    0x868
// off_mem_rsvmap:    0x30
// version:        17
// last_comp_version:    16
// boot_cpuid_phys:    0x0
// size_dt_strings:    0x129
// size_dt_struct:    0x828
/ {
#address-cells = <0x00000002>;
#size-cells = <0x00000002>;
compatible = "riscv-virtio";
model = "riscv-virtio,qemu";
chosen {
bootargs = "console=ttyS0 kernel=/dev/vda ro";
stdout-path = "/uart@10000000";
};
uart@10000000 {
interrupts = <0x0000000a>;
interrupt-parent = <0x00000002>;
clock-frequency = <0x00384000>;
reg = <0x00000000 0x10000000 0x00000000 0x00000100>;
compatible = "ns16550a";
};
virtio_mmio@10008000 {
interrupts = <0x00000008>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10008000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10007000 {
interrupts = <0x00000007>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10007000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10006000 {
interrupts = <0x00000006>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10006000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10005000 {
interrupts = <0x00000005>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10005000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10004000 {
interrupts = <0x00000004>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10004000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10003000 {
interrupts = <0x00000003>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10003000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10002000 {
interrupts = <0x00000002>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10002000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
virtio_mmio@10001000 {
interrupts = <0x00000001>;
interrupt-parent = <0x00000002>;
reg = <0x00000000 0x10001000 0x00000000 0x00001000>;
compatible = "virtio,mmio";
};
cpus {
#address-cells = <0x00000001>;
#size-cells = <0x00000000>;
timebase-frequency = <0x00989680>;
cpu@0 {
device_type = "cpu";
reg = <0x00000000>;
status = "okay";
compatible = "riscv";
riscv,isa = "rv64imafdc";
mmu-type = "riscv,sv48";
clock-frequency = <0x3b9aca00>;
interrupt-controller {
#interrupt-cells = <0x00000001>;
interrupt-controller;
compatible = "riscv,cpu-intc";
linux,phandle = <0x00000001>;
phandle = <0x00000001>;
};
};
};
memory@80000000 {
device_type = "memory";
reg = <0x00000000 0x80000000 0x00000000 0x08000000>;
};
soc {
#address-cells = <0x00000002>;
#size-cells = <0x00000002>;
compatible = "riscv-virtio-soc";
ranges;
interrupt-controller@c000000 {
linux,phandle = <0x00000002>;
phandle = <0x00000002>;
riscv,ndev = <0x0000000a>;
riscv,max-priority = <0x00000007>;
reg-names = "control";
reg = <0x00000000 0x0c000000 0x00000000 0x04000000>;
interrupts-extended = <0x00000001 0x0000000b 0x00000001 0x00000009>;
interrupt-controller;
compatible = "riscv,plic0";
#interrupt-cells = <0x00000001>;
};
clint@2000000 {
interrupts-extended = <0x00000001 0x00000003 0x00000001 0x00000007>;
reg = <0x00000000 0x02000000 0x00000000 0x00010000>;
compatible = "riscv,clint0";
};
};
};

Finishing up

This concludes our update on recent developments in RISC-V QEMU. Stay tuned to the SiFive blog for future updates...