Jan 30, 2023

Booting QEMU Cubieboard from Yocto SD card

This is part 6 of the QEMU Board Emulation post series.

Posts in this post series so far have been focused on the Versatile Express board, since it has wide range of supported peripherals and is easy to start with.

I have recently worked on contributing to Allwinner A10 and Cubieboard support in QEMU. I have added several components (clock controller module and DRAM controller) as well as I2C controller in order to be able to execute U-Boot and SPL code from SD card. By reusing the bootrom approach from Allwinner H3, the Cubieboard can now be started just by supplying the SD card image, and it will perform boot similar to the one executed on a real board.

In this blog post I will focus on Yocto support for Cubieboard and how to use new QEMU support. I will also extend Cubieboard with the previously developed I2C sensor peripheral, to test that application can be shared between Vexpress and Cubieboard.

Jan 15, 2023

QEMU custom devices - Lessons learned

In the Learning Linux device driver development using QEMU blog series I showed examples of memory-mapped and I2C QEMU devices.

In the mean time I have been working on contributing to QEMU. I focused on Allwinner support, specifically on Allwinner A10 support. My idea was to use similar approach to the one implemented for OrangePi-PC and Allwinner-H3, where user can pass an SD card image and QEMU can perform the complete boot sequence.

The payches have been merged and review process is available here:


While working on these patches, the reviewers had very useful comments and suggestions. Based on those, I have also updated the patches used in the Learning Linux device driver development using QEMU blog series, to match the latest QEMU version and improve the coding standard. The updated patches are in the qemu-7.2 branch of the QEMU custom peripherals repository.

Nov 6, 2022

Qt6 with Yocto for Vexpress A9 in QEMU

 Adding meta-qt5 layer and building a simple application that is run on boot.

First the examples, then a simple application

Switch to systemd?

This is part 5 of the QEMU Board Emulation post series.

In the previous post I showed how to use Yocto to build an image for Vexpress-A9 board. Yocto allows us to build a custom distribution and to streamline bootloader and kernel cross-compilation and handling.

In the Linux device driver development post series I showed custom memory-mapped and I2C devices for QEMU, and device driver and userspace application development for those devices. The userspace applications that were presented are usable from command line only, to demonstrate the desired functionality.


In real embedded systems, the use of non-GUI (command line only) applications is enough if the unit has no display. In that case data can be published to the outside world over some interface (usually network interface). We will cover this in some of the future posts.

However, there are a lot of cases where the embedded system has a display, and in that case the userspace application should provide graphical interface. There are different libraries that enable graphical interface to be shown, but in this post we will use Qt, version 6. We will integrate it into Yocto development using the meta-qt6 layer.

Items that will be covered in this post are

Sources for the Yocto build environment for Vexpress-A9 can be found in following repositories:

Qt6 application development

The application is developed in Qt6, using combination of C++ and QML. Qt provides many classes which provide different functionality and simplify development.

One of the most interesting features are the signals and slots mechanism, where asynchronous communication between threads of execution is provided with loose coupling. One class can provide a slot function, essentially a callback, and another class can provide a signal function, and then signal and slot have to be connected. Once signal is emitted, the slot function is called.

We will go into more details soon, and the source code of the developed application can be found in the qt6-sens-app github repository.

Ideas

The goal of this application is to provide graphical interface towards the custom memory-mapped and I2C QEMU devices developed in the Linux device driver development post series.

The user interface should be simple, one part showing items related to the memory-mapped sensor and the other part showing items related to the I2C sensor, thus combining functionalities provided by mmsens-app and i2csens-app.

I will try to reuse most of the functionalities from the existing classes, but updating them to use Qt-specific types (like QString instead of std::string). The biggest change for the existing implemented classes will be that they will inherit from QObject in order to be able to use the signals and slots functionality.

Handling memory-mapped sensor

The mmsens-app has a separate thread where poll syscall is used to block thread execution until sysfs_notify is executed on the interrupt sysfs attribute.

In this case, Qt6 provides the QFileSystemWatcher class which provides similar functionality: the QFileSystemWatcher can be configured to track a state of a certain file until notify event is received, and then can emit the fileChanged signal, when read of data register can be performed.

This way we can just set up correct signals and slots and we don't need to create additional explicit thread.

As for the contol, we should be able to

  • enable memory-mappped sensor and interrupt generation
  • select sampling frequency
  • present last read value

Handling I2C sensor

The i2csens-app has a separate thread which blocks until timeout period expires, and then reads the TEMPERATURE register.

Using Qt we can use the QTimer class. In this case, when QTimer expires it emits the timeout signal, so we can use that to trigger read.

Again, we can only set up correct signals and slots and we don't need to create additional explicit thread.

As for the contol, we should be able to

  • enable I2C sensor
  • show last read I2C value

In this case, since the value is 'temperature', we will use line chart to show last 10 measured values.

QML interaction

QML is used for frontend and C++ for backend implementation. They need to interact and exchange data, and there are multiple ways to do it.

In this case I wanted to make a loose coupling between QML and C++, so I just used signals/slots mechanism for exchanging data between QML and C++.

This is a simple application, so using only signals/slots was possible. In more complex situations some other mechanism presented in the link above may have to be used in order to provide full functionality.

Implemented GUI is shown in the following picture

Application design

The class organization is presented in the following picture. Manager class represents glue between QML and C++ classes, routing signals in both directions. Poller class implements functionality related to the memory-mapped sensor, while I2CHandler class implements functionality related to the I2C sensor.

An example interaction for memory-mapped sensor and I2C sensor are shown in the following pictures.

Using meta-qt6 Yocto layer

In order to simplify cross-compilation we will use Yocto. The meta-qt6 layer provides all needed dependencies.

However, before the layer could be added to the bblayers.conf file, we need to update Yocto to kirkstone version to be compatible with the meta-qt6 layer version 6.4.

Upgrading Yocto to kirkstone

Considering we have only few recipes in the meta-vexpress and meta-vexpress-apps repositories, the upgrade process mostly consisted in updating the layer specification in layer.conf to add kirkstone as supported version, and in updating append/prepend/overrides to use : instead of _ as separator (change that happened in honister).

Adding new layer

After we have updated Yocto to kirkstone, we can add the layer just by updating bblayers.conf with the following line

${TOPDIR}/../sources/meta-qt6 \

Adding qt6-sens-app recipe

The recipe for the qt6-sens-app is similar to the ones used for other userspace applications.

Main difference are the Qt-related items

inherit qt6-cmake

DEPENDS += " qtbase qtdeclarative qtdeclarative-native "
RDEPENDS:${PN} += " qtcharts-qmlplugins ttf-dejavu-sans "

The qtbase and qtdeclarative provide base Qt C++ and QML support. Since we are using QtCharts (QML version) we also need to add qtcharts-qmlplugins.

Base image does not have fonts installed, so I added ttf-dejavu-sans to be able to render text.

Testing

Building and running application

The Yocto image is built using the following command

# framebuffer image build command
$ DISTRO=mistra-framebuffer MACHINE=vexpress-qemu bitbake mistra-image

After image is built, copied to the SD card and QEMU is run, the following output will be visible

# QEMU output
$ qemu-system-arm -M vexpress-a9 -m 1G -kernel u-boot.elf \
                  -drive file=sd.img,format=raw,if=sd \
                  -net nic -net tap,ifname=qemu-tap0,script=no \
                  -serial mon:stdio
[ ... ]
Mistra FrameBuffer 4.0 vexpress-qemu /dev/ttyAMA0
vexpress-qemu login:

The application can be started using

$ qt6-sens-app

The application UI will be shown and we can interact with it.

Simple testing scenario that demonstrates application functionality is shown in the following image

Summary

In this blog post the Yocto image is extended with Qt6 functionality. A new application for interaction with custom memory-mapped and I2C sensor is implemented and it's functionality demonstrated.

Oct 18, 2022

Custom I2C QEMU peripheral and userspace application

This is part 4 of the Linux device driver development post series.

After showing custom memory-mapped sensor and it's handing in previous posts, we will now take a look at I2C peripheral.

In this post we will cover the following things

Custom I2C QEMU peripheral

Emulating I2C peripheral under QEMU is not much harder then emulating a memory-mapped peripheral.

The peripheral needs to be connected to an existing I2C controller, assigned and I2C address and should provide responses when data is written to or read from I2C bus.

Peripheral description

In this example we will implement a simple I2C temperature sensor. Once enabled, it will return a random value in the range of 15.0 to 25.0 with 0.5 degrees (Celsius) step on every read.


The peripheral has following registers

offset name description
0x0 ID Unique ID register, always returning value 0x5A
0x1 CONFIG Configuration register, used to enable temperature temperature
0x2 TEMPERATURE Data register holding current temperature. The value is coded as Q5.1 to be able to hold 0.5 degree step

Bit values of CONFIG register are shown in the following table

name pos dflt description
Reserved 7:1 0 Reserved
EN 0 0 Enable device

Peripheral implementation

The implementation of the peripheral is available in github repository.

Functions of interest are the ones related to data handling. The global structure is defined in the following way

typedef struct I2CSensor {
  /*< private >*/
  I2CSlave i2c;
  /*< public >*/
  uint8_t regs[NR_REGS];  // peripheral registers
  uint8_t count;          // counter used for tx/rx
  uint8_t ptr;            // current register index
} I2CSensor;

Writing to the device

The only register that supports writing is the CTRL register.

In order to set the register that will be written, first I2C write must be the register address (that will update the ptr field), and second write must be the value that should be written.

static int i2c_sens_tx(I2CSlave *i2c, uint8_t data)
{
  I2CSensor *s = I2C_SENS(i2c);

  if (s->count == 0) {
    /* store register address */
    s->ptr = data;
    s->count++;
  } else {
    if (s->ptr == REG_CTRL_OFFSET) {
      s->regs[s->ptr++] = data;
    }
  }
  return 0;
}

Reading from device

Reading process is similar to writing: first one byte must be written to set the address of register to be read, and then the reading can proceed.

All registers support reading so it is just a matter of returning current register value.

/* Called when master requests read */
static uint8_t i2c_sens_rx(I2CSlave *i2c)
{
  I2CSensor *s = I2C_SENS(i2c);
  uint8_t ret = 0xff;

  if (s->ptr < NR_REGS) {
    ret = s->regs[s->ptr++];
  }

  return ret;
}

We also want each read to trigger loading of random value to the TEMPERATURE register. That is achieved by defining an event callback which modifies the s->regs[2] value when I2C_START_RECV event is received.

The register is modified only if device is currently enabled, otherwise a value of 0xff will be stored.

static int i2c_sens_event(I2CSlave *i2c, enum i2c_event event)
{
  I2CSensor *s = I2C_SENS(i2c);

  if (event == I2C_START_RECV) {
    if (s->ptr == REG_TEMPERATURE_OFFSET) {
      if (s->regs[REG_CTRL_OFFSET] & REG_CTRL_EN_MASK) {
        s->regs[REG_TEMPERATURE_OFFSET] = i2c_sens_get_temperature();
      } else {
        s->regs[REG_TEMPERATURE_OFFSET] = 0xff;
      }
    }
  }

  s->count = 0;

  return 0;
}

Integrating into QEMU

Similarly to the Memory-mapped peripheral, the new I2C peripheral must be integrated into the Versatile Express description. This time, instead of attaching to a place in memory map, it needs to be attached to an I2C controller at a selected I2C address.

We can choose either to use an already existing I2C controller, or to also add a new I2C controller in the memory map, and then attach this component to the I2C controller.

In this example we will add a new I2C controller at address 0x10010000 and attach our custom I2C component to that I2C controller. The I2C slave address is chosen as 0x36.

The excerpt from initialization of emulated Versatile Express board is

  dev = sysbus_create_simple(TYPE_VERSATILE_I2C, map[VE_I2C_SENSOR], NULL);
  i2c = (I2CBus *)qdev_get_child_bus(dev, "i2c");
  i2c_slave_create_simple(i2c, "mistra.i2csens", 0x36);

I2C Userspace handling

With the device created in QEMU we can turn to making a userspace application to access the I2C device.

The program flow will be similar to the one used for memory-mapped device

  1. Device should be initialized
  2. After device is initialized, we should periodically read the temperature data

The full userspace application can be found in github.

Initialization

The device initialization now involves setting up the I2C userspace communication. Everything is done using the dev file for the selected I2C controller. In our case it is /dev/i2c-2.

The process can be summarized as

  1. Opening the I2C controller device file
    fd = open("/dev/i2c-2", O_RDWR);
    
  2. Configuring I2C slave address
    ioctl(fd, I2C_SLAVE, 0x36);
    
  3. Enabling device by writing 1 to CTRL register. Since this write should be performed in two steps, an array consisting of CTRL register offset (0x01) and value to be written (1) is passed to the write function
    uint8_t buffer[2] = { 0x01, 0x01 };
    write(fd, buffer, 2);
    

Periodic execution

After the device is initialized, we can use a separate thread to periodically initiate reads from I2C device. Unlike the memory-mapped device, this peripheral has no interrupt generation possibilities, so we need to make periodic execution in some other way.

We will use std::this_thread::sleep_until to enable periodic readouts from the I2C peripheral.

The read process also consists of two steps: writing TEMPERATURE register offset and then reading the TEMPERATURE register value.

while (m_running)
{
  // Read current time so we know when to timeout
  current_time = std::chrono::system_clock::now();

  // read I2C device and print
  // first write address
  uint8_t buffer[1] = { 0x02 };
  write(fd, buffer, 1);
  // then read value
  read(fd, buffer, 1);

  std::cout << "Measured " << (buffer[0] / 2.) << std::endl;

  // sleep_until
  std::this_thread::sleep_until(current_time + std::chrono::seconds(1));
}

Testing in QEMU

i2c-tools

Before testing the userspace application, we need to test that device is integrated properly inside QEMU. For that we can use i2c-tools package (add it Yocto or install in Ubuntu rootfs).

Once i2c-tools are installed, we can use i2cdetect to verify that a peripheral at 0x36 address exists using

$ i2cdetect -y 2
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- 36 -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- -- 

The i2cget can be used to read a value and i2cset to set certain value.

We can check that ID register returns 0x5A

$ i2cget -y 2 0x36 0
0x5a

The last test would be to turn on the device and check that TEMPERATURE register returns different values every time

$ i2cset -y 2 0x36 1 1
$ i2cget -y 2 0x36 2
0x22
$ i2cget -y 2 0x36 2
0x25
$ i2cget -y 2 0x36 2
0x2c

Userspace application

Userspace application can be compiled in the same manner as the memory-mapped application, via external toolchain or using a Yocto SDK generated toolchain:

$ mkdir build && cd build
$ cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain.cmake ..
$ make -j

or

$ source /opt/mistra-framebuffer/3.1/environment-setup-armv7at2hf-neon-mistra-linux-gnueabi
$ mkdir build && cd build
$ cmake ..
$ make -j

Alternatively, application can be added to the Yocto build system with a recipe shown in github.


After the application has been built and copied to the rootfs, it can be started. The application will initialize the I2C peripheral and start printing read temperature value every second.

Summary

In this blog post userspace application development for custom QEMU I2C sensor is presented.

The application encapsulates the I2C userspace access and implements periodic reading of data.

It can be improved with unit tests, as well as more interactive behavior, but it is left for some other time.

Oct 8, 2022

Userspace application for custom QEMU memory-mapped peripheral

This is part 3 of the Linux device driver development post series.

In the previous post structure of a Linux device driver for the custom memory-mapped peripheral in QEMU was shown.

In this post we will cover the following things

Developing userspace application for designed memory-mapped device

The driver that was developed in the previous post provides interface to our custom memory-mapped peripheral. What remains is to make an application that will interact with the hardware using that driver.

The application should be able to initialize the device and then block until interrupts arrive, when it should capture the value from the data register and print it.

Device initialization

In order to initialize the device the application needs to:

  1. turn it on via EN bit;
  2. select desired frequency via FREQ bit;
  3. enable interrupt generation via IEN bit.

Since all of these bits are accessible via implemented sysfs attributes, the application should access this files and perform appropriate writes.

After device is initialized, the application will go into an infinite loop waiting for interrupt to happen.

Interrupt handling (poll)

The title of this subsection might be misleading, since interrupt handling is done in the kernel driver code. However, the device driver only acknowledges the interrupt and clears interrupt flag, so interrupt processing is left to the application code. The notification from device driver to user space is implemented via sysfs_notify function on interrupt sysfs attribute.

In order for user space application to be able to be woken up when interrupt happens we will use poll syscall. It will make application thread to block while waiting for notification from device driver. Once interrupt is received, the application will be woken up and continue processing.

As for the processing, the application should read the data register (the processed value provided by the sysfs attribute) and print the value that was read.

Coding ideas

In this case I opted for using C++/CMake implementation presented in github repository.

The low-level access to sysfs attribute files and poll syscall is implemented in the Poller class. This class handles initialization and interrupt notification so the higher level code can only worry about handling of read data.

The Poller class will provide method to do the device initialization. It will also provide a method to start a thread where it uses poll syscall to wait for notification from sysfs attribute interrupt. Once notification is received, Poller will read the data sysfs attribute and print read value.

After Ctrl+c is pressed the application will call method from Poller to disable interrupt generation and turn off the device.

Building application manually

Using arm-none-linux-gnueabihf

CMake is chosen as the build system. Since we are cross-compiling for ARM architecture, we need to provide the toolchain.cmake file, which describes everything related to cross-compilation toolchain.

An example for arm-none-linux-gnueabihf toolchain is in github repository.

When setting up the build system, we will pass that toolchain file to cmake using -DCMAKE_TOOLCHAIN_PATH. The rest of the process is the standard CMake flow.

$ mkdir build && cd build
$ cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain.cmake ..
$ make -j

The resulting mmsens-app binary file will be in the build directory.

Using Yocto SDK

Yocto can be used to create and SDK which includes all required development libraries and build tools.

In order to build the SDK, process similar to building image (presented in part 4 of QEMU Board emulation) with additinal parameter of -c populate_sdk.

$ source setup-environment build_framebuffer
$ DISTRO=mistra-framebuffer MACHINE=vexpress-qemu bitbake core-image-test -c populate_sdk

The resulting toolchain file will be in build_framebuffer/tmp/deploy/sdk/, the mistra-framebuffer-glibc-x86_64-core-image-test-armv7at2hf-neon-vexpress-qemu-toolchain-3.1.sh.

Toolchain and SDK is installed using

$ ./mistra-framebuffer-glibc-x86_64-core-image-test-armv7at2hf-neon-vexpress-qemu-toolchain-3.1.sh
Mistra FrameBuffer SDK installer version 3.1
============================================
Enter target directory for SDK (default: /opt/mistra-framebuffer/3.1):
You are about to install the SDK to "/opt/mistra-framebuffer/3.1". Proceed [Y/n]?
Extracting SDK...........................................done
Setting it up...done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
 $ . /opt/mistra-framebuffer/3.1/environment-setup-armv7at2hf-neon-mistra-linux-gnueabi

From our mmsens-app we can initiate the build of the application in the following way

$ source /opt/mistra-framebuffer/3.1/environment-setup-armv7at2hf-neon-mistra-linux-gnueabi
$ mkdir build && cd build
$ cmake ..
$ make -j

There is no need to pass the toolchain file since sourcing environment script already does that for us.

Adding application to Yocto

Before we go to testing, we shall look into writing recipe for adding this application to the Yocto build.

The recipe should inherit cmake which will indicate to bitbake that CMake is used for building.

Summary of the recipe is presented here and full recipe can be found in meta-vexpress-apps.

DESCRIPTION = "Memory-mapped sensor userspace application"

SRC_URI = "git://github.com/straxy/mmsens-app.git;protocol=https;branch=main"

S = "${WORKDIR}/git"

inherit cmake

Only other thing needed is to add mmsens-app to IMAGE_INSTALL variable in the image recipe, and the application will be included in the final output image.

After recipe is added, the image can be built and loaded by following instructions from part 4 of QEMU Board emulation.

Testing application

If application is built manually, it needs to be copied to the rootfs of the SD card image following part 2 of QEMU Board emulation.

If application is build as part of the Yocto image, it will be included in the rootfs and located in /usr/bin.

Once application is in rootfs, it just needs to be executed using ./mmsens-app (if not in $PATH) or mmsens-app (if in $PATH).

The application will print the read value from the data register, and can be gracefully stopped by pressing Ctrl+c.

$ ./mmsens-app /sys/class/mmsens/mmsens0
Hello World!
[   51.670787] Interrupt received
Initial 0001
[   52.667802] Interrupt received
0002
[   53.667079] Interrupt received
0003
[   54.667250] Interrupt received
0004
[   55.666935] Interrupt received
0005
[   56.667187] Interrupt received
0006
^C[   57.666989] Interrupt received
0007
$

Running same application from Yocto image inside QEMU

Summary

In this blog post userspace application development for custom QEMU memory mapped sensor is presented.

The application encapsulates the sysfs files access and can block until interrupt is received so it can process it.

It can be improved with unit tests, as well as more interactive behavior, but it is left for some other time.

Aug 14, 2022

Developing Linux device driver for QEMU custom memory-mapped device

This is part 2 of the Linux device driver development post series.

In the previous post I presented the steps for creating a custom memory-mapped peripheral in QEMU.

In this post I will cover the following things

Developing character device driver for the designed memory-mapped device

The driver for the custom memory-mapped device should provide interface for user space applications to use the custom-memory mapped peripheral. This means that all bit-fields can be read and modified by a user-space application.

The device first needs to be 'recognized' by the system for which we will use Device-tree mapping and platform driver structures.

After device is recognized by the system, we need to have some methods to access and modify registers of our custom memory-mapped peripheral, and we will use character device driver structures with sysfs attributes.

Platform driver

Peripherals can be connected to the processor directly (via 'platform' bus, like our memory-mapped peripheral) and also via external busses: I2C, SPI, UART, USB, PCI, etc. Some of these busses support dynamic enumeration of devices (USB, PCI), but for others there needs to be a way to let the system know what is present in those busses.

In standard PCs BIOS is in charge of preparing and providing information about present devices to the operating system on boot.

ARM systems do not have BIOS, so the information must be provided in some other way. Initially, it was done by hardcoding details for each board in the Linux architecture specific code, so when board runs it has all of the information about peripherals that it needs.

However, that also meant that two boards with just minor differences could not use the same Linux kernel image.

To avoid that, Device Tree specification is now used. In Device Tree specifics of a device are described: memory regions, interrupt line numbers, dma channels, as well as key for binding the compatible driver with that device.

This way the device driver gets all relevant information about the way the device is integrated into the system from the Device Tree.

Device Tree description

Device Tree is a tree-like hierarchical description of system devices and busses they are connected to. It is used for describing peripherals that cannot be automatically enumerated.

Device Tree is written in textual form (.dts and .dtsi files, but also C header files can be preprocessed) and need to be translated into binary form (binary blob, .dtb) before they can be used on a board. At boot time Linux kernel (also recent versions of U-Boot) parse the Device Tree blob and try to match corresponding device drivers in order to initialize the system.

Without going into too much details (there are always good materials on Bootlin website), the Device Tree excerpt for our memory-mapped peripheral should look like

iofpga@7,00000000 {
...
    mmsens@18000 {
        compatible = "mistra,mmsens";
        reg = <0x18000 0x1000>;
        interrupts = <29>;
    };
...
};

The compatible string is used by the device driver when probing to match with this device.

Field reg is used to describe memory regions used by the device, in our case it is offset 0x18000 of the CS7 region. Memory range is 0x1000 long.

The interrupt line that is used is noted in the interrupts field.

Platform driver for memory-mapped peripheral

The platform driver provides callbacks that are called when Device Tree is parsed and device is probed, which are then used to get information about the device. The platform_driver structure has following important fields

  • .driver - initialized with driver name and table of compatible strings used for matching driver with the device from the Device Tree
  • .probe callback - called when Device Tree is parsed, to try to register device with the driver
  • .remove callback - called when device is removed (not particularly interesting for platform drivers) or when driver is removed from the system

In the case of our memory-mapped device, platform_driver structure should look like

static struct platform_driver mmsens_driver = {
    .driver = {
        .name = DRIVER_NAME,
        .of_match_table = mmsens_of_match,
    },
    .probe = mmsens_probe,
    .remove = mmsens_remove,
};

NOTE: The of_ prefix comes from 'Open Firmware', since full name of Device Tree is Open Firmware Device Tree.

Match table should contain the compatible string, which (from Device Tree excerpt above) is selected to be "mistra,mmsens"

static const struct of_device_id mmsens_of_match[] = {
    { .compatible = "mistra,mmsens", },
    { /* end of table */ }
};
MODULE_DEVICE_TABLE(of, mmsens_of_match);

The .probe callback should do the following things:

  1. Try to match device from Device Tree with the driver based on the 'compatible' string (of_match_node)
  2. Try to extract memory regions information from Device Tree (platform_get_resource) and remap memory so it is accessible by the driver (devm_ioremap_resource)
  3. Try to extract interrupt lines information from Device Tree (platform_get_irq) and register handler function for that interrupt (devm_request_irq)

Interrupt handler in our case should only clear the IFG flag

static irqreturn_t mmsens_isr(int irq, void *data)
{
    struct mmsens *dev = data;

    pr_info("Interrupt received\n");

    iowrite32(0, dev->base_addr + MMSENS_STATUS_OFFSET);

    return IRQ_HANDLED;
}

In the simplest scenario the .remove callback does not need to do anything, but once we add the character device operations it will change.

Chardev operations

So far, the platform driver only allows us to match the driver with the device when Device Tree description is parsed.

In order to be able to interact with the device and read/write some data to it, we need add another layer, which is character device. The use of character device allows us later on to add more operations, like IOCTL or sysfs attributes, to have even more ways to interact with the device.

Character device operations are executed when character device file (usually under /dev) is accessed, so we need to make sure that information obtained from platform driver framework (base address of the remapped region) can be used within the character device operations. For that purpose, we will create a custom structure which will be stored as private_data and shared between these two frameworks

/**
 * struct mmsens - mmsens device private data structure
 * @base_addr:    base address of the device
 * @irq:    interrupt for the device
 * @dev:    struct device pointer
 * @parent:    parent pointer
 * @cdev:    struct cdev
 * @devt:    dev_t member
 */
struct mmsens {
    void __iomem *base_addr;
    int irq;
    struct device *dev;
    struct device *parent;
    struct cdev cdev;
    dev_t devt;
};

The character device operations structure defines several callbacks

  • .open - used to prepare driver structures for accessing the device
  • .release - cleanup of operations done in .open
  • .read - read operation from the device, usually raw data that is copied to the user space
  • .write - write operation to the device, usually raw data that is copied from the user space

In the case of our memory-mapped device, it should look like

static struct file_operations mmsensdev_fops = {
    .owner = THIS_MODULE,
    .open = mmsens_open,
    .release = mmsens_release,
    .read = mmsens_read,
    .write = mmsens_write,
};

The details of the implementation are available in the github repository.

Sysfs attributes

The character device operations allow only reading or writing of raw data to the device. However, since our device has several registers and supports different operations, we need an additional interface to be able to control it.

That can be achieved using IOCTL callback, or by using sysfs attributes (we will use the latter).

The sysfs attributes are created in the /sys directory when the device is created as separate files. Each file can be used according to the way they are specified (read-only, write-only, read-write) and they can be used to access individual bits/registers, or perform specific operations.

In order to be able to use the sysfs attributes, the character device class must be created, and all devices of that class will appear under that directory.

The attributes can be read and/or written. The <operation>_show callback is used when attribute file is read, <operation>_store callback is used when attribute file is written, and attribute is initialized using static DEVICE_ATTR_RW(<operation>); (there are also the _RO and _WO variants).

In the case of our device, following attributes are supported

  • DEVICE_ATTR_RO(interrupt) - used to obtain interrupt status, and can be used to poll from user space application (more on that in next post)
  • DEVICE_ATTR_RW(enable_interrupt) - used to enable/disable interrupt generation
  • DEVICE_ATTR_RW(enable) - used to enable/disable device
  • DEVICE_ATTR_RW(frequency) - used to select desired sampling frequency
  • DEVICE_ATTR_RO(available_frequencies) - used to show available sampling frequencies
  • DEVICE_ATTR_RO(data) - used to show data in BCD format

If we take a look at the data attribute for instance, we can see that inside it reads the register value and returns string with formatted value

static ssize_t data_show(struct device *child, struct device_attribute *attr, char *buf)
{
    struct mmsens *dev = dev_get_drvdata(child);

    u32 data = ioread32(dev->base_addr + MMSENS_DATA_OFFSET);
    data &= DATA_MASK;

    return sprintf(buf, "%04X\n", data);
}

The details of the implementation are available in github repository.

Building driver out-of-tree

The kernel driver can be provided in two ways: as part of the kernel source code, or as an out-of-tree entity. In first case, using the kernel configuration (menuconfig for instance) it can be selected whether driver will be built into the kernel, or as a separate kernel module (.ko extension). In the out-of-tree build, kernel header files are needed and driver can be built as a kernel module file.

In this case, we will be using the out-of-tree approach. If we have the kernel source code in the $KERNEL_PATH, the Makefile for building the kernel module would look like

obj-m := mmsensdrv.o

SRC := $(shell pwd)

all:
    $(MAKE) ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- -C $(KERNEL_PATH) M=$(SRC)

After executing make command, the mmsensdrv.ko file would be available and can be transferred to the SD card and tested.

Adding driver to Yocto

If we want to add driver to Yocto a new recipe has to be created. The template (skeleton) exists at poky/meta-skeleton/recipes-kernel/hello-mod and we will reuse it.

The recipe for the module should look like

SUMMARY = "Memory-mapped QEMU sensor driver"
DESCRIPTION = "${SUMMARY}"
LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=12f884d2ae1ff87c09e5b7ccc2c4ca7e"

inherit module

SRC_URI = "git://github.com/straxy/mmsens-drv.git;protocol=https;branch=main"
SRCREV = "${AUTOREV}"

S = "${WORKDIR}/git"

# The inherit of module.bbclass will automatically name module packages with
# "kernel-module-" prefix as required by the oe-core build environment.

RPROVIDES_${PN} += "kernel-module-mmsens-drv"

This module also needs to be added to the image recipe so it will be included in the output root filesystem image

IMAGE_INSTALL += "kernel-module-mmsens-drv"

After these changes have been added, the image can be rebuilt and run inside QEMU.

Testing driver

Once Linux kernel is started inside QEMU, the module can be loaded. If mmsensdrv.ko is copied to the SD card manually, it can me loaded into the kernel with

$ insmod mmsensdrv.ko

If driver was included in Yocto image, it will be loaded automatically on boot.

First we can check that mmsensX file exists in the /dev directory

$ ls /dev/mmsens*
/dev/mmsens0

Reading that file should return value 0 since device must be started manually, by setting the EN bit in CTRL register.

$ cat /dev/mmsens0 
mistra.mmsens:DATA: read of value 0x0
0

Next, we can check that appropriate entries exist in the sysfs

$ ls /sys/class/mmsens/mmsens0/
available_frequencies  device                 frequency              subsystem
data                   enable                 interrupt              uevent
dev                    enable_interrupt       power

Finally, we can do the proper testing.

Data incrementing

If we enable the device and try to read data attribute, as well as /dev/mmsens0, we should see that values are changing.

$ echo 1 > /sys/class/mmsens/mmsens0/enable
mistra.mmsens:CTRL: read of value 0x0
mistra.mmsens:CTRL: write of value 0x1
r_ctrl_post_write: Wrote 1 to CTRL
$ sleep 10
$ cat /sys/class/mmsens/mmsens0/data && cat /dev/mmsens0
mistra.mmsens:DATA: read of value 0x10
0010
mistra.mmsens:DATA: read of value 0x10
16

After enabling the device, the data attribute returns the BCD formatted value, while /dev/mmsens0 returns the raw (integer) value, as expected.

Frequency change

The list of available frequencies can be obrained from the available_frequencies attribute and current frequency selection from frequency attribute.

$ cat /sys/class/mmsens/mmsens0/available_frequencies
normal fast
$ cat /sys/class/mmsens/mmsens0/frequency 
mistra.mmsens:CTRL: read of value 0x0
normal

Per specification, normal frequency means that value changes once per second, while fast frequency means that value changes twice per second (every 0.5 seconds).

If we change sampling frequency from normal to fast, we should see that values are changing twice as often.

# Before
$ pushd /sys/class/mmsens/mmsens0
$ cat data && sleep 1 && cat data
0081
0082
# Change
$ echo fast > frequency
# After
$ cat data && sleep 1 && cat data
0085
0087
# Cleanup
$ popd

Interrupt generation

Finally, we should check that interrupts are generated properly. However, since we do not have userspace application that would do something useful with those interrupts, we can use output from /proc/interrupts

# Before enabling interrupt generation
$ cat /proc/interrupts | grep mmsens
 40:          0     GIC-0  61 Level     10018000.mmsens

NOTE: Where is value 29 from our description? First 32 interrupt lines are private interrupts per core, so numbering actually starts from 32. That means that actual interrupt number is 61 (32+29).

We can enable interrupt by writing 1 to enable_interrupt and number of occurences (column 1) increases. It is also visible from QEMU debug code that once interrupt is generated, interrupt handler in device driver clears STATUS register, thus acknowledging interrupt.

# After enabling interrupt generation
$ echo 1 > /sys/class/mmsens/mmsens0/enable_interrupt
mm_sens_update_irq: Interrupt generated
[ 8104.791017] Interrupt received
mistra.mmsens:STATUS: write of value 0x0
r_status_post_write: Wrote 0 to STATUS
mm_sens_update_irq: Interrupt none
$ cat /proc/interrupts | grep mmsens
 40:          1     GIC-0  61 Level     10018000.mmsens

Summary

In this blog post a simple character device platform driver is presented. The driver allows initialization and bit-field access of memory mapped peripheral.

Device Tree description for the memory-mapped device is also shown, which allows device drvier to obtain information about the device that is present.

The driver itself handles interrupt, but we have not gone into processing that event, and if someone would want to use it as it is (for instance with bash script), they could only do a polling approach to handle data.


Next step is to develop a user space application which will be able to initialize device using the driver, as well as receive information about received interrupt and process updated data. This will be done in next blog post in this series.

Jun 25, 2022

Developing custom memory-mapped peripheral for QEMU

This is part 1 of the Linux device driver development post series.

In the previous post I presented some of the goals for doing this work with QEMU.

In this post I will cover the following things

Designing custom memory-mapped device in QEMU

QEMU can emulate many different peripherals. More importantly, it is possible to create new peripherals that are emulated.

That is particularly useful when someone wants to learn device driver development, since that peripheral will be unique, and it would be impossible to find already existing driver.


In this post we will develop a new, custom peripheral for QEMU.

The peripheral will have several 32-bit registers and ability to generate interrupts. It will function as an 4-digit BCD counter. It can be enabled or disabled, generate interrupts on each count, and counting frequency can be selected.

In the next post we will develop Linux device driver that will enable access to this peripheral.

Register map

Register map for the custom peripheral is shown in the following table

offset name description
0x0 CONFIG Configuration register, used to enable component, interrupt generation and select frequency
0x4 STATUS Status register, used to read/clear interrupt flag
0x8 DATA Data register holding current counter value

Bit values of CONFIG register are shown in the following table

name pos dflt description
Reserved 31:3 0 Reserved
FREQ 2 0 Frequency setting:
  • 0 - normal frequency (1 Hz)
  • 1 - fast frequency (2 Hz)
IEN 1 0 Interrupt enable
EN 0 0 Enable device

Bit values of STATUS register are shown in the following table

name pos dflt description
Reserved 31:2 0 Reserved
IFG 1 0 Interrupt flag
Reserved 0 0 Reserved

Bit values of DATA register are shown in the following table

name pos dflt description
Reserved 31:16 0 Reserved
SAMPLE 15:0 0 Current counter value

Fitting into Vexpress-A9 memory map

In order to instantiate and use developed memory-mapped component, we need to integrate it into the Vexpress-A9 memory map and connect it to interrupt controller.

Looking at the memory map for Vexpress-A9, there are several regions that are unused, where we can place our custom component. In this example, I chose offset 0x00018000 in the motherboard peripheral memory map, region CS7. This means that the absolute address is 10018000.

Also, we need an interrupt line for the custom component. Again, looking at the documentation, there are several "reserved" interrupt lines which we can use. In this example, I chose interrupt line 29.

QEMU implementation

Details of implementation of a new memory-mapped device in QEMU will be shown in this section. Main points will be displayed, so someone can use it as instructions for creating a new device.

The device will be described by it's registers and added to the Versatile Express memory map.

Register mapping

QEMU has a very simple and verbose way of describing registers of a component.

For each register, it's offset from the base address is specified, as well as bit-fields that the register consists of. Additionally, for each bitfiled access permissions can be specified and callback functions that are executed before and/or after accessing the register.


Register description for our custom component can be described as

REG32(CTRL, 0x00)
    FIELD(CTRL,     EN,     0,  1)      /* component enable */
    FIELD(CTRL,     IEN,    1,  1)      /* interrupt enable */
    FIELD(CTRL,     FREQ,   2,  1)      /* sampling frequency setting */

REG32(STATUS, 0x04)
    FIELD(STATUS,   IFG,    1,  1)      /* interrupt flag */

REG32(DATA, 0x08)
    FIELD(DATA,     SAMPLE, 0,  16)     /* current value */

The previous code excerpt defines that we have three 32-bit registers called CTRL (offset 0x0 from base address), STATUS (offset 0x4) and DATA (offset 0x8). If we look at the the register CTRL, it has three bit-fields that are used: EN (bit position 0, size 1 bit), IEN (bit position 1, size 1 bit), FREQ (bit position 2, size 1 bit). Similar goes for other two registers.

NOTE: The details of the REG32 and FIELD macros can be found in hw/registerfields.h include file. Thing to keep in mind is that these macros create additional macros (mask, shift, etc.) which we can use later for accessing individual registers and bit-fields.


In order to specify actions that are performed when someone tries to access these registers (read or write), following data must be defined

static const RegisterAccessInfo mm_sens_regs_info[] = {
    {   .name = "CTRL",           .addr = A_CTRL,
        .reset = 0,
        .rsvd = ~(R_CTRL_EN_MASK | R_CTRL_IEN_MASK | R_CTRL_FREQ_MASK),
        .post_write = r_ctrl_post_write,
    },
    {   .name = "STATUS",           .addr = A_STATUS,
        .reset = 0,
        .rsvd = ~R_STATUS_IFG_MASK,
        .post_write = r_status_post_write,
    },
    {   .name = "DATA",         .addr = A_DATA,
        .reset = 0,
        .rsvd = ~R_DATA_SAMPLE_MASK,
        .ro = R_DATA_SAMPLE_MASK,
    },
};

static const MemoryRegionOps mm_sens_reg_ops = {
    .read = register_read_memory,
    .write = register_write_memory,
    .endianness = DEVICE_LITTLE_ENDIAN,
    .valid = {
        .min_access_size = 4,
        .max_access_size = 4,
    }
};

Looking at the second structure, mm_sens_reg_ops, it defines that we are using register read and write functions when accessing this component.

Register functions use the mm_sens_regs_info array defined above, where RegisterAccessInfo structure has several useful fields:

  • name - register name, used for debugging
  • addr - register offset
  • reset - reset value of register
  • ro - read-only bitmask
  • w1c - write one to clear bitmask
  • cor - clear one read bitmask
  • rsvd - reserved bits bitmask
  • pre_write - callback executed before write command
  • post_write - callback executed after write command
  • post_read - callback executed after read command

The post_write functions in the previous code block need to perform certain actions based on the values that are written to the registers.

For instance, after CTRL register bit EN bit is set to 1, the DATA values should start incrementing. Or, after bit FREQ has changed, the frequency of incrementing DATA register should change.

Before we go into details of post_write functions, it would be useful to first explain how is DATA register periodically incremented, as well as how are interrupts implemented in QEMU code.

QEMU timers

QEMU uses timers to enable periodical execution. Timers have a simple API (described in hw/ptimer.h) which we will go over in this section.

Our plan is to increment value of DATA register at two different frequencies: normal (1 Hz) and fast (2 Hz). The value should be incremented only when bit EN in CTRL register is set to 1. The value should also be incremented as BCD value, so each nibble should have values only in the range of 0-9.


Having analyzed our needs, following ptimer functions are of interest

  • ptimer_init - create timer object and define callback that is executed when timer period expires
  • ptimer_set_freq - set timer reload frequency in Hz
  • ptimer_run - start timer and select whether continuous or oneshot mode is used
  • ptimer_stop - stop timer

Based on this, the post_write function for CTRL has two parts.

In the first part FREQ is handled so after every write a check is made if value of FREQ bit has changed, and if it has, timer frequency is updated.

// first part, FREQ handling
...
    new_sfreq = (s->regs[R_CTRL] & R_CTRL_FREQ_MASK) >> R_CTRL_FREQ_SHIFT;

    if (new_sfreq != s->sampling_frequency) {
        s->sampling_frequency = new_sfreq;
        switch (s->sampling_frequency) {
            case FREQ_NORMAL:
                ptimer_set_freq(s->timer, DATA_UPDATE_NORMAL_FREQ);
                break;
            case FREQ_FAST:
                ptimer_set_freq(s->timer, DATA_UPDATE_FAST_FREQ);
                break;
            default:
                DB_PRINT("Unknown frequency %u\n", s->sampling_frequency);
                break;
        }
    }
...

In the second part EN is handled and timer is started/stopped if EN bit is set/reset.

Additionally, if timer is enabled and IEN bit is also set, then evaluation of interrupt generation condition must be made (more on interrupts in next subsection).

// second part, EN/IEN handling
...
    if (s->regs[R_CTRL] & R_CTRL_EN_MASK) {
        /* start timer if not started*/
        ptimer_run(s->timer, 0);

        if (s->regs[R_CTRL] & R_CTRL_IEN_MASK) {
            /* check if alarm should be triggered */
            mm_sens_update_irq(s);
        }
    } else {
        /* stop timer */
        ptimer_stop(s->timer);
    }
...

Increments of DATA register are implemented in the timer callback in the following manner

// DATA incrementing
static void mm_sens_update_data(void *opaque)
{
    MMSensor *s = MM_SENS(opaque);

    s->regs[R_DATA] = s->regs[R_DATA] + 1;
    if ((s->regs[R_DATA] & 0x000fu) > 0x0009u) {
        s->regs[R_DATA] += 0x0006u;
        if ((s->regs[R_DATA] & 0x00f0u) > 0x0090u) {
            s->regs[R_DATA] += 0x0060u;
            if ((s->regs[R_DATA] & 0x0f00u) > 0x0900u) {
                s->regs[R_DATA] += 0x0600u;
                if ((s->regs[R_DATA] & 0xf000u) > 0x9000u) {
                    s->regs[R_DATA] += 0x6000u;
                }
            }
        }
    }

    s->regs[R_STATUS] |= R_STATUS_IFG_MASK;

    mm_sens_update_irq(s);
}

This way the BCD requirement is met and counting will look like in the following diagram

QEMU interrupt handling

Interrupt handling in peripheral in QEMU is performed using the qemu_set_irq function. The function receives an additional parameter which indicates whether interrupt is pending or not. If interrupt is pending (and is not masked in the interrupt controller) it will be raised to the CPU.

In the case of our peripheral, interrupt is pending if both bit IFG in STATUS register and bit IEN in CTRL register are set. This condition has to be checked every time a change happens to any of these two registers, so there is a function that can be reused.

// IRQ handling
static void mm_sens_update_irq(MMSensor *s)
{
    bool pending = s->regs[R_CTRL] & s->regs[R_STATUS] & R_CTRL_IEN_MASK;

    qemu_set_irq(s->irq, pending);
}

Integrating with board file and build system

In order to use the custom memory mapped peripheral, it must be 'placed' in the memory map of the emulated board. Since we are using Versatile Express A9, then it's description must be updated.

Luckily, this is done with one simple command, where we can see chosen base address (0x10018000) and interrupt number (29).

sysbus_create_simple("mistra.mmsens", 0x10018000, pic[29]);

Since build system uses meson and ninja, the new component file is added to the build system in the following manner

softmmu_ss.add(files('mmsens.c'))

The patch file with implementation of the custom component is available in github. The main details of our custom component were explained in previous sections. However, there are standard QEMU structures that also must be used in order to describe VMState, as well as initialization and those can be reused from the patch.

Patch is applied to the QEMU source tree with following command

$ cd $QEMU_SRC
$ patch -p1 < BCD-memory-mapped.patch

After the patch is applied, QEMU must be rebuilt.

Testing developed peripheral

Testing peripheral without appropriate Linux device driver is a bit harder, but not impossible.

We can use the embedded debug prints from the memory-mapped component. Before they can be used, the MM_SENS_ERR_DEBUG definition must be changed from 0 to 1. This way all debug prints from the component will be visible.


U-Boot has integrated commands for reading and writing to memory addresses, so we can use it to try to enable the component, interrupt generation and read current data value.

After QEMU is started with

# Run QEMU with SD card and networking
$ qemu-system-arm -M vexpress-a9 -m 1G -kernel $UBOOT \
                  -drive file=sd.img,format=raw,if=sd \
                  -net nic -net tap,ifname=qemu-tap0,script=no \
                  -serial mon:stdio

U-Boot prompt should be reached by pressing a key.

Following commands are available

  • md <addr> - read data from address addr
  • mw <addr> <val> - write val to address addr

We can first try reading the CTRL and DATA registers

# Run QEMU with SD card and networking
U-Boot> md 0x10018000 1
10018000:mistra.mmsens:CTRL: read of value 0x0
 00000000                               ....
U-Boot> md 0x10018008 1
10018008:mistra.mmsens:DATA: read of value 0x0
 00000000                               ....

The lines starting with 10018000:mistra.mmsens: at debug prints from QEMU, while second lines are written by U-Boot.

In order to enable peripheral, so DATA value starts incrementing, we can write 1 to EN bit in CTRL register. If we read DATA register afterwards, we can see that the values are changing.

# Run QEMU with SD card and networking
U-Boot> mw 0x10018000 1
mistra.mmsens:CTRL: write of value 0x1
r_ctrl_post_write: Wrote 1 to CTRL
U-Boot> md 0x10018008 1
10018008:mistra.mmsens:DATA: read of value 0x1
 00000001                               ....
U-Boot> md 0x10018008 1
10018008:mistra.mmsens:DATA: read of value 0x2
 00000002                               ....

We can also check that the IFG in STATUS register is set. However, interrupt is not triggered since handling of this interrupt is not implemented in U-Boot, which is expected. We will implement interrupt handling in the next blog post, when we develop the Linux device driver for this component.

Summary

In this blog post the process of developing a custom memory-mapped peripheral for QEMU is shown. Main details are described and the complete patch is available with full details of the component.

Using this process many different components can be implemented.


In the next blog post I will show process of development of Linux device driver for this component.