Skip to content

Protocol

Dual Boot

1) Flash SSD by installing etcher 2) Ubuntu and Windows on same machine

Install Salamander4 OS

1) Install Ubuntu 22.04.3 LTS 2) Do everything in build_with_yocto.md

SSH to device

Connect to device with ssh -oHostKeyAlgorithms=+ssh-rsa root@10.10.1.229
or ssh -p 22 root@192.168.1.x (changes often)

LasalClass2 to device

Connect LasalClass2 with Salamander 4, IP of Salamander4 device

Configure bridge for QEMU

This setup allows the virtual machines to communicate with the outside network through the Ethernet connection provided by either the laptop or the docking station.

name device
enp0s31f6 Laptop's Ethernet port
enx4cd717733f80 Docking station's Ethernet port

Reduce latency

We test the system using the Xenomai test suite - latency
- clocktest

Max Latency default

Default latency 

sigma_ibo@pamhal:$ ps -eo pid,psr,comm | grep qemu
   7295  10 start_qemu.sh
   7298  17 qemu-system-x86

latency -h -s -T 600 -g max_latency_default_10min.txt

lat worst is 4070.018

Max Latency with taskset

To isolate CPUs on your host system (Ubuntu), you can add the isolcpus option to the kernel boot configuration. Here are the steps you can follow:

  1. Open the GRUB configuration file with a text editor. You can use the nano editor for this. Execute the following command in your terminal: 
    sudo nano /etc/default/grub
  2. Search for the entry GRUB_CMDLINE_LINUX and add isolcpus=0,1,2,3,4 (or the corresponding CPU numbers you want to isolate). It should then look like this: 
    GRUB_CMDLINE_LINUX="isolcpus=0,1,2,3,4"

  3. Save the changes and close the editor. If you are using nano, you can do this by pressing Ctrl+X, then typing Y to save the changes, and finally pressing Enter to close the editor.

  4. Update GRUB with the following command: 

    sudo update-grub

  5. Reboot your system for the changes to take effect.

Check with: cat /sys/devices/system/cpu/isolated

sigma_ibo@pamhal:~$ cat /sys/devices/system/cpu/online 
0-19
sigma_ibo@pamhal:~$ cat /sys/devices/system/cpu/isolated
0-4

After taskset on CPU4 with qemu_def_2taskset_vsock_nmbridge.sh 

sigma_ibo@pamhal:$ ps -eo pid,psr,comm | grep qemu
   8752   0 start_qemu.sh
   8755   4 qemu-system-x86
latency -h -s -T 600 -g max_latency_with_taskset_10min.txt

lat worst reduced from 4070.018 to 457.545 with stats

Enable Preempt_RT Kernel

Either do everything in kernel-patch.md to patch the kernel and enable Fully Preemptible Kernel (RT), or simply enable Ubuntu Pro's real-time kernel, here.

Info

Before the isolation of CPU x, both kernel threads and user processes were running on this CPU. The user processes included various applications such as msedge, code, bash and qemu-system-x86.

After isolating CPU x, only kernel threads and the qemu-system-x86 process appear to be running on this CPU. There do not appear to be any other user processes running on this CPU.

The isolcpus option prevents the kernel from scheduling normal (non-real-time) processes on the isolated CPUs. However, there are some exceptions:

  • If a process is explicitly set to run on an isolated CPU (for example with taskset), it will run on that CPU even if it is isolated.
  • Some kernel threads can run on isolated CPUs because they are not controlled by the normal scheduler. These include the threads you see in your output, such as kthreadd, migration/4, ksoftirqd/4, kworker/4:0-events and others.
  • Interrupts can be handled on isolated CPUs unless they are explicitly redirected with the irqaffinity option.

latency -T 600

lat worst reduced from 457.545 to 32.216 with stats

Realtime priority

Danger

Setting a real-time priority of 99 for a process means that this process has the highest priority in the system and is executed before all other processes. This can result in other processes, including important system processes, not receiving the CPU time they need to function properly. This can lead to system instability and, in the worst case, to the system becoming unresponsive or "crashing".

It is important to be careful when using real-time priorities and ensure that other important processes still get the CPU time they need. It might be helpful to gradually increase the real-time priority and observe the effects on the system instead of jumping straight to the highest priority.

To see the real-time priorities of all running processes, have a look at thread priorities.

Configuring the system for real-time

For now, PREEMPT_RT is a set of patches that is supposed to be applied on top of mainline Linux. Most Linux distributions do not build it by default, and you will most likely have to do it yourself [3]. How this can be done falls outside the scope of this post, but there are plenty of guides out there. Hopefully in the near future, all of PREEMPT_RT's functionality will be merged in to mainline, and Linux distributions will provide RT-enabled kernels out-of-the-box.

Once you successfully compiled the RT kernel, the default hardware and OS configurations are usually not tuned correctly for RT. The following hardware and OS configurations should likely always be checked and tuned: latency_reduction_steps.md

Third process is for latency minimization

13    2972    2972 qemu-system-x86_64 -M pc,ac FF    -     98
13    2972    2976 qemu-system-x86_64 -M pc,ac FF    -     98
13    2972    3292 qemu-system-x86_64 -M pc,ac FF    -     90
13    2972    8699 qemu-system-x86_64 -M pc,ac FF    -     95

Salamander 4 latency comparison

PCI initalization

sudo modprobe vfio-pci lspci -v lspci -nn sudo sh -c 'echo "5112 2200" > /sys/bus/pci/drivers/vfio-pci/new_id' lspci -v