Latency Analysis

As expected, with the addition of the PREEMPT_RT patch, overall worst-case system latency was reduced significantly in almost every stress scenario. This is especially relevant in some of the most demanding categories, where the real-time kernel was able to achieve comparable latencies to an unloaded system. In these same test cases, the mainline kernel had significant latencies and could not even be considered to be acceptable for many real-time systems. Most notably, the latency in the out-of-memory error condition was very significant and reached well in the milliseconds’ region with the mainline kernel. This was not the case in the real-time kernel where the same latency was still comparable to other categories. Based on these measurements, the reliability and determinism of the real-time kernel are confirmed to hold in terms of experienced scheduling latencies.

The single biggest reason behind the real-time kernel’s superior performance seems to be the conversion of spinlocks to sleeping locks that also implement priority inheritance functionality. This is to be expected as the non-preemptible regions inside the kernel are significantly reduced by this change. Also, the conversion of most interrupt handlers to be threaded and thus subject to scheduling with lower priorities might slightly account for the better latencies measured with the real-time kernel. The reason why this change is probably not so relevant with this particular kernel configuration, as the conversion of spinlocks, is that there were not too many sources of interrupts enabled. Those that were required to be enabled seemed to be anyways quite fast to complete compared to the other sources of latencies. In the end, the latency improvements of PREEMPT_RT probably did not appear to be that impressive when compared to the performance of the mainline kernel which is already quite close to being real-time capable. This is mostly because other significant changes historically introduced by PREEMPT_RT are already merged, so they also benefit the mainline kernel.

Even if the real-time kernel is better than the mainline, it still has some sections that produce the over 150 µs latencies that were consistently measured. Based on the tracing of the kernel internals, some insight into these latencies was acquired. The longest latencies appear to be exclusively caused by timer interrupts, which are always executed in hard interrupt context, and occasional raw spinlocks that are still required. The other significant peaks in the latency histograms show that there are also some additional causes for smaller, but considerable latencies. However, analyzing these reliably by tracing is hard and probably unnecessary if the longer latencies still occur. Also, some of the latency differences between different categories are probably explained by hardware-specific issues. For example, a more demanding load to the system might cause additional cache misses, etc. that cannot be fully mitigated by the PREEMPT_RT patch.

Overall, the findings of this thesis are quite significant, as they show that the PREEMPT_RT patch on RISC-V can be made to work without too much additional effort. With the real-time preemption model enabled, the measured latencies are fairly predictable and generally in an acceptable range for a real-time system. In addition, there do not seem to be any fundamental problems with running the PREEMPT_RT on RISC-V architecture. However, it is possible that if some more advanced kernel features were enabled from the configuration, there might be some fundamental problems with the current RISC-V specific implementation. At least with more enabled features from the kernel, the latency performance could get worse.

Even though performing exact measurements on a Linux system is challenging, the results are very repeatable and sensible. However, the latency measurements will always have some uncertainty as the system is very asynchronous and mainly driven by interrupts. More exact results could have been observed by running each one of the latency measurements for a longer period, even for days. The general shape of the latency response, however, is still captured with the used shorter test duration. One actual source of possible errors in the presented results is the used PREEMPT_RT patch version, as the closest version was for 6.3 kernel. This means, that there is a possibility that up until the official release of this version, some new features could have been added to the kernel which are not handled properly by the PREEMPT_RT. However, this is quite an unlikely situation, and also the tracing results suggest that there should not be any problems regarding this. Anyways, using the results presented in this thesis as a conservative estimate is fine. Finally, it is also possible that there is still some oversight in the system configuration. Otherwise, all other sources of error should be mitigated and taken into account in the measurements.