NVRAM-ified Unixoid

Non-volatile main memory (NVM)

The trend toward fast, byte-addressable non-volatile memory (i.e. NVM in the form of NVRAM) with latencies and write resistance closer to SRAM and DRAM than flash positions this new type of main memory as a possible replacement for established volatile technologies. While on the one hand the non-volatility and low leakage capacity make NVRAM an attractive candidate for new system designs in addition to other advantageous features, on the other hand there are also major challenges, especially for the programming of such systems. For example, power failures in combination with NVRAM to protect the computing status result in control flows that can unexpectedly transform a sequential process into a non-sequential process (cf. REDOS): a program has to deal with its own status from earlier interrupted runs. If programs can be executed directly in the NVRAM, normal volatile main memory (functional) becomes superfluous. Volatile memory can then only be found in the cache and in device/processor registers. An operating system designed for this can dispense with many, if not all, persistence measures that would normally otherwise be implemented and thereby reduce its level of background noise. Considered in detail, this enables energy requirements to be reduced, computing power to be increased and latencies to be reduced. In addition, the elimination of these persistence measures means that an “NVM-only” operating system is leaner than its functionally identical twin of conventional design. On the one hand, this contributes to better analysability of non-functional properties of the operating system and, on the other hand, results in a smaller attack surface or trustworthy computing base.

NVRAM-ification of Linux and FreeBSD

The NEON and PAVE projects assume an “NVM-only” Unix-like universal operating system, namely Linux on the one hand and FreeBSD on the other. Both operating systems are based on functionally identical x86-64 hardware, especially Optane™ technology as NVRAM. They have both been NVRAM-ified so that they themselves, as well as the machine programs they run, execute directly in NVRAM. Basically, only the main memory technology was exchanged for them, that is, DRAM for NVRAM.

In practice, replacing the main memory base (DRAM with NVRAM) has proven to be unexpectedly difficult and time-consuming for both operating systems, but the problems were much more in the BIOS. The bootstrap process proved tricky, as it can only be done from DRAM to work around hardware and firmware limitations of the x86-64 platform. In addition, some legacy device drivers depend on 32-bit addressable memory, while the Optane™ NVRAM populates physical addresses well above 4 GiB. However, as soon as the system is booted it runs completely in NVRAM and no longer uses DRAM.

Note that with “NVRAM-ification” of Linux and FreeBSD, only the main memory platform was replaced for both operating systems: a rigorous technology switch from DRAM to NVRAM was made without taking special precautions for efficient NVRAM operation. Both systems were adopted unchanged, except for the porting measures to be able to switch from DRAM to NVRAM. These system variants ported to NVM should not only provide a basis for the work envisaged in NEON and PAVE, but also enable, in particular, a quantitative and comparative analysis of the performance parameters of common (Unix-like) general-purpose operating systems with respect to DRAM and NVRAM. This means in particular that Linux as well as FreeBSD were not “REDOS-ed” for these investigations, in order to prevent possible and “NVM-only” caused race conditions in the event of power failures by special system-software features running in the background.

We have established a nearly worst-case baseline for the possible performance of such a “NVRAM-ified” general-purpose operating system under heavy load. While the synthetic memory benchmarks showed a wide variety of performance penalties, all of the kernel build benchmarks only revealed a penalty of significantly less than 4x. Thus, the expected degradation was drowned in “system jitter” to a large extent. The power draw increased only insignificantly and even decreased for several workloads, while the overall energy consumption was proportional to the prolonged execution times of the benchmarks.

Considering the higher read/write latencies of Optane™ NVRAM compared to DRAM, the worse performance of these NVM-only Unioxids is not unexpected, but it is already within limits and falls into a range that already makes these solutions, which are unoptimized for NVRAM, suitable for certain application domains. High performance is not always the measure of all things! Sometimes it is more important to guarantee the timeliness of operations, even at the expense of speed. Likewise, it may be the premise to support efficient fault-tolerant operation of the computing system. For the latter, the NVRAM-enabled systems presented here would already not be a bad choice.

Project-related publications