Introduction
There is a growing trend of moving private computations from on-premises to the public cloud and to the edge. However, many individuals, companies, and organizations consider the public cloud and the edge as untrusted environments and are wary to transfer their confidential data and computations to them. Thus, securing data has become a number one business imperative.
Another recent trend is the emergence of a new class of legal requirements for personal-data processing as regulations such as GDPR, Schrems II and HIPPA take effect across the globe. These legal requirements mandate heightened confidentiality guarantees of personal data at all stages: during storage, during transmission and during processing of this data. Therefore, companies that must adhere to these regulations have an additional incentive to provide elevated security of their clients’ personal data.
Recognizing these trends, Cloud Service Providers (CSPs) such as Microsoft Azure adopted Confidential Computing solutions to stay outside of the client “trust boundary” so that they have no visibility into clients’ data. To secure data at all stages of its processing, Confidential Computing relies on Trusted Execution Environment (TEE) technologies. One of the prominent TEEs – available as part of the Azure Confidential Computing offering – is the Intel® Software Guard Extensions (Intel SGX) hardware-based technology, found in latest Intel® Xeon® processors designed for the data center, Edge, and IoT.
Intel SGX allows companies to create so-called SGX enclaves – opaque, encrypted regions of memory isolated from the rest of the environment. The code and data executed in SGX enclaves are considered trusted and better protected even against privileged attacks. The rest of the environment – including privileged software such as the operating system and the hypervisor – is considered untrusted and possibly malicious. Thus, Intel SGX is a perfect solution for companies that want to outsource their computations on sensitive data to the public cloud or to the edge.
Challenges of Intel SGX and how Gramine overcomes them
Being a hardware technology, Intel SGX provides powerful building blocks for application development. Software developers can port their applications to Intel SGX by putting only the security-critical part of the application into the Intel SGX enclave and leaving the non-critical parts outside of the enclave. Several development kits can help ease the task of writing such code; Intel® SGX SDK and Open Enclave SDK are two prominent examples. However, in many real-world scenarios, it is infeasible to write a new application from scratch or to port an existing application manually.
Gramine can help ease this porting burden for developers: Gramine supports the “lift and shift” paradigm for Linux applications, where the whole application is secured in a “push-button” approach, without source-code modification or recompilation. Instead of manually selecting a security-critical part of the application, users can take the whole original application and run it completely inside the Intel SGX enclave with the help of Gramine.
Gramine not only runs Linux applications out of the box, but also provides several tools and infrastructure components for developing end-to-end protected solutions with Intel SGX:
- Support for both local and remote Intel SGX attestation, with the help of RA-TLS and Secret Provisioning components.
- Transparent encryption and integrity protection of files; in particular, the Encrypted Files feature allows security-critical files to be automatically encrypted and decrypted inside the enclave.
- Optional feature of so-called asynchronous (aka exitless) transitions for performance-critical applications because transitions between the enclave and the untrusted environment can be rather slow in Intel SGX.
- Full integration with the GDB debugger for ease of testing, as well as with the Perf and Intel VTune profilers to better understand and optimize performance of workloads running in Intel SGX enclaves.
- Full support of multi-process applications, by providing complete fork/clone/execve implementations; Gramine is one of the few frameworks that support this.
Gramine currently supports many programming languages and frameworks, as well as many kinds of workloads. Gramine supports C/C++, Rust, Google Go, Java, Python, R and other languages, as well as database, AI/ML, webserver and other workloads. The typical performance overhead observed is around 5-20% depending on the workload.[1]
Gramine primer
To run an application in an SGX enclave with Gramine, the application must be accompanied by a so-called manifest file –a simple plain text configuration file. Upon startup, Gramine parses the manifest file and extracts all the necessary information about the application including its external dependencies and Intel SGX enclave properties. The manifest is the single most important file when enabling applications inside Gramine. Ultimately, the security and correct functioning of the application depends on how its manifest file is written.
Here is an example how the manifest can look like for a Python application (abridged for readability):[2]
libos.entrypoint = “python”
fs.mounts = [
{ path = “/lib”, uri = “file:/usr/local/lib/gramine” },
{ type = “tmpfs”, path = “/tmp” }
]
sgx.enclave_size = “1024M”
sgx.thread_num = 32
Typically, the user writes a manifest file as a Jinja-style template, and Gramine provides a tool to render this template into the final manifest. The manifest file contains not only the description of the application itself (its properties, its executables and dependencies, etc.) but also the description of Gramine (its executables and dependencies). In this way, the manifest anchors a specific version of the application as well as the specific version of Gramine to be run inside the SGX enclave, for ease of deployment, attestation and maintenance.
After creation of the manifest file, the Gramine user can invoke tools to generate Gramine- and SGX-specific files. In particular, the manifest template needs to be expanded to the interim version (with e.g. all paths replaced with absolute paths) with the gramine-manifest tool. Then the interim manifest must be signed with the private enclave-signing key, such that the authenticity of the graminized application can be proved later during SGX attestation. This is achieved with the gramine-sgx-sign tool, that produces the final manifest file and the SGX-specific SIGSTRUCT file as outputs. After this step, the bundle < application, Gramine, final manifest file, SIGSTRUCT file > can be shipped to an SGX-enabled deployment platform and run there via gramine-sgx.
Here is a command-line snippet on how to “graminize” the application and run it:
# prepare the manifest file for your app
$ vim python.manifest.template
# generate Gramine- and SGX-specific files
$ gramine-manifest python.manifest.template python.manifest
$ gramine-sgx-sign --key signing.key --manifest python.manifest \
--output python.manifest.sgx
# run Python workload in Gramine
$ gramine-sgx python -c ‘print(“Hello, world”)’
Hello, world
This sequence of steps is also depicted in the diagram below. Note that Gramine supports two-phase signing: the development and testing happens on the development platform (with some dummy signing key) and the actual act of signing happens on a separate signing platform.[3] This split is done to protect the signing key from being stolen; ideally only a dedicated person has access to the signing platform (such a platform may not even have network access).
In addition to the ready-for-use tools like gramine-manifest, The Gramine project also provides the Python API to help users develop their own tools. For example, the user might want to calculate and print the MRENCLAVE measurement based on the Gramine manifest. The tool will look like this:
from graminelibos import get_tbssigstruct
sigstruct = get_tbssigstruct(path_to_manifest, datetime.date.today())
print(sigstruct['enclave_hash'].hex())
This Python API provides a lot of flexibility for developers. It allows to develop complete, use-case specific pipelines to package, sign and distribute graminized applications.
Gramine Shielded Containers (GSC) primer
As can be seen, with core Gramine the user must manually describe the “packaging” of the application in the manifest file. Unfortunately, writing a manifest file can be a tedious trial-and-error process. In particular, the manifest must contain all dependencies of the ported application (shared libraries, configuration files, helper scripts, etc.). In case of e.g. Python workloads, the number of dependencies can easily exceed thousands of files. Luckily, there are already application-packaging solutions that solve this problem; one of them is Docker.
Therefore, the Gramine project provides the GSC tool. GSC takes a base Docker image with the application to protect and a tiny, abridged manifest (that contains only the SGX-specific manifest options that cannot be deduced from the base Docker image). GSC creates the graminized Docker image from the base Docker image in two steps. The first step is gsc build which combines the original Docker image and the Gramine-containing Docker image into one, scans the entire image and adds all detected files to the user-supplied abridged manifest, generating the final manifest. The second step is gsc sign-image which simply signs the graminized Docker image with the private signing key. After that, the graminized Docker image can be added to a Docker registry or directly distributed to SGX-enabled platforms, where the graminized Docker container will run. Thus, GSC provides an easy option to run already-existing Docker images in Gramine SGX enclaves.
Here is a command-line snippet on how to “graminize” a Docker image and run it:
# pull public Docker image from DockerHub
$ docker pull python
# build GSC image from base Python image, using the manifest file
$ gsc build python python.manifest
# sign SGX enclave inside the GSC image
$ gsc sign-image python private.key
# run the GSC image
$ docker run gsc-python –c ‘print(“Hello, world”)’
Hello, world
This sequence of steps is also depicted in the diagram below. As you can see, the process is very similar to the core Gramine usage but applies to Docker images rather than application binaries.
Customers of Gramine
Since its inception in 2014, Gramine has been in experimental use by the academic community, but since its production release in the fall of 2021 Gramine has seen significant customer traction. One notable production usage of Gramine is the ePrescription project in Germany, led by the national digital health agency Gematik GmbH, with IBM as a partner using Gramine to ensure maximum privacy and isolation. Bosch jointly with Intel and Edgeless developed a SGX-protected scalable confidential AI use case for Advanced driver-assistance systems (ADAS) develo.... Among other usages, Tencent and JD Cloud develop SGX-based solutions with Gramine, Eder Labs is working on secure AI/ML solutions with Gramine, Enclaive.io deploys Gramine-based confidential containers, and Cosmian is developing collaborative confidential computing solutions based on Gramine. The Gramine project is also used by the Intel OpenFL project to protect integrity of computations to deliver secure federated learning deployments. Finally, Intel has published Confidential Computing Zoo (CCZoo), a collection of code-ready reference solutions to facilitate developers to build their own end-to-end Confidential Computing solutions easily with Gramine. We recently published another solution brief on Gramine highlighting several use cases.
Gramine Future
Gramine is a growing project and continues to add more features and to increase the number of supported applications and runtimes. Gramine also serves as a research vehicle by several top academic groups for their security research that will shape the future of Confidential Computing.
Thanks to Gramine’s modular architecture, the project is extensible to different operating systems, architectures and environments. As of this writing we are working on extending Gramine to support the Intel® Trust Domain Extensions (Intel® TDX) technology. There is also a customer demand for running SGX workloads that securely communicate with hardware accelerators (devices such as GPU, TPU, etc.) that provide their own trusted execution environments. The respective support for such devices is being added to Gramine; however, it will be the responsibility of the application to secure any communication with the device. As an example, we are working on the proof-of-concept implementation for so-called “unified enclaves”, with SGX on Xeon CPUs and Protected Xe Path (PXP) on Intel’s datacenter GPUs.
We are planning to release new stable versions of Gramine regularly, approximately three times a year. Note that Gramine is an LGPL-licensed open-source project, and we welcome contributions to the core project or any of its related sub-projects.
To try Gramine, you have several options:
- Use an official Gramine Docker image. The Gramine community publishes an official Docker image with the latest stable release of Gramine.
- Use Confidential Computing images of Redis and PyTorch available at Azure Marketplace.
- Deploy Gramine containers with Intel SGX VM nodes on Azure Kubernetes Service (AKS)
- Install Gramine on your machine, by following our Quick Start guide.
For the latest information on our project, you can refer to the links below:
- Gramine project: http://www.gramineproject.io
- GitHub repo: https://github.com/gramineproject/gramine
- Gitter channel: https://gitter.im/gramineproject/community
[1] We should note that Intel SGX and Gramine have specific performance bottlenecks and limitations, therefore some workloads may exhibit poor performance. Refer to the Performance tuning and analysis documentation.
[2] To learn about all manifest options in Gramine, please refer to the corresponding Gramine documentation.
[3] In reality, the gramine-sgx-sign tool is not a perfect fit for two-phase signing because the tool requires the same view of the file system as on the development platform. (We show this tool in the diagram for simplicity.) Users can develop their own tools for two-phase signing by using Gramine’s Python API.