fortanix rust tutorial

2022-02-17 · 12 min read

Source: https://edp.fortanix.com/docs/installation/guide/

Next time, also check out: https://docs.conclave.net/machine-setup.html

Windows SGX setup: https://dqdongg.com/windows/2020/05/31/Windows-sgx.html

Fortanix on Azure DCsv3 VM #

If you don't have a recent Intel CPU that supports SGX, you'll need to develop on a remote machine.

This tutorial will show you how to get the basic toolchain set up for developing Fortanix rust-sgx applications on a remote Azure VM that supports SGX confidential compute.

Create our VM #

We're using Ubuntu 20.04 LTS here. If you use an older Ubuntu version, you will experience problems.

$ az vm create \
	--name sgx-test \
	--resource-group sgx-test_group \
	--size Standard_DC1s_v3 \
	--image Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest

{
  "fqdns": "",
  "id": "/subscriptions/XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX/resourceGroups/sgx-test_group/providers/Microsoft.Compute/virtualMachines/sgx-test",
  "location": "westus",
  "macAddress": "..",
  "powerState": "VM running",
  "privateIpAddress": "10.0.0.4",
  "publicIpAddress": "..",
  "resourceGroup": "sgx-test_group",
  "zones": ""
}

Sanity check for SGX support # 

$ ssh phlip9@XX.XX.XX.XX

# The SGX kernel device should be available
sgx-test:~$ ls -la /dev | grep sgx
drwxr-xr-x  2 root root          80 Apr 26 20:10 sgx
crw-rw-rw-  1 root root     10, 125 Apr 26 20:10 sgx_enclave
crw-rw----  1 root sgx_prv  10, 126 Apr 26 20:10 sgx_provision

# Kernel log should say something about SGX
sgx-test:~$ sudo dmesg | grep -i sgx
[    0.543687] sgx: EPC section 0x2c0000000-0x3bfffffff

If you've built the sgx-detect fortanix tool locally (targetting x86_64-unknown-linux) you can try scp'ing that over. It won't work just yet, since don't all SGX system libs installed.

# Try copying over a pre-built sgx-detect
$ scp ~/dev/fortanix-rust-sgx/target/release/sgx-detect phlip9@XX.XX.XX.XX:~/

sgx-test:~$ ./sgx-detect
./sgx-detect: error while loading shared libraries: libsgx_dcap_ql.so.1: cannot open shared object file: No such file or directory

(Ubuntu/Debian) Install SGX libraries via APT #

Adding deb repos with SGX dependencies.

# For the Intel `libsgx-` dependencies.
$ echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu focal main' | sudo tee /etc/apt/sources.list.d/intel-sgx.list
$ wget -qO - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo apt-key add -

# (Optional) If you need `clang`.
# $ echo "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main" | sudo tee /etc/apt/sources.list.d/llvm-toolchain-focal-10.list
# $ wget -qO - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -

# (Optional) If you're using the Azure DCAP client, `az-dcap-client`.
# $ echo "deb [arch=amd64] https://packages.microsoft.com/ubuntu/20.04/prod focal main" | sudo tee /etc/apt/sources.list.d/msprod.list
# $ wget -qO - https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -

$ sudo apt update

# You should see libsgx things here now.
$ apt search libsgx
libsgx-ae-epid/unknown 2.15.101.1-bionic1 amd64
  Intel(R) Software Guard Extensions QE and PvE
# ..
libsgx-urts-dbgsym/unknown 2.15.101.1-bionic1 amd64
  debug symbols for libsgx-urts

$ sudo apt upgrade

# You may need these if you're doing more than just running `sgx-detect`.
# $ sudo apt install \
# 	clang-10 libssl-dev gdb libsgx-enclave-common libsgx-quote-ex \
# 	libprotobuf17 libsgx-dcap-ql libsgx-dcap-ql-dev az-dcap-client \
# 	open-enclave ninja-build

# SGX dependencies
$ sudo apt install libsgx-enclave-common libsgx-quote-ex libsgx-dcap-ql libsgx-dcap-ql-dev

# Standard dev tools
$ sudo apt install build-essential libssl-dev pkg-config

# Fortanix needs protoc for the AESM client
$ sudo apt install protobuf-compiler

# Install rust
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Add SGX target
$ rustup target add x86_64-fortanix-unknown-sgx

(macOS, build-only) Install SGX deps via Homebrew # 

$ brew install protobuf

# Install rust
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Add SGX target
$ rustup target add x86_64-fortanix-unknown-sgx

Build and run Fortanix sgx-detect tools #

Let's verify our install with sgx-detect.



# # Install fortanix SGX tools (via crates.io)
# $ cargo install fortanix-sgx-tools sgxs-tools
# # Failed b/c published crates are 4 mo out-of-date and need weird nightly
# # version to compile.

# Install fortanix SGX tools (via git repo) since the crates.io version
# is very out-of-date.
$ git clone https://github.com/fortanix/rust-sgx.git --depth=1
$ cargo install --path ./intel-sgx/fortanix-sgx-tools
$ cargo install --path ./intel-sgx/sgxs-tools

$ sgx-detect
Detecting SGX, this may take a minute...
✔  SGX instruction set
  ✔  CPU support
  ✔  CPU configuration
  ✔  Enclave attributes
  ✔  Enclave Page Cache
  SGX features
    ✔  SGX2  ✔  EXINFO  ✘  ENCLV  ✘  OVERSUB  ✔  KSS
    Total EPC size: 4.0GiB
✔  Flexible launch control
  ✔  CPU support
  ？ CPU configuration
  ✔  Able to launch production mode enclave
✔  SGX system software
  ✔  SGX kernel device (/dev/sgx_enclave)
  ✔  libsgx_enclave_common
  ✔  AESM service
  ✔  Able to launch enclaves
    ✔  Debug mode
    ✔  Production mode
    ✔  Production mode (Intel whitelisted)

You're all set to start running SGX programs!


$ echo >> ~/.cargo/config -e '[target.x86_64-fortanix-unknown-sgx]\nrunner = "ftxsgx-runner-cargo"'

(Failed) Attempt at SGX inside WSL2 #

Install protobuf compiler

# Ubuntu/Debian/WSL
$ sudo apt install protobuf-compiler
$ protoc --version

libprotoc 3.6.1

Installing libsgx-dcap-ql-dev

# make sure we're running on focal Ubuntu release
$ lsb_release -a

$ echo "deb https://download.01.org/intel-sgx/sgx_repo/ubuntu focal main" | sudo tee -a /etc/apt/sources.list.d/fortanix.list >/dev/null

$ curl -sSL https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo -E apt-key add -

$ sudo apt update
$ sudo apt install libsgx-dcap-ql-dev

Installing Linux SGX drivers

On Intel's website, find the latest “Intel SGX Linux Release” (not “Intel SGX DCAP Linux Release”) and download the “Intel (R) SGX Installers” for your platform. The package will have driver in the name.

Ubuntu 20.04 Drivers: https://download.01.org/intel-sgx/sgx-linux/2.15.1/distro/ubuntu20.04-server/

Installation Guide: https://download.01.org/intel-sgx/sgx-dcap/1.12.1/linux/docs/Intel_SGX_SW_Installation_Guide_for_Linux.pdf

$ sudo apt install build-essential ocaml automake autoconf libtool libssl-dev dkms

$ wget https://download.01.org/intel-sgx/sgx-linux/2.15.1/distro/ubuntu20.04-server/sgx_linux_x64_driver_1.41.bin

$ chmod a+x sgx_linux_x64_driver_1.41.bin
$ sudo ./sgx_linux_x64_driver_1.41.bin

Error

Error! Your kernel headers for kernel 5.10.93.2-microsoft-standard-WSL2 cannot be found.
Please install the linux-headers-5.10.93.2-microsoft-standard-WSL2 package,
or use the --kernelsourcedir option to tell DKMS where it's located

I conveniently already have this kernel pulled : )

$ sudo ./sgx_linux_x64_driver_1.41.bin --kernelsourcedir

Unfortunately, seems like it won't recognize the path... My guess is that SGX inside WSL2 is probably a dead end. :(

Attempt in Win11 # 

PS > git config --global http.sslCAinfo "C:\Users\phlip9\scoop\apps\mingit-busybox\current\mingw64\ssl\certs\ca-bundle.crt"
PS > scoop update

PS > scoop install openssl

PS > scoop bucket add extras
PS > scoop install protobuf

# make sure our nightly compiler is recent
PS > rustup toolchain install nightly-2021-12-15

# install the SGX toolchain
PS > rustup target add x86_64-fortanix-unknown-sgx --toolchain nightly-2021-12-15

# install the SGX tools
PS > cargo +nightly-2021-12-15 install fortanix-sgx-tools sgxs-tools

Didn't detect OpenSSL library...

EDIT: nvm had to restart whole Windows Terminal app, not just open a new shell... pretty fuckin annoying tbh...

OK, it picked up the OpenSSL library and protobuf compiler. Unfortunately it now fails b/c some file is hard depending on std::os::unix which doesn't exist on windows...

PS > git clone git@github.com:fortanix/rust-sgx.git

PS > cargo +nightly-2021-12-15 build -p fortanix-sgx-tools -p sgxs-tools

Error

error[E0433]: failed to resolve: could not find `unix` in `os`
 --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\info.rs:7:9
  |
7 |     os::unix::ffi::OsStringExt,
  |         ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
 --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\list.rs:4:9
  |
4 |     os::unix::ffi::OsStrExt,
  |         ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
 --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\swaps.rs:7:9
  |
7 |     os::unix::ffi::OsStringExt,
  |         ^^^^ could not find `unix` in `os`

error[E0599]: no function or associated item named `from_vec` found for struct `OsString` in the current scope
   --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\info.rs:130:22
    |
130 |         Ok(OsString::from_vec(ret))
    |                      ^^^^^^^^ function or associated item not found in `OsString`

error[E0599]: no method named `as_bytes` found for reference `&OsStr` in the current scope
  --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\list.rs:55:43
   |
55 |         self.starts_with(path.as_os_str().as_bytes(), |m| &m.source)
   |                                           ^^^^^^^^ method not found in `&OsStr`

error[E0599]: no method named `as_bytes` found for reference `&OsStr` in the current scope
  --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\list.rs:63:43
   |
63 |         self.starts_with(path.as_os_str().as_bytes(), |m| &m.dest)
   |                                           ^^^^^^^^ method not found in `&OsStr`

error[E0599]: no method named `as_bytes` found for reference `&OsStr` in the current scope
  --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\mounts\list.rs:72:49
   |
72 |             let input = func(mount).as_os_str().as_bytes();
   |                                                 ^^^^^^^^ method not found in `&OsStr`

error[E0599]: no function or associated item named `from_vec` found for struct `OsString` in the current scope
   --> C:\Users\phlip9\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-mounts-0.2.4\src\swaps.rs:106:22
    |
106 |         Ok(OsString::from_vec(ret))
    |                      ^^^^^^^^ function or associated item not found in `OsString`

OK with some fixes (hacks) I got it to build

Running sgx-detect

PS > .\target\release\sgx-detect.exe

Detecting SGX, this may take a minute...
thread 'main' panicked at 'Couldn't find sgx_uae_service.dll: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." }', intel-sgx\aesm-client\src\imp\windows.rs:67:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I think I'm missing some stuff installed?

Let's have a look at the Intel SGX SDK for Windows guide: https://cdrdv2.intel.com/v1/dl/getContent/671222?explicitVersion=true

Installing Intel Management Engine Driver (for Win 10 (though I'm on Win 11 loe))

In the installer.zip > Cons (for consumer) > MEI-Only Installer MSI (don't want other BS) > MEISetup.exe

Verifying we have SGX not disabled in the BIOS. Under Advanced/CPU Config/SW Guard Extensions (SGX), select SW Controlled and not Disabled. Curiously no direct Enabled option?

Note: "Trusted Computing" refers to TPM (Trusted Platform Module) and not SGX. I believe it's used for UEFI secure boot or something.

Now let's try to get the SGX platform software and drivers installed.

(One thing that makes me nervous, is that many docs suggest that the SGX platform software should be pre-installed via Windows Update if SGX is supported. However, ls /mnt/c/Windows/System32 | grep -i sgx doesn't show anything...)

Just saw this note here (https://www.intel.com/content/www/us/en/developer/articles/guide/sgx-sdk-installation-guide.html)

Note: You can’t install Intel® SGX PSW installer when Windows OS is installed in Legacy mode and Intel® SGX is configured as “Software Controlled” in BIOS. You need to configure Intel® SGX as “Enabled” in BIOS before installing Intel® SGX PSW.

Does this mean Legacy AND SW Controlled or Legacy OR SW Controlled???

Something to look into? https://github.com/intel/sgx-software-enable Would need to live-boot into a linux distro in UEFI boot mode, then compile and run this I guess.

useful command msinfo32

BIOS Mode: UEFI Secure Boot State: Off (Why??) Total RAM: 16 GB Available RAM: 9.85 GB (??)

Booted into my Pop!-OS live USB. Pulled and built the sgx_enable program. Looks like it ran successfully!!

Booting into the BIOS still shows Software Controlled but I don't know if that's just 'cus the BIOS can't show it properly.

Booting back into Win11, I notice something's changed in msinfo32. Under System Summary > Hardware Resources > Memory, there's a new entry!

ResourceDeviceStatus
0xa0200000-0xa5f7ffffIntel Software Guard Extensions DeviceOK

Opened Windows Update and checked for updates. A "Intel Software Component" looks like it installed? Going to restart just to be sure these things actually installed lmao.

AYYYYY

λ ls /c/Windows/System32/ | grep -i sgx
sgx_dcap_ql.dll*
sgx_dcap_quoteverify.dll*
sgx_enclave_common.dll*
sgx_epid.dll*
sgx_launch.dll*
sgx_platform.dll*
sgx_quote_ex.dll*
sgx_uae_service.dll*
sgx_urts.dll*

Running sgx-detect again

PS > .\target\release\sgx-detect.exe -v

Detecting SGX, this may take a minute...
✔  SGX instruction set
  ✔  CPU support
  ✔  CPU configuration
  ✔  Enclave attributes
  ✔  Enclave Page Cache
  SGX features
    ✘  SGX2  ✘  EXINFO  ✘  ENCLV  ✘  OVERSUB  ✘  KSS
    Total EPC size: 93.5MiB
✘  Flexible launch control
  ✔  CPU support
  ？ CPU configuration
  ✘  Able to launch production mode enclave
✔  SGX system software
  ✘  SGX kernel device
  ✔  libsgx_enclave_common
  ✔  AESM service
  ✔  Able to launch enclaves
    ✔  Debug mode
    ✘  Production mode
    ✔  Production mode (Intel whitelisted)

🕮   Flexible launch control > CPU configuration
Your hardware supports Flexible Launch Control, but whether it's enabled could not be determined. More information might be available by re-running this program with sudo. Would you like to do that?
(not supported yet)

debug: Error reading MSR 3Ah: RDMSR not implemented on Windows

More information: https://edp.fortanix.com/docs/installation/help/#flc-cpu-configuration

🕮   SGX system software > SGX kernel device
The SGX device (/dev/sgx/enclave, /dev/sgx or /dev/isgx) is not present.

It could not be detected whether you have an SGX driver installed. Please make sure the SGX driver is installed and loaded correctly.

debug: Error opening device: Device Driver Path not supported in Windows
debug: Error checking module status: KMOD Status not implemented on Windows

More information: https://edp.fortanix.com/docs/installation/help/#sgx-driver

🕮   SGX system software > Able to launch enclaves > Production mode
The enclave could not be launched. This might indicate a problem with FLC.

debug: failed to load report enclave
debug: cause: The EINITTOKEN provider didn't provide a token
debug: cause: aesm error code Unknown(16388)

More information: https://edp.fortanix.com/docs/installation/help/#run-enclave-prod

You're all set to start running SGX programs!

I think this is OK? I'm pretty sure our CPU isn't recent enough to support FLC (Flexible Launch Control). I also didn't see anything relevant in the BIOS settings. It looks like it was able to launch an enclave in debug mode at least?

Will need to buy newer Intel CPU for more serious development.