2022-04-05 · 7 min read
Earthly replaces Makefile+Dockerfile with a Docker-like DSL. You write Make-like build targets, which can have dependencies on other targets. The contents of each build target is (effectively) a Dockerfile.
The ultimate goal is repeatable, reproducible builds. Repeat a failed build from CI. Seamlessly reproduce your developer environment on a colleague's machine. Etc etc...
Earthly vs Nix #
Both use linux namespaces for isolation. Nix has an absolutely awful DSL language that I have to relearn every time I touch it. Earthfiles are significantly more imperative; they look like a list of Makefile targets with Dockerfile recipes.
Earthly vs Buck / Bazel #
Bazel and Buck provide truely hermetic builds, but they each require complete control over the entire toolchain, so no using
cargo, etc... This is fine for Google/Facebook, but not for smaller teams (IMO).
Earthly claims to strike a more pragmatic balance between truely repeatable and deterministic builds and productive development for a smaller team.
- It appears build steps must explicitly mark files as "artifacts". You must then explicitly copy these artifacts over in dependent build steps.
- Likewise, output artifacts must be explicitly exported to the user's FS from the container FS.
- Exported artifacts are not transitive, so if your dependency explicitly exports to the user's FS, your build step won't unless you also explicitly export.
- Can easily push artifacts to remote destinations, run db migrations, cut releases, etc... as a build step.
- Like normal Dockerfiles, an intermediate build step is cached as a new layer.
- Seems like a solid choice for a CI pipeline and devs occasionally reproducing CI runs locally.
- Builds are nicely isolated from the dev machine, reducing implicit state that's actually necessary for builds to succeed.
- Builds are (mostly) reproducible across environments. Though, my experience with a big Rust project was that this was not a problem.
- I love that build step inputs and outputs are marked explicitly. Following an unfamiliar Earthfile doesn't feel too bad.
- Once you've set up a common development base image, all developers can share good quality tools and configuration without have to independently set up debuggers, profilers, etc...
- Can use a shared build cache like Bazel, Buck, or scc so beefy builds don't have to take forever. CI can also shared this build cache!
- Like Docker, toolchains are stuck inside containers, usually in a way that is inaccessible to other dev tools, like IDEs, debuggers, profilers, etc...
- I worry that basic dev tools like running an LSP might be challenging to set up? Would this be like mounting the dev directory as a volume for the LSP service? Or is this not even a problem at all... idk. I don't use Docker frequently enough to know.
- Some weird idiosyncrasies, presumably due to the Docker layering model. Developers need a solid mental model to avoid committing egregiously large, slow, or uncacheable intermediate layers.
- Example: explicit caching of derived Cargo dependency state in first three build steps: https://github.com/earthly/earthly/blob/main/examples/rust/Earthfile
- Rust has a nice enough runner (
cargo) that you don't hit a lot of the reproducibility problems experienced in other languages *cough* C/C++.
- Caching is still too coarse-grained compared to running stateful native tools like
- Unlike Bazel or Buck, which take full control of the toolchain, it seems more likely to hit non-determinism with Earthly, since we're using language tools which typically don't know or care about reproducibility.
- Get to learn yet another fun and exciting DSL, with its own special and unique patterns and other miscellaneous weirdness.
- Too slow in tight development loop. On my beefy desktop, a no-op build takes at least 5s. On my M1 MBP, a no-op build takes at least 8-10s. This means any non-trivial build step has a minimum 8-10s duration, which is frankly unacceptable. Non-trivial file copying also adds significant overhead.
- Multi-platform builds (targetting amd64/linux) failed on my M1 Mac for inscrutable reasons. Most likely not earthly's fault, but an issue nonetheless.
- Feels unwieldy running integration tests that need access to low-level hardware devices. Our requirement is probably not typical though.
- Remote builder authentication uses mTLS certs. Provisioning these seems like a pain, esp. when I already have ssh secrets provisioned : (
- Overall, Earthly still feels a bit early. I'd check back after another 6mo (that would be Q1 2023 as of writing).
# Ubuntu/Debian/PopOS! $ curl --proto '=https' --tlsv1.3 -sSfL https://pkg.earthly.dev/earthly.pgp \ | gpg --dearmor \ | sudo tee /usr/share/keyrings/earthly.gpg $ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/earthly.gpg] https://pkg.earthly.dev/deb stable main" \ | sudo tee /etc/apt/sources.list.d/earthly.list > /dev/null $ sudo apt update $ sudo apt install earthly # macOS $ brew install earthly
Earthly Remote Build #
Install docker on remote (Ubuntu) #
Install docker if you haven't already
$ sudo mkdir -p /etc/apt/keyrings $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg $ echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \ | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null $ sudo apt update $ sudo apt install docker-ce docker-ce-cli containerd.io \ docker-compose-plugin $ sudo groupadd docker $ sudo usermod -aG docker $USER $ newgrp docker # activate changes? else log out and back in # sanity check docker installation $ docker run --rm hello-world # start docker on restart $ sudo systemctl enable docker.service $ sudo systemctl enable containerd.service
Run earthly/buildkit (remote) #
This runs a modified buildkit listening on TCP port 8372, without authentication.
$ docker run \ --privileged -t -v earthly-tmp:/tmp/earthly:rw \ -e BUILDKIT_TCP_TRANSPORT_ENABLED=true -p 8372:8372 \ earthly/buildkitd:v0.6.15
(FAIL) Run ssh forward (host) #
$ ssh -NL 8372:localhost:8372 firstname.lastname@example.org
FAIL: It appears Earthly tries to do a local build if the buildkit hostname is
localhost : /
Open VM port #
Very unsafe. Do this only for brief sanity testing.
$ az vm open-port --name sgxdev2 --port 8372
TODO: provision TLS certs.
Test remote build #
$ EARTHLY_BUILDKIT_HOST=tcp://my-remote-buildkit:8372 earthly +my-cool-target
Misc. Notes #
Earthfile Syntax #
SAVE ARTIFACT .. AS LOCAL ..+ Only save the output locally if we ask for the specific target OR we use the
SAVE IMAGE foo:latest+ Saves current target layer as a docker image named
SAVE IMAGE --push ..+ Push image to remote repo
RUNcommand defines an "external" command. These will only run if the entire build succeeds. You also need to run the earthly build with
--pushto enable these commands.
release: RUN --push --secret GITHUB_TOKEN=+secrets/GH_TOKEN github-release upload
$ earthly --push +release
Also useful for running things like DB migrations
migrate: FROM +build RUN --push bundle exec rails db:migrate
or terraform apply
apply: RUN --push terraform apply -auto-approve
- Targets in other Earthfile (relative path, in same repo)
build: # ./services/foobar/Earthfile # -> contains `deps` target FROM ./services/foobar+deps # ..
- Import targets from other Earthfile
VERSION 0.6 IMPORT ./services/foobar # .. build: FROM foobar+deps # ..
Run docker commands inside a target using
WITH DOCKER .. END+ Will init a docker daemon that can be used in a
RUNcommand + Recommend using earthly's "docker-in-docker" (dind) container
Pulling a docker image from docker hub
hello: FROM earthly/dind:alpine WITH DOCKER --pull hello-world RUN docker run hello-world END
- Loading an image created by another target
my-hello-world: FROM ubuntu CMD echo "hello world" SAVE IMAGE my-hello:latest hello: FROM earthly/dind:alpine WITH DOCKER --load hello:latest=+my-hello-world RUN docker run hello:latest END