Release Management & Docker Images¶
Release process and tools¶
- GoReleaser Pro is used to do the actual releasing using .goreleaser.yml. GoReleaser Pro is a licensed product that requires separate installation and a license key, which is in the GitHub Workflow configuration and is available in 1Password to DDEV maintainers who need it.
- The Master Build/Release GitHub Action does the actual running of the GoReleaser actions and provides the needed secrets.
GitHub Actions Required Secrets¶
How to add new people to these accounts¶
- AUR is Arch Linux User Repository.
ddev-bin
is athttps://aur.archlinux.org/packages/ddev-bin
. The current maintainer of this is @rfay, who can add co-maintainers. - The chocolatey package. Additional maintainers can be added at (login required)
https://community.chocolatey.org/packages/ddev/ManagePackageOwners
; they could then create tokens to push it. - GitHub requires write access to this repository, either via permissions on the repository or on the org.
- Apple signing and notarization requires access to the DDEV Foundation group on
https://developer.apple.com
. It’s easy enough to add additional people. - Windows signing is an awkward process that requires a dongle. When the current signing certificate expires we definitely want the simpler approach.
- Discord
- Docker
Actual secrets required¶
The following “Repository secret” environment variables must be added to https://github.com/ddev/ddev/settings/secrets/actions:
AMPLITUDE_API_KEY
: Key that enables Amplitude reporting. Environment variable for Make isAmplitudeAPIKey
.AMPLITUDE_API_KEY_DEV
: Key that enables Amplitude reporting for development versions e.g. a PR build. Environment variable for Make isAmplitudeAPIKey
.AUR_EDGE_GIT_URL
: The Git URL for AUR edge (normallyddev-edge-bin
), for examplessh://aur@aur.archlinux.org/ddev-edge-bin.git
.AUR_STABLE_GIT_URL
: The Git URL for AUR stable (normallyddev-bin
), for examplessh://aur@aur.archlinux.org/ddev-bin.git
.AUR_SSH_PRIVATE_KEY
: Private SSH key for theddev-releaser
user. This must be processed into a single line, for example,perl -p -e 's/\n/<SPLIT>/' ~/.ssh/id_rsa_ddev_releaser| pbcopy
.CHOCOLATEY_API_KEY
: API key for Chocolatey.DDEV_GITHUB_TOKEN
: GitHub personal token (repo
scope, classic PAT) that gives access to create releases and push to the Homebrew repositories.DDEV_MACOS_APP_PASSWORD
: Password used for notarization, see signing_tools.DDEV_MACOS_SIGNING_PASSWORD
: Password for the macOS signing key, see signing_tools.DDEV_MAIN_REPO_ORGNAME
: The organization to be used for testing, normallyddev
but it may beddev-test
for the test organization.DDEV_WINDOWS_SIGNING_PASSWORD
: Windows signing password.DOCKERHUB_USERNAME
: Username for pushing tohub.docker.com
or updating image descriptions.DOCKERHUB_TOKEN
: Token for pushing tohub.docker.com
. or updating image descriptions.FURY_ACCOUNT
: Gemfury account that receives package pushes.FURY_TOKEN
: Push token assigned to the above Gemfury account.GORELEASER_KEY
: License key for GoReleaser Pro.HOMEBREW_EDGE_REPOSITORY
: Likeddev/homebrew-ddev-edge
but may beddev-test/homebrew-ddev-edge
.HOMEBREW_STABLE_REPOSITORY
: Likeddev/homebrew-ddev-edge
but may beddev/homebrew-ddev-edge
.
Creating a Release¶
This is completely automated now, so nothing needs to be done unless something goes wrong.
Prerelease Tasks¶
- Create and execute a test plan.
- Make sure
version-history.md
is up to date. - Push the new version of
ddev/ddev-php-base
. - Update
ddev/ddev-webserver
to use the new version ofddev/ddev-php-base
and push it with the proper tag. - Make sure the Docker images are all tagged and pushed.
- Make sure
pkg/versionconstants/versionconstants.go
is all set to point to the new images and tests have been run. -
If the
devcontainer-feature.json
(for GitHub Codespaces) needs to be updated, use thedevcontainer
CLI and aGITHUB_TOKEN
that has power to manage packages (write:packages
scope, classic PAT), change the version in thedevcontainer-feature.json
and run:
Actual Release Creation¶
- Create a release for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
- Make sure you’re about to create the right release tag.
- Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
Post-Release Tasks¶
- After the release has been created, the new gitpod image must be pushed.
cd .gitpod/images && DOCKER_TAG="<YYMMDD>" ./push.sh
- PR to update
.gitpod.yml
with the new image. - PR to update ddev-gitpod-launcher with the new image.
Pushing Docker Images with the GitHub Actions Workflow¶
The easiest way to push Docker images is to use the GitHub Actions workflow, especially if the code for the image is already in the ddev/ddev repository.
Actual release creation¶
- Create a release for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
- Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
- Verify that Homebrew (Linux and macOS) and Chocolatey and AUR are working correctly with the right versions.
You can push all images besides ddev-dbserver
at https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml
You can push ddev-dbserver
images at https://github.com/ddev/ddev/actions/workflows/push-tagged-dbimage.yml
If you need to push from a forked PR, you’ll have to do this from your fork (for example, https://github.com/rfay/ddev/actions/workflows/push-tagged-image.yml
), and you’ll have to specify the branch on the fork. This requires setting the DOCKERHUB_TOKEN
and DOCKERHUB_USERNAME
secrets on the forked PR, for example https://github.com/rfay/ddev/settings/secrets/actions
. You can do the same with ddev-dbserver
at https://github.com/rfay/ddev/actions/workflows/push-tagged-dbimage.yml
for example.
- Visit
https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml
. - Click the “Push tagged image” workflow on the left side of the page.
- Click the “Run workflow” button in the blue section above the workflow runs.
- Choose the branch to build from (usually
master
). - Enter the image (
ddev-webserver
,ddev-php-base
, etc.). - Enter the tag that will be used in
pkg/version/version.go
.
Pushing Docker Images Manually¶
While it’s more error-prone, images can be pushed from the command line:
docker login
with a user that has push privileges.docker buildx create --name ddev-builder-multi --use
or if it already exists,docker buildx use ddev-builder-multi
.cd containers/<image>
.- Before pushing
ddev-webserver
, make sure you’ve pushed a version ofddev-php-base
and updatedddev-webserver
’s Dockerfile to use that as a base. make push VERSION=<release_version> DOCKER_ARGS=--no-cache
for most of the images. Forddev-dbserver
it’smake PUSH=true VERSION=<release_version> DOCKER_ARGS=--no-cache
. There’s a push-all.sh script to update all of them, but it takes forever.ddev-dbserver
images can be pushed withmake PUSH=true VERSION=<release_version> DOCKER_ARGS=--no-cache
from thecontainers/ddev-dbserver
directory.
Maintaining ddev-dbserver
MySQL 5.7 and 8.0 ARM64 Images¶
Sadly, there are no ARM64 Docker images for MySQL 5.7 and 8.0, so we have our own process to maintain ddev/mysql-arm64-images and ddev/xtrabackup-build images for DDEV.
ddev/mysql:5.7
uses Ubuntu 18.04 as the base image, and Ubuntu 18.04 ARM64 hasmysql-server
5.7 in it, so we can install.ddev/mysql:8.0
uses Ubuntu 20.04 as the base image, and Ubuntu 20.04 ARM64 hasmysql-server
8.0 in it, so we can install it from packages.- Unfortunately, the
ddev snapshot
command depends onxtrabackup
8.0 being installed formysql:8.0
. There are no ARM64 packages or binaries provided by Percona forxtrabackup
, so we build it from source with ddev/xtrabackup-build. There’s a catch, however:xtrabackup
’s development cycle lags behindmysql:8.0
’s development cycle, so you can’t build a usableddev/mysql:8.0
image until there’s anxtrabackup
version released. Further, when Ubuntu bumpsmysql-server-8.0
to a new version, there’s no way to use the old one. So the only time that you can maintainddev/mysql:8.0
is when Ubuntu 20.04 has the same version that’s released forpercona-xtrabackup
. (In the case at this writeup, I was finally able to buildpercona-xtrabackup
8.0.28, and the same day Ubuntu bumped its packages to 8.0.29, meaning that it was unusable.) - To build percona-xtrabackup, follow the instructions on ddev/xtrabackup-build. Create a release with the release of Percona xtrabackup, for example
8.0.29-21
. When that succeeds, then there is an upstream xtrabackup to be used in the ddev/mysql:8.0 build. - To build
ddev/mysql
(both 5.7 and 8.0) ARM64 images, follow the instructions on ddev/mysql-arm64-images. After the various files are updated, you can push a new release and the proper images will be pushed. - After building a new set of
ddev/mysql
images, you’ll need to pushddev/ddev-dbserver
with new tags. Make sure to update the ddev/ddev-dbserver
Makefile to set the explicit version of the upstreammysql:8.0
(for example, 8.0.29, if you’ve succeeded in getting 8.0.29 forpercona-xtrabackup
andmysql:8.0
).
Actual Release Docker Image Updates¶
We may not build every image for every point release. If there have been no changes to ddev-traefik-router
or ddev-ssh-agent
, for example, we may not push those and update pkg/version/version.go
on major releases.
But here are the steps for building:
- The
ddev/ddev-php-base
image must be updated as necessary with a new tag before pushingddev-webserver
. You can do this using the process above. - The
ddev/ddev-webserver
Dockerfile mustFROM ddev/ddev-php-base:<tag>
before building/pushingddev-webserver
. But then it can be pushed using either the GitHub Actions or the manual technique. - If you’re bumping
ddev-dbserver
8.0 minor release, follow the upstream Maintaining ddev-dbserver MySQL 5.7 & 8.0 ARM64 Images instructions. - Update
pkg/version/version.go
with the correct versions for the new images, and run all the tests.
Manually Updating Homebrew Formulas¶
Homebrew formulas normally update with the release process, so nothing needs to be done.
If you have to temporarily update the Homebrew formulas, you can do that with a commit to https://github.com/ddev/homebrew-ddev and https://github.com/ddev/homebrew-ddev-edge. The bottles and checksums for macOS (High Sierra) and x86_64_linux are built and pushed to the release page automatically by the release build process (see bump_homebrew.sh). Test brew upgrade ddev
both on macOS and Linux and make sure DDEV is the right version and behaves well.
Manually Updating Chocolatey¶
Normally the release process does okay with pushing to Chocolatey, but at times a failure can happen and it’s not worth doing the whole release process again.
Note that if an existing approved release is being updated you have to have a new version. So for example, if v1.21.3
failed, you’ll need to work with v1.21.3.1
, so make chocolatey VERSION=v1.21.3.1
below.
- Open up Gitpod, https://gitpod.io/#https://github.com/ddev/ddev and
cd /workspace/ddev
git checkout <tag>
sudo apt-get update && sudo apt-get install -y nsis
sudo .ci-scripts/nsis_setup.sh /usr/share/nsis
- Edit the checksum in
tools/chocolateyinstall.ps1
to match the released checksum of theddev-windows-installer
inchecksums.txt
of the release that is being repaired, for example, forv1.21.3
this would be the checksum forddev_windows_installer.v1.21.3.exe
in v1.21.3 checksums.txt. - Edit
url64
intools/chocolateyinstall.ps1
to be the intended actual DDEV download version - edit the version where it appears and edit the GitHub org. For example, if the actual version of DDEV to be downloaded isv1.21.3
then put that there.
make chocolatey VERSION=<tag>
export CHOCOLATEY_API_KEY=key33333
cd .gotmp/bin/windows_amd64/chocolatey
docker run --rm -v $PWD:/tmp/chocolatey -w /tmp/chocolatey linuturk/mono-choco push -s https://push.chocolatey.org/ --api-key "${CHOCOLATEY_API_KEY}"
Manually Updating AUR Repository¶
The AUR repository normally updates with the release process, so nothing needs to be done.
However, you can manually publish the release to the DDEV AUR repository. The README.md in the AUR Git repository (ssh://aur@aur.archlinux.org/ddev-bin.git
or https://aur.archlinux.org/ddev-bin.git
) has instructions on how to update, including how to do it with a Docker container, so it doesn’t have to be done on an ArchLinux or Manjaro VM.
Manually Signing the Windows Installer¶
This is done by the release process, but the manual process is documented here.
This is done automatically by the release build on a dedicated Windows test runner (GitHub Actions runner) named testbot-asus-win10pro
. You would need to do this process manually on that build machine or install the fob on another machine.
After rebooting this machine, sometimes an automated reboot, the password for the security fob has to be re-entered or Windows signing will fail. We do this by opening up tb-win11-06
using Chrome Remote Desktop (or manually physically opening it), opening Git Bash, and cd ~/tmp && signtool sign gsudo.exe
. There happens to be a gsudo.exe
there but it doesn’t matter what you sign—the idea is to pop up the GUI where you enter the password (which is in 1Password).
Basic Instructions¶
- Install the suggested Windows SDK. Only the signing component is required.
- Add the path of the kit binaries to the Windows system PATH,
C:/Program Files (x86)/Windows Kits/10/bin/10.0.22621.0/x64/
. - The keyfob and Safenet Authentication Client must be installed. The best documentation for the Safenet software is at https://support.globalsign.com/ssl/ssl-certificates-installation/safenet-drivers. You must configure the advanced client settings to “Enable single logon” or it will require the password on each run.
- After
make windows_install
theddev-windows-installer.exe
will be in.ddev/bin/windows_amd64/ddev_windows_installer.exe
and you can sign it withsigntool sign ddev-windows-installer.exe
. - If you need to install the GitHub self-hosted Windows runner, do it with the instructions in project settings → Actions → Runners.
- Currently the
actions/cache
runner does not work out of the box on Windows, so you have to install tar and zstd as described in this issue.
We shouldn’t use this high-security keyfob approach to signing on the next go-around with the certs.
It’s way too difficult to manage, and the Safenet software is atrocious.
APT and YUM/RPM Package Management¶
The Linux apt
and yum
/rpm
packages are built and pushed by the nfpms
and furies
sections of the .goreleaser.yml file.
- The actual packages are served by gemfury.com.
- The name of the organization in GemFury is
drud
, managed athttps://manage.fury.io/dashboard/drud
. - Randy Fay, Matt Stein, and Simon Gillis are authorized as owners on this dashboard.
- The
pkg.ddev.com
domain name is set up as a custom alias for our package repositories; seehttps://manage.fury.io/manage/drud/domains
. (Users do not seedrud
anywhere. Although we could have moved to a new organization for this, the existing repositories contain all the historical versions so it made sense to be less disruptive.) - The
pkg.ddev.com
CNAME
is managed in CloudFlare becauseddev.com
is managed there. - The fury.io tokens are in DDEV’s shared 1Password account.
Testing Release Creation¶
When significant changes are made to the .goreleaser.yml
or related configuration, it’s important to be able to test without actually deploying to ddev/ddev/releases
of course. We have two ways to test the configuration; we can run goreleaser
manually for simpler tests, or run a full release on ddev-test/ddev
where needed.
Running goreleaser
manually¶
This approach is great for seeing what artifacts get created, without deploying them.
Prerequisites:
- GoReleaser Pro must be installed
export GORELEASER_KEY=<key>
export GITHUB_REPOSITORY_OWNER=ddev-test
git tag <tagname> # Try to include context like PR number, for example v1.22.8-PR5824
make windows_amd64 windows_arm64 darwin_amd64 darwin_arm64 linux_amd64 linux_arm64 completions
goreleaser release --prepare --nightly --clean
This will create all the artifacts that would have been pushed in the dist
directory. You can copy Linux packages from there to test them manually, download the built tarballs for use elsewhere, install Homebrew package manually, for example:
Creating a test release on ddev-test/ddev
¶
ddev-test/ddev is now set up for actual release testing. It has all or most of the environment variables set up already. It also acts against ddev-test/homebrew-ddev
and ddev-test/homebrew-ddev-edge
so you can test Homebrew publishing.
- Create a branch on
ddev-test/ddev
. - Using the web UI, create a release using that branch as base. The release tag must start with
v1.
. Where possible, please use a release tag that includes context about the PR you are working against, likev1.28.8-PR2022FixStuff
, and include in the release notes a link to the issue. The tag must be a valid Semantic Version tag, so don’t use underscores, etc. - Test out the resulting artifacts that get published or deployed.