We’re quite proud of the automated testing setup we use to develop DDEV. Each pushed commit to the project gets a full set of tests run on macOS, Windows (including WSL2), and Linux.
Originally we tested quite nicely with CircleCI (on Linux), and that did pretty well at finding issues with macOS. But when we started supporting Windows and macOS Apple Silicon we found that we really needed explicit coverage on those machines because there are a lot of behavior differences, so over time we moved everything to GitHub Actions, including self-hosted runners (mostly on Buildkite) that we maintain ourselves for macOS and Windows.
Even though DDEV is written in Go, which is amazingly complete for each platform, things like filesystem case-sensitivity, slashes, and behavior of docker-compose cause plenty of unexpected issues. So we run more than 10 test suites on each push, building and testing the containers on Linux, traditional Windows, WSL2 (both Docker Desktop and Docker CE), and macOS (Apple Silicon and Intel). And it also includes all the permutations for Apache support and the Mutagen features. It’s a lot of computing power, but it’s paid off enormously. All of this shows up very nicely on the DDEV pull request and commits pages, so it’s easy to see at a glance what’s happened with testing.
Our test suite is large and has grown substantially over the lifespan of DDEV, and gets run on every push. Browse the .github/workflows directory and the cmd and pkg test files for a current sense of scale.
The biggest problems we have with this whole setup is it seems to push Docker to the limit, especially on Windows. So we see too often the Docker failures and hangs that users see only infrequently. Currently, this means a lot of random maintenance of Docker on the test machines.
One of our ongoing goals is to try to get the macOS and Windows test runners out of Randy’s house, as this is too dependent on him. Watch the issue to find out more — it’s still open as of this update. Here’s a picture of what most of the macOS and Windows test runners looked like when this was written:
Try out DDEV for yourself, and consider contributing to it as an open source project! We have a contribution guide for this project as well as a Community Guide and Code of Conduct for all of our projects.