CI Aligned Developer Experience
The CI Aligned Developer Experience a.k.a “Makefile Process” is an attempt to simplify and align the processes that run on the Continuous Integration Server (Jenkins) with the processes that the developers run locally. This takes on two distinct workflows that exist for separate user needs. These changes are being driven out by the need to split out the Sirius application containers into smaller, discrete containers, built with officially maintained images.
This benefits the team by delivering:
- Closer Dev/CI parity.
- Node assets compiled by Webpack 4, with packages managed by Yarn for a consistent and simplified build process.
- An up to date and maintainable build process without insecure packages.
- A resilient and scalable centralised logging and alerting platform that we don’t have to maintain ourselves (CloudWatch).
- Use of native Docker features for container monitoring (e.g.
docker logs) - A more straight forward CI process, that is easier to understand and safer to change.
- Less container image maintenance, off the shelf images are maintained for us.
- Ability to develop against a single discrete part of the application, without requiring the full Sirius stack running (Long term goal).
- Less obfuscation and multiple layers of unnecessary abstraction.
Abstraction Layers
- Jenkinsfile: main CI interaction
- Makefile: for build workflows in local and ci
- docker-compose: where possible commands a run in Docker containers, configured with docker-compose for easer reading
- composer.json: running PHP scripts like Behat
- package.json: running npm scripts like
test:watch, wrapping complex underlying command line switches into an easy to replicate step
The design goal for each stage should be it not requiring a the layer above to set environment variables. That way it can be run independently. For example, the npm scripts should be able to be run from your local machine without complex environment variables being set.
The CI Workflow
The CI workflow exists in primarily in two files that reside in the root of the opg-sirius repository.
Jenkinsfile - How the CI workflow is defined, broken down into stages that run in parallel blocks where possible; each step in the Jenkinsfile references a single Make Target.
Makefile - The primary Makefile in the Sirius project, this contains all commands that are executed in the CI environment by the Jenkinsfile. These are predominantly running services, service builds or commands against a service container as defined in the docker-compose.yml. Make targets that are not executed in the CI environment reside in Makefile.dev.mk.
Should you need to reproduce any stage of the CI workflow locally, all you need to do is execute the appropriate make targets as listed in the Jenkinsfile. Some targets are dependant on each other, thus their ordering in the Jenkinsfile.
It is important to note that CI commands will not map any of the usual developer experience ports to the local host, if you have brought up a CI docker-compose stack and want to connect to the frontend you will need to run docker ps and check which local port has been mapped to the frontend-proxy container’s port 8080
The Developer Workflow
The developer workflow differs from the CI workflow in broadly two ways:
- The CI process uses fully built containers for all services that make up the Sirius application stack, all application code is present in the container image and is immutable. The Developer experience mounts the compiled javascript artifacts for LPA and Supervision into the frontend and supervision containers respectively. This allows yarn service containers to watch and recompile any files as they change. Where required ports are mapped from the localhost into the container services to allow easier debugging.
- Tests that run at scale on CI (30+ Selenium instances) run a single instance locally, due to local resources.
PHP Development
To replicate as closely as possible the previous process when you start the development environments the api & membrane applications (back-end & auth-membrane respectively) are bind mounted into the container so you can test against them as normal without rebuilding the images and restarting the containers. However if you use the UI test suites to drive the frontend to test that backend changes have not broken any functionality you will need to rebuild the api & membrane images before running the test suites. This is due to bind mounts on MacOS impacting performance so severely that UI driven testing will not work. Therefore UI tests use a separate docker-compose override file (docker-compose.test.yml) which does not include the bind mounts found in the developer override file (docker-compose.dev.yml).
Getting Started
You can run make help at any point to see help text on the make targets. Make also auto completes so if you type make supervision and hit TAB twice it will show you all the Supervision targets.
make supervision-
supervision-compile-assets supervision-inspec supervision-jest supervision-jest-watch supervision-protractor supervision-protractor-local supervision-serenity-report supervision-watch
Development Environment Setup
Pre-requisites
Ensure you have docker installed
To pull our prebuilt containers you will need to login to Elastic Container Registry. This requires AWS Vault, as described in the New Starter docs.
You can then login to ECR with:
aws-vault exec sirius-operator -- make ecr_login
Setup
Makefile: make dev-setup
This command will run the following commands in order to make your development environment ready to use.
- composer - Install all PHP dependencies locally for frontend, membrane and api
- compile-node-assets - Outputs Compiled Node Assets for LPA & Supervision to Disk
- build-containers - Runs a parallel docker build to produce production ready images
- reset-membrane-database - Clear and recreate the Membrane database
- reset-api-database - Clear and recreate the API database (needed for ingestion to run)
- ingest - Use Built Containers to run Ingestion to create DB Snapshots stored under data-snapshot and additional case data stored under tmp/storage
- dev-up - Start up the Local Development Environment
KNOWN ISSUE
Occasionally on MacOS the compile-node-assets step will fail with Yarn trying to download packages with an error that displays There appears to be trouble with your network connection. several times before the build fails. This seems to be an intermittent issue with yarn which they don’t have a solution for. However clearing your local yarn cache, removing all the node_modules directories, removing all the yarn.lock files, and regenerating them without mapping the container cache directories to your local machine seems to fix the issue. The fix persists after subsequent cleans which is particularly odd.
Should you find yourself with this issue run make fix-local-yarn which should perform all those steps for you.
Starting the development environment
Makefile: make dev-up
Once you have run the initial setup (opg make:dev:setup), you can start the local development docker environment with these commands, these commands will wipe and restore the databases to the last time ingest was run on your system.
To quick start the development environment, which brings up the environment without restoring the databases.
Makefile: make dev-up
Stopping the development environment
Makefile: make dev-stop
Stop your development environment with these commands.
Stopping and removing lingering containers
Makefile: make down
This will stop all services across all potentially running named service stacks, remove any stopped or orphaned containers, and any networks associated with those stacks.
Cleaning/Wiping the development environment
HERE BE MONSTERS:
Running this command will stop your environment, delete all files that are not committed to git, remove all containers and all docker networks. This will require you to run make dev-setup before being able to resume development.
Makefile: make clean
Database Migrations - Ingestion (Database, Test Data & Fixture Setup)
The Ingestion process has been split into two steps, the first will run the main ingestion process which will create database backups stored under the opg-sirius/data-snapshot directory, and DDC case creation XML & PDF files stored under opg-sirius/tmp/storage. Those outputs can then readily and quickly be reused to prime the various test suites with their expected data, and refresh the development environment without the need to run a full ingestion.
Ingestion Setup
This step requires fully built containers to be available locally, and expects you to have successfully run make dev-setup
Makefile: make ingest
This will create local database backups and DDC datagrams.
Known Issue: If ingestion has already been run against a set of containers then it will fail on a subsequent run because the data will already exist, usually failing on the step highlighted below. If you see an error run make down to remove old ingestion container data.
docker-compose run --rm ingestion /sbin/my_init --skip-runit --skip-startup-files --no-kill-all-on-exit -- /app/scripts/database/useringest.sh
*** Running /app/scripts/database/useringest.sh...
=== ingesting users from csv ===
Error with user creation for user brian.maiden@opgtest.com
*** /app/scripts/database/useringest.sh exited with status 1.
make: *** [ingest] Error 1
Workaround Run make reset-api-database which will clear the database before it runs ingestion
Reset API Database
Makefile: make reset-api-database
Reset Membrane Database
Makefile: make reset-membrane-database
This will clear and recreate the database
Restore Database
Makefile: make restore-database
This will restore the previously backed up data sets into the local environment.
Developing Migrations
Ingest
Makefile: make ingest
Ensure that you have rebased and run ingest before developing migrations to ensure your schema is up to date.
API Migration Commands
make doctrine-diff: Generate a migration by comparing your current database to your mapping information.make doctrine-execute: Execute a single migration version up or down manually.make doctrine-generate: Generate a blank migration class.make doctrine-latest: Outputs the latest version number.make doctrine-migrate: Execute a migration to a specified version or the latest available version.make doctrine-status: View the status of a set of migrations.make doctrine-version: Manually add and delete migration versions from the version table.
Add PARAMS="" to pass command line parameters to Doctrine
make doctrine-version PARAMS="--show-versions"
IMPORTANT: When adding a new column to a table, Doctrine will automatically set a default value in the alter statement it generates e.g. ALTER TABLE test ADD new_column DATE DEFAULT NULL. This can potentially lock up the database as the entire test table is rewritten to store the new default value against all existing records. You should manually split the ALTER statement into two e.g. ALTER TABLE test ADD new_column DATE and ALTER TABLE test ALTER COLUMN new_column SET DEFAULT NULL, so that the default value is only set on new records added in the future.
Connecting to the local Postgres database
docker compose exec postgres-api psql --user api --password
Username: api Password: api
Development
Frontend
PHP Behat Test Suites
These test suites rely on the pre-built containers, ensure you have the latest version of your code in the container by rebuilding api with make build-api
The tests are split out as they are in CI.
Makefile: make behat-ddc-sqs
Makefile: make behat-functional
Makefile: make behat-v0
Makefile: make behat-v1-ci
Makefile: make behat-v1-finance
Makefile: make behat-v1-ingestion
Makefile: make behat-v1-supervision
If you want to run all the V1 tests you can execute the command below
Makefile: make behat-v1
Quality Assurance
These are the steps required to build and use a branch locally.
Setup the local branch
Makefile: make qa-setup
This will build the current branch locally start the environment and restore the test data allowing you to test locally.
Running backend Unit/Behat tests locally
make api-unit FEATURES=LOCAL
make api-unit FEATURES=LOCAL API_UNIT_EXTRA="--group=SW-1567"
make api-behat API_BEHAT_PROFILE=functional API_BEHAT_EXTRA="--tags=@SW-1238" FEATURES=LOCAL
make api-behat API_BEHAT_EXTRA="--tags=@SW-1199" FEATURES=LOCAL
Starting a local environment for Quality Assurance
Makefile: make env-up
This will start a previously built environment locally. If you want to restore the database again run make restore-database
Testing a remotely built branch locally
To save time building a branch locally, if it has already been built by CI you can pull the images locally using the build tag.
Pulling the required images locally
Makefile: make env-pull TAG="SFX-25_Supervision_Webpack4__2019-05-03__161"
This will pull all the application images that make up this build locally.
Starting the local environment from specific image tags
Makefile: make env-up TAG="SFX-25_Supervision_Webpack4__2019-05-03__161"
This will start the local environment with the previous pulled application images.