Baseline letter comparison
In order to detect regressions in letter generation, we maintain a “baseline” HTML version of each letter in the opg-sirius repository. By generating a fixed template with fixed data, and fixing the date of generation (using Carbon::setTestNow()) these baseline letters are idempotent and will always produce the same HTML output.
Detecting regressions
In the CI pipeline, the letters are all generated again and their HTML output is compared to the related baseline letter. If there is any difference between the two outputs, the comparison and pipeline will fail. This is because something in the pull request has altered the output of a letter and either requires fixing (to return the letter to its baseline state) or manual testing (to confirm the new output is as expected).
This process can be replicated locally by running make api-letters-compare.
Updating the baselines
If a letter is intentionally changed, the baseline must be updated to reflect that. This can be done automatically by running make api-letters-generate, which will regenerate all baseline letters based on the current state of the repo. You will need to add the changed baselines to your commit.
When updating baseline letters you should check that only the letters you intended to change have new baselines. If any additional baselines have changed, you may have a bug or need to do additional testing to ensure the change is not a regression.
When you run make-api-letters-generate, Sirius will also create a PDF version of each baseline letter in the same folder. These are not committed to the repo (they are explicitly ignored by .gitignore) but can be useful for reviewing changes. It’s advisable to attach the PDF version of any changed baseline letters to the ticket so that other members of the team (including PMs) are clear what the impact of the change has been.