Unable to pull environment images
I have now had a few instances where a build-and-test job has failed because it cannot retrieve the env docker image. Produces a 'manifest unknown' error. e.g. https://gitlab.nektar.info/ccantwel/nektar/-/jobs/19737. The preceding env-build job appeared to complete successfully and push the image to the repository. However, rerunning the failed build-and-test job reproduced the error. Rerunning the env-build step(s) before it (in order) allowed the build job to execute successfully (the default image was also not there). Only observed it occur on full builds. Thoughts I had were: that either the systemd clean-up job which runs on the gitlab server is inadvertently removing images it shouldn't, or my addition of dependencies through the 'needs:' to the tasks is somehow leading to it starting the build job before it had completed the push of the image (but then why would it not run when I just rerun the build job?)
This is tricky as no examples of this were seen during development. Further investigation required. I'll attempt to recreate and will increase the verbosity of the registry cleaning service to see what its doing.