Using Portainer for a Docker CICD pipeline

I mostly self-host my software on docker. I have two ubuntu servers that I use as docker servers mostly for resiliency (I’m thinking of going to docker swarm, but I’ve not got up the motivation to dig into that yet).

The Old Way

I used to manage these services through a massive docker-compose.yml file on each server. Before I streamlined things, the compose file was 1885 lines for about 70 containers. Anything I wanted to do, I would ssh into the machine and make the needed changes. The basic flow would be:

I would keep the docker-compose.yml file in a git repository but that was only for versioning reasons. I posted a previous series on finding a CICD solution for my homelab and settled on woodpecker CI so I wanted to make use of this for my container environment.

The Plan

I was already running portainer and they released an update to their business version that let you trigger changes through an api as well as trigger a redeploy of your containers through a webhook.

What I wanted to achieve was to be able to:

• Clone a repository containing the relevant docker-compose.yml file
• Make the required changes to my containers then commit and push the updated file back to the Gitea repository
• Have Gitea trigger a webhook to Portainer
• Have portainer make the requisite changes to my containers

In order for portainer to be able to control my containers, I would also have to create them through portainer. Luckily portainer supports reading a docker-compose.yml and .env file from a git repository to form a stack so there would not be much effort required there. However, it would be far easier to manage if I split my huge compose file into multiple more manageable files, each in their own repository.

Implementation

I started by going through all of the containers in my original compose file and splitting them out into meaningful separate compose stacks and saving them into their own Gitea repositories. I then had to configure portainer to be triggered from a webhook and Gitea to send a webhook at the time of a commit.

The configuration you need to do is shown in the flowchart below:

• Make sure you have your compose file and .env in a gitea repository
• Open Portainer and go to your docker instance and add a stack
  • Make sure the stack is pulling from the gitea repository
• Copy the url for with webhook
• Create a webhook in your gitea repository with the portainer webhook URL

Now whenever you modify your compose file and commit it to the gitea repository, portainer will redeploy the containers in that compose file with whatever changes you made.

Other Interesting Things

You can also call this webhook using curl - curl --request POST "<WEBHOOK URL>". This means you can deploy changes from scripts, and in my case, from WoodpeckerCI pipelines.

In portainer, you can go to an individual container, enable gitops and get a webhook for that individual container rather than the whole compose file. I use this with woodpecker to manipulate my development environments when I need them.