Using Portainer for a Docker CICD pipeline

Table of Contents

I mostly self-host my software on docker. I have two ubuntu servers that I use as docker servers mostly for resiliency (I’m thinking of going to docker swarm, but I’ve not got up the motivation to dig into that yet).

The Old Way

I used to manage these services through a massive docker-compose.yml file on each server. Before I streamlined things, the compose file was 1885 lines for about 70 containers. Anything I wanted to do, I would ssh into the machine and make the needed changes. The basic flow would be:

flowchart LR; A[ssh docker host]-->B[cd to Docker directory]-->C[docker compose actions]

I would keep the docker-compose.yml file in a git repository but that was only for versioning reasons. I posted a previous series on finding a CICD solution for my homelab and settled on woodpecker CI so I wanted to make use of this for my container environment.

The Plan

I was already running portainer and they released an update to their business version that let you trigger changes through an api as well as trigger a redeploy of your containers through a webhook.

What I wanted to achieve was to be able to:

Clone a repository containing the relevant docker-compose.yml file
Make the required changes to my containers then commit and push the updated file back to the Gitea repository
Have Gitea trigger a webhook to Portainer
Have portainer make the requisite changes to my containers

%%{init: {'themeVariables': { 'edgeLabelBackground': 'white'}}}%% flowchart LR; home[Local Directory] gitea[(Gitea Repository)] portainer[Portainer Instance] gitea-->|1. git clone|home-->|2. commit & push changes|gitea-->|3. webhook|portainer

In order for portainer to be able to control my containers, I would also have to create them through portainer. Luckily portainer supports reading a docker-compose.yml and .env file from a git repository to form a stack so there would not be much effort required there. However, it would be far easier to manage if I split my huge compose file into multiple more manageable files, each in their own repository.

Implementation

I started by going through all of the containers in my original compose file and splitting them out into meaningful separate compose stacks and saving them into their own Gitea repositories. I then had to configure portainer to be triggered from a webhook and Gitea to send a webhook at the time of a commit.

The configuration you need to do is shown in the flowchart below:

Make sure you have your compose file and .env in a gitea repository
Open Portainer and go to your docker instance and add a stack
- Make sure the stack is pulling from the gitea repository
Copy the url for with webhook
Create a webhook in your gitea repository with the portainer webhook URL

Now whenever you modify your compose file and commit it to the gitea repository, portainer will redeploy the containers in that compose file with whatever changes you made.

Other Interesting Things

You can also call this webhook using curl - curl --request POST "<WEBHOOK URL>". This means you can deploy changes from scripts, and in my case, from WoodpeckerCI pipelines.

In portainer, you can go to an individual container, enable gitops and get a webhook for that individual container rather than the whole compose file. I use this with woodpecker to manipulate my development environments when I need them.