Continuous integration (CI) and continuous delivery / deployment (CD) are contemporary best practices for automatically testing, building and deploying software. There are many tools and services for performing CI/CD including Jenkins, Circle CI, GitLab CI and GitHub Actions.

Summary

This article describes how to set up GitLab CI to automatically, securely deploy software using SSH, using a Jekyll website for the Git repository.

A brief outline of the steps, programs and files that are involved:

  • Generate dedicated SSH key
    • ssh-keygen
    • id_rsa-gitlab-ci(.pub)
  • Configure SSH on deployment host and GitLab CI
    • ~/.ssh/authorized_keys
    • ~/.ssh/known_hosts
    • ssh-agent
  • Configure GitLab CI variables
    • GITLAB_CI_DEPLOY_SSH_*
  • Configure GitLab CI through .gitlab-ci.yml
    • image: ruby:alpine
    • bundle install
    • jekyll build
    • rsync
  • Trigger deployment by pushing to GitLab
    • git push origin master

Set up

SSH

Generate dedicated SSH key

First, generate an SSH key for deploying software. Since the private key will be hosted with a third-party, GitLab, generate a dedicated key exclusively for this purpose, to minimize the blast radius if that private key is compromised.

It is important to secure the private key because it will have write access to the production site, eg. generate and store the SSH key in an encrypted disk partition. On macOS, this can be accomplished with Disk Utility.app.

Generate a 4096-bit RSA key with a contextual comment to the file id_rsa-gitlab-ci with the command:

ssh-keygen -t rsa -b 4096 -f id_rsa-gitlab-ci -C "GitLab CI 2021 deploy example.com"

For a complete list of options for the ssh-keygen command, see man ssh-keygen or https://www.ssh.com/academy/ssh/keygen

For automated deployments, the passphrase for the SSH key can be left blank or a passphrase can be used, in which case the sshpass program can be used to fill in the passphrase on deployment.

Add public key to ~/.ssh/authorized_keys on the production server

In order to authenticate SSH sessions using the private key, add the public key to ~/.ssh/authorized_keys on the production server.

For the ssh-keygen example above, the public-key file is id_rsa-gitlab-ci.pub. First, copy it to the production server via scp. If the production server was example.com, then from the local computer where the key was generated, run:

scp id_rsa-gitlab-ci.pub example.com:

Then SSH to the production server:

ssh example.com

On the production server, add the public key to the authorized keys:

cat id_rsa-gitlab-ci.pub >> ~/.ssh/authorized_keys

Get production server’s host key

Once you have successully SSH’d to the production server, obtain the host key from the local computer in ~/.ssh/known_hosts. If the production server was example.com, use grep to search for the host key:

grep example.com .ssh/known_hosts

Which might yield a value like

example.com,93.184.216.34 ecdsa-sha2-nistp256 AAAA1234...

Hang on to this value, it will be used later.

CI/CD Variables

To avoid hard-coding variables and secrets into the source code, use GitLab CI/CD variables. These variables will be available in the GitLab CI pipeline as environment variables and can be used in the pipeline commands.

https://docs.gitlab.com/ee/ci/variables/

To access GitLab CI/CD variables interface, navigate to your Git repository on GitLab and then navigate to:

Settings > CI/CD > Variables > Expand

It can be useful to use a hierarchical namespace convention when naming variables. For example:

GitLab (service provider) > CI (service) > Deploy (pipeline stage) > SSH (deployment-command type)

So that the variable names are prefixed with GITLAB_CI_DEPLOY_SSH_

Then add variables by clicking ‘Add Variable’ and filling in the respective keys and values:

key value
GITLAB_CI_DEPLOY_SSH_PRIVATE_KEY The contents of the SSH private key that was generated above
GITLAB_CI_DEPLOY_SSH_HOST The domain name of the production server, eg. example.com
GITLAB_CI_DEPLOY_SSH_KNOWN_HOSTS The host key that was obtained above from the local ~/.ssh/known_hosts
GITLAB_CI_DEPLOY_SSH_USER The name of the user logging in to the production server
GITLAB_CI_DEPLOY_SSH_DIR The directory on the production server to which the site should be deployed. Relative directories, ie. directories that don’t start with / or ~, are relative to the user’s home directory.

Configure GitLab CI: .gitlab-ci.yml

For running a pipeline in GitLab CI, the following will need to be configured:

  • A Docker image to run commands
  • Jobs to be run as stages
  • The branches and commands for which job(s) should be run.

These are all specified in the repository’s .gitlab-ci.yml file.

The official ruby:alpine Docker image on Docker Hub can be used for building a Jekyll website. The ruby:alpine image seems to be slightly faster than the Debian-based ruby image.

image: ruby:alpine

To minimize CI run time, the build and deploy steps can be combined so that there is only one job that needs to go through the boot process.

stages:
  - deploy

The gem directory can be cached to speed up the gem-installation step on subsequent runs.

cache:
  paths:
    - vendor/ruby

https://docs.gitlab.com/ee/ci/caching/

Next, set up the job, here named deploy and configured to run during the deploy stage and when commits are pushed or merged to the master branch:

deploy:
  stage: deploy
  only:
    - master # only deploy for changes to master

Specify the master branch so that pushes to other branches, like feature branches, aren’t deployed to production. If your primary branch is something other than master, eg. main, use that branch instead.

Then set up the commands to be run for the deploy job under the script key.

Add the dependencies for installing the Jekyll gem and its dependencies:

  script:
    - apk add --no-cache g++ musl-dev make

Install the gems to vendor/:

    - bundle install -j $(nproc) --path vendor

Build the static website:

    - bundle exec jekyll build

Set up SSH and start an SSH agent, installing it if it’s not already installed:

    - 'which ssh-agent || ( apk --update add openssh-client )'
    - mkdir -p ~/.ssh
    - eval $(ssh-agent -s)

Copy the host key into the pipeline’s ~/.ssh/known_hosts, which is more secure than disabling Strict Host Key Checking.

    - echo "$GITLAB_CI_DEPLOY_SSH_KNOWN_HOST" >> ~/.ssh/known_hosts

Add the SSH private key to the SSH agent, so that the pipeline can log in to the production server:

    - echo "$GITLAB_CI_DEPLOY_SSH_PRIVATE_KEY" | ssh-add -

Install rsync if it’s not already installed and use it to copy the built site, _site/, to the production server and site directory:

    - 'which rsync || ( apk --update add rsync )'
    - rsync -av _site/ $GITLAB_CI_DEPLOY_SSH_USER@$GITLAB_CI_DEPLOY_SSH_HOST:$GITLAB_CI_DEPLOY_SSH_DIR/

Putting It All Together

Putting all of the CI configuration and commands together, .gitlab-ci.yml should look something like:

image: ruby:alpine

stages:
  - deploy

cache:
  paths:
    - vendor/ruby

deploy:
  stage: deploy
  only:
    - master
  script:
    - apk add --no-cache g++ musl-dev make
    - bundle install -j $(nproc) --path vendor
    - bundle exec jekyll build
    - 'which ssh-agent || ( apk --update add openssh-client )'
    - 'which rsync || ( apk --update add rsync )'
    - mkdir -p ~/.ssh
    - eval $(ssh-agent -s)
    - echo "$GITLAB_CI_DEPLOY_SSH_KNOWN_HOST" >> ~/.ssh/known_hosts
    - echo "$GITLAB_CI_DEPLOY_SSH_PRIVATE_KEY" | ssh-add -
    - rsync -av _site/ $GITLAB_CI_DEPLOY_SSH_USER@$GITLAB_CI_DEPLOY_SSH_HOST:$GITLAB_CI_DEPLOY_SSH_DIR/

After editing this file in your preferred text editor or IDE, commit the file to Git and push it to GitLab.

If GitLab is the origin remote and master is the default branch, run:

git add .gitlab-ci.yml
git commit -m "Add GitLab CI config"
git push origin master

GitLab is now configured for performing CI/CD of the Jekyll website.

Verify the configured pipeline works by pushing an empty commit:

git commit --allow-empty -m "empty commit to trigger CI build;"
git push origin master

On the repository’s project page on GitLab, navigate to ‘CI/CD’ and you should see a pipeline that corresponds to the empty commit. Click on the pipeline run to see its status in more detail. After clicking on the pipeline run, click on the job, deploy in this case, to see the the output of the deploy job. If there are errors, for example problems with variables, syntax or commands, there should be error messages that will allow for debuggingp. When setting up my first CI pipeline and setting up this particular CI pipeline for the first time, I encountered plenty of errors along the way and used the job output to debug those errors.

Using the smaller alpine image speeds up CI runs. GitLab free plans are limited to 400 minutes of CI run time per month– while setting up / debugging, you can stay within your quota by monitoring the run times of individual jobs and how much of the monthly quota has been used.

Conclusion

With some configuration, deployment variables and SSH assets and practices, GitLab CI can be set up to automatically build and securely deploy a website or other software when commits are pushed to the default branch. Automating this workflow helps minimize mistakes when deploying software to production while also reducing effort.

References