CI/CD - Advanced

This section covers advanced CI/CD features: matrix builds, retries on failure, concurrency management, run conditions, external runners, and auto-scaling in Kubernetes.

Basic concepts (workflow, jobs, triggers, artifacts, cache) are described in the CI/CD Pipelines section.

Matrix Builds

When you need to test a project across multiple language versions, operating systems, or configurations - use a matrix. GitRiver will create a separate job for each combination.

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu, alpine]
        rust: ['1.80', '1.93', stable]
      fail_fast: false
      max_parallel: 4
    image: rust:${{ matrix.rust }}
    steps:
      - run: cargo test

This creates 6 jobs (2 OS x 3 Rust versions). Values are accessible via ${{ matrix.key }}.

• fail_fast - if true (default), remaining jobs are cancelled on the first failure. Set to false to wait for all.
• max_parallel - parallelism limit (unlimited by default).

Adding and Excluding Combinations

strategy:
  matrix:
    os: [ubuntu, alpine]
    rust: [stable, nightly]
    include:
      - os: ubuntu
        rust: nightly
        features: experimental
    exclude:
      - os: alpine
        rust: nightly

include adds combinations with additional variables. exclude removes unwanted ones.

---

Job Run Conditions

By Variable Value

Run deploy only from the main branch:

jobs:
  deploy:
    if: $CI_COMMIT_BRANCH == "main"
    steps:
      - run: ./deploy.sh

On Previous Job Failure

Send a notification if tests failed:

jobs:
  notify:
    needs: [test]
    if: failure()
    steps:
      - run: curl -X POST $SLACK_WEBHOOK -d '{"text":"Tests failed!"}'

Available Expressions

Expression	When it triggers
$CI_COMMIT_BRANCH == "main"	Exact string comparison
$CI_COMMIT_TAG =~ /^v\d+/	Regex match
success()	All previous jobs succeeded
failure()	At least one previous job failed
always()	Always (even if the pipeline is cancelled)
cancelled()	The pipeline was cancelled

Run Rules (rules)

Rules are a more flexible alternative to conditions. They are evaluated top to bottom; the first match determines what happens to the job.

Why they’re needed: when a single if condition isn’t enough - for example, deploy automatically for main, wait for manual trigger when src/ changes, and don’t run for everything else.

rules:
  - if: "$CI_COMMIT_BRANCH == 'main'"
    when: always
  - changes:
      - "src/**"
    when: manual
  - when: never

when value	What happens
always	The job runs automatically
never	The job does not run
manual	The job waits for manual trigger (“Run” button in the interface)
on_success	Runs if dependencies succeeded (default)
on_failure	Runs if dependencies failed

---

Retries on Failure (retry)

If a job may fail due to an unstable network or flaky tests - configure automatic retries:

jobs:
  test:
    retry:
      max: 2
      when: [script_failure, stuck_or_timeout_failure]
    steps:
      - run: npm test

• max - maximum retries (0-2)
• when - which errors to retry on:
  • script_failure - command returned a non-zero exit code
  • stuck_or_timeout_failure - job got stuck or exceeded the timeout
  • always - on any error (empty list = always)

---

Concurrency Management (concurrency)

Problem: with rapid consecutive pushes, multiple pipelines run simultaneously, and deploys may execute in parallel - this is dangerous.

Solution: concurrency group. Jobs with the same group do not run in parallel.

concurrency:
  group: deploy-$CI_COMMIT_BRANCH
  cancel_in_progress: true

If cancel_in_progress: true - when a new run starts, the previous one is automatically cancelled. This is useful for CI tests: why wait for an old result when there’s already a new push.

---

Allowed Failure (allow_failure)

If a job is not critical for the pipeline (e.g., a linter you’re just introducing):

jobs:
  lint:
    allow_failure: true
    steps:
      - run: cargo clippy -- -D warnings

This job’s failure won’t make the pipeline red - it will remain green.

---

Interruptible Jobs (interruptible)

To have a job automatically cancelled on a new push to the same ref:

jobs:
  test:
    interruptible: true
    steps:
      - run: cargo test

Works together with concurrency - on a new push to the branch, the old job is interrupted.

---

Passing Data Between Jobs (outputs)

A job can pass values to subsequent jobs. For example, one job determines the version, and another uses it during deploy.

Writing - via the $CI_OUTPUT file:

jobs:
  prepare:
    steps:
      - run: |
          VERSION=$(cat VERSION)
          echo "version=$VERSION" >> $CI_OUTPUT

  deploy:
    needs: [prepare]
    steps:
      - run: echo "Deploying version $NEEDS_PREPARE_OUTPUTS_VERSION"

The variable name is formed automatically: NEEDS_<JOB_NAME>_OUTPUTS_<KEY> (all uppercase).

---

External Runners

By default, GitRiver executes CI jobs on the same host via Docker. For scaling or isolation, you can connect external runners - separate machines that pick up jobs from the server.

Registering a Runner

1. Open Administration -> “Runners”
2. Click “Add runner”
3. Specify the name, description, and labels (e.g.: linux, gpu, arm64)
4. Save the token - it is shown only once

Installing a Runner

On the machine where the runner will execute:

gitriver-runner --token <TOKEN> --server https://git.example.com

The runner polls the server regularly (every 3 seconds by default), picks up jobs with matching labels, and executes them in Docker.

Routing Jobs by Labels

In the workflow, specify which runner should execute the job:

jobs:
  gpu-test:
    runs_on: [self-hosted, linux, gpu]
    steps:
      - run: python train.py

The job will be assigned to a runner that has all the specified labels.

---

K8s Auto-Scaling Runners

For automatic runner scaling in Kubernetes - GitRiver creates K8s Jobs as CI tasks arrive and deletes pods after completion.

Setup

1. Connect a cluster: Administration -> “Clusters” -> “Add cluster”
   • Specify the API URL and CA certificate, or upload a kubeconfig
   • Click “Test connection” to verify
2. Create a configuration: Administration -> “K8s Runners” -> “Add”
3. Specify:
   • Cluster - which cluster to create pods in
   • Namespace - which namespace
   • Docker image - image for the runner
   • Label (runs_on) - label for job routing
   • Max pods - limit on simultaneous pods
   • CPU/RAM resources, tolerations, node selector (optional)

How It Works

1. A CI job with the label runs_on: [k8s] enters the queue
2. GitRiver creates a K8s Job in the specified cluster/namespace
3. The pod starts, picks up the job, executes it, and sends the result
4. After completion, the pod is deleted

---

Web Terminal

The web terminal allows you to connect to a running CI container in real time - for debugging, checking the filesystem state, or running commands manually.

When Available

• The job is running (Running)
• The job has a Docker image specified (image:)
• You have write access to the repository

How to Use

1. Open a running job in the “CI/CD” tab
2. Click the “Terminal” button (appears next to the logs)
3. An interactive terminal opens in the browser
4. You can execute any commands inside the container

The terminal supports full PTY: Tab completion, arrow keys for history navigation, Ctrl+C to interrupt.

Limitations

• The terminal closes automatically when the job finishes
• Terminal size: 24x80 characters
• Available only for Docker jobs (not for jobs without image:)

---

Multiple Workflow Files

A single repository can have multiple workflows:

.gitriver/
  workflows/
    ci.yml          - tests on every push
    release.yml     - build on tag push
    nightly.yml     - nightly tests on schedule
    deploy.yml      - deploy to production

On push, GitRiver checks all files and runs those whose triggers matched.

Maximum 20 workflow files per repository.