Skip to content

CI integration

The best place to run dvforge upload is the last step of the CI job that runs your regression. That way every nightly lands in DV Forge without anyone remembering to send it. This page gives ready-to-adapt snippets for GitHub Actions, GitLab CI, and Jenkins, and explains the choices behind them: when the step runs, where the token comes from, and which build id to use.

The pattern

Every snippet below follows the same five rules:

  1. Always run the upload, even when the regression failed. Failing regressions are the ones you most need in DV Forge. Use if: always(), when: always, or post { always { … } }.
  2. Never let reporting fail the build. The regression result should decide whether the build is red, not the upload. Use continue-on-error, allow_failure, or || echo ….
  3. Take the token from the CI secret store as the DVFORGE_TOKEN environment variable. Never pass it with --token, and never commit it.
  4. Use the CI build id as --external-id. It stays the same when you retry a job in the same build, so a retried upload resumes the same regression instead of creating a second one.
  5. Send branch and revision details with --branch, --rev, --scm, and --ci-url, so the regression page links back to the build.

Installing the CLI on your runners

The snippets assume dvforge is already on the runner's PATH, for example the standalone binary installed in your runner image or on your simulation farm's shared tools path. Your DV Forge contact provides the package or binary; see Getting the CLI.

Snippets

Define DVFORGE_TOKEN as a repository or organization secret, and DVFORGE_URL as a variable (for example https://api.daruur.cloud).

name: nightly-regression

on:
  schedule: [{ cron: "0 2 * * *" }]
  workflow_dispatch:

jobs:
  regress:
    runs-on: [self-hosted, sim]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # full history, so --rev-order (commit count) is meaningful

      - name: Run regression
        run: make regress OUT=out # writes out/results.xml and out/logs/**/*.log

      - name: Upload results to DV Forge
        if: always() # failing regressions are the interesting ones
        continue-on-error: true # never let reporting fail the build
        env:
          DVFORGE_URL: ${{ vars.DVFORGE_URL }}
          DVFORGE_TOKEN: ${{ secrets.DVFORGE_TOKEN }}
        run: |
          dvforge upload \
            --name "nightly ${{ github.run_number }}" \
            --external-id "gh-${{ github.repository_id }}-${{ github.run_id }}" \
            --branch "${{ github.ref_name }}" \
            --rev "${{ github.sha }}" \
            --rev-order "$(git rev-list --count HEAD)" \
            --scm git \
            --ci-provider github \
            --ci-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
            --junit out/results.xml \
            --logs 'out/logs/**/*.log' \
            --only-failing-logs

Re-running a workflow keeps the same github.run_id, so a re-run resumes the same regression.

Define DVFORGE_TOKEN as a masked CI/CD variable. If you also mark it protected, it is only available to pipelines on protected branches and tags.

variables:
  DVFORGE_URL: https://api.daruur.cloud

dvforge-upload:
  stage: report
  tags: [sim] # a runner with dvforge installed
  needs: [regress] # the job that produced out/results.xml and out/logs/
  when: always # upload failing pipelines too
  allow_failure: true # reporting must not fail the pipeline
  script:
    - >
      dvforge upload
      --name "nightly $CI_PIPELINE_IID"
      --external-id "gl-$CI_PROJECT_ID-$CI_PIPELINE_ID"
      --branch "$CI_COMMIT_REF_NAME"
      --rev "$CI_COMMIT_SHA"
      --scm git
      --ci-provider gitlab
      --ci-url "$CI_PIPELINE_URL"
      --junit out/results.xml
      --logs 'out/logs/**/*.log'
      --only-failing-logs

The regression job must pass out/ to this job, for example as artifacts. Retrying a job keeps the same CI_PIPELINE_ID.

Store the project token as a Secret text credential, here with the id dvforge-token. This example is for a Perforce shop: the changelist number doubles as the revision order.

pipeline {
  agent { label 'sim' }
  environment {
    DVFORGE_URL = 'https://api.daruur.cloud'
  }
  stages {
    stage('Regress') {
      steps {
        sh 'make regress OUT=out'   // writes out/results.xml and out/logs/**/*.log
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'dvforge-token', variable: 'DVFORGE_TOKEN')]) {
        // Single quotes: let the shell expand $DVFORGE_TOKEN, not Groovy.
        sh '''
          dvforge upload \
            --name "nightly ${BUILD_NUMBER}" \
            --external-id "jenkins-${JOB_NAME}-${BUILD_NUMBER}" \
            --branch "${BRANCH_NAME:-main}" \
            --scm p4 \
            --rev "${P4_CHANGELIST}" \
            --rev-order "${P4_CHANGELIST}" \
            --ci-provider jenkins \
            --ci-url "${BUILD_URL}" \
            --junit out/results.xml \
            --logs 'out/logs/**/*.log' \
            --only-failing-logs || echo "dvforge upload failed (exit $?); rerun to resume"
        '''
      }
    }
  }
}

The single-quoted sh '''…''' block lets the shell, not Groovy, expand $DVFORGE_TOKEN, which is how Jenkins recommends handling secrets.

Choosing the external id and name

The --external-id decides which regression an upload belongs to. Pick a value that is:

  • Unique per regression run. Two different nightlies must never share one, or the second adds its jobs to the first.
  • Stable across retries of the same run. A retried CI job must reuse it, so the upload resumes instead of duplicating.

The CI's build or pipeline id meets both. Prefix it with the CI system and the repository or job (as above), so ids from different sources never collide. Avoid attempt numbers and timestamps.

The --name is only a display label in the app, for example nightly 1234. Uploading again with the same external id and a different name renames the regression.

Branch and revision details

Flag Shown or used
--branch Shown under the regression name. Drives the branch filter, branch-scoped triage rules, and the recurred badge, which compares runs on the same branch.
--rev Shown (shortened) next to the branch. Up to 128 characters: a commit SHA or a changelist.
--scm git, p4, svn, or hg.
--rev-order A number that increases with each revision: a Perforce changelist, or git rev-list --count HEAD. Recorded with the regression; not shown in the app today.
--ci-url Shown as a CI build link on the regression page. Must start with http:// or https://.
--ci-provider A free-text label such as github, gitlab, or jenkins. Recorded with the regression.
--build-target The default build target for jobs whose JUnit doesn't name one.

For git rev-list --count HEAD to be meaningful, check out the full history (fetch-depth: 0 in GitHub Actions; in GitLab, set GIT_DEPTH to 0).

Running on every nightly

Schedule the regression job itself; the upload step rides along:

  • GitHub Actions: a schedule trigger with a cron expression, as above.
  • GitLab CI: a pipeline schedule on the project.
  • Jenkins: a triggers { cron('H 2 * * *') } block in the pipeline.

Upload every run, including the green ones. Pass rates need the passing jobs, and the recurred badge relies on complete runs in which a signature didn't appear.

Tips

  • Validate the JUnit in a pull request. Run the upload with --dry-run (no token needed) to catch malformed XML, missing files, or ambiguous log matches before the nightly runs.
  • Check the exit code if you want an alert. Exit 2 means the command or input is wrong; 3 means a token or permission problem; 4 means the server was unreachable or rejected the upload. See Exit codes.
  • Rerun to resume. If the upload step failed with exit 4, rerunning the same step with the same external id picks up where it stopped.
  • Using a data plane? Set DVFORGE_URL to the data plane's URL. Nothing else changes. See Data planes.
  • Several results files? Repeat --junit and --logs; everything goes into one regression.