CI integration¶
The best place to run dvforge upload is the last step of the CI job that runs
your regression. That way every nightly lands in DV Forge without anyone
remembering to send it. This page gives ready-to-adapt snippets for GitHub
Actions, GitLab CI, and Jenkins, and explains the choices behind them: when the
step runs, where the token comes from, and which build id to use.
The pattern¶
Every snippet below follows the same five rules:
- Always run the upload, even when the regression failed. Failing
regressions are the ones you most need in DV Forge. Use
if: always(),when: always, orpost { always { … } }. - Never let reporting fail the build. The regression result should decide
whether the build is red, not the upload. Use
continue-on-error,allow_failure, or|| echo …. - Take the token from the CI secret store as the
DVFORGE_TOKENenvironment variable. Never pass it with--token, and never commit it. - Use the CI build id as
--external-id. It stays the same when you retry a job in the same build, so a retried upload resumes the same regression instead of creating a second one. - Send branch and revision details with
--branch,--rev,--scm, and--ci-url, so the regression page links back to the build.
Installing the CLI on your runners
The snippets assume dvforge is already on the runner's PATH, for
example the standalone binary installed in your runner image or on your
simulation farm's shared tools path. Your DV Forge contact provides the
package or binary; see Getting the CLI.
Snippets¶
Define DVFORGE_TOKEN as a repository or organization secret, and
DVFORGE_URL as a variable (for example https://api.daruur.cloud).
name: nightly-regression
on:
schedule: [{ cron: "0 2 * * *" }]
workflow_dispatch:
jobs:
regress:
runs-on: [self-hosted, sim]
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history, so --rev-order (commit count) is meaningful
- name: Run regression
run: make regress OUT=out # writes out/results.xml and out/logs/**/*.log
- name: Upload results to DV Forge
if: always() # failing regressions are the interesting ones
continue-on-error: true # never let reporting fail the build
env:
DVFORGE_URL: ${{ vars.DVFORGE_URL }}
DVFORGE_TOKEN: ${{ secrets.DVFORGE_TOKEN }}
run: |
dvforge upload \
--name "nightly ${{ github.run_number }}" \
--external-id "gh-${{ github.repository_id }}-${{ github.run_id }}" \
--branch "${{ github.ref_name }}" \
--rev "${{ github.sha }}" \
--rev-order "$(git rev-list --count HEAD)" \
--scm git \
--ci-provider github \
--ci-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
--junit out/results.xml \
--logs 'out/logs/**/*.log' \
--only-failing-logs
Re-running a workflow keeps the same github.run_id, so a re-run resumes
the same regression.
Define DVFORGE_TOKEN as a masked CI/CD variable. If you also mark it
protected, it is only available to pipelines on protected branches and
tags.
variables:
DVFORGE_URL: https://api.daruur.cloud
dvforge-upload:
stage: report
tags: [sim] # a runner with dvforge installed
needs: [regress] # the job that produced out/results.xml and out/logs/
when: always # upload failing pipelines too
allow_failure: true # reporting must not fail the pipeline
script:
- >
dvforge upload
--name "nightly $CI_PIPELINE_IID"
--external-id "gl-$CI_PROJECT_ID-$CI_PIPELINE_ID"
--branch "$CI_COMMIT_REF_NAME"
--rev "$CI_COMMIT_SHA"
--scm git
--ci-provider gitlab
--ci-url "$CI_PIPELINE_URL"
--junit out/results.xml
--logs 'out/logs/**/*.log'
--only-failing-logs
The regression job must pass out/ to this job, for example as
artifacts. Retrying a job keeps the same CI_PIPELINE_ID.
Store the project token as a Secret text credential, here with the id
dvforge-token. This example is for a Perforce shop: the changelist
number doubles as the revision order.
pipeline {
agent { label 'sim' }
environment {
DVFORGE_URL = 'https://api.daruur.cloud'
}
stages {
stage('Regress') {
steps {
sh 'make regress OUT=out' // writes out/results.xml and out/logs/**/*.log
}
}
}
post {
always {
withCredentials([string(credentialsId: 'dvforge-token', variable: 'DVFORGE_TOKEN')]) {
// Single quotes: let the shell expand $DVFORGE_TOKEN, not Groovy.
sh '''
dvforge upload \
--name "nightly ${BUILD_NUMBER}" \
--external-id "jenkins-${JOB_NAME}-${BUILD_NUMBER}" \
--branch "${BRANCH_NAME:-main}" \
--scm p4 \
--rev "${P4_CHANGELIST}" \
--rev-order "${P4_CHANGELIST}" \
--ci-provider jenkins \
--ci-url "${BUILD_URL}" \
--junit out/results.xml \
--logs 'out/logs/**/*.log' \
--only-failing-logs || echo "dvforge upload failed (exit $?); rerun to resume"
'''
}
}
}
}
The single-quoted sh '''…''' block lets the shell, not Groovy, expand
$DVFORGE_TOKEN, which is how Jenkins recommends handling secrets.
Choosing the external id and name¶
The --external-id decides which regression an upload belongs to. Pick a
value that is:
- Unique per regression run. Two different nightlies must never share one, or the second adds its jobs to the first.
- Stable across retries of the same run. A retried CI job must reuse it, so the upload resumes instead of duplicating.
The CI's build or pipeline id meets both. Prefix it with the CI system and the repository or job (as above), so ids from different sources never collide. Avoid attempt numbers and timestamps.
The --name is only a display label in the app, for example
nightly 1234. Uploading again with the same external id and a different name
renames the regression.
Branch and revision details¶
| Flag | Shown or used |
|---|---|
--branch |
Shown under the regression name. Drives the branch filter, branch-scoped triage rules, and the recurred badge, which compares runs on the same branch. |
--rev |
Shown (shortened) next to the branch. Up to 128 characters: a commit SHA or a changelist. |
--scm |
git, p4, svn, or hg. |
--rev-order |
A number that increases with each revision: a Perforce changelist, or git rev-list --count HEAD. Recorded with the regression; not shown in the app today. |
--ci-url |
Shown as a CI build link on the regression page. Must start with http:// or https://. |
--ci-provider |
A free-text label such as github, gitlab, or jenkins. Recorded with the regression. |
--build-target |
The default build target for jobs whose JUnit doesn't name one. |
For git rev-list --count HEAD to be meaningful, check out the full history
(fetch-depth: 0 in GitHub Actions; in GitLab, set GIT_DEPTH to 0).
Running on every nightly¶
Schedule the regression job itself; the upload step rides along:
- GitHub Actions: a
scheduletrigger with a cron expression, as above. - GitLab CI: a pipeline schedule on the project.
- Jenkins: a
triggers { cron('H 2 * * *') }block in the pipeline.
Upload every run, including the green ones. Pass rates need the passing jobs, and the recurred badge relies on complete runs in which a signature didn't appear.
Tips¶
- Validate the JUnit in a pull request. Run the upload with
--dry-run(no token needed) to catch malformed XML, missing files, or ambiguous log matches before the nightly runs. - Check the exit code if you want an alert. Exit 2 means the command or input is wrong; 3 means a token or permission problem; 4 means the server was unreachable or rejected the upload. See Exit codes.
- Rerun to resume. If the upload step failed with exit 4, rerunning the same step with the same external id picks up where it stopped.
- Using a data plane? Set
DVFORGE_URLto the data plane's URL. Nothing else changes. See Data planes. - Several results files? Repeat
--junitand--logs; everything goes into one regression.