The dvforge CLI¶
dvforge upload sends one regression to DV Forge: your JUnit results become
jobs, and the simulation logs it can match to those jobs are uploaded alongside
them. This page covers installing and configuring the CLI, how it reads JUnit
and finds logs, and how to control batching, waiting, and resuming. For a
flag-by-flag list, see the CLI reference.
Getting the CLI¶
Your DV Forge contact provides the CLI in one of two forms:
-
A Python package for Python 3.11 or later. Its only dependency is the
httpxHTTP library. Install it into a virtual environment or withpipx: -
A standalone binary for RHEL, Rocky, and Alma Linux 8 and 9 on x86_64, for CI agents that don't have Python 3.11. It needs nothing else installed:
Check the installation:
Agents that mount /tmp noexec
At start-up the standalone binary unpacks itself into $TMPDIR (default
/tmp). If /tmp is mounted noexec, set TMPDIR to a directory that
allows execution.
Configuration¶
The CLI reads three settings, from flags or environment variables. A flag overrides its environment variable.
| Setting | Environment variable | Flag | Default |
|---|---|---|---|
| Upload URL | DVFORGE_URL |
--url |
http://localhost:8000 |
| Project token | DVFORGE_TOKEN |
--token |
none |
| Project | DVFORGE_PROJECT |
--project ORG/SLUG |
the token's project |
- Set
DVFORGE_URLtohttps://api.daruur.cloud, or to your data plane's URL if your project uses one. The default only suits a local test server. - Set
DVFORGE_TOKENto a project token with theingest:writescope. Tokens look likedvf_<8 hex characters>_<secret>. A token with only thereadscope is refused (exit code 3). See Projects, roles and tokens. - You rarely need
DVFORGE_PROJECT: a project token already names its project. If you do set it, it must name the token's project.
Prefer environment variables for the token
Command-line flags are visible to other users in ps. Put the token in
DVFORGE_TOKEN, ideally from your CI's secret store. The CLI sends it as
an Authorization: Bearer header and never prints it.
A worked example¶
Suppose your regression writes out/results.xml and one log per test under
out/logs/. After the run:
export DVFORGE_URL=https://api.daruur.cloud
export DVFORGE_TOKEN=dvf_1a2b3c4d_...
dvforge upload \
--name "nightly 1234" \
--external-id "ci-1234" \
--junit out/results.xml \
--logs 'out/logs/**/*.log' \
--only-failing-logs \
--branch main \
--rev "$(git rev-parse HEAD)" \
--scm git
The CLI writes progress and warnings to stderr and a summary to stdout:
dvforge: batch 1/1: 1000 jobs, 42 logs
regression 7c0d… (created): nightly 1234 [ci-1234]
jobs sent: 1000 in 1 batch(es) - pass 958, fail 33, error 4, timeout 5
logs: 42 uploaded (42 new, 0 already stored) (958 passing logs skipped)
status: complete
On a re-run the first line says (updated) instead of (created), batches
that already arrived are reported as already ingested, and logs that are
already stored count as already stored.
To check what the CLI would send without sending anything, add --dry-run.
It needs no token and makes no network calls, but it still reads the JUnit,
matches and hashes the logs, and prints every request it would make:
dry run: nothing will be sent
POST /v1/ingest/regressions
{"branch": "main", "external_id": "ci-1234", "name": "nightly 1234", ...}
PUT /v1/blobs/787ed16a… <- out/logs/crc_test__77.log (60 B, gzip)
POST /v1/ingest/regressions/<id>/jobs batch 1/1: 3 jobs Idempotency-Key: 2695a43d…
first job: {"status": "pass", "test": "fifo.fifo_smoke_test", ...}
POST /v1/ingest/regressions/<id>/complete {"status": "complete"}
would send 3 jobs in 1 batch(es) (pass 1, fail 1, timeout 1); 2 logs to upload (1 passing logs skipped)
How JUnit becomes jobs¶
Pass one or more JUnit XML files with --junit (repeat the flag for several
files; they all go into the same regression). The root element must be
<testsuites>, <testsuite>, or <testcase>. Suites may be nested. Every
<testcase> becomes one job.
Test name¶
The job's test name is classname.name. If the testcase has no classname
(the class attribute is also accepted), if the classname equals the name, or
if the name already starts with classname., the test name is just name.
Test names are clipped to 255 characters. A testcase with neither a name nor a
classname is skipped.
Status¶
| In the testcase | Job status |
|---|---|
An <error> element (it outranks <failure>) |
error |
A <failure> element |
fail |
A <skipped> element |
skipped |
| None of these | pass |
A <failure> or <error> whose type contains timeout or timed out |
timeout |
A <failure> or <error> whose type contains killed, or is oom or sigkill |
killed |
A status property set to pass, fail, error, timeout, killed, or skipped |
that status (overrides everything above) |
For a job that didn't pass, the failure message is the first line of the
<failure> or <error> element's message attribute, or else the first line
of its text, up to 2,000 characters. If a failing job has no message and no
log, the element's type is used as its message.
Properties¶
DV-specific facts come from <property> elements. Names are
case-insensitive. Properties set on a <testsuite> apply to every testcase
inside it, including nested suites. A property on the testcase overrides the
suite's value.
| Property | Job field | Notes |
|---|---|---|
seed, random_seed, sv_seed, ntb_random_seed |
Seed | Checked in that order. Also read from a seed attribute on the testcase, or from an explicit seed_1234, seed=1234, or seed-1234 in the test name. Up to 255 characters. |
component |
Component | Used for the By component breakdown. Up to 255 characters. |
simulator |
Simulator | Falls back to --simulator. Up to 64 characters. Recorded only; it doesn't choose the log parser. |
simulator_version |
Simulator version | Up to 64 characters. |
parser |
Parser hint | Names the log parser to use, for example questa. See Simulators and log formats. |
build_target |
Build target | Falls back to the regression's --build-target. Up to 255 characters. |
config |
Configuration | Up to 1,024 characters. |
host |
Host | Falls back to the <testsuite hostname> attribute. Up to 255 characters. |
plusargs |
Plusargs | Split on whitespace, up to 200 entries. |
cycles |
Simulated cycles | Integer, 0 to 2⁶³−1. Used to pick a signature's fewest cycles log. |
sim_time_fs |
Simulation time | Integer femtoseconds, 0 to 2⁶³−1. |
exit_code |
Exit code | Signed 32-bit integer. Unsigned forms such as 4294967295 or 0xC0000005 are converted to their signed value. |
status |
Status | Overrides the status from the JUnit elements (see above). |
log, logfile, log_file, log_path, logpath, sim_log |
Log file | Points at this test's log. Read from the testcase only, never inherited from a suite. |
Integer properties accept decimal, 0x hexadecimal, and _ digit separators.
A value that isn't a number, or is out of range, is dropped rather than
failing the upload.
The CLI also reads these testcase attributes:
| Attribute | Job field |
|---|---|
time |
Runtime in seconds. Dropped unless it is a finite number between 0 and 10⁹. |
timestamp |
Start time (ISO 8601). |
The regression's start time is the earliest <testsuite timestamp> across all
the JUnit files. If there is none, DV Forge uses the time the regression was
opened.
How logs are matched to tests¶
The CLI attaches at most one log file to each job, and uses each log file at most once. For each testcase, the first rule that finds a file wins.
1. An explicit pointer in the JUnit. One of the log properties above, a
[[ATTACHMENT|path]] marker in <system-out> or <system-err>, or a line in
those elements that is nothing but a path ending in .log or .log.gz
(optionally prefixed log:). Relative paths are resolved against the JUnit
file's directory first, then the current directory. Pointers work without
--logs.
2. File names under --logs. Pass one or more --logs patterns. Quote
them so the shell doesn't expand them; ** matches any depth, and a directory
means every file under it. The CLI compares the testcase's name attribute
(not the classname) with each file's base name, minus .gz and a .log,
.txt, or .out extension. The match is case-insensitive and must fall on
word boundaries, so test doesn't match contest_x.log. Candidates are
ranked:
- Exactly the name and the seed, for example
fifo_test__1234.logorfifo_test.1234.log. - Contains the name and the seed, for example
run_fifo_test_seed1234.log. - Exactly the name (and the seed isn't in the file name, or the test has no seed).
- Contains the name.
- Otherwise, the last two parent directories contain the name (and the seed),
for example
regress/fifo_test/seed_1234/sim.log.
Better ranks claim files first, so fifo_test never takes
fifo_test_long__1.log from fifo_test_long. If several files tie for a
test's best rank, the CLI warns that the match is ambiguous and attaches
nothing to that test. Files that no test claims are counted and ignored. A
pattern that matches no files produces a warning.
Logs that are already gzipped on disk are sent as they are; others are compressed on the fly. In both cases the stored hash is that of the uncompressed content.
Choosing which logs to send¶
| Flag | Effect |
|---|---|
--only-failing-logs |
Upload logs only for fail, error, timeout, and killed jobs. Recommended: DV Forge parses only failing logs. |
--no-logs |
Send results only. Ignores --logs and the JUnit log pointers. Can't be combined with --logs or --only-failing-logs. |
--max-log-mb N |
Skip, with a warning, any log larger than N MiB uncompressed. Default 256, which is also the server's limit. |
A log that can't be read is skipped with a warning. A job whose log was skipped is still sent; it is bucketed by its failure message instead.
Batches and concurrency¶
Jobs go up in batches of --batch-size jobs (default and maximum 5,000). For
each batch, the CLI first uploads the logs that batch references, then sends
the batch. --concurrency sets how many logs are hashed and uploaded in
parallel (default 8, maximum 64).
Network errors, HTTP 5xx responses, and HTTP 429 are retried up to five
attempts per request, with exponential backoff and jitter. A Retry-After
header from the server is honored, up to 60 seconds.
Waiting for processing¶
By default the CLI exits as soon as DV Forge has accepted every batch.
Processing (parsing failures and assigning signatures) continues in the
background. Add --wait to make the CLI poll each batch until it is
processed:
- If every batch finishes, the CLI completes the regression and exits 0.
- If any batch fails to process, the CLI exits 4 and doesn't complete the regression.
- If batches are still processing after
--wait-timeoutseconds (default 600), the CLI exits 4. The batches keep processing.
Use --wait when a later CI step reads the results back, for example through
the API.
Uploading one regression in parts¶
Some flows produce one regression's results in several pieces, for example
sharded runs on different machines. Upload each piece with the same
--external-id and add --no-complete to every upload except the last. The
regression stays live until the final upload completes it.
Jobs that repeat across parts
A job's identity is derived from the regression's external id, the test name, the seed, and how many earlier jobs in the same upload had that test and seed. If two parts both report the same test with the same seed, the second one is treated as a duplicate and dropped. Give each part distinct test and seed combinations.
You don't strictly need --no-complete: uploading to a completed regression
reopens it. But leaving it open keeps the status accurate while parts are
still arriving.
Resuming an interrupted upload¶
Rerun the exact same command. Every step is safe to repeat:
- Before uploading a log, the CLI asks whether DV Forge already has it, and skips it if so.
- The regression is found again by its external id.
- Job ids and batch keys depend only on the inputs, never on time, so a resent
batch is recognized as
already ingested.
If you press Ctrl-C, the CLI stops with exit code 130 and the message "interrupted — rerun the same command to resume". After a server or network failure (exit code 4), it prints "rerun the same command to resume; completed batches are not re-ingested".
Keep these two rules in mind:
- The first copy of a job wins. If an earlier attempt sent a job without
its log, a rerun that now has the log doesn't attach it. To change jobs that
already arrived, upload under a new
--external-id. - Don't change log files mid-upload. If a log changes between hashing and upload, the CLI stops with an error. Rerun once the file is stable.
Uploading through a data plane¶
If your project is bound to a data plane, set
DVFORGE_URL to the data plane's URL. The command, the token, and the output
stay the same.
If the data plane can't reach DV Forge while you upload, it keeps your batches and the completion in a durable queue and forwards them in order once the connection is back. The CLI reports this and still exits 0:
dvforge: batch 2/3: 5000 jobs, 12 logs (spooled by the data plane)
dvforge: note: the control plane was unreachable; the data plane queued 2 batch(es) and the completion and forwards them in order once it is back
...
status: complete (spooled on the data plane)
Two limits apply:
--waitcan't wait for queued batches, because they don't have a batch id yet. The CLI warns about this.- Opening the regression can't be queued, because the CLI needs the regression id it returns. If DV Forge is unreachable at that moment, the CLI retries and then exits 4.
Exit codes¶
| Code | Meaning | What to do |
|---|---|---|
| 0 | Success, including --dry-run and uploads queued by a data plane. |
Nothing. |
| 2 | Usage error: bad or conflicting flags, a missing or unreadable JUnit file, malformed or unsafe XML, no testcases, or no credentials. | Fix the command or the input. Nothing was sent. |
| 3 | Authentication or permission error (HTTP 401 or 403). | Check DVFORGE_TOKEN and its scope. Also check DVFORGE_URL: a project bound to a data plane refuses direct uploads with a 403, and a data plane refuses projects not bound to it. |
| 4 | Server or network error after retries, any other API error, a batch that failed to process, or a --wait timeout. |
Rerun the same command to resume. |
| 130 | Interrupted (Ctrl-C). | Rerun the same command to resume. |
Limits and clipping¶
The CLI keeps an upload from being rejected over one oversized value:
| What | Limit | When exceeded |
|---|---|---|
--name, --external-id, --branch, --build-target |
255 characters | Usage error (exit 2) |
--rev |
128 characters | Usage error (exit 2) |
--ci-provider |
64 characters | Usage error (exit 2) |
--ci-url |
2,048 characters; must start with http:// or https:// |
Usage error for length; the server refuses other schemes (exit 4) |
| Test name, seed, component, build target, host | 255 characters | Clipped |
| Simulator, simulator version, parser hint | 64 characters | Clipped |
| Configuration | 1,024 characters | Clipped |
| Failure message | 2,000 characters (first line only) | Clipped |
| Plusargs | 200 entries | Extra entries dropped |
| Runtime | 0 to 10⁹ seconds | Dropped |
| Cycles, simulation time | 0 to 2⁶³−1 | Dropped |
| Exit code | Signed 32-bit | Dropped |
| Jobs per batch | 5,000 | Usage error if --batch-size is larger (exit 2) |
| Log size | 256 MiB uncompressed (--max-log-mb); a pre-gzipped file over 64 MiB on disk |
Skipped with a warning |
JUnit safety¶
JUnit files are written by the code under test, so the CLI treats them as
untrusted. It refuses any file that contains a DOCTYPE or ENTITY
declaration (exit code 2). Legitimate JUnit reports never need one. Deeply
nested suites are handled without risk of a crash.