Signatures¶
A signature is one distinct failure in your project. Every failing job you send to DV Forge is reduced to a fingerprint, and failures with the same fingerprint land in the same signature, whatever regression, test or seed they came from. The same bug in nightly-1 and nightly-9 is one signature. This page explains how failures are bucketed, what the classification means, and how to read the Signatures list and a signature's page.
How failures are bucketed¶
Signatures are project-wide: there is one signature per distinct failure across all regressions in a project, not one per regression. Each new failure either joins an existing signature or creates a new one.
The headline¶
DV Forge buckets a failure by its headline: the one structured message the failure is keyed by. For a parsed log, that's the first message at the highest severity present (error or fatal). Messages a tool prints only because an earlier error ended the run (a summary such as "exiting due to N errors") don't headline a log that also contains the cause. When no parser finds a structured message, DV Forge uses the one-line failure message sent with the result, or an error-looking line picked from the log.
See Simulators and log formats for what each parser reads.
The fingerprint¶
The fingerprint is computed from four parts of the headline:
| Part | What's used |
|---|---|
| Severity | info, warning, error or fatal |
| Message id | The tool or UVM message id (for example SCBD), normalized. A UVM component's full name used as an id keys by its last segment. |
| Message text | The message, normalized (see below) |
| Source file | The basename of the file that reported it. Directories are dropped. |
The instance hierarchy path is not part of the fingerprint. The same
checker instantiated in two environments (uvm_test_top.env.scbd and
uvm_test_top.env2.scbd) reporting the same message is one failure.
Normalization removes run-specific values from the message text so they can't split one bug into many signatures:
- times: simulation times with a unit, wall-clock times, dates
- addresses and data: hex values, sized and based Verilog literals
(
32'hDEAD_BEEF), plain numbers - seeds and test names echoed into the message (
seed=8812,+UVM_TESTNAME=…, and the job's own test name) - paths (only the file name is kept) and instance indexes in hierarchical names
(
u_core3andu_core0match;if[2]andif[5]match) - letter case, decoration such as
**or->, and extra whitespace
Identity is kept: severity, message id, assertion and check names (a_ack1 and
a_ack2 stay different), signal names and wording.
For example, these two headlines produce the same fingerprint:
UVM_ERROR /proj/tb/env/axi_scoreboard.sv(212) @ 12500 ns: uvm_test_top.env.scbd [SCBD] Data mismatch at addr 0x1f40: exp 32'hdeadbeef got 32'h0 @ 12500 ns (seed=8812)
UVM_ERROR /work/run7/tb/env/axi_scoreboard.sv(212) @ 99100 ns: uvm_test_top.env2.scbd [SCBD] Data mismatch at addr 0x2a00: exp 32'h12345678 got 32'hffffffff @ 99100 ns (seed=17)
Both reduce to this normalized template:
The template is severity | message id | message | file, with - for a part
the headline doesn't have. You can see each signature's template on its page.
If your testbench prints a run-specific value the built-in rules don't recognize (a transaction tag, a port number that varies by seed), a project admin can add a mask for it. See Bucketing settings.
Bucketing versions¶
The rules behind the fingerprint are versioned. Each signature records the
bucketing version it was keyed under; you see it in the signature header as
bucketing v3, and the current engine version on
Bucketing settings. When DV Forge improves the rules, the
version goes up and every affected project is re-bucketed automatically.
Signatures keep their ids, triage state, merges and split rules through a
re-bucket.
Classification¶
Every signature has a classification: what kind of problem it is.
| Classification | Shown as | Meaning |
|---|---|---|
rtl |
RTL | Design bug in the RTL |
dv |
DV | Testbench / verification environment |
infra |
Infra | Farm, license, disk or network |
tool |
Tool | Simulator or tool crash |
unknown |
Unclassified | Not classified yet |
Who set it¶
Next to the classification, the signature page shows who set it:
| Shown as | Source |
|---|---|
| set by rule | DV Forge's first guess, made once when the signature was created |
| set by a person | A triager picked it on the signature page |
| set by a triage rule | A project's triage rule with a Classify action |
The first guess is deliberately conservative. It picks:
- Infra for license checkout failures, out-of-memory and killed processes, full disks, scheduler run limits and shared-filesystem (NFS) errors.
- DV for null class handle accesses and UVM testbench setup errors (test not found, factory can't create a type, objection count below zero).
- Tool for simulator crashes (segmentation faults, core dumps, internal errors).
- RTL only for a simulator-reported assertion failure whose source file is
under an
rtl/,hdl/ordesign/directory. - Unclassified for everything else, including scoreboard mismatches, timeouts and compile errors.
A person's choice always wins. Once someone sets the classification, triage rules never overwrite it (the rule run records the action as skipped). A triage rule can replace the first guess or another rule's classification.
The Signatures list¶
Open Signatures in the sidebar. The list shows every live signature in the selected project. Merged signatures and signatures with no remaining hits are hidden. The total count is shown at the top right (total, or matching when a filter is on), and the list updates live as new results arrive.
Columns¶
| Column | What it shows |
|---|---|
| Signature | The title (links to the signature page), plus a recurred badge if the signature has recurred |
| Class | The classification |
| Status | The triage status, plus the root-cause agent's state on its latest run (agent running, agent done, agent failed) if one was started |
| Hits | Failing jobs bucketed here, across the whole project |
| Regressions | How many regressions hit it |
| First seen, Last seen | When it was first and last seen, by regression start time |
Search, filter and sort¶
The filter bar above the list has:
- Title contains: a case-insensitive substring of the title (up to 200 characters).
- Classification: All or one classification.
- Status: All or one triage status.
- Sort by: Last seen (default), First seen, Hits or Status changed. Every sort is newest or largest first. You can also select the Hits, First seen and Last seen column headers to sort.
Select Apply to filter, and Reset to go back to the defaults. Filters live in the page URL, so you can bookmark or share a filtered list.
The list shows 50 signatures per page, with Previous and Next below it.
Tip
For richer filtering (assignee, labels, dates, hit counts and more), bulk actions and CSV export, use the triage queue and the filter language.
If the project has no signatures yet, the list says so and offers Ingest regression. Signatures appear once a regression with failing jobs is sent to the project; see Sending results.
A signature's page¶
Select a signature's title to open its page. Anyone with access to the project can view it.
Header¶
The small line above the title shows:
- Signature and the signature key, a 16-character fingerprint
- the bucketing version (
bucketing v3) - the severity and the message id, if the headline has them
Below it are the title, the triage status badge, the agent state badge (if a root-cause run was started), the recurred badge (if it has recurred), and the Classification picker with who set it.
The buttons under the header are Run root-cause (see Root-cause agent), and for triagers Merge into… and Split… (see Merge, split and suggestions). If the agent has run on this signature, view latest run → opens that run.
The triage controls (status, assignee, labels, bug link, watchers), the discussion, attachments and activity timeline are described in Working a signature.
Stats¶
| Card | What it counts |
|---|---|
| Hits | Failing jobs bucketed here, across the project |
| Regressions | Regressions that hit it |
| First seen | When it was first seen. in this regression → opens that regression. Hover the value for the exact time. |
| Last seen | When it was last seen, with a link to that regression |
| Recurrences | How many times it has recurred, and when it last did |
First and last seen use the regression's start time (or its upload time if it has no start time), not the time DV Forge processed it. Results uploaded out of order still give the true first and last sighting.
Normalized template¶
Select Normalized template (Show / Hide) to see the text every failure in this signature reduced to. It's the quickest way to see why two failures did or didn't land together.
History¶
History lists the regressions this signature hit, newest first (the latest 50):
| Column | What it shows |
|---|---|
| Regression | The regression name (links to it), with branch and revision underneath when known |
| Started | When the regression started |
| Hits | Failing jobs in that regression bucketed here |
| Tests | Distinct tests among those jobs |
| Shortest failing job | Links to the log of the shortest failing job. shortest log when one job has both the fewest simulated cycles and the shortest runtime; otherwise fewest cycles and fastest. |
The shortest failing log is usually the easiest place to start debugging.
Recent failures¶
Recent failures lists the individual failing jobs bucketed here, newest first (the latest 50): Test (with the error line underneath), Regression, Seed, Kind, When, and a log link.
Merged from and Split rules¶
If other signatures were merged into this one, Merged from lists them. If this signature has split rules, Split rules lists each substring and the child signature it routes to. Both are covered in Merge, split and suggestions.
Edit the title and classification¶
You need the triager role. A merged signature can't be edited.
To rename a signature:
- Select the pencil icon (Rename) next to the title.
- Type the new title (up to 512 characters; it can't be empty).
- Select Save. Press Esc or select Cancel to discard.
The title is for people. Renaming doesn't change how failures are bucketed.
To reclassify a signature, pick a value in the Classification drop-down. The change saves immediately and the label next to it changes to set by a person. From then on, triage rules leave the classification alone.
Recurrence¶
A signature recurs when it comes back after an absence: it's hit in a regression after at least one completed regression on the same branch ran without hitting it.
Precisely, a hit in regression R counts as a recurrence when:
- the signature was seen before R, and
- at least one completed regression on R's branch started after that earlier sighting and before R.
Regressions are ordered by start time (or upload time). Regressions with no branch are compared with each other. Only completed regressions count as an absence: one that is still receiving results, or was aborted, doesn't.
Where you see it:
- a recurred badge on the Signatures list and the signature page (recurred ×3 when it has recurred more than once; hover for the last time),
- the Recurrences card on the signature page,
- a recurred badge next to the signature on that regression's page,
- the activity timeline ("Came back after an absence"),
- the triage rule trigger A signature comes back after an absence (see Triage rules).
Recurrence is recomputed from history whenever signatures change shape (merge, unmerge, split, re-bucket, or a regression being deleted).
Recurred versus regressed¶
Recurrence is about absence, and it ignores triage state. Regressed is a triage status: DV Forge moves a Resolved signature to Regressed when a regression that ran after it was resolved hits it again.
- A signature nobody has resolved can recur many times and never regress.
- A resolved signature hit again by the very next regression regresses, even if no clean completed regression ran in between (so it doesn't recur).