Mutation-Based Testing Technologies Close the “Quality Gap” in Functional Verification for Complex Chip Designs

Leading-edge chip designs are verified by sophisticated and diverse verification environments, the complexity of which rivals or exceeds that of the design itself. Despite advances in stimulus generation and coverage measurement techniques, existing tools do not tell the engineer how good the testbench is at propagating the effects of bugs to observable points or detecting incorrect operation that indicates the presence of bugs. As a result, decisions about where to focus verification efforts, how to methodically improve the environment, or whether it is robust enough to catch most potential bugs are often based on partial data or “gut-feel” assessments. Thus the application of mutation-based testing techniques is emerging as a viable approach to measuring effectiveness and driving improvement in all aspects of functional verification quality for simulation-based environments.

Existing Methods
Functional verification consumes a significant portion of the time and resources devoted to the typical design project. As chips continue to grow in size and complexity, designers must increasingly rely on a dedicated verification team to ensure that systems fully meet their specifications.

Verification engineers have at their disposal a set of dedicated tools and methodologies for verification automation and quality improvement. In spite of this, functional logic errors remain a significant cause of project delays and re-spins. A key reason is that two important aspects of verification environment quality—the ability to propagate an effect of a bug to an observable point and the ability to observe the faulty effect and thus detect the bug—cannot be analyzed or measured. Existing methods such as code coverage and functional coverage largely ignore these two aspects, allowing functional errors to escape the verification process despite excellent coverage scores.

At its core, code coverage is a simple measure of the ability of the stimulus to activate the logic in the design, where “activate” means execute every line, toggle every signal, traverse every path, or some similarly discrete activity. While this is a necessary condition—you can’t find a bug if you don’t “touch” the code related to the bug—it is certainly not sufficient to expose the presence of all or even most problems in a design. Code coverage says nothing about the ability of the verification environment to propagate an effect of a bug once activated or to detect its presence assuming propagation is achieved. Verification engineers thus accept that while code coverage provides interesting data, it is a poor measure of overall verification environment quality.

Functional coverage is generally more interesting and necessary in its own right. In basic terms, it provides a way to determine if all important areas of functionality have been exercised, where “important” is defined in various ways, such as “all operational states,” “all functional sequences,” or the like. The rub is that, by definition, functional coverage is subjective and inherently incomplete. The functional areas (functional coverage “points”) to be checked are defined by engineers and typically based on the design specification, which thoroughly describes how a design should operate, but does not provide a comprehensive view of how it should not operate.

Continue Reading