Functional qualification: a technical brief

By Mark Hampton, Certess Inc.

Introduction

If there were a bug in your design, could the verification environment find it? Functional qualification is the first technology to provide an objective answer to this fundamental question. It is an important addition to the solutions available for the increasingly challenging task of delivering functionally correct silicon on time and on budget. Functional qualification enables the rapid improvement and cost reduction of verification. The core technology underlying functional qualification is mutation analysis [1]. Mutation analysis has been actively researched for over 30 years in the software testing community (with, among others, PIMS [2,3], Mothra [4,5,6], Proteum [7], Jester [8], MuJava [9], Jumble [10], etc.) but Certess provides a commercial tool (Certitude) that uses this technology within the electronic design automation (EDA) space.

Functional qualification overview

To be effective, verification must ensure that designs are shipped without critical bugs. To find a design bug, three things must occur during the execution of the verification environment:

The bug must be activated; i.e. the code containing the bug is exercised.
The bug must be propagated to an observable point; e.g. the outputs of the design.
The bug must be detected; i.e. behavior is checked and a failure indicated.

Traditional EDA technologies have focused on item 1, activating the bug. Techniques such as code coverage and functional coverage can help ensure that design code is well-activated, but they cannot guarantee that design bugs will be propagated. Nor can they guarantee that the bugs will be detected by the checkers, assertions or comparison against a reference model.

Functional qualification automatically inserts artificial bugs into the design and determines if the verification environment can detect these bugs. A known artificial bug that cannot be detected points to a verification weakness. If an artificial bug cannot be detected, there is evidence that actual design bugs would also not be detected by the verification environment. A functional qualification tool, such as Certitude from Certess, helps the user understand the nature of these verification weaknesses. Functional qualification is able to provide new information to the verification engineer (verifier). For the first time, verifiers can measure the ability of their verification environments to propagate and check potential design bugs. Put more bluntly, Certitude is the first tool to measure the quality of their work comprehensively.

A functional qualification run (or "a qualification run") is performed in two possible modes:

Metric mode: In this mode, Certitude provides an objective and comprehensive score of verification quality. This score can be used to benchmark different verification environments.
Verification Improvement mode: In this mode, Certitude identifies specific faults that cannot be detected so that the verification environment can be improved, and to ensure there are no real design bugs coupled to this fault.

For most engineers, functional qualification is a new concept, which naturally raises questions. The following sections are intended to provide answers to the most common concerns.

Why functional qualification is important

The functional correctness of ICs is essential to final product quality. Designers make mistakes when implementing designs " this is why so much resource is invested in verification. Typically, over 50% of the development effort is invested in functional verification [11,12] " with some authors, such as Bergeron, claiming numbers up to 70% of the design effort [13]. The verification environment is used to measure the functional quality of the design code. Ideally, verification would check that the design behaves correctly with all possible input scenarios. Unfortunately, this is a very complex problem. For typical industrial designs, there are so many possible input scenarios that only a small sample of them is actually applied during verification.

The verification environment (or testbench) is constructed using one or more programming languages. The verification code is complex, and implementation errors will occur. In typical IC projects, the risk of functional defects in the design must be minimized with a limited amount of resources. It would cost too much to build another verification environment to check the implementation of the initial verification environment. Therefore, the industry has looked for more cost-effective means of ensuring the functional quality of the verification environment.

The software industry had been developing programs for several decades [2,5,7,9] before the hardware design community started using Hardware Description Languages (HDL). It was therefore a natural step for the hardware community to adopt techniques from the software community. For example, the software community uses the notion of code coverage (first published by Miller and Maloney [14]) to identify missing test cases. Thus, in the 1990's, code coverage was broadly adopted in IC development projects.

Nevertheless, it can be shown that code coverage is only a partial measure of the verification environment's functionality.

For proof, consider the following examples:
There are many types of code coverage (e.g., statement [15], branch [16], expression [17], toggle, path [18]), and typically only a subset of possible code coverage statistics is considered. Another example is statement coverage. Statement coverage only measures whether each statement was executed at least once in the simulator by at least one test case. In this way, code coverage can be thought of as a "sampling" of the behavior of the input sequences.

Another example of the limitations of code coverage is the lack of temporal information. For actual verification to occur, the environment must exercise a path through the design (from inputs to checkers) and these temporal inter-process relationships are not present in code coverage information. It is quite possible that code is covered as a side effect, and that the related functionality can't impact the result of the individual test case being executed.

Furthermore, code coverage does not indicate if the output behavior is being checked correctly.

Despite the limitations of code coverage, many companies have used it in an attempt to measure verification and, indirectly, the engineers doing the verification work. One of the consequences of measuring people is that it changes their behavior. In general, people want metrics to indicate they are doing a good job. Measuring verification partially, with code coverage, had a perverse effect in the industry, as engineers focused on creating test cases to improve code coverage numbers.

The amount of verification code grew rapidly. In the 1990's, this led to the emergence of Hardware Verification Languages (HVLs), such as OpenVera and 'e'. These HVL methodologies exploit pseudo-randomization and constraint solving, which together increase verification productivity.

But these methodologies make it very difficult for an engineer to understand the behavior of any individual test case (or scenario). Since analyzing code coverage is a time-consuming activity that is typically performed after most of the test cases have been implemented, it became impractical to understand individual test case behavior in an HVL methodology.

These challenges led to the introduction of functional coverage [19,20]. Functional coverage addresses some of the shortcomings of code-coverage within an HVL methodology by providing an indication of design behaviors that have been exercised. This provides needed feedback on the behavior of test cases generated in an HVL methodology. Functional coverage is typically used as an abstract form of code coverage. But functional coverage is explicitly coded by an engineer, so it is likely to be incomplete and is a subjective measure.

Furthermore, like code-coverage, functional coverage still does not indicate if the design output behavior is being checked correctly.

In spite of its shortcomings, the industry has largely adopted this subjective measure of verification quality (functional coverage). Functional coverage does allow some of the temporal inter-process relationships of the design behavior to be measured, but it is quite possible that an individual functional coverage point is hit as a side effect, and the related functionality can't impact the result of the scenario being executed. This is perhaps even more likely to occur when the input sequence is pseudo-randomized. The huge number of possible paths through the design, the likelihood of coding mistakes in the functional coverage code and its subjective nature mean it is unrealistic as a measure of completeness.

To partially offset the subjective nature of functional coverage, vendors recommend using code coverage techniques as a secondary check, a measure whose limitations have already been discussed. In addition, both metrics have similar problems, which means that they cannot adequately compensate for each other's weaknesses.

Functional coverage is widely used today, and once again the consequence of measuring people is that it changes their behavior. Measuring verification subjectively, with functional coverage, again has had an undesirable side-effect, as engineers more and more focus on hitting functional coverage points that they are themselves defining.

Functional qualification is a radical change in how verification is measured. Good verification is capable of finding design bugs. Functional qualification provides new information by actively inserting artificial design bugs into the design and measuring the ability of the verification environment to find those bugs. For the first time, verification can be measured in terms of its ability to detect design bugs.

Today, organizations spend over 50% of their human resources doing verification. In other words, more effort is going into checking IP than into creating it. In contrast, new IP is the main source of revenue for most semiconductor companies.

The first step in getting the cost of verification under control is introducing an accurate measure of verification completeness. This needs to be automated and objective. This is what functional qualification delivers, and this is why it is needed urgently.

Correlating artificial bugs with design bugs

Functional qualification is derived from the theory of mutation analysis. In mutation analysis, a set of small changes (called faults or mutations) are identified for a given software program. Each mutation is introduced independently into the program, and the verification environment is executed to see if it can be propagated to the outputs of the program.

The "artificial bugs" inserted into the design by Certitude are like the faults in mutation analysis. However, functional qualification is different from mutation analysis in several ways:

Functional qualification is concerned with the ability of the verification environment to detect the fault, whereas for mutation analysis only propagation is considered. For a fault to be detected, there must be a check made so that at least one test case fails.
In functional qualification, a fault can be detected without propagating to the primary outputs (if a test case fails with the fault). Thus, functional qualification supports not only black-box verification but also white-box verification and assertions.
Functional qualification uses a radically different process for performing the mutation analysis. This provides relevant results much faster than previous mutation analysis procedures. More details are presented in the section, "Certitude Innovations Addressing Performance Concerns".
Functional qualification is being used in hardware verification as opposed to software verification.

The fault models (or types of artificial bugs) used in functional qualification are very similar to the ones used in mutation analysis. For the purposes of functional qualification, the HDL design is considered to be similar to a software program. The faults are syntactically correct, small functional changes to the design behavior. They are typically based on the simplest errors that could be introduced when writing code.

For example, an operator may be substituted with a different operator:

a = b or c;

becomes

a = b and c;

There are many possible fault models. Certitude includes such faults as forcing a conditional expression to true or false, inverting conditional expressions, removing assignments, removing clauses from case/switch statements, switching operators, changing constants etc. For every synthesizable statement, there are one or more associated faults.

Certitude uses a deterministic set of fault models so that for a specific design instance, the same faults will always be created (the faults are not inserted randomly). The faults are based on the syntax of the design code, not its semantics, so Certitude does not need to understand the design specification. Functional qualification can therefore be applied in any application space, such as interfaces, microprocessors, networking, image processing, etc.

For hardware engineers, it is sometimes easier to understand the fault models for mutation analysis by first considering the evolution of fault models used in the fault grading of manufacturing vectors. The most common fault model is known as the stuck-at-'1' and stuck-at-'0' fault model. In a netlist representation of the circuit, each net is independently forced to a logic '1' or logic '0' while the manufacturing vectors are run. If the output behavior changes when a fault is introduced, the fault can be detected on a tester.

Fault simulation for manufacturing test vector grading required a fault model that measured the ability of those manufacturing test vectors to find physical defects in the IC. Mutation analysis requires a fault model that can measure the ability of verification to find programming errors. The initial concepts are similar, but have resulted in two distinct fields of research " fault grading in hardware and mutation analysis in software.

When researchers initially considered the fault model for fault grading, they studied the types of manufacturing defects that occur in real ICs. One of the common faults is bridging, where a wire connects with another wire due to a manufacturing defect. With bridging faults, the number of faults increases dramatically as design size increases. Further research demonstrated that manufacturing test vectors capable of detecting a high proportion of the stuck-at-'1' and stuck-at-'0' faults were also effective in detecting most bridging faults. This is an important point to keep in mind: In fault grading, the fault model is not explicitly modeling all possible manufacturing faults. The fault model instead provides a good measure of the ability of manufacturing test vectors to find real manufacturing faults. In the field of mutation analysis, researchers have arrived at analogous conclusion.

At the end of the 1970's, the first mutation analysis system was developed for the Fortran language with PIMS [2,3]. This led to a great deal of experimentation in the field of mutation analysis. The experimental results demonstrated the applicability of mutation analysis in measuring the effectiveness of the input sequences used in software testing. This led to further research to explain why the technique was so effective.

While it is possible that a programmer makes a simple error, similar to a fault, it is also likely that a programming error will be much more complex. Fixing a programming error may require many changes to the code to produce the correct behavior. The simple faults used in mutation analysis can be considered as first-order faults that represent the simplest syntactically correct mistakes a programmer could make. More complex programming errors can be thought of as higher order faults. For example, a second-order fault could be reproduced through a combination of two first-order faults. A real bug could be an even higher order fault. Similar to fault grading, where there is a relationship between stuck-at faults and bridging faults, mutation analysis researchers hypothesized that there was a coupling effect between first- and higher order faults.

The coupling effect [21,22] claims that a verification environment capable of finding most first-order faults would also be very effective in finding higher order faults. In the year 2000, Wah provided a theoretical basis to the coupling effect [23,24].

It is important to realize the limitations of mutation analysis. Mutation analysis does not claim that a verification environment capable of finding all first-order faults is perfect. There are two major reasons why mutation analysis is not an exhaustive proof:

The number of possible first-order faults is extremely large. To make mutation analysis a more tractable problem, the concept of selective mutation was introduced [25]. Selective mutation uses a subset of the imaginable fault models. This technique has been demonstrated experimentally to ensure a very high quality of testing, while radically reducing the number of faults that must be analyzed. This means that a small portion of higher order faults could go unnoticed even if the complete subset of faults based on selective mutation can be detected. This is similar to the selection of the stuck-at-'1' or '0' fault subset in manufacturing fault grading.
One of the potential errors in the design being verified is code omission. If the designer forgot to write some code, faults can't be inserted in code that is not present. This is captured in the competent programmer assumption [26,27] cited in the mutation analysis literature, which claims the program being tested is largely correct and therefore most of the functionality is present during the verification. In the case of functional qualification, there must also be checking of the output behavior. This implies that there is an independent implementation of the expected design behavior, so the chance of the same functionality not being implemented in both the verification environment and design is small. However, there is a small possibility that this may occur.

In the verification improvement mode of functional qualification, the coupling from an undetected fault to a potential design bug is more complex than the traditional coupling effect presented in the mutation analysis literature. When considering an undetected fault in functional qualification, the engineer interprets the nature of the design functionality that is not verified and then improves the verification environment. This creative process cannot be formalized and is very powerful. The engineer uses the undetected fault as a clue to find the root cause of the verification weakness. Through root cause analysis and the resulting verification improvements, an even larger and more diverse range of potential design bugs will be coupled to the previously undetected fault.

Mutation analysis provides a solid theoretical basis for the bug detection abilities of functional qualification. Certitude uses selective mutation. The fault models used have evolved through years of applying functional qualification to industrial design projects. The current fault models have proven to be sufficiently robust for world-leading industrial verification teams.

The future of functional qualification

An objective measurement of verification quality improves the recognition of verification expertise. With functional qualification, verification engineers (verifiers) have immediate feedback on their work, and can grow their skill set much faster. Management is able to adopt classical process improvement initiatives within the organization that optimize the cost vs. risk in each verification project.

Early adopters of functional qualification have used this information to accelerate the improvement of verification, achieving higher productivity, higher quality and lower costs. Because verification is such a critical process in IC design, these organizations now have a strategic advantage over their competition.

In the longer term, we believe functional qualification will become "common sense" amongst the verification community. Verification is sometimes perceived as a second-choice career path, partly because verification has not been objectively measured. Functional qualification will help clarify the professionalism and value of verification expertise.

Research in verification has for a long time tended to focus on automating the generation of stimuli. One of the primary reasons for this is that code coverage provided an objective (although partial) goal. Functional qualification introduces a new perspective wherein the goal of verification is to propagate potential design bugs through a path in the system behavior, reaching a checker that can indicate a failure. This new perspective will lead to a more balanced approach in EDA research where technologies can be benchmarked more effectively.

Functional qualification and code coverage

Functional qualification is a radically different technology from code coverage, and subsumes code coverage.

There is no longer a need to use code coverage if functional qualification has been adopted. A qualification score of 100% would also imply a 100% statement and branch coverage score. For example faults that remove an assignment provide information about the statement coverage while also measuring if bugs in the statement can be detected.

Another example are conditional faults which fix a condition to true or false. This provides the branch coverage information while also measuring if bugs in the condition can be detected.

Functional qualification and functional coverage

Functional coverage is typically used with pseudo-randomized test case generation. If pseudo-randomization is being used, functional coverage provides an effective feedback mechanism for debugging scenarios as they are developed.

Functional coverage does not, however, provide effective goals for verification:

The coverage points are subjective.
The coverage points do not consider paths from inputs to checkers.
Important temporal relationships between coverage points are not captured " the order in which coverage points are covered in individual test cases is more important than "hitting" individual coverage points.

Functional coverage is extremely labor intensive. A minimal amount of functional coverage can aid in the debug and development of test case scenarios, but the bulk of the effort should be in building scenarios that correlate with test plan objectives.

Functional qualification can be used instead of functional coverage to ensure that the test plan was well executed. Functional coverage can then be used as a debug aid during the development of scenarios. This can significantly reduce the functional coverage coding effort.

Functional qualification and assertion based verification (ABV)

In the context of functional qualification an assertion is treated like a checker in the verification environment. Assertions may detect a fault by causing the testcase execution to return a fail result.

Faults that are not detected may indicate missing assertions or assertions that contain coding errors.

In an ABV flow Certitude's metric mode can be used to measure the effectiveness of assertions.

Today Certitude fully supports the use of assertions with simulation. In the future we will support the functional qualification of assertions when they are used as properties in formal verification.

Functional qualification and the intelligent testbench

In 1996, out of a discussion between EDA industry experts came the notion of the "Intelligent Testbench," introduced by G. Smith from Dataquest [26] (cf. [27] for a definition). This term has evolved in meaning, and in 2008, it has been used to describe two different types of testbench implementation:

Graph or rule-based descriptions of the verification plan combined with algorithms that generate test cases from the graph or rule set [28]. The graph or rule-based description of the verification plan can be seen as a formalization of the test plan into an executable specification. There is still a need to check if the generated test cases are effective in activating, propagating and checking potential design bugs, so functional qualification is a complimentary technology.
Techniques for automatically generating test cases that target code coverage or functional coverage goals. These techniques will only be effective if the goal of the test case generation takes into account the activation, propagation and checking of potential design bugs. There is a considerable risk that automatically generating a test case that focuses on a specific code coverage or functional coverage point will not produce an effective test case and will instead provide a false sense of security. This is because code coverage and functional coverage of the design do not provide a sufficient goal to ensure the test case generated exercises a new path from design inputs to a checker. Functional qualification could provide an effective goal for these techniques by requiring the automatically generated test cases to activate, propagate and detect faults. This field of research is know as mutation-based testing in the software research community [29].

Functional qualification and ESL

The successor of System-Level Design [30], Electronic System-Level (ESL) [31] design tools, uses a higher level of abstraction for design entry (e.g., UML, SystemC, C) than the current Register Transfer Level (RTL)-based flows. There will always be a need for verification when the design behavior is captured in a formal representation. Verification is a critical part of ESL development.

Mutation analysis has been applied at many different levels of abstraction, for example, UML, XML, state machines, etc. Functional qualification can again leverage directly from the existing mutation analysis research to provide support for ESL development.

The algorithms and know-how developed at Certess for functional qualification are equally applicable at more abstract levels of description. The performance of functional qualification is further enhanced as the design abstraction increases because the total number of faults is reduced. The ease of analysis of functional qualification results is also further enhanced as the design abstraction increases because there are fewer implementation details, therefore the faults are easier to correlate with the specification.

Functional qualification represents a new category of EDA product that is also required in ESL development.

Hierarchical verification

Verification is performed at different levels of the design hierarchy. This allows more productive verification because each level of verification can focus on specific classes of potential design flaws.

Every bug found and fixed at a lower level will not need to be discovered at a higher level of verification. For bugs found at the unit level, the time saving can be anywhere from many man-hours to many man-days compared to finding it at a higher level of integration. There are huge productivity gains to be had by efficiently increasing the quality of verification at lower levels in the design hierarchy.

One of the central challenges in realizing the benefits of hierarchical verification is driving efficient and continuous improvement at each level of verification. There needs to be an independent and objective measure of verification quality to ensure each level is achieving optimal quality within the time and resource constraints. Functional qualification allows an objective and complete measure of the verification quality to enable management of the hierarchical verification.

Verification closure

When considering verification in an industrial context, it is important to keep in mind real-world project constraints. First, verification cannot guarantee the design is perfect " to begin with, the specification may be wrong " so some problems can only be found during system integration. Second, there are a limited amount of time and resources for performing verification " the objective is therefore not to be perfect, but to efficiently minimize the risk of project failure.

An EDA tool cannot replace a human's ability to judge project risk. This knowledge is captured in a test plan. The test plan should drive verification closure rather than any single metric. Through root cause analysis, verification weaknesses (found by functional qualification) result in improved planning and execution for the current and future projects.

Functional qualification can be seen as a way of calibrating the verification environment. A long-term objective for functional verification would be:

Implement and execute the test plan.
Run functional qualification in Verification Improvement mode once to confirm there are no important verification weaknesses.
Run functional qualification in Metric Mode as a final check and to build a database of qualification scores for ongoing process improvement and project management.

It will take many more improvements in verification technology and methodology before real industrial projects reach this point of maturity. Functional qualification provides the metrics necessary to move toward the point of zero time spent in verification closure and minimal verification cost.

Common metric

In theory functional qualification can measure the quality of static/formal verification and dynamic verification. By extending Certitude to support formal tools, it would be possible to have a common metric for these complimentary verification technologies. This would allow more effective use of a combined verification strategy in which some aspects of the design are verified formally while other aspects are verified with simulation.

Functional qualification can also be extended to support the use of emulators and FPGA prototypes.

Functional qualification holds the promise of providing a common metric for all verification technologies, thus facilitating more efficient methodologies combining the strengths of multiple verification technologies.

Design reuse

Design IP is increasingly re-used to get products to market rapidly and cost-effectively. This trend is ongoing and, as system complexity grows, it becomes increasingly difficult to perform system-level verification and validation. There is a growing reliance on IP quality, and the functional quality of IP is directly linked to the quality of its functional verification.

Functional qualification will allow new levels of IP quality to be achieved in the future, thus ensuring final products will reach market more rapidly at a lower cost.

Organization-wide improvements

Within an IC development organization, verification is one of the most expensive design activities. If we consider the engineers' salaries, training costs, CPUs, EDA licenses, prototyping, project delays, missed market windows, silicon masks, reduced fabrication efficiency, etc., for large companies, the cost can be in the hundreds of millions of dollars.

This cost is also rapidly growing. Verification is already the most resource-intensive activity. The future competitiveness of each semiconductor provider will increasingly depend on how effectively it makes use of the available verification resources.

Yet functional verification is one of the design activities that has lacked an objective, independent measurement. It has therefore been difficult to manage efficient quality improvements. Functional qualification is a key building block in bringing science into the verification process. Functional qualification provides the independent and objective measurement that can drive improved efficiency throughout the organization.

As multiple groups adopt functional qualification, there is an opportunity to build a user community to share insights into current methodologies and emerging methodologies, and to get alignment in the critical goal of reducing verification costs by improving quality.

The qualifier

Several decades ago, a single engineer performed the design engineer's role, the verification engineer's role and the backend place-and-route engineer's role.

In the 1990's, the engineers became specialized in RTL design. The term "designer" reflects this. One of the major reasons was the introduction of EDA technology for synthesis.

In the last decade, there has been a further specialization, with the verification engineer focusing on the functional verification activity. The term "verifier" reflects this. One of the major reasons this occurred was the introduction of EDA technology for verification, e.g., HVLs.

There is a growing realization in the industry that the typical separation of the design and verification activities is not leading to the most effective solution. There is a need to put quality at the center of the IC development activity, and this is leading to a further specialization. The emerging EDA category of functional qualification is enabling a specialization in this activity, leading to the role of the "qualifier."

In some organizations, this has already resulted in engineers working across teams to help use functional qualification, but perhaps more importantly, acting as agents of change and increasing links between projects and groups so that the organization improves global verification efficiency and thus minimizes costs.

Conclusion

Functional qualification is able to provide new information. For the first time, we can measure the ability of verification environments to propagate and check potential design bugs. Functional qualification measures the ability of the verification environment to exercise paths from design inputs to checkers. Put more bluntly, Certitude is the first tool to measure the quality of verification work comprehensively.

The first step in getting the cost of verification under control is to introduce an accurate measure of verification completeness. This is what functional qualification delivers, and the reason it is needed urgently.

Mutation analysis provides a solid theoretical basis for the bug detection abilities of functional qualification. The deterministic fault models used by Certitude have been validated through years of applying functional qualification to industrial design projects.

Functional qualification leverages over 30 years of research in the field of mutation analysis for software. Furthermore, we developed innovative, patent-protected, improvements to address fundamental performance issues.

After addressing the performance concerns of mutation analysis, we focused attention on improving the ease of results analysis for the user. Certitude presents the most valuable information early, and this filters out most of the potential noise. The result is a technology that provides a solid return on time invested.

Project teams require time to act on the results of the qualification. Management allocates this time because improving quality of the verification environment reduces overall development costs. Companies make this effort because it is more cost-effective to improve the verification quality than ignore the qualification information and risk potential bugs escaping into the final product.

Functional qualification is complimentary with today's verification practices, and is well positioned to evolve with the foreseeable evolutions in the development process, such as hierarchical verification, ESL, the intelligent testbench and design IP reuse. Functional qualification can help move the industry toward a quality-centric design process.

Like all EDA technologies, functional qualification is not a silver bullet. The key is building a methodology. An important factor for effective large-scale deployment of Certitude is the support and collaboration of an internal champion. Efficiently leveraging this new information source requires a fresh perspective on verification improvement. We have been helping clients do this for over four years, Certess Field Application Engineers ensure clients adopt the technology effectively.

Functional qualification provides an objective measurement that helps teams improve their verification skills faster. When deployed, this accelerates the entire organization's ability to deliver working chips on time.

Acknowledgments

This document would not have been possible without the feedback and motivation of Luis Morales. Florian Letombe not only reviewed the document but prepared the extensive bibliography. Joerg Grosse provided valuable feedback on the early drafts. The final draft was reviewed by Michael Lyons. Thanks a lot, guys!

Bibliography

[1] R. A. DeMillo, R. J. Lipton and F. G. Sayward. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer, 11(4):34-41, 1978.
[2] T. A. Budd, R. A. DeMillo, R. J. Lipton and F. G. Sayward. The design of a prototype mutation system for program testing. In National Computer Conference (NCC'78), pages 623-627, 1978.
[3] R. J. Lipton and F. G. Sayward. The Status of Research on Program Mutation. In Digest for the Workshop on Software Testing and Test Documentation, pages 355-373, 1978.
[4] A. J. Offutt and K. N. King. A Fortran 77 interpreter for mutation analysis. In Symposium on Interpreters and Interpretive Techniques (PLDI'87), pages 177-188, 1987.
[5] A. J. Offutt. Automatic Test Data Generation. PhD thesis, Georgia Institute of Technology, Atlanta, GA, 1988.
[6] Richard A. DeMillo and A. Jefferson Offutt. Constraint-Based Automatic Test Data Generation. IEEE Transactions on Software Engineering, 17(9):900-910, 1991.
[7] M. E. Delamaro and J. C. Maldonado. Proteum-A Tool for the Assessment of Test Adequacy for C Programs. In Conference on Performability in Computing Systems (PCS'96), pages 79-95, 1996.
[8] I. Moore. Jester - a JUnit Test Tester. In eXtreme Programming and Flexible Processes in Software Engineering (XP'00), pages 84-87, 2000.
[9] Y.-S. Ma, J. Offutt and Y. R. Kwon. MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97-133, 2005.
[10] S. A. Irvine, T. Pavlinic, L. Trigg, J. G. Cleary, S. Inglis and M. Utting. Jumble Java Byte Code to Measure the Effectiveness of Unit Tests. In 3rd Mutation Testing Workshop (Mutation'07), pages 169-175, 2007.
[11] G. J. Myers. The Art of Software Testing. John Wiley & Sons, Inc., 1979.
[12] B. Korel. Automated Software Test Data Generation. IEEE Transactions on Software Engineering, 16(8):870--879, 1990.
[13] J. Bergeron. Writing Testbenches: Functional Verification of HDL Models. Kluwer Academic, 2000.
[14] J. C. Miller and C. J. Maloney. Systematic mistake analysis of digital computer programs. Communications of the ACM, 6(2):58-63, 1963.
[15] S. Ntafos. A Comparison of Some Structural Testing Strategies. IEEE Transactions on Software Engineering, 14(6):868-874, 1988.
[16] M. Roper. Software Testing. McGraw-Hill Book Company, 1994.
[17] W. E. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, 8(4):371-379, 1982.
[18] B. Beizer. Software Testing Techniques. International Thomson Computer Press, 1990.
[19] A. Gupta, S. Malik and P. Ashar. Toward Formalizing a Validation Methodology Using Simulation Coverage. In Design Automation Conference (DAC'97), pages 740-745, 1997.
[20] M. Benjamin, D. Geist, A. Hartman, G. Mas, R. Smeets and Y. Wolfsthal. A Study in Coverage-Driven Test Generation. In Design Automation Conference (DAC'99), pages 970-975, 1999.
[21] T. A. Budd. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT, 1980.
[22] L. J. Morell. A Theory of Error-Based Testing. PhD thesis, University of Maryland, College Park, MD, 1984.
[23] K. S. How Tai Wah. A Theoretical Study of Fault Coupling. Software Testing, Verification and Reliability, 10(1):3-45, 2000.
[24] K. S. How Tai Wah. An analysis of the coupling effect I: single test data. Science of Computer Programming, 48(2-3):119-161, 2003.
[25] A. P. Mathur. Performance, effectiveness, and reliability issues in software testing. In 15th Annual International Computer Software and Applications Conference (COMPSAC'91), pages 604 - 605, 1991.
[26] G. Smith. What happened to the intelligent test bench?. In 9th IEEE International High-Level Design Validation and Test Workshop (HLDVT'04), pages 189-189, 2004.
[27] E. R. Hnatek. Practical Reliability of Electronic Equipment and Products. CRC Press, 2002.
[28] T. Toyama and A. Ohnishi. Rule-based Verification of Scenarios with Pre-conditions and Post-conditions. In 13th IEEE International Conference on Requirements Engineering (RE'05), pages 319-328, 2005.
[29] K. N. King and A. J. Offutt. A Fortran Language System for Mutation-Based Software Testing. Software-Practice and Experience, 21(7):685-718, 1991.
[30] P.H.A. van der Putten, J.P.M. Voeten, M.C.W. Geilen and M.P.J. Stevens. System level design methodology. In IEEE Computer Society Workshop, pages 11-16, 1998.
[31] B. Bailey, G. Martin and A. Piziali. ESL Design and Verification: A Prescription for Electronic System Level Methodology. Morgan Kaufmann/Elsevier, 2007.

About the Author:
Mark Hampton, is the CTO at Certess Inc.

Tell us what you think of this page:

3/5 stars (29 votes)

Click here to view the top rated pages