The EXACT tool you need: Test the test

In our last quarterly issue we reported on EXTESSY AG’s tool for requirements and data exchange, EXERPT. Since that article appeared, EXERPT was awarded one of the Innovation Prizes 2008 during the CeBIT trade fair in Hanover, Germany. In an earlier issue, we had interviewed founder and President, Prof. Gert Bikker, who spoke to us about the powerful AUTOSAR development and co-simulation platform, ACE. The EXTESSY portfolio is full of innovation and technology. This article features the next tool in that line-up, EXACT.

The topic of test and test management is a challenging one in many industrial sectors. There are now OEM´s with foresight who are searching for a tool solution for this use. In exemplary fashion, Audi Electronics Venture GmbH (AEV) has developed EXACT (EXtessy Automated Conformance Test) to address the issue. EXTESSY AG holds the exclusive distribution and merchandising rights to this product, and strongly promotes the on-going development. The cooperation between AEV and EXTESSY has proven to be a highly successful one.

The testing of software modules at the model or code level is a critical point in the development process. There is an even greater potential for test automation from the perspective of developers who receive these models from suppliers.

The EXTESSY tool EXACT provides developers with a comprehensive range of automation options in one integrated testing tool. The automation of test cases is one of the key features of EXACT. The tool contains time-saving features like the reusability input data of any kind in the recurring cycles of product development. This tool was developed in collaboration (AEV with EXTESSY) and is an immensely powerful testing tool – an automation and administration framework for modeling, implementation and documenting customer-specific testing processes.

How it works

Models such as Simulink and TargetLink are developed based on requirements that are covered by test cases. Defined test runs use these test cases to validate the model. EXACT supports this scenario with an uncluttered GUI. Users have said that one of the tool’s advantages is its simple manageability.

EXACT – case study

Various test cases specify inputs that are related to a predefined output. Presumably the expected output is to be compared to the real output. This comparison of the expected result to the real output is conducted automatically by EXACT in each test case.

Test results often have to be converted manually into test reports. EXACT allows the automatic test report generation in HTML, PDF or XML format.

In summary, the challenges in the field of test and test management are easy to describe – comparison of test data and reporting test results often is time consuming, error prone and difficult to automate. Variations of test runs are nearly as time-consuming and costly as complete new tests. Reports are not generated automatically. EXACT offers an easy to use solution to these challenges with time saving processes, error avoidance by simplifying variations of test runs, and an automated test report generation.

EXACT uses stimulation data from various sources such as vehicle dynamic scenarios, model generated data or CTE test files as input. Stimulation data can be modified by manipulation of noise, offset, gain, etc. This allows the same input data to be used for different test cases. The system under test can be a model, software or processor-in-the-loop (MiL, SiL or PiL). The system is simulated with the predefined input data.

During analysis the real output data is compared to the expected reference data. When the simulation is executed, the expected output has been predefined. EXACT compares the real output data to the predefined output data and uses predefined methods for the analysis and illustration of the expected vs. actual results.

According to the predefined tolerance criteria, a test report is generated which displays test results in text and/or graphical form. For each output signal different tolerance criteria can be defined. E.g. step 1-5 have a tolerance value of “0”, ongoing steps 6-10 have a tolerance value of “0,5”.

EXACT Workflow

The core benefits of EXACT can be summarized in three categories:

1. Test Administration:

The combination of test case parameters which define a complete test configuration can be varied – for example, the usage of the same test case with different manipulations or analysis methods. Various analysis functions are built in and can be extended by customized analysis methods. Different analysis methods can be used to test the model under different conditions to receive a variance comparison.

2. Automated Testing

EXACT uses the predefined test configuration to execute test cases with the system under test and analyzes the resulting output. These data are used as the basis for the test report generation.

3. Automatic Generation of Test Reports

EXACT generates PDF and HTML test reports including the graphical representation of signal input, manipulation, reference data and comparison results. Additionally, the report results can be saved as XML data to be processed for on-going development activities.

Certification

The release 1.2.0.8 of EXACT was certified for BMW by TÜV (the German Technical Control Association) according to the PTB-norm (software as an inspection device). This certification of EXACT was an important advancement for EXACT, which shows that the development of EXACT at Audi Electronics Venture GmbH is based on accepted development practices. EXACT has been continually improved with the input from the TÜV certification process and on-going development activities at Audi AG and BMW AG.

Application at BMW AG

EXACT was certified for the purpose as a test process environment for the Validation Suite of BMW AG. This ensures a qualified method applied software development tools.

“The BMW Validation Suite systematically evaluates on bit-level TargetLink®-models (“Model-in-the-Loop”), generated C-Code on the development PCs (“Software-in-the-Loop”) and generated C-Code for the target processor (“Processor-in-the-Loop”). Therefore, the BMW Validation Suite ensures that the specified behaviors of models are reproduced exactly on the target processor and serves the expected result of BMW AG. The validation of the software development tools significantly reduces the error rate in the software-development process. With EXACT we have a qualified test tool for the huge amount of tests we perform.”

Dr. Stefan-Alexander Schneider / BMW Group Project VASE