The tutorials in this chapter are based on simple programs written in C. To run them, you need the C compiler. The chapter has the following sections:
“Setting Up the Tutorials”, shows you how to run the script that creates the files needed for the tutorials.
“Tutorial #1: Analyzing a Single Test”, takes you through the steps of performing coverage analysis for a single test.
“Tutorial #2: Analyzing a Test Set”, discusses creating additional tests to achieve full coverage.
“Tutorial #3: Optimizing a Test Set”, explains how to fine-tune a test set to eliminate redundant tests.
If you are going to run these tutorials, you must run them in order; each tutorial builds on the results of previous tutorials.
If at any time a command syntax is not clear, enter the following:
cvcov help commandname |
Enter the following commands to set up the tutorials:
% cp -r /usr/demos/WorkShop/Tester /usr/tmp/tutorial % cd /usr/tmp/tutorial % echo ABCDEFGHIJKLMNOPQRSTUVWXYZ > alphabet % make -f Makefile.tutorial copyn
This moves some scripts and source files used in the tutorial to /usr/tmp/tutorial, creates a test file named alphabet , and makes a simple program, copyn, which copies n bytes from a source file to a target file.
To see how the program works, try a simple test by typing:
% copyn alphabet targetfile 10 % cat targetfile ABCDEFGHIJ
You should see the first 10 bytes of alphabet copied to targetfile.
Tutorial #1 discusses the following topics:
This is the first step in providing test coverage. The user defines the instrumentation criteria in an instrumentation file.
Enter the following to see the instrumentation directives in the file tut_instr_file used in the tutorials:
% cat tut_instr_file COUNTS -bbcounts -fpcounts -branchcounts CONSTRAIN main, copy_file
We will be getting all counting information (blocks, functions, branches, and arcs) for the two functions specified in the CONSTRAIN directive, main and copy_file.
Enter the following command to instrument copyn:
% cvcov runinstr -instr_file tut_instr_file copyn /lib32/rld /usr/lib32/libssrt.so /usr/lib32/libss.so /usr/lib32/libc.so.1 cvcov: Instrument "copyn" of version "0" succeeded.Directory ver##0 has been created by default. This contains the instrumented executable, copyn.pixie, and other instrumentation data.
A test defines the program and arguments to be run, instrument directory, executables, and descriptive information about the test.
Enter the following to make a test:
% cvcov mktest -cmd "copyn alphabet targetfile 20"
You will see the following message:
cvcov: Made test directory: "/var/tmp/tutorial/test0000"
Directory test0000 has been created by default. It contains a single file, TDF, the test description file.
Note: The directory /var/tmp is linked to /usr/tmp. Enter the following to get a textual listing of the test:
% cvcov cattest test0000 Test Info Settings --------------------------------------------------------- Test /var/tmp/tutorial/test0000 Type single Description Command Line copyn alphabet targetfile 20 Number of Exes 1 Exe List copyn Instrument Directory /var/tmp/tutorial/ Experiment List
To run a test, we use technology from the WorkShop Performance Analyzer. The instrumented process is set to run, and a monitor process (cvmon ) captures test coverage data by interacting with the WorkShop process control server (cvpcs).
Enter the following command:
% cvcov runtest test0000
You will see the following message:
cvcov: Running test "/var/tmp/tutorial/test0000" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet targetfile 20
Now the directory test0000 contains the directory exp##0, which contains the results of the first test experiment.
You can analyze test coverage data many ways. In this tutorial, we will illustrate a simple top-down approach. We will start at the top to get a summary of overall coverage, proceed to the function level, and go finally to the actual source lines.
Enter the following to get the summary:
% cvcov lssum test0000
You will see the display shown in Example 2-1.
% cvcov lssum test0000 % cvcov lssum test0000 Coverages Covered Total % Coverage Weight --------------------------------------------------------------------- Function 3 3 100.00% 0.400 Source Line 22 33 66.67% 0.200 Branch 0 10 0.00% 0.200 Arc 8 18 44.44% 0.200 Block 24 49 48.98% 0.000 Weighted Sum 62.22% 1.000Although both functions have been covered, there is incomplete coverage for source lines, branches, arcs, and blocks.
Note: Items are highlighted on your screen to emphasize null coverage. As a convention in this manual, highlighting or user input is in boldface. Enter the following to look at the line count information for the main function:
% cvcov lssource main test0000
This produces a source listing annotated with counts, shown in Example 2-2.
% cvcov lssource main test0000 Counts Source -------------------------------------------------------------------- #include <stdio.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #define OPEN_ERR 1 #define NOT_ENOUGH_BYTES 2 #define SIZE_0 3 int copy_file(); main (int argc, char *argv[]) 1 { int bytes, status; 1 if( argc < 4){ 0 printf("copyn: Insufficient arguments.\n"); 0 printf("Usage: copyn f1 f2 bytes\n"); 0 exit(1); } 1 if( argc > 4 ) { 0 printf("Error: Too many arguments\n"); 0 printf("Usage: copyn f1 f2 bytes\n"); 0 exit(1); } 1 bytes = atoi(argv[3]); 1 if(( status = copy_file(argv[1], argv[2], bytes)) >0){ 0 switch ( status) { case SIZE_0: 0 printf("Nothing to copy\n"); 0 break; case NOT_ENOUGH_BYTES: 0 printf("Not enough bytes\n"); 0 break; case OPEN_ERR: 0 printf("File open error\n"); 0 break; } 0 exit(1); } 1 } int copy_file( source, destn, size) char *source, *destn; int size; 1 { char *buf; int fd1, fd2; struct stat fstat; 1 if( (fd1 = open( source, O_RDONLY)) <= 0){ 0 return OPEN_ERR; } 1 stat( source, &fstat); 1 if( size <= 0){ 0 return SIZE_0; } 1 if( fstat.st_size < size){ 0 return NOT_ENOUGH_BYTES; } 1 if( (fd2 = creat( destn, 00777)) <= 0){ 0 return OPEN_ERR; } 1 buf = (char *)malloc(size); 1 read( fd1, buf, size); 1 write( fd2, buf, size); 1 return 0; }Notice that the 0-counted lines appear in a highlight color. In this example, the lines with 0 counts occur where there is an error condition. This is our first good look at branch and block coverage at the source line level. The branch and block coverage in the summary are at the assembly language level.
In the second tutorial, we are going to create additional tests with the objective of achieving 100% overall coverage. From examining the source code in Example 2-2, it seems that the 0-count lines in main and copy_file are due to error-checking code that is not tested by test0000.
| Note: This tutorial needs test0000, which was created in the previous tutorial. |
The script tut_make_testset is supplied to demonstrate how to set up this test set.
Enter sh -x tut_make_testset to run the script.
Example 2-3 shows the first portion of the script (as it runs), in which the individual tests are created. The tut_make_testset script uses mktest to create eight additional tests. The tests test0001 and test0002 pass too few and too many arguments, respectively. test0003 attempts to copy from a nonexistent file named no_file. test0004 attempts to pass 0 bytes, which is illegal. test0005 attempts to copy 20 bytes from a file called not_enough, which contains only one byte. In test0006, we attempt to write to a directory without proper permission. test0007 tries to copy too many bytes. In test0008, we attempt to copy from a file without read permission.
Example 2-3. tut_make_testset Script: Making Individual Tests
% sh -x tut_make_testset + cvcov mktest -cmd copyn alphabet target -des not enough arguments cvcov: Made test directory: "/var/tmp/tutorial/test0001" + cvcov mktest -cmd copyn alphabet target 20 extra_arg \ -des too many arguments cvcov: Made test directory: "/var/tmp/tutorial/test0002" + cvcov mktest -cmd copyn no_file target 20 -des cannot access file cvcov: Made test directory: "/var/tmp/tutorial/test0003" + cvcov mktest -cmd copyn alphabet target 0 -des pass bad size arg cvcov: Made test directory: "/var/tmp/tutorial/test0004" + echo a + 1> not_enough + cvcov mktest -cmd copyn not_enough target 20 -des not enough data \ (less bytes than requested) in original file cvcov: Made test directory: "/var/tmp/tutorial/test0005" + cvcov mktest -cmd copyn alphabet /usr/bin/target 20 \ -des cannot create target executable due to permission problems cvcov: Made test directory: "/var/tmp/tutorial/test0006" + ls -ld /usr/bin drwxr-xr-x 3 root sys 3584 May 12 18:25 /usr/bin + cvcov mktest -cmd copyn alphabet targetfile 200 -des size arg too big cvcov: Made test directory: "/var/tmp/tutorial/test0007" + cvcov mktest -cmd copyn /usr/adm/sulog targetfile 20 \ -des no read permission on source file cvcov: Made test directory: "/var/tmp/tutorial/test0008"
After the individual tests are created, the script uses mktset to make a new test set and addtest to include the new tests in the set. Example 2-4 shows the portion of the script in which the test set is created and the individual tests are added to the test set.
Example 2-4. tut_make_testset Script: Making and Adding to the Test Set
+ cvcov mktset -des full coverage testset -testname tut_testset cvcov: Made test directory: "/var/tmp/tutorial/tut_testset" + cvcov addtest test0000 tut_testset cvcov: Added "/var/tmp/tutorial/test0000" to "tut_testset" + cvcov addtest test0001 tut_testset cvcov: Added "/var/tmp/tutorial/test0001" to "tut_testset" + cvcov addtest test0002 tut_testset cvcov: Added "/var/tmp/tutorial/test0002" to "tut_testset" + cvcov addtest test0003 tut_testset cvcov: Added "/var/tmp/tutorial/test0003" to "tut_testset" + cvcov addtest test0004 tut_testset cvcov: Added "/var/tmp/tutorial/test0004" to "tut_testset" + cvcov addtest test0005 tut_testset cvcov: Added "/var/tmp/tutorial/test0005" to "tut_testset" + cvcov addtest test0006 tut_testset cvcov: Added "/var/tmp/tutorial/test0006" to "tut_testset" + cvcov addtest test0007 tut_testset cvcov: Added "/var/tmp/tutorial/test0007" to "tut_testset" + cvcov addtest test0008 tut_testset cvcov: Added "/var/tmp/tutorial/test0008" to "tut_testset"
Enter cvcov cattest tut_testset to check that the new test set was created correctly.
This is shown in Example 2-5. The index numbers in brackets in the subtest list are used to identify the individual tests as part of a test set. This index is used to list the contribution of each test.
Example 2-5. Contents of the New Test Set
% cvcov cattest tut_testset Test Info Settings ---------------------------------------------------------- Test /var/tmp/tutorial/tut_testset Type set Description full coverage testset Number of Exes 1 Exe List copyn Number of Subtests 9 Subtest List [0] /var/tmp/tutorial/test0000 [1] /var/tmp/tutorial/test0001 [2] /var/tmp/tutorial/test0002 [3] /var/tmp/tutorial/test0003 [4] /var/tmp/tutorial/test0004 [5] /var/tmp/tutorial/test0005 [6] /var/tmp/tutorial/test0006 [7] /var/tmp/tutorial/test0007 [8] /var/tmp/tutorial/test0008 Experiment ListEnter the following to run the tests in the test set:
% cvcov runtest tut_testset
By applying the runtest command to the test set, we can run all the tests together. See Example 2-6. When you run a test set, only tests without results are run; tests that already have results will not be run again. In this case, test0000 has already been run. If you need to rerun a test, you can do so using the -force flag.
Example 2-6. Running the New Test Set
% cvcov runtest tut_testset cvcov: Running test "/var/tmp/tutorial/test0000" ... cvcov: Running test "/var/tmp/tutorial/test0001" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet target copyn: Insufficient arguments. Usage: copyn f1 f2 bytes cvcov: Running test "/var/tmp/tutorial/test0002" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet target 20 extra_arg Error: Too many arguments Usage: copyn f1 f2 bytes cvcov: Running test "/var/tmp/tutorial/test0003" ... /var/tmp/tutorial//ver##0/copyn.pixie no_file target 20 File open error cvcov: Running test "/var/tmp/tutorial/test0004" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet target 0 Nothing to copy cvcov: Running test "/var/tmp/tutorial/test0005" ... /var/tmp/tutorial//ver##0/copyn.pixie not_enough target 20 Not enough bytes cvcov: Running test "/var/tmp/tutorial/test0006" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet /usr/bin/target 20 File open error cvcov: Running test "/var/tmp/tutorial/test0007" ... /var/tmp/tutorial//ver##0/copyn.pixie alphabet targetfile 200 Not enough bytes cvcov: Running test "/var/tmp/tutorial/test0008" ... /var/tmp/tutorial//ver##0/copyn.pixie /usr/adm/sulog targetfile 20 File open error
Enter cvcov lssum tut_testset to list the summary for the test set.
Example 2-7 shows the results of the tests in the new test set with lssum.
Example 2-7. Examining the Results of the New Test Set
% cvcov lssum tut_testsetCoverages Covered Total % Coverage Weight --------------------------------------------------------------------- Function 3 3 100.00% 0.400 Source Line 33 33 100.00% 0.200 Branch 9 10 90.00% 0.200 Arc 18 18 100.00% 0.200 Block 46 49 93.88% 0.000 Weighted Sum 98.00% 1.000
Note: Block (basic block) weight will always be different based depending on compile options and compiler versions. Enter cvcov lssource main tut_testset to see the coverage for the individual source lines as shown in Example 2-8.
Example 2-8. Source with Counts
% cvcov lssource main tut_testset Counts Source -------------------------------------------------------------------- #include <stdio.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #define OPEN_ERR 1 #define NOT_ENOUGH_BYTES 2 #define SIZE_0 3 int copy_file(); main (int argc, char *argv[]) 9 { int bytes, status; 9 if( argc < 4){ 1 printf("copyn: Insufficient arguments.\n"); 1 printf("Usage: copyn f1 f2 bytes\n"); 1 exit(1); } 8 if( argc > 4 ) { 1 printf("Error: Too many arguments\n"); 1 printf("Usage: copyn f1 f2 bytes\n"); 1 exit(1); } 7 bytes = atoi(argv[3]); 7 if(( status = copy_file(argv[1], argv[2], bytes)) >0){ 6 switch ( status) { case SIZE_0: 1 printf("Nothing to copy\n"); 1 break; case NOT_ENOUGH_BYTES: 2 printf("Not enough bytes\n"); 2 break; case OPEN_ERR: 3 printf("File open error\n"); 3 break; } 6 exit(1); } 1 } int copy_file( source, destn, size) char *source, *destn; int size; 7 { char *buf; int fd1, fd2; struct stat fstat; 7 if( (fd1 = open( source, O_RDONLY)) <= 0){ 2 return OPEN_ERR; } 5 stat( source, &fstat); 5 if( size <= 0){ 1 return SIZE_0; } 4 if( fstat.st_size < size){ 2 return NOT_ENOUGH_BYTES; } 2 if( (fd2 = creat( destn, 00777)) <= 0){ 1 return OPEN_ERR; } 1 buf = (char *)malloc(size); 1 read( fd1, buf, size); 1 write( fd2, buf, size); 7 return 0; }As you look at the source code, notice that all lines are covered.
Enter cvcov lssource -asm main tut_testset to see the coverage for the individual assembly lines.
When we list the assembly code using lssource -asm, we find that not all blocks and branches are covered at the assembly level. This is due to compilation with the -g flag, which adds debugging code that can never be executed.
Enter cvcov lsline tut_testset to see the coverage at the source line level. Notice that 100% of the lines have been covered.
Tester lets you look at the individual test coverages in a test set. When you put together a set of tests, you may want to improve the efficiency of your coverage by eliminating redundant tests. The lsfun, lsblock, and lsarc commands all have the -contrib option, which displays coverage result contributions by individual tests. We will now look at the contributions by tests for the test set we just ran, tut_testset.
| Note: This tutorial needs tut_testset and all its subtests; these were created in the previous tutorial. |
Enter cvcov lsfun -contrib -pretty tut_testset to see the function coverage test contribution.
Example 2-9, shows how the test set covers functions. Note that the subtests are identified by index numbers; use cattest if you need to map these results back to the test directories.
Example 2-9. Test Contributions by Function
% cvcov lsfun -contrib -pretty tut_testset Functions Files Counts ------------------------------------------ main copyn.c 9 copy_file copyn.c 7 main rld_startup.c 1 Functions Files [0] [1] [2] [3] [4] [5] ------------------------------------------------------------------------ main copyn.c 1 1 1 1 1 1 copy_file copyn.c 1 0 0 1 1 1 main rld_startup.c 1 0 0 0 0 0 Functions Files [6] [7] [8] --------------------------------------------------- main copyn.c 1 1 1 copy_file copyn.c 1 1 1 main rld_startup.c 0 0 0
At the function level, each test covers both functions except for Tests [1] and [2]. The information here is not sufficient to tell us if we have optimized the test set. To do this, we must look at contributions at the arc and block levels. Tester shows arc and block coverage information by test when you apply the -contrib flag to lsarc and lsblock, respectively.
Enter cvcov lsarc -contrib -pretty tut_testset to see the arc coverage test contribution.
Example 2-10, shows the individual test contributions. Notice that Tests [5] and [7] have identical coverage to each other; so do Tests [3] and [8].
We can get additional information by looking at block coverage, confirming our hypothesis about redundant tests.
Example 2-10. Arc Coverage Test Contribution Portion of Report
% cvcov lsarc -contrib -pretty tut_testset Callers Callees Line Files Counts -------------------------------------------------------------- main copy_file 27 copyn.c 7 main printf 17 copyn.c 1 main printf 18 copyn.c 1 main __exit 19 copyn.c 1 main printf 22 copyn.c 1 main printf 23 copyn.c 1 main __exit 24 copyn.c 1 main atoi 26 copyn.c 7 main printf 30 copyn.c 1 main printf 33 copyn.c 2 main printf 36 copyn.c 3 main __exit 39 copyn.c 6 copy_file _open 50 copyn.c 7 copy_file _stat 53 copyn.c 5 copy_file _creat 60 copyn.c 2 copy_file malloc 63 copyn.c 1 copy_file _read 65 copyn.c 1 copy_file _write 66 copyn.c 1 Callers Callees Line Files [0] [1] [2] [3] [4] [5] ----------------------------------------------------------------------------------------- main copy_file 27 copyn.c 1 0 0 1 1 1 main printf 17 copyn.c 0 1 0 0 0 0 main printf 18 copyn.c 0 1 0 0 0 0 main __exit 19 copyn.c 0 1 0 0 0 0 main printf 22 copyn.c 0 0 1 0 0 0 main printf 23 copyn.c 0 0 1 0 0 0 main __exit 24 copyn.c 0 0 1 0 0 0 main atoi 26 copyn.c 1 0 0 1 1 1 main printf 30 copyn.c 0 0 0 0 1 0 main printf 33 copyn.c 0 0 0 0 0 1 main printf 36 copyn.c 0 0 0 1 0 0 main __exit 39 copyn.c 0 0 0 1 1 1 copy_file _open 50 copyn.c 1 0 0 1 1 1 copy_file _stat 53 copyn.c 1 0 0 0 1 1 copy_file _creat 60 copyn.c 1 0 0 0 0 0 copy_file malloc 63 copyn.c 1 0 0 0 0 0 copy_file _read 65 copyn.c 1 0 0 0 0 0 copy_file _write 66 copyn.c 1 0 0 0 0 0 Callers Callees Line Files [6] [7] [8] ----------------------------------------------------------------------- main copy_file 27 copyn.c 1 1 1 main printf 17 copyn.c 0 0 0 main printf 18 copyn.c 0 0 0 main __exit 19 copyn.c 0 0 0 main printf 22 copyn.c 0 0 0 main printf 23 copyn.c 0 0 0 main __exit 24 copyn.c 0 0 0 main atoi 26 copyn.c 1 1 1 main printf 30 copyn.c 0 0 0 main printf 33 copyn.c 0 1 0 main printf 36 copyn.c 1 0 1 main __exit 39 copyn.c 1 1 1 copy_file _open 50 copyn.c 1 1 1 copy_file _stat 53 copyn.c 1 1 0 copy_file _creat 60 copyn.c 1 0 0 copy_file malloc 63 copyn.c 0 0 0 copy_file _read 65 copyn.c 0 0 0 copy_file _write 66 copyn.c 0 0 0
Enter the following to see the test contribution to block coverage:
% cvcov lsblock -contrib -pretty tut_testset
If you examine the results, you will see that Tests [5] and [7] and Tests [3] and [8] are identical.
Now we can try to tune the test set. If we can remove tests with redundant coverage and still achieve the equivalent overall coverage, then we have tuned our test set successfully. Since the arcs and blocks covered by Test [7] are also covered by Test [5], we can remove either one of them without affecting the overall coverage. The same analysis holds true for Tests [3] and [8].
Delete test0007 and test0008 as shown in Example 2-11. Then rerun the test set and look at its summary.
Note that the coverage is retabulated without actually rerunning the tests. The test summary shows that overall coverage is unchanged, thus confirming our hypothesis.
Example 2-11. Test Set Summary after Removing Tests [8] and [7]
% cvcov deltest test0008 tut_testset cvcov: Deleted "/var/tmp/tutorial/test0008" from "tut_testset" % cvcov deltest test0007 tut_testset cvcov: Deleted "/var/tmp/tutorial/test0007" from "tut_testset" % cvcov runtest tut_testset cvcov: Running test "/var/tmp/tutorial/test0000" ... cvcov: Running test "/var/tmp/tutorial/test0001" ... cvcov: Running test "/var/tmp/tutorial/test0002" ... cvcov: Running test "/var/tmp/tutorial/test0003" ... cvcov: Running test "/var/tmp/tutorial/test0004" ... cvcov: Running test "/var/tmp/tutorial/test0005" ... cvcov: Running test "/var/tmp/tutorial/test0006" ... % cvcov lssum tut_testset Coverages Covered Total % Coverage Weight ------------------------------------------------------------------------------ Function 3 3 100.00% 0.400 Source Line 33 33 100.00% 0.200 Branch 9 10 90.00% 0.200 Arc 18 18 100.00% 0.200 Block 48 52 92.31% 0.000 Weighted Sum 98.00% 1.000