- For "all-pairs testing", look up http://en.wikipedia.org/wiki/All-pairs_testing and http://www.pairwise.org/. You should maybe also read "Pairwise Testing: A Best Practice That Isn’t" at http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf; and think about (really) random testing.
- PICT is a tool by Microsoft which makes all-pairs testing practical by adding numerous features like constraints and "invalid values". PICT can be downloaded at http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi. You can currently read the user's guide online at http://www.amibugshare.com/pict/help.html and an article at http://msdn.microsoft.com/en-us/library/cc150619.aspx.
- StringTemplate is a Template Engine - see http://www.stringtemplate.org/ and http://www.antlr.org/wiki/display/ST/StringTemplate+3.0+Printable+Documentation.
PICT model.pict | PICT2Code code.template > code.cs
model.pict is a PICT definition of the test variables. Here is a simple example:
P1: 0, 1, 100
P2: null, "SomeValue"
P3: 0.0m, 1.0m, 2.0m
(Full combinatorial testing would create 3*3*2 = 18 test cases for these values. All-pairs testing creates 9).
code.template is a StringTemplate template describing the code. Here is an example:
[Test]
public void Test_$P1_AsIdent$_$P2_AsIdent$_$P3_AsIdent$() {
new MyClass().Method($P1$, $P2$, $P3$);
throw new NotImplementedException("Asserts are missing!");
}
public void Test_$P1_AsIdent$_$P2_AsIdent$_$P3_AsIdent$() {
new MyClass().Method($P1$, $P2$, $P3$);
throw new NotImplementedException("Asserts are missing!");
}
This looks like "real code", except for the $...$ segments which mirror the variables defined in model.pict. The segments come in two variants:
- $P1$ etc. will be replaced with a value from the test model.
- $P1_AsIdent$ etc. will be replaced with a string that contains only letters and digits.
[Test]
public void Test_P1_is_100_P2_is_null_P3_is_0_0m() {
new MyClass().Method(100, null, 0.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
public void Test_P1_is_1_P2_is_null_P3_is_2_0m() {
new MyClass().Method(1, null, 2.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
public void Test_P1_is_0_P2_is__SomeValue__P3_is_0_0m() {
new MyClass().Method(0, "SomeValue", 0.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
public void Test_P1_is_100_P2_is_null_P3_is_0_0m() {
new MyClass().Method(100, null, 0.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
public void Test_P1_is_1_P2_is_null_P3_is_2_0m() {
new MyClass().Method(1, null, 2.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
public void Test_P1_is_0_P2_is__SomeValue__P3_is_0_0m() {
new MyClass().Method(0, "SomeValue", 0.0m);
throw new NotImplementedException("Asserts are missing!");
}
[Test]
...
You get the idea.
Finally, here is the source code of PICT2Code:
1 using System;
2 using System.IO;
3 using System.Text.RegularExpressions;
4 using Antlr3.ST;
5
6 namespace PICT2Code {
7 class PICT2CodeMain {
8 class ErrorListener : IStringTemplateErrorListener {
9 public void Error(string msg, Exception e) {
10 Console.Error.WriteLine(msg + ": " + e.Message);
11 }
12 public void Warning(string msg) {
13 Console.Error.WriteLine(msg);
14 }
15 }
16
17 static void Main(string[] args) {
18 if (args.Length < 1) {
19 Usage("Missing parameters");
20 Environment.Exit(1);
21 }
22 var codeTemplateFile = args[0];
23 string codeTemplate;
24 using (var tr = new StreamReader(codeTemplateFile)) {
25 codeTemplate = tr.ReadToEnd();
26 }
27
28 using (var pictOutput = args.Length > 1
29 ? new StreamReader(args[1]) : Console.In) {
30 string[] varnames = ReadLine(pictOutput);
31 var nonIdentChars = new Regex("[^a-zA-Z0-9_]");
32 for (;;) {
33 string[] data = ReadLine(pictOutput);
34 if (data == null) {
35 break;
36 }
37 var st = new StringTemplate(codeTemplate)
38 { ErrorListener = new ErrorListener() };
39 for (int i = 0; i < data.Length; i++) {
40 st.SetAttribute(varnames[i], data[i]);
41 st.SetAttribute(varnames[i] + "_AsIdent",
42 varnames[i] + "_is_"
43 + nonIdentChars.Replace(data[i], "_"));
44 }
45 Console.WriteLine(st.ToString());
46 }
47 }
48 }
49
50 private static void Usage(string msg) {
51 Console.Error.WriteLine(msg);
52 Console.Error.WriteLine();
53 Console.Error.WriteLine(@"
54 Usage: PICT2Code <template file> [<pict output file>]
55
56 Typical calls are
57 PICT model.pict | PICT2Code code.template > code.cs
58 and
59 PICT model.pict /e:tests.old > tests.new
60 PICT2Code code.template tests.new > code.cs
61 ");
62 }
63
64 private static string[] ReadLine(TextReader pictOutput) {
65 string line = pictOutput.ReadLine();
66 return line == null ? null : line.Split('\t');
67 }
68 }
69 }
70
Happy testing!
Thanks for the helpful article and the interesting references. I agree that you often cannot test all combinations of a functionality, because there are too many of them. So, you have to extract a rage of test cases. Pair-wise or t-wise methods can help here. However, you should use it wisely and you should not “forget” to test the main tests cases, because sometimes this technique may produce not enough of them, e.g., “when highly probable combinations get too little attention”. IMHO, either you could specify those test cases manually or when using the PICT tool for example you may set the more important parameters (equivalence classes) to be covered as triplets or more.
ReplyDeleteIn addition the code generation and well defined templates save a lot of manual work when creating the tests.