1 The table below contains some statistics about all the different parameter
2 configurations we ran learnlib with. The
\emph{RivestSchapire
} learner using
3 the Random test method resulted in an incorrect model being learned. When the
4 \emph{KearnsVazirani
} learner using the WMethod tester wasn't able to learn a
5 model, this is due the learner hitting a non-deterministic path. This
6 problem hasn't anything to do with the actual learner and is the result of a
7 uncaught error situation in the adapter. This shows us that a leaner can be
8 used to test software, since we discovered a bug in our adapter. Due to time
9 constrains we were not able to fix this bug. When
\emph{WMethod
} is selected
10 as the testing method
\emph{RivestSchapire
} is also able to learn the correct
11 model.
\emph{WMethod
} does however increase the time needed to learn the
12 model significantly, when a different learner is used there is no reason not to
13 use the Random testing method.
16 \begin{tabular
}{lllccc
}
18 Alphabet & Method & Test method & States & Time \\
20 small & LStar & Random &
2 &
12 sec \\
21 small & TTT & Random &
2 &
5 sec \\
22 small & RivestSchapire & Random &
2 &
6 sec \\
23 small & KearnsVazirani & Random &
2 &
5 sec \\
24 small & LStar & WMethod &
2 &
35 sec \\
25 small & TTT & WMethod &
2 &
32 sec \\
26 small & RivestSchapire & WMethod &
2 &
33 sec \\
27 small & KearnsVazirani & WMethod &
2 &
33 sec \\
29 partial & LStar & Random &
4 &
18 sec \\
30 partial & TTT & Random &
4 &
16 sec \\
31 partial & RivestSchapire & Random &
4 &
13 sec \\
32 partial & KearnsVazirani & Random &
4 &
22 sec \\
33 partial & LStar & WMethod &
4 &
384 sec \\
34 partial & TTT & WMethod &
4 &
390 sec \\
35 partial & RivestSchapire & WMethod &
4 &
384 sec \\
36 partial & KearnsVazirani & WMethod &
4 &
383 sec \\
38 full & LStar & Random &
5 &
44 sec \\
39 full & TTT & Random &
5 &
25 sec \\
40 full & RivestSchapire & Random &
4 &
12 sec \\
41 full & KearnsVazirani & Random &
5 &
19 sec \\
42 full & LStar & WMethod &
5 &
2666 sec \\
43 full & TTT & WMethod &
5 &
2632 sec \\
44 full & RivestSchapire & WMethod &
5 &
2638 sec \\
45 full & KearnsVazirani & WMethod & - & - \\
48 \caption{Learning parameters and resulting model size.
}