a4/question3.tex

   1 The table below contains some statistics about all the different parameter
   2 configurations we ran learnlib with. The \emph{RivestSchapire} learner using
   3 the Random test method resulted in an incorrect model being learned. When the
   4 \emph{KearnsVazirani} learner using the WMethod tester wasn't able to learn a
   5 model, this is due the learner hitting a non-deterministic path.  This
   6 problem hasn't anything to do with the actual learner and is the result of a
   7 uncaught error situation in the adapter. This shows us that a leaner can be
   8 used to test software, since we discovered a bug in our adapter. Due to time
   9 constrains we were not able to fix this bug. When \emph{WMethod} is selected
  10 as the testing method \emph{RivestSchapire} is also able to learn the correct
  11 model. \emph{WMethod} does however increase the time needed to learn the
  12 model significantly, when a different learner is used there is no reason not to
  13 use the Random testing method.
  14
  15 \begin{table}[H]
  16         \begin{tabular}{lllccc}
  17                 \toprule
  18                 Alphabet & Method & Test method & States & Time \\
  19                 \midrule
  20                 small & LStar & Random & 2 & 12 sec \\
  21                 small & TTT & Random & 2 & 5 sec \\
  22                 small & RivestSchapire & Random & 2 & 6 sec \\
  23                 small & KearnsVazirani & Random & 2 & 5 sec \\
  24                 small & LStar & WMethod & 2 & 35 sec \\
  25                 small & TTT & WMethod & 2 & 32 sec \\
  26                 small & RivestSchapire & WMethod & 2 & 33 sec \\
  27                 small & KearnsVazirani & WMethod & 2 & 33 sec \\
  28
  29                 partial & LStar & Random & 4 & 18 sec \\
  30                 partial & TTT & Random & 4 & 16 sec \\
  31                 partial & RivestSchapire & Random & 4 & 13 sec \\
  32                 partial & KearnsVazirani & Random & 4 & 22 sec \\
  33                 partial & LStar & WMethod & 4 & 384 sec \\
  34                 partial & TTT & WMethod & 4 & 390 sec \\
  35                 partial & RivestSchapire & WMethod & 4 & 384 sec \\
  36                 partial & KearnsVazirani & WMethod & 4 & 383 sec \\
  37
  38                 full & LStar & Random & 5 & 44 sec \\
  39                 full & TTT & Random & 5 & 25 sec \\
  40                 full & RivestSchapire & Random & 4 & 12 sec \\
  41                 full & KearnsVazirani & Random & 5 & 19 sec \\
  42                 full & LStar & WMethod & 5 & 2666 sec \\
  43                 full & TTT & WMethod & 5 & 2632 sec \\
  44                 full & RivestSchapire & WMethod & 5 & 2638 sec \\
  45                 full & KearnsVazirani & WMethod & - & - \\
  46                 \bottomrule
  47         \end{tabular}
  48         \caption{Learning parameters and resulting model size.}
  49 \end{table}