a4/question3.tex

   1 The table below contains some statistics about all the different parameter configurations we ran learnlib with.
   2 The \emph{RivestSchapire} learner using the Random test method resulted in an incorrect model being learned.
   3 When the \emph{KearnsVazirani} learner using the WMethod tester wasn't able to learn a model, this is due the learner hitting a non-deterministic path.
   4 This problem hasn't anything to do with the actual learner and is the result of a uncaught error situation in the adapter.
   5 This shows us that a leaner can be used to test software, since we discovered a bug in our adapter.
   6 Due to time constrains we were not able to fix this bug.
   7 When \emph{WMethod} is selected as the testing method \emph{RivestSchapire} is also able to learn the correct model.
   8 \emph{WMethod} does however increase the time needed to learn the model significantly, when a different learner is used there is no reason not to use the Random testing method.
   9
  10 \begin{longtable}{| l | l | l | c | c | c |}
  11         \caption{Learning parameters and resulting model size.} \\\hline
  12         Alphabet & Method & Test method & States & Time \\\hline \hline
  13         small & LStar & Random & 2 & 12 sec \\\hline
  14         small & TTT & Random & 2 & 5 sec \\\hline
  15         small & RivestSchapire & Random & 2 & 6 sec \\\hline
  16         small & KearnsVazirani & Random & 2 & 5 sec \\\hline
  17         small & LStar & WMethod & 2 & 35 sec \\\hline
  18         small & TTT & WMethod & 2 & 32 sec \\\hline
  19         small & RivestSchapire & WMethod & 2 & 33 sec \\\hline
  20         small & KearnsVazirani & WMethod & 2 & 33 sec \\\hline
  21
  22         partial & LStar & Random & 4 & 18 sec \\\hline
  23         partial & TTT & Random & 4 & 16 sec \\\hline
  24         partial & RivestSchapire & Random & 4 & 13 sec \\\hline
  25         partial & KearnsVazirani & Random & 4 & 22 sec \\\hline
  26         partial & LStar & WMethod & 4 & 384 sec \\\hline
  27         partial & TTT & WMethod & 4 & 390 sec \\\hline
  28         partial & RivestSchapire & WMethod & 4 & 384 sec \\\hline
  29         partial & KearnsVazirani & WMethod & 4 & 383 sec \\\hline
  30
  31         full & LStar & Random & 5 & 44 sec \\\hline
  32         full & TTT & Random & 5 & 25 sec \\\hline
  33         full & RivestSchapire & Random & 4 & 12 sec \\\hline
  34         full & KearnsVazirani & Random & 5 & 19 sec \\\hline
  35         full & LStar & WMethod & 5 & 2666 sec \\\hline
  36         full & TTT & WMethod & 5 & 2632 sec \\\hline
  37         full & RivestSchapire & WMethod & 5 & 2638 sec \\\hline
  38         full & KearnsVazirani & WMethod & - & - \\\hline
  39 \end{longtable}
  40