a1/3eval.tex

   1 \subsection{Exit criteria}
   2 \label{sec:exitcriteria}
   3 The testing will be complete and the SUT will pass if and only if all the
   4 functionality tests are a success. Functionality tests are very important since
   5 the SUT has to adhere to a specification of a computer-to-computer protocol and
   6 therefore the impact of a defect in the system is very high.
   7
   8 Other exit criteria are schedule deadline and cost limits. When these limits
   9 are reached the testing will terminate but there is no confidence in the
  10 quality of the product.
  11
  12 \subsection{Test ware}
  13 As stated in Section \ref{section:levels} a basic \emph{manual test} is performed in order to detect a faulty project early on.
  14 If the SUT fails all second stage test cases we will try to expand the checklist in order to make sure that a similar faulty SUT can be detected during the first stage of testing.
  15
  16 The packets composed according to \textit{RFC793} as described in Section \ref{section:testenv} are saved for possible reuse.
  17 Since they are created by hand this is a labor intensive task.
  18 It makes sense to keep them around in case the SUT fails certification and the client would request reevaluation of a updated version of the SUT.
  19 If the SUT does in fact pass certification the packets could still be reused in the event an other client would request \textit{RFC793} certification of a similar product.
  20 The same is true for the \emph{Java TCP driven echo server}.
  21
  22 \subsection{Issue tracking}
  23 Whenever an issue is detected (a test fails) this issue needs to be recorded
  24 such that this incident is added to the test report. It can then be determined
  25 if the incident was the result of a fault in the SUT or was caused by something
  26 else, e.g. an error in the test.
  27
  28 The record of a incident needs to hold all relevant data such that the incident
  29 can easily be reproduced later. Therefore each incident will be recorded in an
  30 incident report, which will include the following details:
  31 \begin{itemize}
  32         \item When the issue was detected and by whom.
  33         \item Which test detected the issue.
  34         \item Expected results.
  35         \item Actual Results.
  36         \item Relevant logs, screenshots or other data which is needed in
  37                         addition to the test itself to reproduce the incident and which
  38                         might be beneficial to it's resolution.
  39         \item Other areas of the SUT which might be affected by the incident or by
  40                         a change resulting from the incident.
  41 \end{itemize}
  42