2approach.tex

   1 The test process consists of several stages. The results of the first stage,
   2 planning, are descibed in this document. On the basis of this document actual
   3 test cases will be designed. Afterwards the actual tests will be implemented
   4 and executed. The results of these tests will then be evaluated against the
   5 exit criteria found in Section~\ref{sec:exitcriteria} and depending on the
   6 outcome of these evaluations further tests might be deemed necessary.
   7
   8 \subsection{Quality Characteristics}
   9 The quality characteristics that are to be tested are described using the
  10 ISO/IEC 25010 \cite{iso25010} as a guideline. In the following sections we will
  11 discuss the relevant \textit{product quality} and \textit{quality in use}
  12 characteristics.
  13
  14 \subsubsection{Product quality}
  15 Product quality is divided into eight main categories which are divided into
  16 several subcategories. Below we will discuss the qualities which are relevant
  17 to the SUT.
  18 \begin{itemize}
  19         \item \textbf{Functional suitability}\\
  20         As described in Section~\ref{sec:risks} the SUT is core functionality of
  21         the networking capable system. Because many other systems running on the
  22         system could rely on it it is very important that the SUT is functionality
  23         suitable.  Therefore all three sub characteristics of Functional
  24         Suitability (\textit{Functional completeness, Functional correctness,
  25         Functional appropriateness}) are of vital importance. As was previously
  26         mentioned in Section~\ref{sec:risks} extra emphasis should be placed on
  27         testing \emph{Functional Correctness} as recovery from Failures in
  28         computer-to-computer systems is problematic.
  29         \item \textbf{Performance efficiency} \label{sec:perf_eff}\\
  30         As the SUT runs as a service on a system with other programs it must have
  31         efficient \emph{resource utilisation}. It can not contain any memory leaks
  32         or use other resources more than necessary.
  33         \item \textbf{Compatibility}\\
  34         \emph{Interoperability} is the key feature of the SUT as it's purpose is to
  35         communicate with other systems implementing the TCP protocol. Therefore it
  36         is of vital importance that the SUT implements the TCP protocol correctly.
  37         Furthermore it is very important that the SUT can \emph{co-exist} with
  38         other programs on the system it runs on, since it is used as a service by
  39         those programs. This means that the SUT has to handle preemption as well as
  40         having multiple programs requesting it's services at once.
  41         \item \textbf{Reliability}\\
  42         As stated before, the SUT is used as a core service, this means it has to
  43         be very \emph{mature}. It needs to behave as expected under normal working
  44         conditions. As it can continually be requested the SUT needs to have
  45         constant
  46         \emph{availability}. As the SUT relies on a potentially unreliable channel
  47         to send and receive data it needs to be \emph{fault tolerant}. The SUT
  48         needs to properly handle errors in received data or complete unavailability
  49         of the underlying channel.
  50 \end{itemize}
  51 This leaves four categories which are not relevant the SUT. Below we will
  52 shortly discuss per category why these are not relevant. \emph{Maintainability}
  53 is an important aspect of any software system, however for the SUT it is not a
  54 core aspect, as it is first and foremost of importance that the implementation
  55 is correct, furthermore TCP does not change often. \emph{Usability} isn't a
  56 core aspect either, as the SUT is not used directly by humans, but is a service
  57 which is addressed when another program needs it. \emph{Portability} isn't
  58 either as the SUT is installed on a system once and intended to work on that
  59 system. \emph{Security} isn't a feauture of the SUT either, systems using the
  60 SUT can add their own security mechanisms on top of it.
  61
  62
  63 \subsubsection{Quality in use}
  64 Quality in use is dived into five subcategories. Below we will discuss the
  65 categories which are relevant to the SUT.
  66 \begin{itemize}
  67         \item \textbf{Effectiveness}\\
  68         This is the core aspect of the SUT, users (other programs) need to be able
  69         to effectively use the SUT to send and receive data.
  70         \item \textbf{Efficiency}\\
  71         This issue has already been covered above under
  72         ``performance efficiency''~\ref{sec:perf_eff}.
  73         \item \textbf{Satisfaction}\\
  74         It is important that programs using the SUT can \emph{trust} that the SUT
  75         provides the promised services. This means that data is send and received
  76         reliably and the SUT provides clear and unambiguous errors when this
  77         service
  78         can not be provided.
  79         \item \textbf{Context Coverage}\\
  80         The SUT needs to behave as expected in all specified contexts
  81         (\emph{context completeness}).
  82 \end{itemize}
  83 This leaves \emph{freedom from risk}, which we consider not relevant as the SUT
  84 itself does not pose any risks, and correct implementation (which is covered in
  85 the other categories) gives clear guarantees to programs using the services of
  86 the SUT.
  87
  88 \subsection{Levels and types of testing} \label{levels}
  89 The client will deliver a product for certification. This means our team will
  90 only conduct \emph{acceptance testing} and assume that the client who requested
  91 certification has conducted \emph{unit}, \emph{module} and \emph{integration testing}.
  92 We will only be conducting \emph{black-box testing} and the client is not required to handover any
  93 source-code.
  94
  95 Initially we will conduct a few basic \emph{manual tests} based on
  96 experience acquired from previous certification requests (\emph{error guessing}). If
  97 the product fails these basic tests we immediately reject it and seize all further
  98 activities.
  99
 100 If the product is not rejected after the basic \emph{manual tests} we will proceed with the
 101 second stage of testing. For these follow-up tests we will use \emph{equivalence partitioning}
 102 to reduce the number of test cases. Every test case will result in a test report.
 103 If any of the test cases fail the product is rejected. In order to deliver
 104 usable feedback to the client we will still produce a test report.
 105
 106 \subsubsection{Manual tests}
 107
 108 The basic tests mentioned in Section \ref{levels} are conducted using a
 109 checklist. If any of the checks fail we immediately reject the product.
 110
 111 \begin{enumerate}
 112         \item Is the product complete?
 113         \item Does the product come with a manual or quick start guide?
 114         \item Is it possible to get the product in a usable state?
 115         \item Can we use the product to initiate a connection in a corruption
 116         free
 117         environment?
 118 \end{enumerate}
 119
 120 These\emph{ manual tests} are performed in order to ensure that the client has delivered a usable product.
 121
 122 \subsubsection{Test generation}
 123
 124 For the second state of testing we first use \emph{equivalence partitioning} to reduce the
 125 overall number of test cases as mentioned in Section \ref{levels}. At the highest level we can define the following equivalent \emph{input classes} for the SUT.
 126
 127 \begin{enumerate}
 128         \item Valid requests:
 129                 \begin{enumerate}
 130                         \item Single request.
 131                         \item Multiple requests.
 132                 \end{enumerate}
 133         \item Invalid requests:
 134                 \begin{enumerate}
 135                         \item Single request.
 136                         \item Multiple requests.
 137                 \end{enumerate}
 138 \end{enumerate}
 139
 140 For these requests we can introduce more cases by applying \emph{equivalence partitioning}
 141 to the different packets that are sent during one request.
 142
 143 \begin{enumerate}
 144         \item Packets received in order.
 145         \item Packets received out of order.
 146 \end{enumerate}
 147
 148 For each individual packet we can specify the follow \emph{equivalent classes}.
 149
 150 \begin{enumerate}
 151         \item Valid packet.
 152         \item Corrupted packet.
 153         \item Missing packets.
 154 \end{enumerate}
 155
 156 A \emph{decision table} is used in order to ensure all different equivalent classes are covered in our tests.
 157 Valid packets can only exist in one form, the valid form and there is
 158 no need to divide these into additional classes.
 159 For the invalid packets we will construct a test case where only one parameter is made invalid.
 160 For the invalid value of this parameter we use \emph{boundary value analysis} to reduce the total
 161 number of test cases.
 162 The SUT input parameters correspond to different fields in the network packets.
 163 The following fields and there boundary values are considered during the testing.
 164
 165 \begin{enumerate}
 166         \item Checksum: valid, invalid.
 167         \item Header: valid, even or odd number of bits corrupted.
 168         \item Payload: valid, even or odd number of bits corrupted.
 169 \end{enumerate}
 170
 171 \subsubsection{Test environment and automatization}
 172 \label{section:testenv}
 173 %%%Java\footnote{\url{http://www.java.com}} TCP driven echo server that executed
 174 %%%on a virtualized Ubuntu system\footnote{\url{http://www.ubuntu.com}} running on
 175 %%%the error behaviour custom iptables output policies have to be set
 176 %%%Listing~\ref{listing:iptables}. This is needed because the kernel by default
 177 %%%closes all connections from unknown sources and the manually created TCP
 178 %%%packets used in testing the implementation are from a source unknown to the
 179 %%%kernel.
 180 %%%
 181 %%%\begin{lstlisting}[label={listing:iptables},caption={settings iptables}]
 182 %%%Chain OUTPUT (policy ACCEPT)
 183 %%%target       prot    opt     source  destination
 184 %%%ACCEPT       tcp     --      anywhere        anywhere        tcp flags:PSH/PSH
 185 %%%DROP tcp     --      anywhere        anywhere        tcp flags:RST/RST
 186 %%%\end{lstlisting}
 187
 188 All the tools we are going to use together with the SUT gives us the following
 189 collection of software.
 190
 191 \begin{itemize}
 192         \item Windows\footnote{\url{http://www.microsoft.com/en-us/windows}}, used as a host OS.
 193         \item Ubuntu\footnote{\url{http://www.ubuntu.com}}, used as the guest OS running the SUT.
 194         \item VirtualBox\footnote{\url{https://www.virtualbox.org/}}, used to run the guest OS containing the SUT.
 195         \item Wireshark\footnote{\url{https://www.wireshark.org/}}, used on the guest in order to capture and analyze network
 196                 traffic.
 197         \item Bit-Twist\footnote{\url{http://bittwist.sourceforge.net/}}, used to prepare network packets.
 198         \item Java\footnote{\url{http://www.java.com}} TCP driven echo server.
 199 \end{itemize}
 200
 201 All test will be conducted in a virtual environment. We will use VirtualBox to
 202 run a Linux distribution with the product installed.
 203 The Linux distribution in question is Ubuntu.
 204 All the tests are performed from within the VirtualBox environment.
 205 When testing network transmissions we will only analyze the packets sent/received to/from the SUT. The host system is disconnected from the Internet or any other network in
 206 order to prevent unnecessary traffic.
 207 % Dit is niet nodig omdat het via loopback gaat
 208 %       Zeker weten? de SUT ontvangt ook niet loopback packets toch?
 209
 210 For each test case (except for the \emph{manual tests}) a file containing previously
 211 captured network traffic will be replayed using Wireshark and sent to the \emph{Java TCP driven echo server}. We will use Bit-Twist
 212 to update the prepared packets with the MAC address of the guest network
 213 adapter and provide them with a valid source address. This updating step is needed because the kernel would otherwise reject the packets and prevent them from reaching the SUT. The response packets sent by the \emph{Java TCP driven echo server} and passing trough the SUT will be recorded,
 214 analyzed and validated according to the \textit{RFC793} specification.
 215 The valid packets are build manually from the \textit{RFC793} specification.
 216 Invalid packets are generated from this valid traffic using Bit-Twist.
 217 The boundary values for the different parameters (fields in packets) are determined by hand. Automated scripts are used in order to generate packets with some fields replaced with these
 218 \emph{boundary values}.