-Hello world
-%6
-%7
-%8
-%9
-%10
-%11
-%12
+The test process consists of several stages. The results of the first stage,
+planning, are descibed in this document. On the basis of this document actual
+test cases will be designed. Afterwards the actual tests will be implemented
+and executed. The results of these tests will then be evaluated against the
+exit criteria found in Section~\ref{sec:exitcriteria} and depending on the
+outcome of these evaluations further tests might be deemed necessary.
+
+\subsection{Quality Characteristics}
+The quality characteristics that are to be tested are described using the
+ISO/IEC 25010~\cite{iso25010} as a guideline. In the following sections we will
+discuss the relevant \textit{product quality} and \textit{quality in use}
+characteristics.
+
+\subsubsection{Product quality}
+Product quality is divided into eight main categories which are divided into
+several subcategories. Below we will discuss the qualities which are relevant
+to the SUT.
+\begin{itemize}
+ \item \textbf{Functional suitability}\\
+ As described in Section~\ref{sec:risks} the SUT is core functionality of
+ the networking capable system. Because many other systems running on the
+ system could rely on it it is very important that the SUT is functionality
+ suitable. Therefore all three sub characteristics of Functional
+ Suitability (\textit{Functional completeness, Functional correctness,
+ Functional appropriateness}) are of vital importance. As was previously
+ mentioned in Section~\ref{sec:risks} extra emphasis should be placed on
+ testing \emph{Functional Correctness} as recovery from Failures in
+ computer-to-computer systems is problematic.
+ \item \textbf{Performance efficiency} \label{sec:perf_eff}\\
+ As the SUT runs as a service on a system with other programs it must have
+ efficient \emph{resource utilisation}. It can not contain any memory leaks
+ or use other resources more than necessary.
+ \item \textbf{Compatibility}\\
+ \emph{Interoperability} is the key feature of the SUT as it's purpose is to
+ communicate with other systems implementing the TCP protocol. Therefore it
+ is of vital importance that the SUT implements the TCP protocol correctly.
+ Furthermore it is very important that the SUT can \emph{co-exist} with
+ other programs on the system it runs on, since it is used as a service by
+ those programs. This means that the SUT has to handle preemption as well as
+ having multiple programs requesting it's services at once.
+ \item \textbf{Reliability}\\
+ As stated before, the SUT is used as a core service, this means it has to
+ be very \emph{mature}. It needs to behave as expected under normal working
+ conditions. As it can continually be requested the SUT needs to have
+ constant
+ \emph{availability}. As the SUT relies on a potentially unreliable channel
+ to send and receive data it needs to be \emph{fault tolerant}. The SUT
+ needs to properly handle errors in received data or complete unavailability
+ of the underlying channel.
+\end{itemize}
+This leaves four categories which are not relevant the SUT. Below we will
+shortly discuss per category why these are not relevant. \emph{Maintainability}
+is an important aspect of any software system, however for the SUT it is not a
+core aspect, as it is first and foremost of importance that the implementation
+is correct, furthermore TCP does not change often. \emph{Usability} isn't a
+core aspect either, as the SUT is not used directly by humans, but is a service
+which is addressed when another program needs it. \emph{Portability} isn't
+either as the SUT is installed on a system once and intended to work on that
+system. \emph{Security} isn't a feature of the SUT either, systems using the
+SUT can add their own security mechanisms on top of it.
+
+
+\subsubsection{Quality in use}
+Quality in use is dived into five subcategories. Below we will discuss the
+categories which are relevant to the SUT.
+\begin{itemize}
+ \item \textbf{Effectiveness}\\
+ This is the core aspect of the SUT, users (other programs) need to be able
+ to effectively use the SUT to send and receive data.
+ \item \textbf{Efficiency}\\
+ This issue has already been covered above under
+ ``performance efficiency''~\ref{sec:perf_eff}.
+ \item \textbf{Satisfaction}\\
+ It is important that programs using the SUT can \emph{trust} that the SUT
+ provides the promised services. This means that data is send and received
+ reliably and the SUT provides clear and unambiguous errors when this
+ service
+ can not be provided.
+ \item \textbf{Context Coverage}\\
+ The SUT needs to behave as expected in all specified contexts
+ (\emph{context completeness}).
+\end{itemize}
+This leaves \emph{freedom from risk}, which we consider not relevant as the SUT
+itself does not pose any risks, and correct implementation (which is covered in
+the other categories) gives clear guarantees to programs using the services of
+the SUT.
+
+\subsection{Levels and types of testing} \label{section:levels}
+The client will deliver a product for certification. This means our team will
+only conduct \emph{acceptance testing} and assume that the client who requested
+certification has conducted \emph{unit}, \emph{module} and \emph{integration
+testing}. We will only be conducting \emph{black-box testing} and the client
+is not required to handover any source-code.
+
+Initially we will conduct a few basic \emph{manual tests} based on experience
+acquired from previous certification requests (\emph{error guessing}). If the
+product fails these basic tests we immediately reject it and seize all further
+activities.
+
+If the product is not rejected after the basic \emph{manual tests} we will
+proceed with the second stage of testing. For these follow-up tests we will use
+\emph{equivalence partitioning} to reduce the number of test cases. Every test
+case will result in a test report. If any of the test cases fail the product
+is rejected. In order to deliver usable feedback to the client we will still
+produce a test report.
+
+\subsubsection{Manual tests}
+
+The basic tests mentioned in Section~\ref{section:levels} are conducted using a
+checklist. If any of the checks fail we immediately reject the product.
+
+\begin{enumerate}
+ \item Is the product complete?
+ \item Does the product come with a manual or quick start guide?
+ \item Is it possible to get the product in a usable state?
+ \item Can we use the product to initiate a connection in a corruption
+ free
+ environment?
+\end{enumerate}
+
+These\emph{ manual tests} are performed in order to ensure that the client has
+delivered a usable product.
+
+\subsubsection{Test generation}
+
+For the second state of testing we first use \emph{equivalence partitioning} to
+reduce the overall number of test cases as mentioned in
+Section~\ref{section:levels}. At the highest level we can define the following
+equivalent \emph{input classes} for the SUT.
+
+\begin{enumerate}
+ \item Valid requests:
+ \begin{enumerate}
+ \item Single request.
+ \item Multiple requests.
+ \end{enumerate}
+ \item Invalid requests:
+ \begin{enumerate}
+ \item Single request.
+ \item Multiple requests.
+ \end{enumerate}
+\end{enumerate}
+
+For these requests we can introduce more cases by applying \emph{equivalence
+partitioning} to the different packets that are sent during one request.
+
+\begin{enumerate}
+ \item Packets received in order.
+ \item Packets received out of order.
+\end{enumerate}
+
+For each individual packet we can specify the follow \emph{equivalent classes}.
+
+\begin{enumerate}
+ \item Valid packet.
+ \item Corrupted packet.
+ \item Missing packets.
+\end{enumerate}
+
+A \emph{decision table} is used in order to ensure all different equivalent
+classes are covered in our tests. Valid packets can only exist in one form,
+the valid form and there is no need to divide these into additional classes.
+For the invalid packets we will construct a test case where only one parameter
+is made invalid. For the invalid value of this parameter we use \emph{boundary
+value analysis} to reduce the total number of test cases. The SUT input
+parameters correspond to different fields in the network packets. The
+following fields and there boundary values are considered during the testing.
+
+\begin{enumerate}
+ \item Checksum: valid, invalid.
+ \item Header: valid, even or odd number of bits corrupted.
+ \item Payload: valid, even or odd number of bits corrupted.
+\end{enumerate}
+
+\subsubsection{Test environment and automatization}
+\label{section:testenv}
+All the tools we are going to use together with the SUT gives us the following
+collection of software
+
+\begin{itemize}
+ \item Windows\footnote{\url{http://www.microsoft.com/en-us/windows}}, used
+ as a host OS.
+ \item Ubuntu\footnote{\url{http://www.ubuntu.com}}, used as the guest OS
+ running the SUT.
+ \item VirtualBox\footnote{\url{https://www.virtualbox.org/}}, used to run
+ the guest OS containing the SUT.
+ \item Wireshark\footnote{\url{https://www.wireshark.org/}}, used on the
+ guest OS in order to capture and analyze network traffic.
+ \item Bit-Twist\footnote{\url{http://bittwist.sourceforge.net/}}, used to
+ prepare network packets.
+ \item Java\footnote{\url{http://www.java.com}} TCP driven echo server.
+\end{itemize}
+
+\begin{figure}[H]
+ \label{fig:sut}
+ \centering
+ \includegraphics[width=0.5\linewidth]{SUTsetup.eps}
+ \caption{Test environment}
+\end{figure}
+
+
+All test will be conducted in a virtual environment. We will use VirtualBox to
+run a Linux distribution with the product installed. The Linux distribution in
+question is Ubuntu. All the tests are performed from within the VirtualBox
+environment. When testing network transmissions we will only analyze the
+packets sent/received to/from the SUT. The host system is disconnected from the
+Internet or any other network in order to prevent unnecessary traffic.
+% Dit is niet nodig omdat het via loopback gaat
+% Zeker weten? de SUT ontvangt ook niet loopback packets toch?
+% Ja maar we geven de host gewoon geen interface?
+
+For each test case (except for the \emph{manual tests}) a file containing
+previously captured network traffic will be replayed using Wireshark and sent
+to the \emph{Java TCP driven echo server}. We will use Bit-Twist to update the
+prepared packets with the MAC address of the guest network adapter and provide
+them with a valid source address. This updating step is needed because the
+kernel would otherwise reject the packets and prevent them from reaching the
+SUT. The response packets sent by the \emph{Java TCP driven echo server} and
+passing trough the SUT will be recorded, analyzed and validated according to
+the \textit{RFC793} specification. The valid packets are build manually from
+the \textit{RFC793} specification. Invalid packets are generated from this
+valid traffic using Bit-Twist. The boundary values for the different
+parameters (fields in packets) are determined by hand. Automated scripts are
+used in order to generate packets with some fields replaced with these
+\emph{boundary values}.
repositories / tt2015.git / blobdiff