X-Git-Url: https://git.martlubbers.net/?a=blobdiff_plain;ds=sidebyside;f=2approach.tex;h=17c9de7a805471cea04a62b9e35a10f487750ca6;hb=30ccc9d8b967addf6f4b49e2fc81755f0f5230fe;hp=0b06e1e81ceadaa4fdb27e6b27021ef63717cf85;hpb=12f34d11f316bcb7b2cd63732515d55d37a1d6ad;p=tt2015.git diff --git a/2approach.tex b/2approach.tex index 0b06e1e..17c9de7 100644 --- a/2approach.tex +++ b/2approach.tex @@ -1,39 +1,225 @@ - -%6 -%7 -%8 -%9 -%10 -%11 -%12 - The test process consists of several stages. The results of the first stage, planning, are descibed in this document. On the basis of this document actual -test cases will be designed. Afterwards the actual tests will be implemented and -executed. The results of these tests will then be evaluated against the exit -criteria (see Section~\ref{sec:exitcriteria}) and depending on the outcome of -these evaluations further tests might be deemed necessary. +test cases will be designed. Afterwards the actual tests will be implemented +and executed. The results of these tests will then be evaluated against the +exit criteria found in Section~\ref{sec:exitcriteria} and depending on the +outcome of these evaluations further tests might be deemed necessary. \subsection{Quality Characteristics} The quality characteristics that are to be tested are described using the -ISO/IEC 25010 \cite{iso25010}. In the following sections we will discuss the -relevant \textit{product quality} and \textit{quality in use} characteristics. +ISO/IEC 25010~\cite{iso25010} as a guideline. In the following sections we will +discuss the relevant \textit{product quality} and \textit{quality in use} +characteristics. \subsubsection{Product quality} -Product quality is divided into eight main categories which are divided into several -subcategories. Below we will discuss the qualities which are relevant to the SUT. +Product quality is divided into eight main categories which are divided into +several subcategories. Below we will discuss the qualities which are relevant +to the SUT. \begin{itemize} - \item \textbf{Functional suitability}\\ - As descibed in Section~\ref{sec:risks} the SUT is core functioanlity of the - networking capable system. Because many other systems running on the - system could rely on it it is very important that the SUT is functionaly - suitable. Therefore all three subcharacteristics of Functional Suitability - (\textit{Functional completeness, Functional correctness, Functional - appropriateness}) are of vital importance. As was previously mentioned - in Section~\ref{sec:risks} extra emphasis should be placed on testing - \emph{Functional Correctness} as recovery from Failures in + \item \textbf{Functional suitability}\\ + As described in Section~\ref{sec:risks} the SUT is core functionality of + the networking capable system. Because many other systems running on the + system could rely on it it is very important that the SUT is functionality + suitable. Therefore all three sub characteristics of Functional + Suitability (\textit{Functional completeness, Functional correctness, + Functional appropriateness}) are of vital importance. As was previously + mentioned in Section~\ref{sec:risks} extra emphasis should be placed on + testing \emph{Functional Correctness} as recovery from Failures in computer-to-computer systems is problematic. + \item \textbf{Performance efficiency} \label{sec:perf_eff}\\ + As the SUT runs as a service on a system with other programs it must have + efficient \emph{resource utilisation}. It can not contain any memory leaks + or use other resources more than necessary. + \item \textbf{Compatibility}\\ + \emph{Interoperability} is the key feature of the SUT as it's purpose is to + communicate with other systems implementing the TCP protocol. Therefore it + is of vital importance that the SUT implements the TCP protocol correctly. + Furthermore it is very important that the SUT can \emph{co-exist} with + other programs on the system it runs on, since it is used as a service by + those programs. This means that the SUT has to handle preemption as well as + having multiple programs requesting it's services at once. + \item \textbf{Reliability}\\ + As stated before, the SUT is used as a core service, this means it has to + be very \emph{mature}. It needs to behave as expected under normal working + conditions. As it can continually be requested the SUT needs to have + constant + \emph{availability}. As the SUT relies on a potentially unreliable channel + to send and receive data it needs to be \emph{fault tolerant}. The SUT + needs to properly handle errors in received data or complete unavailability + of the underlying channel. +\end{itemize} +This leaves four categories which are not relevant the SUT. Below we will +shortly discuss per category why these are not relevant. \emph{Maintainability} +is an important aspect of any software system, however for the SUT it is not a +core aspect, as it is first and foremost of importance that the implementation +is correct, furthermore TCP does not change often. \emph{Usability} isn't a +core aspect either, as the SUT is not used directly by humans, but is a service +which is addressed when another program needs it. \emph{Portability} isn't +either as the SUT is installed on a system once and intended to work on that +system. \emph{Security} isn't a feature of the SUT either, systems using the +SUT can add their own security mechanisms on top of it. + + +\subsubsection{Quality in use} +Quality in use is dived into five subcategories. Below we will discuss the +categories which are relevant to the SUT. +\begin{itemize} + \item \textbf{Effectiveness}\\ + This is the core aspect of the SUT, users (other programs) need to be able + to effectively use the SUT to send and receive data. + \item \textbf{Efficiency}\\ + This issue has already been covered above under + ``performance efficiency''~\ref{sec:perf_eff}. + \item \textbf{Satisfaction}\\ + It is important that programs using the SUT can \emph{trust} that the SUT + provides the promised services. This means that data is send and received + reliably and the SUT provides clear and unambiguous errors when this + service + can not be provided. + \item \textbf{Context Coverage}\\ + The SUT needs to behave as expected in all specified contexts + (\emph{context completeness}). +\end{itemize} +This leaves \emph{freedom from risk}, which we consider not relevant as the SUT +itself does not pose any risks, and correct implementation (which is covered in +the other categories) gives clear guarantees to programs using the services of +the SUT. + +\subsection{Levels and types of testing} \label{section:levels} +The client will deliver a product for certification. This means our team will +only conduct \emph{acceptance testing} and assume that the client who requested +certification has conducted \emph{unit}, \emph{module} and \emph{integration +testing}. We will only be conducting \emph{black-box testing} and the client +is not required to handover any source-code. + +Initially we will conduct a few basic \emph{manual tests} based on experience +acquired from previous certification requests (\emph{error guessing}). If the +product fails these basic tests we immediately reject it and seize all further +activities. + +If the product is not rejected after the basic \emph{manual tests} we will +proceed with the second stage of testing. For these follow-up tests we will use +\emph{equivalence partitioning} to reduce the number of test cases. Every test +case will result in a test report. If any of the test cases fail the product +is rejected. In order to deliver usable feedback to the client we will still +produce a test report. + +\subsubsection{Manual tests} + +The basic tests mentioned in Section~\ref{section:levels} are conducted using a +checklist. If any of the checks fail we immediately reject the product. + +\begin{enumerate} + \item Is the product complete? + \item Does the product come with a manual or quick start guide? + \item Is it possible to get the product in a usable state? + \item Can we use the product to initiate a connection in a corruption + free + environment? +\end{enumerate} + +These\emph{ manual tests} are performed in order to ensure that the client has +delivered a usable product. +\subsubsection{Test generation} + +For the second state of testing we first use \emph{equivalence partitioning} to +reduce the overall number of test cases as mentioned in +Section~\ref{section:levels}. At the highest level we can define the following +equivalent \emph{input classes} for the SUT. + +\begin{enumerate} + \item Valid requests: + \begin{enumerate} + \item Single request. + \item Multiple requests. + \end{enumerate} + \item Invalid requests: + \begin{enumerate} + \item Single request. + \item Multiple requests. + \end{enumerate} +\end{enumerate} + +For these requests we can introduce more cases by applying \emph{equivalence +partitioning} to the different packets that are sent during one request. + +\begin{enumerate} + \item Packets received in order. + \item Packets received out of order. +\end{enumerate} + +For each individual packet we can specify the follow \emph{equivalent classes}. + +\begin{enumerate} + \item Valid packet. + \item Corrupted packet. + \item Missing packets. +\end{enumerate} + +A \emph{decision table} is used in order to ensure all different equivalent +classes are covered in our tests. Valid packets can only exist in one form, +the valid form and there is no need to divide these into additional classes. +For the invalid packets we will construct a test case where only one parameter +is made invalid. For the invalid value of this parameter we use \emph{boundary +value analysis} to reduce the total number of test cases. The SUT input +parameters correspond to different fields in the network packets. The +following fields and there boundary values are considered during the testing. + +\begin{enumerate} + \item Checksum: valid, invalid. + \item Header: valid, even or odd number of bits corrupted. + \item Payload: valid, even or odd number of bits corrupted. +\end{enumerate} + +\subsubsection{Test environment and automatization} +\label{section:testenv} +All the tools we are going to use together with the SUT gives us the following +collection of software + +\begin{itemize} + \item Windows\footnote{\url{http://www.microsoft.com/en-us/windows}}, used + as a host OS. + \item Ubuntu\footnote{\url{http://www.ubuntu.com}}, used as the guest OS + running the SUT. + \item VirtualBox\footnote{\url{https://www.virtualbox.org/}}, used to run + the guest OS containing the SUT. + \item Wireshark\footnote{\url{https://www.wireshark.org/}}, used on the + guest OS in order to capture and analyze network traffic. + \item Bit-Twist\footnote{\url{http://bittwist.sourceforge.net/}}, used to + prepare network packets. + \item Java\footnote{\url{http://www.java.com}} TCP driven echo server. \end{itemize} -\subsubsection{Quality in use} \ No newline at end of file +\begin{figure}[H] + \label{fig:sut} + \centering + \includegraphics[width=0.5\linewidth]{SUTsetup.eps} + \caption{Test environment} +\end{figure} + + +All test will be conducted in a virtual environment. We will use VirtualBox to +run a Linux distribution with the product installed. The Linux distribution in +question is Ubuntu. All the tests are performed from within the VirtualBox +environment. When testing network transmissions we will only analyze the +packets sent/received to/from the SUT. The host system is disconnected from the +Internet or any other network in order to prevent unnecessary traffic. +% Dit is niet nodig omdat het via loopback gaat +% Zeker weten? de SUT ontvangt ook niet loopback packets toch? +% Ja maar we geven de host gewoon geen interface? + +For each test case (except for the \emph{manual tests}) a file containing +previously captured network traffic will be replayed using Wireshark and sent +to the \emph{Java TCP driven echo server}. We will use Bit-Twist to update the +prepared packets with the MAC address of the guest network adapter and provide +them with a valid source address. This updating step is needed because the +kernel would otherwise reject the packets and prevent them from reaching the +SUT. The response packets sent by the \emph{Java TCP driven echo server} and +passing trough the SUT will be recorded, analyzed and validated according to +the \textit{RFC793} specification. The valid packets are build manually from +the \textit{RFC793} specification. Invalid packets are generated from this +valid traffic using Bit-Twist. The boundary values for the different +parameters (fields in packets) are determined by hand. Automated scripts are +used in order to generate packets with some fields replaced with these +\emph{boundary values}.