-
-%6
-%7
-%8
-%9
-%10
-%11
-%12
-
The test process consists of several stages. The results of the first stage,
planning, are descibed in this document. On the basis of this document actual
-test cases will be designed. Afterwards the actual tests will be implemented and
-executed. The results of these tests will then be evaluated against the exit
-criteria (see Section~\ref{sec:exitcriteria}) and depending on the outcome of
-these evaluations further tests might be deemed necessary.
+test cases will be designed. Afterwards the actual tests will be implemented
+and executed. The results of these tests will then be evaluated against the
+exit criteria found in Section~\ref{sec:exitcriteria} and depending on the
+outcome of these evaluations further tests might be deemed necessary.
\subsection{Quality Characteristics}
The quality characteristics that are to be tested are described using the
-ISO/IEC 25010 \cite{iso25010}. In the following sections we will discuss the
-relevant \textit{product quality} and \textit{quality in use} characteristics.
+ISO/IEC 25010 \cite{iso25010} as a guideline. In the following sections we will
+discuss the relevant \textit{product quality} and \textit{quality in use}
+characteristics.
\subsubsection{Product quality}
-Product quality is divided into eight main categories which are divided into several
-subcategories. Below we will discuss the qualities which are relevant to the SUT.
+Product quality is divided into eight main categories which are divided into
+several subcategories. Below we will discuss the qualities which are relevant
+to the SUT.
\begin{itemize}
- \item \textbf{Functional suitability}\\
- As descibed in Section~\ref{sec:risks} the SUT is core functioanlity of the
- networking capable system. Because many other systems running on the
- system could rely on it it is very important that the SUT is functionaly
- suitable. Therefore all three subcharacteristics of Functional Suitability
- (\textit{Functional completeness, Functional correctness, Functional
- appropriateness}) are of vital importance. As was previously mentioned
- in Section~\ref{sec:risks} extra emphasis should be placed on testing
- \emph{Functional Correctness} as recovery from Failures in
+ \item \textbf{Functional suitability}\\
+ As described in Section~\ref{sec:risks} the SUT is core functionality of
+ the networking capable system. Because many other systems running on the
+ system could rely on it it is very important that the SUT is functionality
+ suitable. Therefore all three sub characteristics of Functional
+ Suitability (\textit{Functional completeness, Functional correctness,
+ Functional appropriateness}) are of vital importance. As was previously
+ mentioned in Section~\ref{sec:risks} extra emphasis should be placed on
+ testing \emph{Functional Correctness} as recovery from Failures in
computer-to-computer systems is problematic.
+ \item \textbf{Performance efficiency} \label{sec:perf_eff}\\
+ As the SUT runs as a service on a system with other programs it must have
+ efficient \emph{resource utilisation}. It can not contain any memory leaks
+ or use other resources more than necessary.
+ \item \textbf{Compatibility}\\
+ \emph{Interoperability} is the key feature of the SUT as it's purpose is to
+ communicate with other systems implementing the TCP protocol. Therefore it
+ is of vital importance that the SUT implements the TCP protocol correctly.
+ Furthermore it is very important that the SUT can \emph{co-exist} with
+ other programs on the system it runs on, since it is used as a service by
+ those programs. This means that the SUT has to handle preemption as well as
+ having multiple programs requesting it's services at once.
+ \item \textbf{Reliability}\\
+ As stated before, the SUT is used as a core service, this means it has to
+ be very \emph{mature}. It needs to behave as expected under normal working
+ conditions. As it can continually be requested the SUT needs to have
+ constant
+ \emph{availability}. As the SUT relies on a potentially unreliable channel
+ to send and receive data it needs to be \emph{fault tolerant}. The SUT
+ needs to properly handle errors in received data or complete unavailability
+ of the underlying channel.
+\end{itemize}
+This leaves four categories which are not relevant the SUT. Below we will
+shortly discuss per category why these are not relevant. \emph{Maintainability}
+is an important aspect of any software system, however for the SUT it is not a
+core aspect, as it is first and foremost of importance that the implementation
+is correct, furthermore TCP does not change often. \emph{Usability} isn't a
+core aspect either, as the SUT is not used directly by humans, but is a service
+which is addressed when another program needs it. \emph{Portability} isn't
+either as the SUT is installed on a system once and intended to work on that
+system. \emph{Security} isn't a feauture of the SUT either, systems using the
+SUT can add their own security mechanisms on top of it.
+
+\subsubsection{Quality in use}
+Quality in use is dived into five subcategories. Below we will discuss the
+categories which are relevant to the SUT.
+\begin{itemize}
+ \item \textbf{Effectiveness}\\
+ This is the core aspect of the SUT, users (other programs) need to be able
+ to effectively use the SUT to send and receive data.
+ \item \textbf{Efficiency}\\
+ This issue has already been covered above under
+ ``performance efficiency''~\ref{sec:perf_eff}.
+ \item \textbf{Satisfaction}\\
+ It is important that programs using the SUT can \emph{trust} that the SUT
+ provides the promised services. This means that data is send and received
+ reliably and the SUT provides clear and unambiguous errors when this
+ service
+ can not be provided.
+ \item \textbf{Context Coverage}\\
+ The SUT needs to behave as expected in all specified contexts
+ (\emph{context completeness}).
\end{itemize}
+This leaves \emph{freedom from risk}, which we consider not relevant as the SUT
+itself does not pose any risks, and correct implementation (which is covered in
+the other categories) gives clear guarantees to programs using the services of
+the SUT.
+
+\subsection{Levels and types of testing} \label{levels}
+The client will deliver a product for certification. This means our team will
+only conduct \emph{acceptance testing} and assume that the client who requested
+certification has conducted \emph{unit}, \emph{module} and \emph{integration testing}.
+We will only be conducting \emph{black-box testing} and the client is not required to handover any
+source-code.
+
+Initially we will conduct a few basic \emph{manual tests} based on
+experience acquired from previous certification requests (\emph{error guessing}). If
+the product fails these basic tests we immediately reject it and seize all further
+activities.
+
+If the product is not rejected after the basic \emph{manual tests} we will proceed with the
+second stage of testing. For these follow-up tests we will use \emph{equivalence partitioning}
+to reduce the number of test cases. Every test case will result in a test report.
+If any of the test cases fail the product is rejected. In order to deliver
+usable feedback to the client we will still produce a test report.
+
+\subsubsection{Manual tests}
+
+The basic tests mentioned in Section \ref{levels} are conducted using a
+checklist. If any of the checks fail we immediately reject the product.
+
+\begin{enumerate}
+ \item Is the product complete?
+ \item Does the product come with a manual or quick start guide?
+ \item Is it possible to get the product in a usable state?
+ \item Can we use the product to initiate a connection in a corruption
+ free
+ environment?
+\end{enumerate}
+
+These\emph{ manual tests} are performed in order to ensure that the client has delivered a usable product.
+
+\subsubsection{Test generation}
+
+For the second state of testing we first use \emph{equivalence partitioning} to reduce the
+overall number of test cases as mentioned in Section \ref{levels}. At the highest level we can define the following equivalent \emph{input classes} for the SUT.
+
+\begin{enumerate}
+ \item Valid requests:
+ \begin{enumerate}
+ \item Single request.
+ \item Multiple requests.
+ \end{enumerate}
+ \item Invalid requests:
+ \begin{enumerate}
+ \item Single request.
+ \item Multiple requests.
+ \end{enumerate}
+\end{enumerate}
+
+For these requests we can introduce more cases by applying \emph{equivalence partitioning}
+to the different packets that are sent during one request.
+
+\begin{enumerate}
+ \item Packets received in order.
+ \item Packets received out of order.
+\end{enumerate}
+
+For each individual packet we can specify the follow \emph{equivalent classes}.
+
+\begin{enumerate}
+ \item Valid packet.
+ \item Corrupted packet.
+ \item Missing packets.
+\end{enumerate}
+
+A \emph{decision table} is used in order to ensure all different equivalent classes are covered in our tests.
+Valid packets can only exist in one form, the valid form and there is
+no need to divide these into additional classes.
+For the invalid packets we will construct a test case where only one parameter is made invalid.
+For the invalid value of this parameter we use \emph{boundary value analysis} to reduce the total
+number of test cases.
+The SUT input parameters correspond to different fields in the network packets.
+The following fields and there boundary values are considered during the testing.
+
+\begin{enumerate}
+ \item Checksum: valid, invalid.
+ \item Header: valid, even or odd number of bits corrupted.
+ \item Payload: valid, even or odd number of bits corrupted.
+\end{enumerate}
+
+\subsubsection{Test environment and automatization}
+\label{section:testenv}
+%%%Java\footnote{\url{http://www.java.com}} TCP driven echo server that executed
+%%%on a virtualized Ubuntu system\footnote{\url{http://www.ubuntu.com}} running on
+%%%the error behaviour custom iptables output policies have to be set
+%%%Listing~\ref{listing:iptables}. This is needed because the kernel by default
+%%%closes all connections from unknown sources and the manually created TCP
+%%%packets used in testing the implementation are from a source unknown to the
+%%%kernel.
+%%%
+%%%\begin{lstlisting}[label={listing:iptables},caption={settings iptables}]
+%%%Chain OUTPUT (policy ACCEPT)
+%%%target prot opt source destination
+%%%ACCEPT tcp -- anywhere anywhere tcp flags:PSH/PSH
+%%%DROP tcp -- anywhere anywhere tcp flags:RST/RST
+%%%\end{lstlisting}
+%%%All the tools we are going to use together with the SUT gives us the following
+%%%collection of software.
+
+\begin{enumerate}
+ \item Windows, used as a host OS.
+ \item Ubuntu, used as the guest OS running the SUT.
+ \item VirtualBox, used to run the guest OS containing the SUT.
+ \item Wireshark, used on the guest in order to capture and analyze network
+ traffic.
+ \item Bit-Twist, used to prepare network packets.
+\end{enumerate}
+
+All test will be conducted in a virtual environment. We will use VirtualBox to
+run a Linux distribution with the product installed.
+The Linux distribution in question is Ubuntu.
+All the tests are performed from within the VirtualBox environment.
+When testing network transmissions we will only analyze the packets sent/received to/from the SUT. The host system is disconnected from the Internet or any other network in
+order to prevent unnecessary traffic.
+% Dit is niet nodig omdat het via loopback gaat
+% Zeker weten? de SUT ontvangt ook niet loopback packets toch?
+
+For each test case (except for the \emph{manual tests}) a file containing previously
+captured network traffic will be replayed using Wireshark. We will use Bit-Twist
+to update the prepared packets with the MAC address of the guest network
+adapter. The response packets coming from the SUT will be recorded and
+analyzed and validated according to the \textit{RFC793} specification.
+The valid packets are build manually from the \textit{RFC793} specification.
+Invalid packets are generated from this valid traffic using Bit-Twist.
+The boundary values for the different parameters (fields in packets) are determined by hand. Automated scripts are used in order to generate packets with some fields replaced with these
+\emph{boundary values}.
-\subsubsection{Quality in use}
\ No newline at end of file
+% Dit is mooier om footnotes van te maken en te gebruiken als het voor het
+% eerst gerefereerd is
+% WAT HOE?
+\begin{enumerate}
+ \item VirtualBox, https://www.virtualbox.org/
+ \item Whireshark, https://www.wireshark.org/
+ \item Bit-Twist, http://bittwist.sourceforge.net/
+\end{enumerate}
repositories / tt2015.git / blobdiff