\subsection{Preflight checklist}
-
Before actual certification is commenced we perform a manual test using a
checklist.
If any of the checks fail we immediately reject the product.
+The checklist is given in the table below. All commands in \texttt{monospace}
+are to be run in a terminal. Commands prefixed with a \texttt{\#} should be run
+with root permissions. Commands prefixed with a \texttt{\$} should be run with
+user permissions.
-\begin{enumerate}
- \item Is the product complete?
- \item Does the product come with a manual or quick start guide?
- \item Is it possible to get the product in a usable state?
- \item Can we use the product to initiate a connection in a corruption
- free environment?
-\end{enumerate}
-
-\textbf{DIT MOET VERANDERD WORDEN, HET PRODUCT ZOU AFGEKEURD WORDEN!}
+\begin{longtable}{|l|rp{.8\linewidth}|}
+ \hline
+ Check 1 & \multicolumn{2}{l|}{Get the SUT in a workable state.}\\
+ \hline
+ \multirow{3}{*}{Course of action}
+ & 1. & Import the VirtualBox image into VirtualBox.\\
+ & 2. & Boot the vm.\\
+ & 3. & Verify the SUT booted successfully and the network modules are
+ loaded.\\
+ \hline
+ Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\
+ \hline\hline
+ Check 2 & \multicolumn{2}{l|}{Verify the SUT is complete.}\\
+ \hline
+ \multirow{5}{*}{Course of action}
+ & 1. & Boot the SUT as in \emph{Check 1}.\\
+ & 2. & Verify the loopback device exists by running
+ \texttt{\$ ifconfig}.\\
+ & 3. & Verify the \emph{echo-server} is present on the system by running
+ \texttt{\$ file code/server/Main.java}\\
+ & 4. & Verify \emph{Scapy} is present on the system by running
+ \texttt{\$ scapy}.\\
+ & 5. & Verify all scripts used for testing are present on the system.\\
+ \hline
+ Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\
+ \hline\hline
+ Check 3 & \multicolumn{2}{l|}{Initialize the testing environment..}\\
+ \hline
+ \multirow{6}{*}{Course of action}
+ & 1. & Boot the SUT as in \emph{Check 1}.\\
+ & 2. & Setup iptables by executing
+ \texttt{\# code/iptables.sh}~\footnote{The IPTables script ensures
+ that the OS does not drop packets due to an the unknown source.}\\
+ & 3. & Navigate to the working directory by running
+ \texttt{\$ cd /home/student/tt2015}\\
+ & 4. & Compile the echo server by running
+ \texttt{\# cd code/server \&\& make}\\
+ & 5. & Start the echo server by running
+ \texttt{\# cd code/server \&\& java Main}\\
+ & 6. & Generate all test cases by running
+ \texttt{\$ python code/client/gen.py}\\
+ \hline
+ Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\
+ \hline\hline
+ Check 4 & \multicolumn{2}{l|}{Test the tool environment.}\\
+ \hline
+ \multirow{3}{*}{Course of action}
+ & 1. & Initialize the SUT as in \emph{Check 3}\\
+ & 2. & Execute the test script by running
+ \texttt{\# code/client/helloworld.py}\\
+ & 3. & Verify the console displays a success message.\\
+ \hline
+ Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\
+ \hline\hline
+ Check 5 & \multicolumn{2}{l|}{All test inputs and scripts are present.}\\
+ \hline
+ \multirow{2}{*}{Course of action}
+ & 1. & Boot the SUT as in \emph{Check 1}.\\
+ & 2. & Verify that the test generation script is present by running
+ \texttt{\$ file code/client/test.py}\\
+ \hline
+ Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\
+ \hline
+ \caption{Preflight checklist\label{tbl:preflight}}
+\end{longtable}
\subsection{Testing of SUT}
-
-The SUT is a series of services for other computer programs with no end-user
-facing
-interface. Therefore the SUT will be tested solely by calling it's services
-through various automated scripts. An automated test suite will be available
-which executes all these automated scripts and aggregates their results to
-asses whether or not the SUT has passed the test.
+The SUT is a series of services for other computer programs with no end-user
+facing interface. Therefore the SUT will be tested solely by calling it's
+services through various automated scripts. An automated test suite will be
+available which executes all these automated scripts and aggregates their
+results to asses whether or not the SUT has passed the test.
The implementation of the SUT is tested using black box testing techniques. A
series of tests asses the correctness of the implementation with regards to the
-TCP specification. These tests are specified in Table~\textbf{referentie naar
-tests-tabel}. The test cases aim to cover the most interesting parts of the
-TCP specification.
+TCP specification. These tests are specified in Table~\ref{tbl:testpairs}. The test cases aim to cover the most interesting parts of the TCP
+specification.
+
+In this test suite the behavior of the outwards (network) interface of the SUT
+is assessed. This is, the behavior of the SUT as
+observed by another system on the network trying to communicate with (a system
+running on the same host as the SUT using) the SUT. This is achieved by running
+man
+echo-server on the system which runs the SUT (a virtual machine) which
+echos back all messages received to the sender. The test scripts will send
+packets to the echo-server and then checks the received
+response to asses whether or not the SUT is preforming as expected.
To cover the TCP specification as complete as possible while still maintaining
a feasible test suite the tests are divided into equivalence partitions. Below
these partitions are given.
\begin{enumerate}
- \item \emph{Number of packets} in request\footnotemark
- \footnotetext{A request is considered establishing a connection
- (handshake) and any n number of payloadpackts}
+ \item \emph{Number of segments} in request~\footnote{A request is
+ considered establishing a connection (handshake) and a number of
+ payload segments}
+ \begin{enumerate}
+ \item 0 payload segments
+ \item 1 payload segments
+ \item n=small payload segments (1 byte)
+ \item n=big payload segments (65495 bytes)
+ \end{enumerate}
+ \item \emph{source port}
+ \begin{enumerate}
+ \item Correct
+ \item Incorrect
+ \end{enumerate}
+ \item \emph{destination port}
\begin{enumerate}
- \item 0 payload packets
- \item 1 payload packet
- \item n=small payload packets
- \item n=big payload packets
+ \item Correct
+ \item Incorrect
\end{enumerate}
- \item Correct or Incorrect \emph{source port}
- \item Correct or Incorrect \emph{Destination port}
- \item Bits flipped in \emph{Payload}
+ \item Bit errors in \emph{payload}
\begin{enumerate}
\item Correct payload
- \item Payload with even number of bits flipped
- \item Payload with odd number of bits flipped
+ \item Payload with bit flips that do not show in checksum
+ \item Payload with bit flips that do show in checksum
+ \end{enumerate}
+ \item \emph{checksum}
+ \begin{enumerate}
+ \item Correct
+ \item Incorrect
+ \end{enumerate}
+ \item \emph{Segment order}
+ \begin{enumerate}
+ \item Correct
+ \item Out of order
+ \item Missing Segments
\end{enumerate}
- \item Correct or Incorrect \emph{checksum}
- \item Packets received in or out of order, or missing packets
\end{enumerate}
-\textbf{hier iets over waarom deze partities relevant zijn!}
+These partitions were chosen since they correspond to key parts of the TCP
+specification.
+
+%
+% één na laatste packket, moet dit B+3 of B+2 zijn?
+%
+
+TCP segments are send over a TCP connection from a \emph{source} to a \emph{destination port}. Therefore segments which are received that have a
+source or destination port set to an incorrect value should not be regarded
+as segments belonging to the connection by the SUT.
+
+TCP uses a \emph{checksum} to catch any error introduced in headers, when this
+checksum does not match the actual computed checksum the SUT should
+disregard the received segment.
+
+The TCP checksum is also an inherently weak one, as it is simply the
+bitwise negation of the addition, in ones complement arithmetic,
+of all 16 bit words in the header and data of the segment (excluding the
+checksum itself). Therefore any \emph{bit error} where the ones complement value
+of one word
+increases by one, and the value of another decreases by one, is undetected.
+The SUT should exhibit the same behavior and accept packets where these type
+of bit errors occur.
+
+TCP guarantees that segments are delivered \emph{in order}
+, even when they are received
+out of order and that missing segments are resend. The SUT should
+exhibit the same behavior. If segments are received out of order it should
+either reassemble them when the missing packet has arrived or request them to
+be resend when the Missing segments should be re-requested (by ACK-ing
+the correct sequence number).
+
+A single request consists of a number of packets that sent a
+to the \emph{echo-server} and back. The TCP specification state that such a
+transaction requires the following messages.
+
+\begin{center}
+ Script $-$ SYN $A$ $\rightarrow$ SUT \\
+ Script $\leftarrow$ SYN-ACK $(A+1)$ $B$ $-$ SUT \\
+ Script $-$ ACK $(A+1)$ $(B+1)$ $\rightarrow$ SUT \\
+ Script $-$ ACK-PUSH $(A+1)$ $(B+2)$ \emph{msg} $\rightarrow$ SUT \\
+ Script $\leftarrow$ ACK $(A+2+msg_{length})$ $(B+3)$ $-$ SUT \\
+ Script $\leftarrow$ ACK-PUSH $(A+2+msg_{length})$ $(B+3)$ \emph{msg} $-$ SUT
+\end{center}
+
+\bigskip
Partitions 2 to 6 are tested using pairwise testing to keep the number of test
cases feasible. The pairs are then all *except some where it does not make sense
to do so) tested with the different request sizes of partition 1.
-This is expressed in the table below.
+This is expressed in Table~\ref{tbl:testpairs}. In this table the first five
+columns represent the different options for the partitions 2 to 6 of the above
+enumeration. The last four columns are the different number segments as
+described in the partition 1 of the above enumeration. These cells identify
+individual test cases by a number. An \xmark in the cell indicates that this
+test case can not be created as it is not possible with that number of segments
+(eg. sending segments out of order when the number of segments is 1).
-\vspace{3mm}
-\textbf{Hier daadwerkelijke decision tabel}
-\vspace{3mm}
+\setcounter{TCC}{1}
+\begin{table}[H]
+ \centering
+ \begin{tabular}{|l|l|l|l|l|l||l|l|l|l|}
+ \hline
+ & \multicolumn{9}{c|}{\textbf{Partition}}\\
+ \hline
+ & 4 & 5 & 3 & 6 & 2 & 1a & 1b & 1c & 1d\\
+ \hline\hline
+ \multirow{9}{*}{Instance}
+ & a & a & a & a & a & \doTCC & \doTCC & \doTCC & \doTCC\\
+ & a & b & b & c & b & \xmark & \xmark & \doTCC & \doTCC\\
+ & c & a & b & a & b & \xmark & \xmark & \doTCC & \doTCC\\
+ & c & b & a & c & a & \xmark & \xmark & \doTCC & \doTCC\\
+ & b & a & b & c & a & \xmark & \xmark & \doTCC & \doTCC\\
+ & b & b & a & b & b & \xmark & \xmark & \doTCC & \doTCC\\
+ & c & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\
+ & b & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\
+ & a & b & b & b & a & \xmark & \xmark & \doTCC & \doTCC\\
+ \hline
+ \end{tabular}
+\caption{Combinations of test cases}
+\label{tbl:testpairs}
+\end{table}
\subsection{Quality, completeness and coverage of tests}
As always, $100\%$ completeness is not feasible, therefore test cases are
carefully selected to cover the most interesting parts of the TCP specification
-to ensure a test suite.
+to ensure a complete but feasible test suite.
+
+To further increase the coverage of the test suites tests are randomized. The
+tests which test the handling of \emph{bit errors}, changes in the \emph{packet
+order} and \emph{dropped packets} randomize where they introduce an error. The
+test suite runs these tests multiple times to increase the likelihood that they
+discover a fault which is only present when an error occurs in a certain
+position.
To further decrease the number of tests needed test cases are divided into
equivalence partitions and the combination of cases as described in
-Table~(\textbf{referentie naar decision table}) ensures that all partitions are
+Table~\ref{tbl:testpairs} ensures that all partitions are
covered and the number of individual tests is still feasible.
+
+%
+% wat ik ook probeer ik krijg de eerste collum
+% zijn tekst niet verticaal gecentered
+%
+
+
+\subsection{Test suite}
+
+Before executing the test suite the test environment has to be initialized.
+
+\begin{enumerate}
+ \item Boot the vm using VirtualBox.
+ \item Setup iptables by executing \texttt{\# code/iptables.sh}
+ \item Navigate to the working directory by running
+ \texttt{\$ cd /home/student/tt2015}
+ \item Start the echo server by running
+ \texttt{\# cd code/server \&\& Java Main}
+\end{enumerate}
+
+\subsubsection{Preflight checks}
+The we do the preflight checks as defined in Table~\ref{tbl:preflight}.
+
+\subsubsection{Test Cases}
+If the SUT passes the preflight checks the actual test cases can be executed.
+Table~\ref{tbl:testcases} shows the expected results of each of the test cases
+described in Table~\ref{tbl:testpairs}.
+
+\setcounter{TCC}{1}
+\begin{table}[H]
+ \centering
+ \begin{tabular}{|l|p{.7\linewidth}|}
+ \hline
+ Test number & Expected results\\
+ \hline\hline
+ \doTCC & An ACK\# of the send sequence number + 1.\\ \hline
+ \doTCC & An ACK\# of the sequence number of the last send segment + the
+ size of the payload of that segment.\\ \hline
+ \doTCC & An ACK\# of the sequence number of the last send segment + the
+ size of the payload of that segment.\\ \hline
+ \doTCC & An ACK\# of the sequence number of the last send segment + the
+ size of the payload of that segment.\\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\
+ $\vdots$ & \\
+ \setcounter{TCC}{14}
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The segment is not attributed to the current connection
+ and therefore no ACK\# is received. \\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The segment is not attributed to the current connection
+ and therefore no ACK\# is received. \\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \doTCC & The ACK\# for the SEQ\# of the first segments which is
+ corrupted is received for each consecutive segment send.\\ \hline
+ \end{tabular}
+\caption{Expected results of test cases}
+\label{tbl:testcases}
+\end{table}
+
+% Bij Ramons afwezigheid
+% Paul Vitero (linkerkant lange gang)
+% verdieping Mercator
+
+
+%\begin{longtable}{|p{.2\linewidth}|p{.8\linewidth}|}
+ %\hline
+ %Nr & 1 \\\hline
+ %Title & Single valid request with 1byte payload. \\\hline
+ %Input & Generated packets. \\\hline
+ %Expected output & Packets echoed back by Echo-Server. \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/1.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+ %Nr & 2 \\\hline
+ %Title & Single valid request with 65495bytes payload. \\\hline
+ %Input & Generated packets. \\\hline
+ %Expected output & Packets echoed back by Echo-Server. \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/2.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+ %Nr & 3 \\\hline
+ %Title & 5 valid requests with 1byte payload. \\\hline
+ %Input & Generated packets. \\\hline
+ %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/3.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+ %Nr & 4 \\\hline
+ %Title & 5 valid requests with 65495bytes payload. \\\hline
+ %Input & Generated packets with 65495bytes payload. \\\hline
+ %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/4.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+ %Nr & 5 \\\hline
+ %Title & 5 valid requests with 1byte payload sent out of order. \\\hline
+ %Input & Generated packets with 1byte payload, two packets are swapped in position. \\\hline
+ %Expected output & All requests sent up to and including
+ %the swapped packet with the lowest sequence number, the remaining packets are dropped. \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/5.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+ %Nr & 6 \\\hline
+ %Title & Request with corrupted source port. \\\hline
+ %Input & Generated packets with 1byte payload, in these packets the source port number is increased by one. \\\hline
+ %Expected output & - \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+%
+ %Nr & 6 \\\hline
+ %Title & Request with corrupted destination port. \\\hline
+ %Input & Generated packets with 1byte payload, in these packets the destination port number is increased by one. \\\hline
+ %Expected output & - \\\hline
+ %\multirow{2}{*}{Course of action}
+ %& 1. Use the steps listed above in order to start the SUT. \\
+ %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline
+ %Valid trace & Verify that the script prints 'Success'. \\\hline
+ %\hline
+%
+%\end{longtable}
+
+%\begin{tabularx}{\linewidth}{| l | X|}
+%\hline
+%Nr & 1 \\\hline
+%Title & Single valid request. \\\hline
+%Input & Pcap file with prerecorded valid packets. \\\hline
+%Expected output & Pcap file with valid response to request. \\\hline
+%Course of action & \begin{enumerate}
+% \item Execute \emph{./scripts/tests/case1-single-valid.sh}
+% \item Load \emph{output/case1.pcap} with ...
+%\end{enumerate} \\\hline
+%Valid trace & \begin{enumerate}
+% \item \textbf{Hier packets benoemen?}
+%\end{enumerate} \\\hline
+%\end{tabularx}
+%
+%\begin{tabularx}{\linewidth}{| l | X|}
+% \hline
+% Nr & 2 \\\hline
+%Title & Single request with corrupted checksum. \\\hline
+% Input & Pcap file used as \emph{test-case 1} input. \\\hline
+% Expected output & No response from SUT, logs with rejected packets. \\\hline
+% Course of action & \begin{enumerate}
+% \item Load input pcap file into ....
+% \item Corrupt checksum of loaded packets.
+% \item Save resulting packets as pcap file.
+% \item Load new pcap file into ...
+% \item Replay new pcap file.
+% \item Record SUT response using...
+% \item Extract log with rejected packets.
+% \item Save recorded packets as a pcap file.
+% \item Analyze packets in resulting file.
+% \end{enumerate} \\\hline
+% Valid trace & \begin{enumerate}
+% \item \textbf{Aangeven welke packets corrupted zijn?}
+% \end{enumerate} \\\hline
+%\end{tabularx}
+%\end{table}
repositories / tt2015.git / blobdiff