X-Git-Url: https://git.martlubbers.net/?a=blobdiff_plain;f=a2%2F1cases.tex;h=79da29da4cd4f4d74757a782098f6e0d9cb1c121;hb=c469ce23f042b86d8a90c6eb6f56530a2f5b498f;hp=b3ea384166d33af4208a9da195e9b12696afebd1;hpb=775f8c86c43002e35e1f769034c5660ed72c10fa;p=tt2015.git diff --git a/a2/1cases.tex b/a2/1cases.tex index b3ea384..79da29d 100644 --- a/a2/1cases.tex +++ b/a2/1cases.tex @@ -1,109 +1,160 @@ - -\pagebreak -\begin{table} \subsection{Preflight checklist} - Before actual certification is commenced we perform a manual test using a checklist. If any of the checks fail we immediately reject the product. -The checklist is given in the table below. - -\begin{tabularx}{\linewidth}{| l | X|} - \hline -Check 1 & Get the SUT in a workable state. \\\hline -Course of action & \begin{enumerate} - \item Import the VirtualBox image into VirtualBox. - \item Boot the vm. - \item Verify the SUT booted succesfully and the network module(s) are loaded. -\end{enumerate} \\\hline -Passed & \textit{Yes/No} \\\hline \hline -Check 2 & Verify the SUT is complete. \\\hline -Course of action & \begin{enumerate} - \item Use the course of action for \emph{Check 1} in order to boot the SUT. - \item Verify the loopback device exists using \emph{ifconfig} - \item Verify the \emph{echo-server} is present on the system. - \item Verify \emph{Scrappy} is present on the system. - \item Verify all scripts used for testing are present on the system. -\end{enumerate} \\\hline -Passed & \textit{Yes/No} \\\hline \hline -Check 3 & Initialize the testing environment. \\\hline -Course of action & \begin{enumerate} - \item Boot the SUT (see \emph{Check 1}). - \item Setup iptables by executing \emph{./scripts/setup/iptables.sh} - \item Setup ... by executing \emph{./scripts/setup/...} -\end{enumerate} \\\hline -Passed & \textit{Yes/No} \\\hline -\end{tabularx} -\end{table} +The checklist is given in the table below. All commands in \texttt{monospace} +are to be run in a terminal. Commands prefixed with a \texttt{\#} should be run +with root permissions. Commands prefixed with a \texttt{\$} should be run with +user permissions. -\begin{table} -\begin{tabularx}{\linewidth}{| l | X |} -\hline -Check 4 & Test the tool environment. \\\hline -Course of action & \begin{enumerate} - \item Boot and initialize the SUT (see \emph{Check 4}) - \item Execute the \emph{./scripts/tests/case0-helloworld.sh} script. - \item Execute the command \emph{cat logs/case0.log}. - \item Verify the console displays a success message. -\end{enumerate} \\\hline -Passed & \textit{Yes/No} \\\hline \hline -Check 5 & All test inputs and scripts are present. \\\hline -Course of action & \begin{enumerate} - \item Boot the SUT (see \emph{Check 1}) - \item Execute the command \emph{ls scripts/tests} - \item Verify that there is a file named case$X$ for each test case $X$ - \item Use \emph{ls packets} - \item Verify that there is a file named \emph{single.pcap} and \emph{multi.pcap}. -\end{enumerate} \\\hline -Passed & \textit{Yes/No} \\\hline -\end{tabularx} -\end{table} - -\newpage +\begin{longtable}{|l|rp{.8\linewidth}|} + \hline + Check 1 & \multicolumn{2}{l|}{Get the SUT in a workable state.}\\ + \hline + \multirow{3}{*}{Course of action} + & 1. & Import the VirtualBox image into VirtualBox.\\ + & 2. & Boot the vm.\\ + & 3. & Verify the SUT booted successfully and the network modules are + loaded.\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline\hline + Check 2 & \multicolumn{2}{l|}{Verify the SUT is complete.}\\ + \hline + \multirow{5}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Verify the loopback device exists by running + \texttt{\$ ifconfig}.\\ + & 3. & Verify the \emph{echo-server} is present on the system by running + \texttt{\$ file code/server/Main.java}\\ + & 4. & Verify \emph{Scapy} is present on the system by running + \texttt{\$ scapy}.\\ + & 5. & Verify all scripts used for testing are present on the system.\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline\hline + Check 3 & \multicolumn{2}{l|}{Initialize the testing environment..}\\ + \hline + \multirow{5}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Setup iptables by executing + \texttt{\# code/iptables.sh}~\footnote{The IPTables script ensures + that the OS does not drop packets due to an the unknown source.}\\ + & 3. & Navigate to the working directory by running + \texttt{\$ cd /home/student/tt2015}\\ + & 4. & Compile the echo server by running + \texttt{\# cd code/server \&\& make \&\& cd -}\\ + & 5. & Start the echo server by running + \texttt{\# cd code/server \&\& java Main}\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline\hline + Check 4 & \multicolumn{2}{l|}{Test the tool environment.}\\ + \hline + \multirow{3}{*}{Course of action} + & 1. & Initialize the SUT as in \emph{Check 3}\\ + & 2. & Execute the test script by running + \texttt{\# code/client/helloworld.py}\\ + & 3. & Verify the console displays a success message.\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline\hline + Check 5 & \multicolumn{2}{l|}{All test inputs and scripts are present.}\\ + \hline + \multirow{2}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Verify that the test generation script is present by running + \texttt{\$ file code/client/test.py}\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline + \caption{Preflight checklist\label{tbl:preflight}} +\end{longtable} \subsection{Testing of SUT} - -The SUT is a series of services for other computer programs with no end-user -facing -interface. Therefore the SUT will be tested solely by calling it's services -through various automated scripts. An automated test suite will be available -which executes all these automated scripts and aggregates their results to -asses whether or not the SUT has passed the test. +The SUT is a series of services for other computer programs with no end-user +facing interface. Therefore the SUT will be tested solely by calling it's +services through various automated scripts. An automated test suite will be +available which executes all these automated scripts and aggregates their +results to asses whether or not the SUT has passed the test. The implementation of the SUT is tested using black box testing techniques. A series of tests asses the correctness of the implementation with regards to the TCP specification. These tests are specified in Table~\textbf{referentie naar -tests-tabel}. The test cases aim to cover the most interesting parts of the -TCP specification. +tests-tabel}. The test cases aim to cover the most interesting parts of the TCP +specification. To cover the TCP specification as complete as possible while still maintaining a feasible test suite the tests are divided into equivalence partitions. Below these partitions are given. \begin{enumerate} - \item \emph{Number of packets} in request\footnotemark - \footnotetext{A request is considered establishing a connection - (handshake) and any n number of payloadpackts} + \item \emph{Number of segments} in request~\footnote{A request is + considered establishing a connection (handshake) and a number of + payload segments} + \begin{enumerate} + \item 0 payload segments + \item 1 payload segments + \item n=small payload segments (1 byte) + \item n=big payload segments (65495 bytes) + \end{enumerate} + \item \emph{source port} + \begin{enumerate} + \item Correct + \item Incorrect + \end{enumerate} + \item \emph{destination port} \begin{enumerate} - \item 0 payload packets - \item 1 payload packet - \item n=small payload packets - \item n=big payload packets + \item Correct + \item Incorrect \end{enumerate} - \item Correct or Incorrect \emph{source port} - \item Correct or Incorrect \emph{Destination port} - \item Bits flipped in \emph{Payload} + \item Bit errors in \emph{payload} \begin{enumerate} \item Correct payload - \item Payload with even number of bits flipped - \item Payload with odd number of bits flipped + \item Payload with bit flips that do not show in checksum + \item Payload with bit flips that do show in checksum + \end{enumerate} + \item \emph{checksum} + \begin{enumerate} + \item Correct + \item Incorrect + \end{enumerate} + \item \emph{Packet order} + \begin{enumerate} + \item Correct + \item Out of order + \item Missing packets \end{enumerate} - \item Correct or Incorrect \emph{checksum} - \item Packets received in or out of order, or missing packets \end{enumerate} -\textbf{hier iets over waarom deze partities relevant zijn! Waarom odd en -even number of bits flipped bijv interessant?} +These partitions were chosen since they correspond to key parts of the TCP +specification. + +TCP segments are send over a TCP connection from a \emph{source} to a \emph{destination port}. Therefore segments which are received that have a +source or destination port set to an incorrect value should not be regarded +as segments belonging to the connection by the SUT. + +TCP uses a \emph{checksum} to catch any error introduced in headers, when this +checksum does not match the actual computed checksum the SUT should +disregard the received segment. + +The TCP checksum is also an inherently weak one, as it is simply the +bitwise negation of the addition, in ones complement arithmetic, +of all 16 bit words in the header and data of the segment (excluding the +checksum itself). Therefore any \emph{bit error} where the ones complement value +of one word +increases by one, and the value of another decreases by one, is undetected. +The SUT should exhibit the same behavior and accept packets where these type +of bit errors occur. + +TCP guarantees that segments are delivered \emph{in order} +,even when they are received +out of order and that missing segments are resend. The SUT should +exhibit the same behavior. If segments are received out of order it should +either reassemble them when the missing packet has arrived or request them to +be resend when the Missing segments should be re-requested (by ACK-ing +the correct sequence number). + \bigskip Partitions 2 to 6 are tested using pairwise testing to keep the number of test @@ -112,18 +163,28 @@ to do so) tested with the different request sizes of partition 1. This is expressed in Table~\ref{table:testpairs}. -\begin{table} -\begin{tabular} {p{2cm} | l | p{2cm} | l | p{1cm} || c | c | p{1.3cm} | p{1.3cm}} -\textbf{Payload} & \textbf{checksum} & \textbf{destination port} & \textbf{order} & \textbf{source port} & \textbf{payload 0} & \textbf{payload 1} & \textbf{payload small} & \textbf{payload big} \\ \hline \hline -Correct & Correct & Correct & In order & Correct & \cmark & \cmark & \cmark & \cmark \\ \hline -Correct & Incorrect & Incorrect & Missing packet & Incorrect & \xmark & \xmark & \cmark & \cmark \\ \hline -odd\# of bits flipped & Correct & Incorrect & Out of order & Incorrect & \xmark & \xmark & \cmark & \cmark \\ \hline -odd\# of bits flipped & Incorrect & Correct & Missing packet & Correct & \xmark & \xmark & \cmark & \cmark \\ \hline -even\# of bits flipped & Correct & Incorrect & Missing packet & Correct & \xmark & \xmark & \cmark & \cmark \\ \hline -even\# of bits flipped & Incorrect & Correct & Out of order & Incorrect & \xmark & \xmark & \cmark & \cmark \\ \hline -odd\# of bits flipped & Incorrect & Incorrect & In order & Incorrect & \cmark & \cmark & \cmark & \cmark \\ \hline -even\# of bits flipped & Incorrect & Incorrect & In order & Incorrect & \cmark & \cmark & \cmark & \cmark \\ \hline -Correct & Incorrect & Incorrect & Out of order & Correct & \xmark & \xmark & \cmark & \cmark \\ +\newcounter{TCC} +\setcounter{TCC}{1} +\newcommand{\doTCC}{\theTCC \stepcounter{TCC}} +\begin{table}[H] + \centering + \begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|} + \hline + & \multicolumn{10}{c|}{\textbf{Partition}}\\ + \hline + & \# & 4 & 5 & 3 & 6 & 2 & 1a & 1b & 1c & 1d\\ + \hline\hline + \multirow{9}{*}{Instance} + & 1 & a & a & a & a & a & \doTCC & \doTCC & \doTCC & \doTCC\\ + & 2 & a & b & b & c & b & \xmark & \xmark & \doTCC & \doTCC\\ + & 3 & c & a & b & a & b & \xmark & \xmark & \doTCC & \doTCC\\ + & 4 & c & b & a & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & 5 & b & a & b & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & 6 & b & b & a & b & b & \xmark & \xmark & \doTCC & \doTCC\\ + & 7 & c & b & b & a & b & \doTCC & \doTCC & \doTCC & \doTCC\\ + & 8 & b & b & b & a & b & \doTCC & \doTCC & \doTCC & \doTCC\\ + & 9 & a & b & b & b & a & \xmark & \xmark & \doTCC & \doTCC\\ + \hline \end{tabular} \caption{Combinations of test cases} \label{table:testpairs} @@ -165,47 +226,107 @@ covered and the number of individual tests is still feasible. % wat ik ook probeer ik krijg de eerste collum % zijn tekst niet verticaal gecentered % -\begin{table} \subsection{Test cases} -\begin{tabularx}{\linewidth}{| l | X|} -\hline -Nr & 1 \\\hline -Title & Single valid request. \\\hline -Input & Pcap file with prerecorded valid packets. \\\hline -Expected output & Pcap file with valid response to request. \\\hline -Course of action & \begin{enumerate} - \item Execute \emph{./scripts/tests/case1-single-valid.sh} - \item Load \emph{output/case1.pcap} with ... -\end{enumerate} \\\hline -Valid trace & \begin{enumerate} - \item \textbf{Hier packets benoemen?} -\end{enumerate} \\\hline -\end{tabularx} - -\begin{tabularx}{\linewidth}{| l | X|} +Before every test case use the following steps to initialize the testing environment. + +\begin{enumerate} + \item Boot the vm using VirtualBox. + \item Setup iptables by executing \texttt{\# code/iptables.sh} + \item Navigate to the working directory by running \texttt{\$ cd /home/student/tt2015} + \item Start the echo server by running \texttt{\# cd code/server \&\& java Main} +\end{enumerate} + +\begin{longtable}{|p{.2\linewidth}|p{.8\linewidth}|} \hline + Nr & 1 \\\hline + Title & Single valid request with 1byte payload. \\\hline + Input & Generated packets. \\\hline + Expected output & Packets echoed back by Echo-Server. \\\hline + \multirow{2}{*}{Course of action} + & 1. Use the steps listed above in order to start the SUT. \\ + & 2. Execute the script by running \texttt{\# code/client/tests/1.py} \\\hline + Valid trace & Verify that the script prints 'Success'. \\\hline + \hline + Nr & 2 \\\hline - Title & Single request with corrupted checksum. \\\hline - Input & Pcap file used as \emph{test-case 1} input. \\\hline - Expected output & No response from SUT, logs with rejected packets. \\\hline - Course of action & \begin{enumerate} - \item Load input pcap file into .... - \item Corrupt checksum of loaded packets. - \item Save resulting packets as pcap file. - \item Load new pcap file into ... - \item Replay new pcap file. - \item Record SUT response using... - \item Extract log with rejected packets. - \item Save recorded packets as a pcap file. - \item Analyze packets in resulting file. - \end{enumerate} \\\hline - Valid trace & \begin{enumerate} - \item \textbf{Aangeven welke packets corrupted zijn?} - \end{enumerate} \\\hline -\end{tabularx} -\end{table} + Title & Single valid request with 65495bytes payload. \\\hline + Input & Generated packets. \\\hline + Expected output & Packets echoed back by Echo-Server. \\\hline + \multirow{2}{*}{Course of action} + & 1. Use the steps listed above in order to start the SUT. \\ + & 2. Execute the script by running \texttt{\# code/client/tests/2.py} \\\hline + Valid trace & Verify that the script prints 'Success'. \\\hline + \hline + + Nr & 3 \\\hline + Title & 5 valid requests with 1byte payload. \\\hline + Input & Generated packets. \\\hline + Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + \multirow{2}{*}{Course of action} + & 1. Use the steps listed above in order to start the SUT. \\ + & 2. Execute the script by running \texttt{\# code/client/tests/3.py} \\\hline + Valid trace & Verify that the script prints 'Success'. \\\hline + \hline + + Nr & 4 \\\hline + Title & 5 valid requests with 65495bytes payload. \\\hline + Input & Generated packets with 65495bytes payload. \\\hline + Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + \multirow{2}{*}{Course of action} + & 1. Use the steps listed above in order to start the SUT. \\ + & 2. Execute the script by running \texttt{\# code/client/tests/4.py} \\\hline + Valid trace & Verify that the script prints 'Success'. \\\hline + \hline + + Nr & 5 \\\hline + Title & 5 valid requests with 1byte payload sent out of order. \\\hline + Input & Generated packets with 1byte payload, two packets are swapped in position. \\\hline + Expected output & All requests sent up to and including + the swapped packet with the lowest sequence number, the remaining packets are dropped. \\\hline + \multirow{2}{*}{Course of action} + & 1. Use the steps listed above in order to start the SUT. \\ + & 2. Execute the script by running \texttt{\# code/client/tests/5.py} \\\hline + Valid trace & Verify that the script prints 'Success'. \\\hline + \hline +\end{longtable} -\newpage +%\begin{tabularx}{\linewidth}{| l | X|} +%\hline +%Nr & 1 \\\hline +%Title & Single valid request. \\\hline +%Input & Pcap file with prerecorded valid packets. \\\hline +%Expected output & Pcap file with valid response to request. \\\hline +%Course of action & \begin{enumerate} +% \item Execute \emph{./scripts/tests/case1-single-valid.sh} +% \item Load \emph{output/case1.pcap} with ... +%\end{enumerate} \\\hline +%Valid trace & \begin{enumerate} +% \item \textbf{Hier packets benoemen?} +%\end{enumerate} \\\hline +%\end{tabularx} +% +%\begin{tabularx}{\linewidth}{| l | X|} +% \hline +% Nr & 2 \\\hline +%Title & Single request with corrupted checksum. \\\hline +% Input & Pcap file used as \emph{test-case 1} input. \\\hline +% Expected output & No response from SUT, logs with rejected packets. \\\hline +% Course of action & \begin{enumerate} +% \item Load input pcap file into .... +% \item Corrupt checksum of loaded packets. +% \item Save resulting packets as pcap file. +% \item Load new pcap file into ... +% \item Replay new pcap file. +% \item Record SUT response using... +% \item Extract log with rejected packets. +% \item Save recorded packets as a pcap file. +% \item Analyze packets in resulting file. +% \end{enumerate} \\\hline +% Valid trace & \begin{enumerate} +% \item \textbf{Aangeven welke packets corrupted zijn?} +% \end{enumerate} \\\hline +%\end{tabularx} +%\end{table}