X-Git-Url: https://git.martlubbers.net/?a=blobdiff_plain;f=a2%2F1cases.tex;h=d339dfa57f64eb3a237f455e7bcd6e0ca107cfb1;hb=77f330957957adaaa9bef237609c6d81e9c3bda7;hp=18420a7e09e73dff808657041eb37542c84ba9d5;hpb=9d533c0f740c7f826b75286f309be9f7f3a3674d;p=tt2015.git diff --git a/a2/1cases.tex b/a2/1cases.tex index 18420a7..d339dfa 100644 --- a/a2/1cases.tex +++ b/a2/1cases.tex @@ -7,84 +7,71 @@ are to be run in a terminal. Commands prefixed with a \texttt{\#} should be run with root permissions. Commands prefixed with a \texttt{\$} should be run with user permissions. -\begin{table}[H] -\begin{tabularx}{\linewidth}{|l|X|} +\begin{longtable}{|l|rp{.8\linewidth}|} \hline - Check 1 & Get the SUT in a workable state. \\ + Check 1 & \multicolumn{2}{l|}{Get the SUT in a workable state.}\\ \hline - Course of action & - \begin{enumerate} - \item Import the VirtualBox image into VirtualBox. - \item Boot the vm. - \item Verify the SUT booted successfully and the network modules - are loaded. - \end{enumerate}\\ - \hline Passed & \textit{Yes/No} \\ + \multirow{3}{*}{Course of action} + & 1. & Import the VirtualBox image into VirtualBox.\\ + & 2. & Boot the vm.\\ + & 3. & Verify the SUT booted successfully and the network modules are + loaded.\\ + \hline + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ \hline\hline - Check 2 & Verify the SUT is complete. \\\hline - Course of action & - \begin{enumerate} - \item Boot the SUT as in \emph{Check 1}. - \item Verify the loopback device exists by running - \texttt{\$ ifconfig}. - \item Verify the \emph{echo-server} is present on the system. - \item Verify \emph{Scapy} is present on the system by running - \texttt{\$ scapy}. - \item Verify all scripts used for testing are present on the - system. - \end{enumerate}\\ + Check 2 & \multicolumn{2}{l|}{Verify the SUT is complete.}\\ + \hline + \multirow{5}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Verify the loopback device exists by running + \texttt{\$ ifconfig}.\\ + & 3. & Verify the \emph{echo-server} is present on the system by running + \texttt{\$ file code/server/Main.java}\\ + & 4. & Verify \emph{Scapy} is present on the system by running + \texttt{\$ scapy}.\\ + & 5. & Verify all scripts used for testing are present on the system.\\ \hline - Passed & \textit{Yes/No}\\ - \hline \hline - Check 3 & Initialize the testing environment.\\ + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline\hline + Check 3 & \multicolumn{2}{l|}{Initialize the testing environment..}\\ \hline - Course of action & - \begin{enumerate} - \item Boot the SUT as in \emph{Check 1}. - \item Setup iptables by executing - \texttt{\# code/iptables.sh}~\footnote{ The IPTables script - ensures that the OS does not drop packets due to an the - unknown source.} - \item Navigate to the working directory by running - \texttt{\$ cd /home/student/tt2015} - \item Compile the echo server by running - \texttt{\# cd code/server \&\& make \&\& cd -} - \item Start the echo server by running - \texttt{\# cd code/server \&\& java Main} - \end{enumerate}\\ + \multirow{6}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Setup iptables by executing + \texttt{\# code/iptables.sh}~\footnote{The IPTables script ensures + that the OS does not drop packets due to an the unknown source.}\\ + & 3. & Navigate to the working directory by running + \texttt{\$ cd /home/student/tt2015}\\ + & 4. & Compile the echo server by running + \texttt{\# cd code/server \&\& make}\\ + & 5. & Start the echo server by running + \texttt{\# cd code/server \&\& java Main}\\ + & 6. & Generate all test cases by running + \texttt{\$ python code/client/gen.py}\\ \hline - Passed & \textit{Yes/No}\\ + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ \hline\hline - Check 4 & Test the tool environment.\\ + Check 4 & \multicolumn{2}{l|}{Test the tool environment.}\\ \hline - Course of action & - \begin{enumerate} - \item Initialize the SUT as in \emph{Check 3} - \item Execute the test script by running - \texttt{\# code/client/helloworld.py}. - \item Verify the console displays a success message. - \end{enumerate}\\ + \multirow{3}{*}{Course of action} + & 1. & Initialize the SUT as in \emph{Check 3}\\ + & 2. & Execute the test script by running + \texttt{\# code/client/helloworld.py}\\ + & 3. & Verify the console displays a success message.\\ \hline - Passed & \textit{Yes/No}\\ + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ \hline\hline - Check 5 & All test inputs and scripts are present. \\\hline - % DIT MOET NOG EVEN ANDERS - Course of action & - \begin{enumerate} - \item Boot the SUT as in \emph{Check 1}. - \item Execute the command \texttt{ls scripts/tests} - \item Verify that there is a file named case$X$ for each test case $X$ - \item Use \emph{ls packets} - \item Verify that there is a file named \texttt{single.pcap} and - \texttt{multi.pcap}. - \end{enumerate}\\ + Check 5 & \multicolumn{2}{l|}{All test inputs and scripts are present.}\\ \hline - Passed & \textit{Yes/No}\\ + \multirow{2}{*}{Course of action} + & 1. & Boot the SUT as in \emph{Check 1}.\\ + & 2. & Verify that the test generation script is present by running + \texttt{\$ file code/client/test.py}\\ \hline -\end{tabularx} -\end{table} - -\newpage + Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ + \hline + \caption{Preflight checklist\label{tbl:preflight}} +\end{longtable} \subsection{Testing of SUT} The SUT is a series of services for other computer programs with no end-user @@ -95,8 +82,7 @@ results to asses whether or not the SUT has passed the test. The implementation of the SUT is tested using black box testing techniques. A series of tests asses the correctness of the implementation with regards to the -TCP specification. These tests are specified in Table~\textbf{referentie naar -tests-tabel}. The test cases aim to cover the most interesting parts of the TCP +TCP specification. These tests are specified in Table~\ref{tbl:testpairs}. The test cases aim to cover the most interesting parts of the TCP specification. To cover the TCP specification as complete as possible while still maintaining @@ -104,52 +90,124 @@ a feasible test suite the tests are divided into equivalence partitions. Below these partitions are given. \begin{enumerate} - \item \emph{Number of packets} in request~\footnote{A request is considered - establishing a connection (handshake) and a number of payload packets} + \item \emph{Number of segments} in request~\footnote{A request is + considered establishing a connection (handshake) and a number of + payload segments} \begin{enumerate} - \item 0 payload packets - \item 1 payload packet - \item n=small payload packets - \item n=big payload packets + \item 0 payload segments + \item 1 payload segments + \item n=small payload segments (1 byte) + \item n=big payload segments (65495 bytes) \end{enumerate} - \item Correct or incorrect \emph{source port} - \item Correct or incorrect \emph{destination port} - \item Bits flipped in \emph{payload} + \item \emph{source port} + \begin{enumerate} + \item Correct + \item Incorrect + \end{enumerate} + \item \emph{destination port} + \begin{enumerate} + \item Correct + \item Incorrect + \end{enumerate} + \item Bit errors in \emph{payload} \begin{enumerate} \item Correct payload - \item Payload with even number of bits flipped - \item Payload with odd number of bits flipped + \item Payload with bit flips that do not show in checksum + \item Payload with bit flips that do show in checksum + \end{enumerate} + \item \emph{checksum} + \begin{enumerate} + \item Correct + \item Incorrect + \end{enumerate} + \item \emph{Segment order} + \begin{enumerate} + \item Correct + \item Out of order + \item Missing Segments \end{enumerate} - \item Correct or Incorrect \emph{checksum} - \item Packets received in or out of order, or missing packets \end{enumerate} -\textbf{hier iets over waarom deze partities relevant zijn! Waarom odd en -even number of bits flipped bijv interessant?} +These partitions were chosen since they correspond to key parts of the TCP +specification. A single request consists of a number of packets that sent a +to the \emph{echo-server} and back. The TCP specification state that such a +transaction requires the following messages. + +\begin{center} + Script $-$ SYN $A$ $\rightarrow$ SUT \\ + Script $\leftarrow$ SYN-ACK $(A+1)$ $B$ $-$ SUT \\ + Script $-$ ACK $(A+1)$ $(B+1)$ $\rightarrow$ SUT \\ + Script $-$ ACK-PUSH $(A+1)$ $(B+2)$ \emph{msg} $\rightarrow$ SUT \\ + Script $\leftarrow$ ACK $(A+2+msg_{length})$ $(B+3)$ $-$ SUT \\ + Script $\leftarrow$ ACK-PUSH $(A+2+msg_{length})$ $(B+3)$ \emph{msg} $-$ SUT +\end{center} + +% +% één na laatste packket, moet dit B+3 of B+2 zijn? +% + +TCP segments are send over a TCP connection from a \emph{source} to a \emph{destination port}. Therefore segments which are received that have a +source or destination port set to an incorrect value should not be regarded +as segments belonging to the connection by the SUT. + +TCP uses a \emph{checksum} to catch any error introduced in headers, when this +checksum does not match the actual computed checksum the SUT should +disregard the received segment. + +The TCP checksum is also an inherently weak one, as it is simply the +bitwise negation of the addition, in ones complement arithmetic, +of all 16 bit words in the header and data of the segment (excluding the +checksum itself). Therefore any \emph{bit error} where the ones complement value +of one word +increases by one, and the value of another decreases by one, is undetected. +The SUT should exhibit the same behavior and accept packets where these type +of bit errors occur. + +TCP guarantees that segments are delivered \emph{in order} +,even when they are received +out of order and that missing segments are resend. The SUT should +exhibit the same behavior. If segments are received out of order it should +either reassemble them when the missing packet has arrived or request them to +be resend when the Missing segments should be re-requested (by ACK-ing +the correct sequence number). + \bigskip Partitions 2 to 6 are tested using pairwise testing to keep the number of test cases feasible. The pairs are then all *except some where it does not make sense to do so) tested with the different request sizes of partition 1. -This is expressed in Table~\ref{table:testpairs}. - -\begin{table} -% DIT MAAK IK NOG EVEN MOOI - \begin{tabularx}{\linewidth}{p{2cm} | l | p{2cm} | l | p{1cm} || c | c | p{1.3cm} | p{1.3cm}} - \textbf{Payload} & \textbf{checksum} & \textbf{destination port} & \textbf{order} & \textbf{source port} & \textbf{payload 0} & \textbf{payload 1} & \textbf{payload small} & \textbf{payload big} \\ \hline \hline - Correct & Correct & Correct & In order & Correct & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline - Correct & Incorrect & Incorrect & Missing packet & Incorrect & \xmark & \xmark & \checkmark & \checkmark \\ \hline - odd\# of bits flipped & Correct & Incorrect & Out of order & Incorrect & \xmark & \xmark & \checkmark & \checkmark \\ \hline - odd\# of bits flipped & Incorrect & Correct & Missing packet & Correct & \xmark & \xmark & \checkmark & \checkmark \\ \hline - even\# of bits flipped & Correct & Incorrect & Missing packet & Correct & \xmark & \xmark & \checkmark & \checkmark \\ \hline - even\# of bits flipped & Incorrect & Correct & Out of order & Incorrect & \xmark & \xmark & \checkmark & \checkmark \\ \hline - odd\# of bits flipped & Incorrect & Incorrect & In order & Incorrect & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline - even\# of bits flipped & Incorrect & Incorrect & In order & Incorrect & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline - Correct & Incorrect & Incorrect & Out of order & Correct & \xmark & \xmark & \checkmark & \checkmark \\ -\end{tabularx} +This is expressed in Table~\ref{tbl:testpairs}. In this table the first five +columns represent the different options for the partitions 2 to 6 of the above +enumeration. The last four columns are the different number segments as +described in the partition 1 of the above enumeration. These cells identify +individual test cases by a number. An \xmark in the cell indicates that this +test case can not be created as it is not possible with that number of segments +(eg. sending segments out of order when the number of segments is 1). + +\setcounter{TCC}{1} +\begin{table}[H] + \centering + \begin{tabular}{|l|l|l|l|l|l||l|l|l|l|} + \hline + & \multicolumn{9}{c|}{\textbf{Partition}}\\ + \hline + & 4 & 5 & 3 & 6 & 2 & 1a & 1b & 1c & 1d\\ + \hline\hline + \multirow{9}{*}{Instance} + & a & a & a & a & a & \doTCC & \doTCC & \doTCC & \doTCC\\ + & a & b & b & c & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & a & b & a & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & b & a & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & b & a & b & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & b & b & a & b & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\ + & b & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\ + & a & b & b & b & a & \xmark & \xmark & \doTCC & \doTCC\\ + \hline + \end{tabular} \caption{Combinations of test cases} -\label{table:testpairs} +\label{tbl:testpairs} \end{table} \subsection{Quality, completeness and coverage of tests} @@ -176,11 +234,18 @@ test suite can be clearly assessed. As always, $100\%$ completeness is not feasible, therefore test cases are carefully selected to cover the most interesting parts of the TCP specification -to ensure a test suite. +to ensure a complete but feasible test suite. + +To further increase the coverage of the test suites tests are randomized. The +tests which test the handling of \emph{bit errors}, changes in the \emph{packet +order} and \emph{dropped packets} randomize where they introduce an error. The +test suite runs these tests multiple times to increase the likelihood that they +discover a fault which is only present when an error occurs in a certain +position. To further decrease the number of tests needed test cases are divided into equivalence partitions and the combination of cases as described in -Table~\ref{table:testpairs} ensures that all partitions are +Table~\ref{tbl:testpairs} ensures that all partitions are covered and the number of individual tests is still feasible. @@ -190,9 +255,149 @@ covered and the number of individual tests is still feasible. % -\subsection{Test cases} +\subsection{Test suite} + +Before executing the test suite the test environment has to be initialized. + +\begin{enumerate} + \item Boot the vm using VirtualBox. + \item Setup iptables by executing \texttt{\# code/iptables.sh} + \item Navigate to the working directory by running + \texttt{\$ cd /home/student/tt2015} + \item Start the echo server by running + \texttt{\# cd code/server \&\& Java Main} +\end{enumerate} + +\subsubsection{Preflight checks} +The we do the preflight checks as defined in Table~\ref{tbl:preflight}. + +\subsubsection{Test Cases} +If the SUT passes the preflight checks the actual test cases can be executed. +Table~\ref{tbl:testcases} shows the expected results of each of the test cases +described in Table~\ref{tbl:testpairs}. + +\setcounter{TCC}{1} +\begin{table}[H] + \centering + \begin{tabular}{|l|p{.7\linewidth}|} + \hline + Test number & Expected results\\ + \hline\hline + \doTCC & An ACK\# of the send sequence number + 1.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ + $\vdots$ & \\ + \setcounter{TCC}{14} + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The segment is not attributed to the current connection + and therefore no ACK\# is received. \\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The segment is not attributed to the current connection + and therefore no ACK\# is received. \\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \end{tabular} +\caption{Expected results of test cases} +\label{tbl:testcases} +\end{table} + +% Bij Ramons afwezigheid +% Paul Vitero (linkerkant lange gang) +% verdieping Mercator + + +%\begin{longtable}{|p{.2\linewidth}|p{.8\linewidth}|} + %\hline + %Nr & 1 \\\hline + %Title & Single valid request with 1byte payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/1.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 2 \\\hline + %Title & Single valid request with 65495bytes payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/2.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 3 \\\hline + %Title & 5 valid requests with 1byte payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/3.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 4 \\\hline + %Title & 5 valid requests with 65495bytes payload. \\\hline + %Input & Generated packets with 65495bytes payload. \\\hline + %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/4.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 5 \\\hline + %Title & 5 valid requests with 1byte payload sent out of order. \\\hline + %Input & Generated packets with 1byte payload, two packets are swapped in position. \\\hline + %Expected output & All requests sent up to and including + %the swapped packet with the lowest sequence number, the remaining packets are dropped. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/5.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 6 \\\hline + %Title & Request with corrupted source port. \\\hline + %Input & Generated packets with 1byte payload, in these packets the source port number is increased by one. \\\hline + %Expected output & - \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% +% + %Nr & 6 \\\hline + %Title & Request with corrupted destination port. \\\hline + %Input & Generated packets with 1byte payload, in these packets the destination port number is increased by one. \\\hline + %Expected output & - \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% +%\end{longtable} -%\begin{table} %\begin{tabularx}{\linewidth}{| l | X|} %\hline %Nr & 1 \\\hline @@ -230,5 +435,3 @@ covered and the number of individual tests is still feasible. % \end{enumerate} \\\hline %\end{tabularx} %\end{table} - -\newpage