X-Git-Url: https://git.martlubbers.net/?a=blobdiff_plain;f=a2%2F1cases.tex;h=ce1031d4b15aaf262da4e490ce4abb19f502d70a;hb=bb19840764d2703e9c09f9d044072b51700d32ec;hp=78ae16890f1d9482693ecc115dd9a56a1c464f9f;hpb=29f7409053932da5c991fa7adfc1467e26b6d7bd;p=tt2015.git diff --git a/a2/1cases.tex b/a2/1cases.tex index 78ae168..ce1031d 100644 --- a/a2/1cases.tex +++ b/a2/1cases.tex @@ -35,7 +35,7 @@ user permissions. \hline\hline Check 3 & \multicolumn{2}{l|}{Initialize the testing environment..}\\ \hline - \multirow{5}{*}{Course of action} + \multirow{6}{*}{Course of action} & 1. & Boot the SUT as in \emph{Check 1}.\\ & 2. & Setup iptables by executing \texttt{\# code/iptables.sh}~\footnote{The IPTables script ensures @@ -43,9 +43,11 @@ user permissions. & 3. & Navigate to the working directory by running \texttt{\$ cd /home/student/tt2015}\\ & 4. & Compile the echo server by running - \texttt{\# cd code/server \&\& make \&\& cd -}\\ + \texttt{\# cd code/server \&\& make}\\ & 5. & Start the echo server by running \texttt{\# cd code/server \&\& java Main}\\ + & 6. & Generate all test cases by running + \texttt{\$ python code/client/gen.py}\\ \hline Passed & \multicolumn{2}{l|}{\textit{Yes/No}}\\ \hline\hline @@ -119,11 +121,11 @@ these partitions are given. \item Correct \item Incorrect \end{enumerate} - \item \emph{Packet order} + \item \emph{Segment order} \begin{enumerate} \item Correct \item Out of order - \item Missing packets + \item Missing Segments \end{enumerate} \end{enumerate} @@ -161,33 +163,37 @@ Partitions 2 to 6 are tested using pairwise testing to keep the number of test cases feasible. The pairs are then all *except some where it does not make sense to do so) tested with the different request sizes of partition 1. -This is expressed in Table~\ref{table:testpairs}. +This is expressed in Table~\ref{tbl:testpairs}. In this table the first five +columns represent the different options for the partitions 2 to 6 of the above +enumeration. The last four columns are the different number segments as +described in the partition 1 of the above enumeration. These cells identify +individual test cases by a number. An \xmark in the cell indicates that this +test case can not be created as it is not possible with that number of segments +(eg. sending segments out of order when the number of segments is 1). -\newcounter{TCC} \setcounter{TCC}{1} -\newcommand{\doTCC}{\theTCC \stepcounter{TCC}} \begin{table}[H] \centering - \begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|} + \begin{tabular}{|l|l|l|l|l|l||l|l|l|l|} \hline - & \multicolumn{10}{c|}{\textbf{Partition}}\\ + & \multicolumn{9}{c|}{\textbf{Partition}}\\ \hline - & \# & 4 & 5 & 3 & 6 & 2 & 1a & 1b & 1c & 1d\\ + & 4 & 5 & 3 & 6 & 2 & 1a & 1b & 1c & 1d\\ \hline\hline \multirow{9}{*}{Instance} - & 1 & a & a & a & a & a & \doTCC & \doTCC & \doTCC & \doTCC\\ - & 2 & a & b & b & c & b & \xmark & \xmark & \doTCC & \doTCC\\ - & 3 & c & a & b & a & b & \xmark & \xmark & \doTCC & \doTCC\\ - & 4 & c & b & a & c & a & \xmark & \xmark & \doTCC & \doTCC\\ - & 5 & b & a & b & c & a & \xmark & \xmark & \doTCC & \doTCC\\ - & 6 & b & b & a & b & b & \xmark & \xmark & \doTCC & \doTCC\\ - & 7 & c & b & b & a & b & \doTCC & \doTCC & \doTCC & \doTCC\\ - & 8 & b & b & b & a & b & \doTCC & \doTCC & \doTCC & \doTCC\\ - & 9 & a & b & b & b & a & \xmark & \xmark & \doTCC & \doTCC\\ + & a & a & a & a & a & \doTCC & \doTCC & \doTCC & \doTCC\\ + & a & b & b & c & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & a & b & a & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & b & a & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & b & a & b & c & a & \xmark & \xmark & \doTCC & \doTCC\\ + & b & b & a & b & b & \xmark & \xmark & \doTCC & \doTCC\\ + & c & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\ + & b & b & b & a & b & \xmark & \doTCC & \doTCC & \doTCC\\ + & a & b & b & b & a & \xmark & \xmark & \doTCC & \doTCC\\ \hline -\end{tabular} + \end{tabular} \caption{Combinations of test cases} -\label{table:testpairs} +\label{tbl:testpairs} \end{table} \subsection{Quality, completeness and coverage of tests} @@ -214,11 +220,18 @@ test suite can be clearly assessed. As always, $100\%$ completeness is not feasible, therefore test cases are carefully selected to cover the most interesting parts of the TCP specification -to ensure a test suite. +to ensure a complete but feasible test suite. + +To further increase the coverage of the test suites tests are randomized. The +tests which test the handling of \emph{bit errors}, changes in the \emph{packet +order} and \emph{dropped packets} randomize where they introduce an error. The +test suite runs these tests multiple times to increase the likelihood that they +discover a fault which is only present when an error occurs in a certain +position. To further decrease the number of tests needed test cases are divided into equivalence partitions and the combination of cases as described in -Table~\ref{table:testpairs} ensures that all partitions are +Table~\ref{tbl:testpairs} ensures that all partitions are covered and the number of individual tests is still feasible. @@ -228,92 +241,149 @@ covered and the number of individual tests is still feasible. % -\subsection{Test cases} +\subsection{Test suite} -Before every test case use the following steps to initialize the testing environment. +Before executing the test suite the test environment has to be initialized. \begin{enumerate} \item Boot the vm using VirtualBox. \item Setup iptables by executing \texttt{\# code/iptables.sh} - \item Navigate to the working directory by running \texttt{\$ cd /home/student/tt2015} - \item Start the echo server by running \texttt{\# cd code/server \&\& java Main} + \item Navigate to the working directory by running + \texttt{\$ cd /home/student/tt2015} + \item Start the echo server by running + \texttt{\# cd code/server \&\& Java Main} \end{enumerate} -\begin{longtable}{|p{.2\linewidth}|p{.8\linewidth}|} - \hline - Nr & 1 \\\hline - Title & Single valid request with 1byte payload. \\\hline - Input & Generated packets. \\\hline - Expected output & Packets echoed back by Echo-Server. \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/1.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - Nr & 2 \\\hline - Title & Single valid request with 65495bytes payload. \\\hline - Input & Generated packets. \\\hline - Expected output & Packets echoed back by Echo-Server. \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/2.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - Nr & 3 \\\hline - Title & 5 valid requests with 1byte payload. \\\hline - Input & Generated packets. \\\hline - Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/3.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - Nr & 4 \\\hline - Title & 5 valid requests with 65495bytes payload. \\\hline - Input & Generated packets with 65495bytes payload. \\\hline - Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/4.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - Nr & 5 \\\hline - Title & 5 valid requests with 1byte payload sent out of order. \\\hline - Input & Generated packets with 1byte payload, two packets are swapped in position. \\\hline - Expected output & All requests sent up to and including - the swapped packet with the lowest sequence number, the remaining packets are dropped. \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/5.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - Nr & 6 \\\hline - Title & Request with corrupted source port. \\\hline - Input & Generated packets with 1byte payload, in these packets the source port number is increased by one. \\\hline - Expected output & - \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - - - Nr & 6 \\\hline - Title & Request with corrupted destination port. \\\hline - Input & Generated packets with 1byte payload, in these packets the destination port number is increased by one. \\\hline - Expected output & - \\\hline - \multirow{2}{*}{Course of action} - & 1. Use the steps listed above in order to start the SUT. \\ - & 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline - Valid trace & Verify that the script prints 'Success'. \\\hline - \hline - -\end{longtable} +\subsubsection{Preflight checks} +The we do the preflight checks as defined in Table~\ref{tbl:preflight}. + +\subsubsection{Test Cases} +If the SUT passes the preflight checks the actual test cases can be executed. +Table~\ref{tbl:testcases} shows the expected results of each of the test cases +described in Table~\ref{table:testpairs}. + +\setcounter{TCC}{1} +\begin{table}[H] + \centering + \begin{tabular}{|l|p{.7\linewidth}|} + \hline + Test number & Expected results\\ + \hline\hline + \doTCC & An ACK\# of the send sequence number + 1.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & An ACK\# of the sequence number of the last send segment + the + size of the payload of that segment.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ + $\vdots$ & \\ + \setcounter{TCC}{14} + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The segment is not attributed to the current connection + and therefore no ACK\# is received. \\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The segment is not attributed to the current connection + and therefore no ACK\# is received. \\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \doTCC & The ACK\# for the SEQ\# of the first segments which is + corrupted is received for each consecutive segment send.\\ \hline + \hline + \end{tabular} +\caption{Expected results of test cases} +\label{table:preflightresults} +\end{table} + +% Bij Ramons afwezigheid +% Paul Vitero (linkerkant lange gang) +% verdieping Mercator + + +%\begin{longtable}{|p{.2\linewidth}|p{.8\linewidth}|} + %\hline + %Nr & 1 \\\hline + %Title & Single valid request with 1byte payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/1.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 2 \\\hline + %Title & Single valid request with 65495bytes payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/2.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 3 \\\hline + %Title & 5 valid requests with 1byte payload. \\\hline + %Input & Generated packets. \\\hline + %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/3.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 4 \\\hline + %Title & 5 valid requests with 65495bytes payload. \\\hline + %Input & Generated packets with 65495bytes payload. \\\hline + %Expected output & Packets echoed back by Echo-Server, in the same order as the client sent them. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/4.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 5 \\\hline + %Title & 5 valid requests with 1byte payload sent out of order. \\\hline + %Input & Generated packets with 1byte payload, two packets are swapped in position. \\\hline + %Expected output & All requests sent up to and including + %the swapped packet with the lowest sequence number, the remaining packets are dropped. \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/5.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% + %Nr & 6 \\\hline + %Title & Request with corrupted source port. \\\hline + %Input & Generated packets with 1byte payload, in these packets the source port number is increased by one. \\\hline + %Expected output & - \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% +% + %Nr & 6 \\\hline + %Title & Request with corrupted destination port. \\\hline + %Input & Generated packets with 1byte payload, in these packets the destination port number is increased by one. \\\hline + %Expected output & - \\\hline + %\multirow{2}{*}{Course of action} + %& 1. Use the steps listed above in order to start the SUT. \\ + %& 2. Execute the script by running \texttt{\# code/client/tests/6.py} \\\hline + %Valid trace & Verify that the script prints 'Success'. \\\hline + %\hline +% +%\end{longtable} %\begin{tabularx}{\linewidth}{| l | X|} %\hline