\section{Requirements}
\subsection{Introduction}
As almost every plan for an application starts with a set of requirements so
will this application. Requirements are a set of goals within different
categories that will define what the application has to be able to do and they
are traditionally defined at the start of the project and not expected to
change much. In the case of this project the requirements were a lot more
flexible because there was only one person doing the programming and there was
a weekly meeting to discuss the matters and most importantly discuss the
required changes. Because of this a lot of initial requirements are removed and
a some requirements were added in the process. The list below shows the
definitive requirements and also the suspended requirements.

There are two types of requirements, functional and non-functional
requirements. Functional requirements are requirements that describe a certain
function in the technical sense. Non-functional requirements describe a
property. Properties can be for example efficiency, portability or
compatibility. To make us able to refer to them later we give the
requirements unique codes. As for the definitive requirements a verbose
explanation is also provided.

\subsection{Functional requirements}
\subsubsection{Original functional requirements}
\begin{itemize}
	\item[I1:] The system should be able to crawl several source types.
		\begin{itemize}
			\item[I1a:] Fax/email.
			\item[I1b:] XML feeds.
			\item[I1c:] RSS feeds.
			\item[I1d:] Websites.
		\end{itemize}
	\item[I2:] Apply low level matching techniques on isolated data.
	\item[I3:] Insert data in the database.
	\item[I4:] The system should have a user interface to train crawlers
		that is usable by someone without a particular computer science
		background.
	\item[I5:] The system should be able to report to the employee when a
		source has been changed too much for successful crawling.
\end{itemize}

\subsubsection{Definitive functional requirements}
Requirement I2 is the one requirement that is dropped completely, this is due
to time constraints. The time limitation is partly because we chose to
implement certain other requirements like an interactive intuitive user
interface around the core of the pattern extraction program. Below are all
definitive requirements.
\begin{itemize}
	\item[F1:] The system should be able to crawl RSS feeds.

		This requirement is an adapted version of the compound
		requirements I1a-I1d. We limited the source types to crawl to
		strict RSS because of the time constraints of the project. Most
		sources require an entirely different strategy and therefore we
		could not easily combine them. An explanation why we chose RSS
		feeds can be found in Section~\ref{sec:whyrss}.

	\item[F2:] Export the data to a strict XML feed.

		This requirement is an adapted version of requirement I3, this
		is also done to limit the scope. We chose to not interact
		directly with the database or the \textit{Temporum}. The
		application however is able to output XML data that is
		formatted following a string XSD scheme so that it is easy to
		import the data in the database or \textit{Temporum} in a
		indirect way.
	\item[F3:] The system should have a user interface to create crawlers
		that is usabl for someone without a particular computer science
		background.

		This requirement is formed from I4. Initially the user
		interface for adding and training crawlers was done via a
		web interface that was user friendly and usable by someone
		without a particular computer science background as the
		requirement stated. However in the first prototypes the control
		center that could test, edit and remove crawlers was a command
		line application and thus not very usable for the general
		audience. This combined requirements asks for a single control
		center that can do all previously described tasks with an
		interface that is usable without prior knowledge in computer
		science.
	\item[F4:] Report to the user or maintainer when a source has been
		changed too much for successful crawling.

		This requirement was also present in the original requirements
		and has not changed. When the crawler fails to crawl a source,
		this can be due to any reason, a message is sent to the people
		using the program so that they can edit or remove the faulty
		crawler. Updating without the need of a programmer is essential
		in shortening the feedback loop explained in
		Figure~\ref{feedbackloop}.
\end{itemize}

\subsection{Non-functional requirements}
\subsubsection{Original functional requirements}
\begin{itemize}
	\item[O1:] Integrate in the existing system used by Hyperleap.
	\item[O2:] The system should work in a modular fashion, thus be able
		to, in the future, extend the program.
\end{itemize}

\subsubsection{Definitive functional requirements}
\begin{itemize}
	\item[N1:] Work in a modular fashion, thus be able to, in the future,
		extend the program.

		The modularity is very important so that the components can be
		easily extended and components can be added. Possible
		extensions are discussed in Section~\ref{sec:discuss}.
	\item[N2:] Operate standalone on a server.

		Non-functional requirement O1 is dropped because we want to
		keep the program as modular as possible and via an XML
		interface we still have a very intimate connection with the
		database without having to maintain a direct connection. The
		downside of an indirect connection instead of a direct
		connection is that the specification is much more rigid. If the
		system changes the specification the backend program should
		also change.
\end{itemize}

\section{Application overview}
The workflow of the application can be divided into several components or
steps. The overview of the application is visible in Figure~\ref{appoverview}.
The nodes are applications or processing steps and the arrows denote
information flow or movement between nodes.
\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{appoverview.pdf}
	\caption{Overview of the application\label{appoverview}}
\end{figure}

\subsection{Frontend}
\subsubsection{General description}
The frontend is a web interface that is connected to the backend system which
allows the user to interact with the backend. The frontend consists of a basic
graphical user interface which is shown in Figure~\ref{frontendfront}. As the
interface shows, there are three main components that the user can use.  There
is also a button for downloading the XML\@. The \textit{Get xml} button is a
quick shortcut to make the backend to generate XML\@. The button for grabbing the
XML data is only for diagnostic purposes located there. In the standard
workflow the XML button is not used. In the standard workflow the server
periodically calls the XML output option from the command line interface of the
backend to process it.

\begin{figure}[H]
	\includegraphics[width=\linewidth]{frontendfront.pdf}
	\caption{The landing page of the frontend\label{frontendfront}}
\end{figure}

\subsubsection{Repair/Remove crawler}
This component lets the user view the crawlers and remove the crawlers from the
crawler database. Doing one of these things with a crawler is as simple as
selecting the crawler from the dropdown menu and selecting the operation from
the other dropdown menu and pressing \textit{Submit}.

Removing the crawler will remove the crawler completely from the crawler
database and the crawler will be unrecoverable. Editing the crawler will open a
similar screen as when adding the crawler. The details about that screen will
be discussed in Section~\ref{addcrawler}. The only difference is that the
previous trained patterns are already made visible in the training interface
and can thus be adapted to change the crawler for possible source changes for
example.

\subsubsection{Add new crawler}
\label{addcrawler}
The addition or generation of crawlers is the key feature of the program and it
is the intelligent part of the system since it includes the graph optimization
algorithm to recognize user specified patterns in the new data. First, the user
must fill in the static form that is visible on top of the page. This for
example contains general information about the venue together with some crawler
specific values such as crawling frequency. After that the user can mark
certain points in the table as being of a category. Marking text is as easy as
selecting the text and pressing the according button. The text visible in the
table is a stripped down version of the original RSS feeds \texttt{title} and
\texttt{summary} fields. When the text is marked it will be highlighted in the
same color as the color of the button text. The entirety of the user interface
with a few sample markings is shown in Figure~\ref{frontendfront}. After the
marking of the categories the user can preview the data or submit. Previewing
will run the crawler on the RSS feed in memory and the user can revise the
patterns if necessary. Submitting will send the page to the backend to be
processed. The internals of what happens after submitting is explained in
detail in Figure~\ref{appinternals} together with the text.

\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{crawlerpattern.pdf}
	\caption{A view of the interface for specifying the pattern. Two %
entries are already marked.\label{frontendfront}}
\end{figure}

\subsubsection{Test crawler}
The test crawler component is a very simple non interactive component that
allows the user to verify if a crawler functions properly without having to
access the database via the command line utilities. Via a dropdown menu the
user selects the crawler and when submit is pressed the backend generates a
results page that shows a small log of the crawler, a summary of the results
and most importantly the results itself. In this way the user can see in a few
gazes if the crawler functions properly. Humans are very fast in detecting
patterns and therefore the error checking goes very fast. Because the log of
the crawl operation is shown this page can also be used for diagnostic
information about the backends crawling system. The logging is in depth and
also shows possible exceptions and is therefore also usable for the developers
to diagnose problems.

\subsection{Backend}
\subsubsection{Program description}
The backend consists of a main module and a set of libraries all written in
\textit{Python}\cite{Python}. The main module is embedded in an apache
HTTP-server\cite{apache} via the \textit{mod\_python} apache
module\cite{Modpython}. The module \textit{mod\_python} allows handling for
python code via HTTP and this allows us to integrate neatly with the
\textit{Python} libraries. We chose \textit{Python} because of the rich set of
standard libraries and solid cross platform capabilities. We chose specifically
for \textit{Python} 2 because it is still the default \textit{Python} version
on all major operating systems and stays supported until at least the year
$2020$. This means that the program can function at least for 5 full years. The
application consists of a main \textit{Python} module that is embedded in the
HTTP-server. Finally there are some libraries and there is a standalone program
that does the periodic crawling.

\subsubsection{Main module}
The main module is the program that deals with the requests, controls the
frontend, converts the data to patterns and sends the patterns to the crawler.
The module serves the frontend in a modular fashion. For example the buttons
and colors can be easily edited by a non programmer by just changing the
appropriate values in a text file. In this way even when conventions change the
program can still function without intervention of a programmer that needs to
adapt the source.

\subsubsection{Libraries}
The libraries are called by the main program and take care of all the hard
work. Basically the libraries are a group of python scripts that for example
minimize the graphs, transform the user data into machine readable data, export
the crawled data to XML and much more.

\subsubsection{Standalone crawler}
The crawler is a program, also written in Python, that is used by the main
module and technically is part of the libraries. The property in which the
crawler stands out is the fact that it also can run on its own. The crawler has
to run periodically by a server to literally crawl the websites. The main
module communicates with the crawler when it is queried for XML data, when a
new crawler is added or when data is edited. The crawler also offers a command
line interface that has the same functionality as the web interface of the
control center.

The crawler saves all the data in a database. The database is a simple
dictionary where all the entries are hashed so that the crawler knows which
ones are already present in the database and which ones are new. In this way
the crawler does not have to process all the old entries when they appear in
the feed. The RSS' GUID could also have been used but since it is an optional
value in the feed not every feed uses the GUID and therefore it is not reliable
to use it. The crawler also has a function to export the database to XML
format. The XML output format is specified in an XSD\cite{Xsd} file for minimal
ambiguity. 

\subsubsection{XML \& XSD}
XML is a file format that can describe data structures. XML can be accompanied
by an XSD file that describes the format. An XSD file is in fact just another
XML file that describes the format of XML files. Almost all programming
languages have an XML parser built in and therefore it is a very versatile
format that makes the eventual import to the database very easy. The most used
languages also include XSD validation to detect XML errors, validity and
completeness of XML files. This makes interfacing with the database and
possible future programs even more easy. The XSD scheme used for this
programs output can be found in the appendices in Algorithm~\ref{scheme.xsd}. The
XML output can be queried via the HTTP interface that calls the crawler backend
to crunch the latest crawled data into XML\@. It can also be acquired directly
from the crawlers command line interface.
