\subsubsection{Edit/Remove crawler}
This component lets the user view the crawlers and remove the crawlers from the
-database. Removing the crawler is as simple as selecting it from the dropdown
-list and pressing the remove button. Editing the crawler is done in the same
-fashion but then pressing the edit button. The editing of the crawlers is
-basically the same as adding a new crawler other then that the previous pattern
-is already visible and can be adapted if for example the structure has changed.
+database. Doing one of these things with a crawler is as simple as selecting
+the crawler from the dropdown menu and selecting the operation from the
+other dropdown menu and pressing \textit{Submit}.
+Removing the crawler will remove the crawler completely from the crawler
+database and the crawler will be unrecoverable. Editing the crawler will open a
+similar screen as when adding the crawler. The details about that screen will
+be discussed in ~\ref{addcrawler}. The only difference is that the previous
+trained patterns are already made visible in the training interface and can
+thus be adapted to change the crawler for possible source changes for example.
\subsubsection{Add new crawler}
+\label{addcrawler}
+TODOOO
\subsubsection{Test crawler}
\subsection{Backend}
\subsubsection{Program description}
The backend consists of a main module and a set of libraries all written in
-\textit{Python}\cite{Python}. The main module can,
-and is, be embedded in an apache
-webserver\footnote{\url{https://httpd.apache.org/}} via the
-\textit{mod\_python} apache module\cite{Modpython}. The module
-\textit{mod\_python} allows the webserver to execute Python code in
-the webserver. We chose Python because of the rich set of standard libraries
-and solid cross platform capabilities. We chose Python 2 because it is still
-the default Python version on all major operating systems and stays supported
-until at least the year 2020 meaning that the program can function safe at
-least 5 full years. The application consists of a main Python module that is
-embedded in the webserver. Finally there are some libraries and there is a
-standalone program that does the periodic crawling.
+\textit{Python}\cite{Python}. The main module can, and is, be embedded in an
+apache webserver\cite{apache} via the \textit{mod\_python} apache
+module\cite{Modpython}. The module \textit{mod\_python} allows the webserver to
+execute Python code in the webserver. We chose Python because of the rich set
+of standard libraries and solid cross platform capabilities. We chose Python 2
+because it is still the default Python version on all major operating systems
+and stays supported until at least the year 2020 meaning that the program can
+function safe at least 5 full years. The application consists of a main Python
+module that is embedded in the webserver. Finally there are some libraries and
+there is a standalone program that does the periodic crawling.
\subsubsection{Main module}
The main module is the program that deals with the requests, controls the
The crawler also has a function to export the database to XML format. The XML
format is specified in an XSD\cite{Xsd} file for minimal ambiguity.
-An XSD file is a file that precisely describes the field the XML file uses.
-As almost every programming language contains an XML library most of the
-languages also contain a XSD library that allows the XML library to parse files
-according to the scheme and in that way the programmer knows exactly what to
-expect as XML fields. The XSD file for this program can be found in the
-appendices in Listing~\ref{scheme.xsd}.
+\subsubsection{XML \& XSD}
+XML is a file format that can describe data structures. XML can be accompanied
+by an XSD file that describes the format. An XSD file is in fact just another
+XML file that describes the format of a class of XML files. Because almost all
+programming languages have an XML parser built in it is a very versatile format
+that makes the importing to the database very easy. The most used languages
+also include XSD validation to detect XML errors, validity and completeness of
+XML files. This makes interfacing with the database and possible future
+programs very easy. The XSD scheme used for this programs output can be found
+in the appendices in Listing~\ref{scheme.xsd}. The XML output can be queried
+via a http interface that calls the crawler backend to crunch the latest
+crawled data into XML.
repositories / bsc-thesis1415.git / blobdiff