\section{Conclusion}
\begin{center}
	\textit{Is it possible to shorten the feedback loop for repairing and %
adding crawlers by making a system that can create, add and maintain crawlers %
for RSS feeds}
\end{center}

The short answer to the problem statement made in the introduction is yes. We
can shorten the loop for repairing and adding crawlers which our system. The
system can provide the necessary tools for a user with no particular
programming skills to generate crawlers and thus the number of interventions
where a programmer is needed is greatly reduced. Although we have solved the
problem we stated the results are not strictly positive. This is because a if
the problem space is not large the interest of solving the problem is also not
large, this basically means that there is not much data to apply the solution
on.

Although the research question is answered the underlying goal of the project
has not been completely achieved. The application is an intuitive system that
allows users to manage crawlers and for the specific domain, RSS feeds. By
doing that it does shorten the feedback loop but only for RSS feeds. In the
testing phase on real world data we stumbled on a small problem. Lack of RSS
feeds and misuse of RSS feeds leads to a domain that is significantly smaller
then first theorized and therefore the application solves only a very small
portion.

Lack of RSS feeds is a problem because a lot of entertainment venues have no
RSS feeds available for the public. Venues either using different techniques to
publish their data or do not publish their data at all via a structured source
besides their website. This shrinks the domain quite a lot. Taking pop music
venues as an example. In a certain province of the Netherlands we can find
about $25$ venues that have a website and only $3$ have a RSS feed.
Extrapolating this information combined with information from other regions we
can safely say that less then $10\%$ of the venues even has a RSS feed.

The second problem is misuse of RSS feeds. RSS feeds are very structured due to
their limitations on possible fields. We found that a lot of venues that are
using a RSS feed seem not to be content with the limitations and try to bypass
such limitations by misusing the protocol. A common misuse is to use the
publication date field to put the date of the actual event in. When loading
such a RSS feed into a general RSS feed reader the outcome is very strange
because a lot of events will have a publishing date in the future and therefore
messing up the order in your program. The misplacement of key information leads
to lack of key information in the expected fields and by that lower overall
extraction performance.

The second most occurring common misuse is to use HTML formatted text in the RSS
feeds text fields. The algorithm is designed to detect and extract information
via patterns in plain text and the performance on HTML is very bad compared to
plain text. A text field with HTML is almost useless to gather information
from because they usually include all kinds of information in other modalities
then text. Via a small study on a selection of RSS feeds($N=10$) we found that
about $50\%$ of the RSS feeds misuse the protocol in such a way that extraction
of data is almost impossible. This reduces the domain of good RSS feeds to less
then $5\%$ of the venues.

\section{Discussion \& Future Research}
\label{sec:discuss}
% low level stuff
The application we created does not apply any techniques on the extracted
data fields. The application is built only to extract and not to process the
labeled data fields with text. When we would combine the information about the
global structure and information about structure in a marked area we increase
performance in two ways. A higher levels of performance are reached due to the
structural information of marked areas. Hereby extra knowledge as extra
constraint while matching the data in marked areas. The second increase in
performance of the application is because the error detection happens more
quickly. Faster error detection is possible because when the match is correct
at a global level it can still contain wrong information at the lower marked
field level. Applying matching techniques on the marked fields afterwards can
generate feedback that could also be useful for the global level of data
extraction. 

% combine RSS HTML
Another use or improvement could be combining the forces of HTML and RSS. Some
specifically structured HTML sources could be converted into a tidy RSS feed
and still get processed by this application. In this way, with an extra
intermediate step, the extraction techniques can still be used. HTML sources
most likely have to be generated from a source by the venue because there has
to be a very consistent structure in the data. Websites with such great
structure are usually generated from a CMS. This will enlarge the domain for
the application significantly since almost all websites use CMS to publish
their data. 

% Re-use user interface
The interface of the program could also be re-used. When conversion between
HTML and RSS feeds is not possible but one has a technique to extract patterns
in a similar way then this application it is also possible to embed it in the
current application. Due to the modularity of the application extending the
application with other matching techniques is very easy.
