these tasks and therefore often serve incomplete information. Because of
the complexity of getting complete information there are not many websites
trying to bundle entertainment information into a complete and consistent
-database. Hyperleap\footnote{\url{http://hyperleap.nl}} tries to achieve goal
+database. Hyperleap\footnote{\url{http://hyperleap.nl}} tries to achieve goal
of serving complete and consistent information.
\section{Hyperleap \& Infotainment}
combination of the words \textit{information} and \textit{entertainment}. It
represents a combination of factual information and subjectual information
(entertainment) within a certain category or field. In the case of Hyperleap
-the category is the leisure industry, leisur industry encompasses all facets of
+the category is the leisure industry, leisure industry encompasses all facets of
entertainment going from cinemas, theaters, concerts to swimming pools, bridge
competitions and conferences. Within the entertainment industry factual
information includes, but is not limited to, information such as starting time,
information and trivia.
Hyperleap manages the largest database containing \textit{infotainment} about
-the leisure industry in the Netherlands and surroundings. The database contains
-over $10.000$ categorized events on average per week and their venue database
-contains over $54.000$ venues delivering the leisure activities ranging from
-theaters and music venues to petting zoos and fastfood restaurants. All the
-subjectual information is obtained or created by Hyperleap and all factual
-information is gathered from different sources, quality checked and therefore
-very reliable. Hyperleap is the only company in its kind that has such high
-quality information. The \textit{infotainment} is presented via several
-websites specialized per genre or category and some sites attract over
+the leisure industry focussed on the Netherlands and surrounding regions. The
+database contains over $10.000$ categorized events on average per week and
+their venue database contains over $54.000$ venues delivering the leisure
+activities ranging from theaters and music venues to petting zoos and fast food
+restaurants. All the subjectual information is obtained or created by Hyperleap
+and all factual information is gathered from different sources, quality checked
+and therefore very reliable. Hyperleap is the only company in its kind that has
+such high quality information. The \textit{infotainment} is presented via
+several websites specialized per genre or category and some sites attract over
$500.000$ visitors per month.
\section{Information flow}
The reason why Hyperleap is the only in its kind with the high quality data is
because Hyperleap spends a lot of time and resources on quality checking, cross
comparing and consistency checking before the data enters the database. To
-achieve this the data is inserted in the database in several different stages
-that are visualized in Figure~\ref{informationflow} as an information flow
-diagram using nodes as processing or publishing steps and arrows are
-information flow.
+achieve this, the data will go through several different stages before it
+enters the database. These stages are visualized in
+Figure~\ref{informationflow} as an information flow diagram. In this diagram
+the nodes are processing steps and the arrows denote information transfer or
+flow.
\begin{figure}[H]
\label{informationflow}
\centering
- \scalebox{0.7}{
- \digraph[]{graph111}{
- rankdir=TB;
- node [shape="rectangle",fontsize=10,nodesep=0.7,ranksep=0.75,width=1];
- edge [weight=5.];
- i0 [label="Website"];
- i1 [label="Email"];
- i2 [label="Fax"];
- i3 [label="RSS/Atom"];
- p1 [label="Preproccessing"];
- p2 [label="Temporum: Postproccesing"];
- o1 [label="Database: Insertion"];
- o2 [label="TheAgenda"];
- o3 [label="BiosAgenda"];
- o4 [label="..."];
- node [width=5]; p1 p2 o1;
- i0 -> p1;
- i1 -> p1;
- i2 -> p1;
- i3 -> p1;
- p1 -> p2;
- p2 -> o1;
- o1 -> o2;
- o1 -> o3;
- o1 -> o4;
- }
- }
+ \includegraphics[scale=0.7]{informationflow.eps}
+ \strut\\
+ \strut\\
\caption{Information flow Hyperleap database}
\end{figure}
-\paragraph{Sources}
-The information that enters the database has to be quality checked. There are
-several criteria the information and the source have to comply to before any of
-them can enter the database. For example the source has to be reliable,
-consistent and free by licence whereas the event entries have to have at least
-the following fields:
-\begin{itemize}
- \item[What]
- The \textit{What} field is the field that describes the content, content is
- a very broad definition. In practice it can be describing the concert tour
- name, theater show title, movie title, festival title and many more.
- \item[Where]
- The \textit{Where} field is the location of the event. The location is often
- omitted because the organization behind source presenting the information
- think it is obvious. This field can also include different sublocations.
- For example when a pop concert venue has their own building but in the
- summer they organize a festival in some park. This data is often assumed
- to be trivial and inherent but in practice this is not the case. In this
- example for an outsider only the name of the park is often not enough.
- \item[When]
- The \textit{When} field is the time and date of the event. Hyperleap wants
- to have at minimum the date, start time and end time. In the field end
- times are often omitted because they are not fixed or the organization
- think it is obvious.
-\end{itemize}
-
+\newpage
+\subsection*{Sources}
+The information that enters the database has to be quality checked and this
+checking starts at the source of the data. There are several criteria the
+information from the source have to comply to before it can enter the
+database. The prerequisites for a source are for example the fact that the
+source has to be reliable, consistent and free by licence. Event information
+from a source must have at least the following fields of information.\\
+\textbf{What:}\\
+The \textit{What} field is the field that describes the content, content is a
+very broad definition. In practice it can be describing the concert tour name,
+theater show title, movie title, festival title and many more.\\
+\textbf{Where:}\\
+The \textit{Where} field is the location of the event. The location is often
+omitted because the organization behind source presenting the information think
+it is obvious. This field can also include different sub locations. For
+example when a pop concert venue has their own building but in the summer they
+organize a festival in some park. This data is often assumed to be trivial and
+inherent but in practice this is not the case. In this example for an outsider
+only the name of the park is often not enough.\\
+\textbf{When:}\\
+The \textit{When} field is the time and date of the event. Hyperleap wants
+to have at minimum the date, start time and end time. In the field end
+times are often omitted because they are not fixed or the organization
+think it is obvious.\\
+\strut\\
Venues often present incomplete data entries that do not comply to the
-requirements explained before.
-
-The first step in the information flow is the source. Hyperleap processes
-different sources and every source has different characteristics. Sources can
-be modern sources like websites or social media but even today a lot of
+requirements explained before. Within the information flow categorizing and
+grading the source is the first step. Hyperleap processes different sources and
+source types and every source has different characteristics. Sources can be
+modern sources like websites or social media but even today a lot of
information arrives at Hyperleap via flyers, fax or email. As sources vary in
content structure sources also vary in reliability. For example the following
entry is an example of a very well structured and probably generated, and thus
probably also reliable, event description. The entry can originate for example
-from a RSS feed.
+from the title of an entry in a RSS feed. The example has a clear structure and
+almost all information required is available directly from the entry.
\begin{flushleft}
- \texttt{2015-05-20, 18:00-23:00 - \textit{Foobar} presenting their new CD in
- combination with a show. Location: small salon.}
+ \texttt{2015-05-20, 18:00-23:00 - \textit{Foobar} presenting their new%
+CD in combination with a show. Location: small salon.}
\end{flushleft}
-An example of a terrible entry could be for example the following text that
-could originate from a flyer or social media post.
+An example of a terrible item could be for example the following text that
+could originate from a flyer or social media post. This example lacks a
+precise date, time and location and is therefore hard for people to grasp at
+first, let alone for a machine. When someone wants to get the full information
+he has to tap in different resources which might not always be available.
\begin{flushleft}
- \texttt{\textit{Foobar} playing to celebrate their CD release in the park
- tomorrow evening.}
+ \texttt{\textit{Foobar} playing to celebrate their CD release in the%
+park tomorrow evening.}
\end{flushleft}
-This example lacks a precise date, time and location and is therefore hard for
-people to grasp at first, let alone for a machine. When someone wants to get
-the full information he has to tap in different resources which might not
-always be available.
-
-\paragraph{Crawler}
+\subsection*{Crawler}
When the source has been determined and classified the next step is
-periodically crawling the source. At the moment the crawling happens using a
-couple of different methods. Some sites are very structured and a programmer
-creates a program that can visit the website systematically and extract that
-information. Some other sources are very hard to programmatically crawled and
-they are visited by employees. The programmed crawlers are always specifically
-created for one or a couple sources and when the source changes for example
-structure the programmer has to adapt the crawler which is costly. Information
-from the different crawlers then goes to the \textit{Temporum}.
-
-\paragraph{Temporum}
+periodically crawling the source. At the moment the crawling happens using two
+main methods.\\
+\textbf{Manual crawling:} Manual crawling is basically letting an employee
+access the source and put the information directly in the database. This often
+happens with non digital sources and with very sudden events or event changes
+such as surprise concerts or event cancellation.\\
+\textbf{Automatic crawling:} Some sites are very structured and a programmer
+can create a program that can visit the website systematically and
+automatically to extract all the new information. Not all digital sources are
+suitable to be crawled automatically and will still need manual crawling. The
+programmed crawlers are always specifically created for one or a couple sources
+and when the source changes for example structure the programmer has to adapt
+the crawler which is costly. Information from the all the crawlers goes first
+to the \textit{Temporum}.
+
+\subsection*{Temporum}
The \textit{Temporum} is a big bin that contains raw data extracted from
different sources and has to be post processed to be suitable enough for the
actual database. This post-processing encompasses several possible tasks. The
first task is to check the validity of the entry. This is not done for every
entry but random samples are taken and these entries will be tested to see
if the crawler is not malfunctioning and there is no nonsense in the
-data.
+data.
+
The second step is matching the entry to several objects. Objects in the
broadest sense can basically be anything, in the case of a theater show
performance the matched object can be the theater show itself, describing the
tour the band is doing. When the source is a ticket vendor the object is the
venue where the event takes place since the source itself is not a venue.
Many of these tasks are done by a human or with aid of a computer program.
-The \textit{Temporum} acts as a safety net because no data is going straigt
+The \textit{Temporum} acts as a safety net because no data is going straight
through to the database. It is first checked, matched and validated.
-\paragraph{Database \& Publication}
+\subsection*{Database \& Publication}
When the data is post processed it is entered in the final database. The
database contains all the events that happened in the past and all the events
that are going to happen in the future. The database also contains information
\begin{figure}[H]
\label{feedbackloop}
\centering
- \scalebox{0.5}{
- \digraph[]{graph112}{
- rankdir=LR;
- node [shape="rectangle"];
- source [label="Source"];
- crawler [label="Crawler"];
- temporum [label="Temporum"];
- employee [label="User"];
- programmer [label="Programmer"];
- database [label="Database"];
- source -> crawler;
- crawler -> temporum;
- temporum -> user;
- user -> database;
- user -> programmer [constraint=false,style="dotted"];
- user -> crawler [constraint=false,style="dashed"];
- programmer -> crawler [constraint=false,style="dotted"];
- }
- }
+ \includegraphics[scale=0.5]{feedbackloop.eps}
+ \strut\\\strut\\
\caption{Feedback loop for malfunctioning crawlers}
\end{figure}
interface to a crawler generation system that is able to crawl RSS\cite{Rss}
and Atom\cite{Atom} publishing feeds. The interface provides the user with
point and click interfaces meaning that no computer science background is
-needed to use the interface, to create, modify, test and remove crawlers. The
+needed to use the interface, to create, modify, test and remove crawlers. The
current Hyperleap backend system that handles the data can, via an interface,
generate XML feeds that contain the crawled data. The structure of the
application is very modular and generic and therefore it is easy to change
things in the program without having to know a lot about the programming
language used. This is because for example all visual things like buttons are
-defined in a human readable text file. In practice it means that one person,
+defined in a human readable text file. In practice it means that one person,
not by definition a programmer, can be instructed to change the structure and
this can also greatly reduce programmer intervention time.
The actual problem statement then becomes:
\begin{center}
- \textit{Is it possible to shorten the feedback loop for repairing and adding
- crawlers by making a system that can create, add and maintain crawlers for
- RSS feeds}
+ \textit{Is it possible to shorten the feedback loop for repairing and%
+adding crawlers by making a system that can create, add and maintain crawlers%
+for RSS feeds}
\end{center}
\section{RSS/Atom}
\section{Why RSS?}
\label{sec:whyrss}
We described a lot of different source formats like HTML, fax/email, RSS and
-XML. Because of the limited scope of the project and the time planned for it
+XML. Because of the limited scope of the project and the time planned for it
we had to remove some of the input formats because they all require different
-techniques. For example when the input source is in HTML format, most probably
+techniques. For example when the input source is in HTML format, most probably
a website, then there can be a great deal of information extraction be
automated using the structural information which is characteristic for HTML.
For fax/email however there is almost no structural information and most of the
fields with vertical bars, commas, whitespace and more non text characters.
These field separators and keywords can be hard for a computer to detect but
people are very good in detecting these. With one look they can identify the
-characters and keywords and build a pattern in their head. Another reason we
+characters and keywords and build a pattern in their head. Another reason we
chose RSS is their temporal consistency, RSS feeds are almost always generated
and because of that the structure of the entries is very unlikely to change.
Basically the RSS feeds only change structure when the CMS that generates it
$V$ is a set of named nodes and $E$ is the set of edges defined by $E\subseteq
V\times V$. An edge $e\in E$ can be seen as $(v1, v2)$ where $v1,v2\in V$ and
can be interpreted as in a figure as an arrow between node $v1$ and node $v2$.
-Multiple connections between two nodes are possible. For example the graph
+Multiple connections between two nodes are possible. For example the graph
visualized in Figure~\ref{graphexample} can be mathematically described as:
$$G=(\{n1, n2, n3, n4\}, \{(n1, n2), (n2, n1), (n2, n3), (n3, n4), (n1, n4)\}$$
In the example graph in Figure~\ref{exampledawg} the final nodes are visualized
with a double circle as node shape. In this example it is purely cosmetic
because $n6$ is a final node anyways because there are no arrows leading out.
-But this does not need to be the case, for example in $G=(\{n1, n2, n3\},
+But this does not need to be the case, for example in $G=(\{n1, n2, n3\},
\{(n1, n2), (n2, n3)\}, \{n2, n3\})$ there is a distinct use for the final node
marking. Graph $G$ accepts the words \textit{a,ab} and to simplify the graph
node $n2$ and $n3$ are final. Finally $v_0$ describes the initial node, this is
repositories / bsc-thesis1415.git / blobdiff