\scalebox{0.7}{
\digraph[]{graph111}{
rankdir=TB;
- node [shape="rectangle",fontsize=10,nodesep=0.7,ranksep=0.75,width=1]
- edge [weight=5.]
- i0 [label="Website"]
- i1 [label="Email"]
- i2 [label="Fax"]
- i3 [label="RSS/Atom"]
- p1 [label="Preproccessing"]
- p2 [label="Temporum: Postproccesing"]
- o1 [label="Database: Insertion"]
- o2 [label="TheAgenda"]
- o3 [label="BiosAgenda"]
- o4 [label="..."]
- p1, p2, o1 [width=5];
- i0 -> p1
- i1 -> p1
- i2 -> p1
- i3 -> p1
- p1 -> p2
- p2 -> o1
- o1 -> o2
- o1 -> o3
- o1 -> o4
+ node [shape="rectangle",fontsize=10,nodesep=0.7,ranksep=0.75,width=1];
+ edge [weight=5.];
+ i0 [label="Website"];
+ i1 [label="Email"];
+ i2 [label="Fax"];
+ i3 [label="RSS/Atom"];
+ p1 [label="Preproccessing"];
+ p2 [label="Temporum: Postproccesing"];
+ o1 [label="Database: Insertion"];
+ o2 [label="TheAgenda"];
+ o3 [label="BiosAgenda"];
+ o4 [label="..."];
+ node [width=5]; p1 p2 o1;
+ i0 -> p1;
+ i1 -> p1;
+ i2 -> p1;
+ i3 -> p1;
+ p1 -> p2;
+ p2 -> o1;
+ o1 -> o2;
+ o1 -> o3;
+ o1 -> o4;
}
}
\end{figure}
a very broad definition. In practice it can be describing the concert tour
name, theater show title, movie title, festival title and many more.
\item[Where]
- The \textit{Where} field is the location of the event. This is ofter
- omitted because the organization think it is obvious. This field can also
- include different sublocations. For example when a pop concert venue has
- their own building but in the summer they organize a festival in some park.
- This data is often assumed to be trivial and inherent but in practice this
- is not the case. In this example for an outsider only the name of the park
- is often not enough.
+ The \textit{Where} field is the location of the event. The location is often
+ omitted because the organization behind source presenting the information
+ think it is obvious. This field can also include different sublocations.
+ For example when a pop concert venue has their own building but in the
+ summer they organize a festival in some park. This data is often assumed
+ to be trivial and inherent but in practice this is not the case. In this
+ example for an outsider only the name of the park is often not enough.
\item[When]
The \textit{When} field is the time and date of the event. Hyperleap wants
to have at minimum the date, start time and end time. In the field end
always be available.
\paragraph{Crawler}
-The second steps is the crawling of the websites. This happens currently in a
-couple of methods. Some sites are very structured and a programmer creates a
-program that can visit the website systematically and extract that information.
-Some other sources are very hard to programmatically crawled and they are
-visited by employees. The programmed crawlers are always specifically created
-for one or a couple sources and when the source changes for example structure
-the programmer has to adapt the crawler which is costly. Information from the
-different crawlers then goes to the \textit{Temporum}.
+When the source has been determined and classified the next step is
+periodically crawling the source. At the moment the crawling happens using a
+couple of different methods. Some sites are very structured and a programmer
+creates a program that can visit the website systematically and extract that
+information. Some other sources are very hard to programmatically crawled and
+they are visited by employees. The programmed crawlers are always specifically
+created for one or a couple sources and when the source changes for example
+structure the programmer has to adapt the crawler which is costly. Information
+from the different crawlers then goes to the \textit{Temporum}.
\paragraph{Temporum}
The \textit{Temporum} is a big bin that contains raw data extracted from
different sources and has to be post processed to be suitable enough for the
-actual database. This processing encompasses several possible tasks. The first
-task is to check the validity of the entry. This is a very shallow test to
-check if the crawler is not malfunctioning and there is no nonsense in the
-data. Most of the data is not directly checked for validity, the data is
-skimmed for strange things but not every datapoint is checked. The second step
-is matching the entry to several objects. For example the entry has to be
-matched to a certain venue when its source is a ticket vendor who sells tickets
-for multiple venues. Another example is that the event is a pop concert and is
-part of a big tour. Many of these tasks are done alone by or with aid of a
-computer program. Almost no data is going straight through the
-\textit{Temporum} and this property makes the \textit{Temporum} a safety net
-for all the incoming data.
+actual database. This post-processing encompasses several possible tasks. The
+first task is to check the validity of the entry. This is not done for every
+entry but random samples are taken and these entries will be tested to see
+if the crawler is not malfunctioning and there is no nonsense in the
+data.
+The second step is matching the entry to several objects. Objects in the
+broadest sense can basically be anything, in the case of a theater show
+performance the matched object can be the theater show itself, describing the
+show for all locations. In case of a concert it can be that the object is a
+tour the band is doing. When the source is a ticket vendor the object is the
+venue where the event takes place since the source itself is not a venue.
+Many of these tasks are done by a human or with aid of a computer program.
+The \textit{Temporum} acts as a safety net because no data is going straigt
+through to the database. It is first checked, matched and validated.
\paragraph{Database \& Publication}
When the data is post processed it is entered in the final database. The
more.
\section{Goal \& Research question}
-The second step in the information flow is crawling the sources and apply the
-preprocessing. This is a expensive task because it requires a programmer to be
-hired because currently all crawlers are programmed for one, or a couple,
-specific sources. Because a big group of sources often changes this very
-expensive and has a long feedback loop. When a source changes it is first
-preprocessed in the old way, send to the \textit{Temporum} and checked by a
-human and matched. The human then notices the error in the data and contacts
-the programmer. The programmer then has to reprogram the specific crawler to
-the new structure. This feedback loop, shown in Figure~\ref{feedbackloop} can
-take days and can be the reason for gaps or faulty information in the database.
+Crawling the sources and applying the preprocessing is the most expensive task
+in the entire information flow because it requires a programmer to be hired.
+Only programmers currently can create crawlers and design preprocessing jobs
+because all current programs are specifically designed to do one thing, most of
+the time for one source. Because a big group of sources often changes this task
+becomes very expensive and has a long feedback loop. When a source changes the
+source is first preprocessed in the old way, send to the \textit{Temporum} and
+checked by a human and matched. The human then notices the error in the data
+and contacts the programmer. The programmer then has to reprogram the specific
+crawler to the new structure. This feedback loop, shown in
+Figure~\ref{feedbackloop} can take days and can be the reason for gaps and
+faulty information in the database.
\begin{figure}[H]
\caption{Feedback loop for malfunctioning crawlers}
\label{feedbackloop}
\scalebox{0.5}{
\digraph[]{graph112}{
rankdir=LR;
- node [shape="rectangle"]
- source [label="Source"]
- crawler [label="Crawler"]
- temporum [label="Temporum"]
- user [label="User"]
- programmer [label="Programmer"]
- database [label="Database"]
- source -> crawler
- crawler -> temporum
- temporum -> user
- user -> database
- user -> programmer [constraint=false,style="dotted"]
- user -> crawler [constraint=false,style="dashed"]
- programmer -> crawler [constraint=false,style="dotted"]
+ node [shape="rectangle"];
+ source [label="Source"];
+ crawler [label="Crawler"];
+ temporum [label="Temporum"];
+ employee [label="User"];
+ programmer [label="Programmer"];
+ database [label="Database"];
+ source -> crawler;
+ crawler -> temporum;
+ temporum -> user;
+ user -> database;
+ user -> programmer [constraint=false,style="dotted"];
+ user -> crawler [constraint=false,style="dashed"];
+ programmer -> crawler [constraint=false,style="dotted"];
}
}
\end{figure}
Figure~\ref{feedbackloop} removing the dotted arrows by dashed arrow.
For this project an application has been developed that can provide an
-interface to a crawler system that is able to crawl RSS\cite{Rss} and
-Atom\cite{Atom} publishing feeds. The interface also provides the user with
-point and click interfaces to create, modify, test and remove crawlers. The
-Hyperleap back end can, via this interface, generate XML feeds that contain the
-crawled data. For editing the structure and contents of the program a
-programmer is in theory also not necessary because all the things someone wants
-to change are located in a single file that is human readable. In practice it
-means that one person, not by definition a programmer, can be instructed to
-change the structure and this can also greatly reduce programmer intervention
-time.
+interface to a crawler generation system that is able to crawl RSS\cite{Rss}
+and Atom\cite{Atom} publishing feeds. The interface provides the user with
+point and click interfaces meaning that no computer science background is
+needed to use the interface, to create, modify, test and remove crawlers. The
+current Hyperleap backend system that handles the data can, via an interface,
+generate XML feeds that contain the crawled data. The structure of the
+application is very modular and generic and therefore it is easy to change
+things in the program without having to know a lot about the programming
+language used. This is because for example all visual things like buttons are
+defined in a human readable text file. In practice it means that one person,
+not by definition a programmer, can be instructed to change the structure and
+this can also greatly reduce programmer intervention time.
\section{RSS/Atom}
RSS/Atom feeds, from now on called RSS feeds, are publishing feeds in the XML
\xmlcode{exrss.xml}
\end{listing}
-The RSS feeds are mostly used by news sites to publish their articles, the feed
+RSS feeds are mostly used by news sites to publish their articles, the feed
then only contains the headlines and if the user that is reading the feed is
interested he can click the so called deeplink and he is then sent to the full
website containing the article. Users often use programs that bundle user
\centering
\scalebox{0.7}{
\digraph[]{graphexample}{
- rankdir=LR
- n1 -> n2 [dir="none"]
- n2 -> n3 [dir="none"]
- n2 -> n3 [dir="none"]
+ rankdir=LR;
+ n1 -> n2 [dir="none"];
+ n2 -> n3 [dir="none"];
+ n2 -> n3 [dir="none"];
}
}
\end{figure}
\centering
\scalebox{0.7}{
\digraph[]{dgexample}{
- rankdir=LR
- n1 -> n2
- n2 -> n1
+ rankdir=LR;
+ n1 -> n2;
+ n2 -> n1;
}
}
\end{figure}
\centering
\scalebox{0.7}{
\digraph[]{dagexample}{
- rankdir=LR
- n01 -> n02
- n02 -> n03
- n03 -> n01
- n11 -> n12
- n12 -> n13
- n12 -> n14
+ rankdir=LR;
+ n01 -> n02;
+ n02 -> n03;
+ n03 -> n01;
+ n11 -> n12;
+ n12 -> n13;
+ n12 -> n14;
}
}
\end{figure}
\scalebox{0.7}{
\digraph[]{graph21}{
rankdir=LR;
- n1,n2,n3,n4,n5 [shape="circle"];
+ node [shape="circle"]; n1 n2 n3 n4 n5;
n6 [shape="doublecircle"];
n1 -> n2 [label="a"];
n2 -> n3 [label="b"];
repositories / bsc-thesis1415.git / blobdiff