\centering
\includegraphics[width=\linewidth]{backend.eps}
\strut\\
- \caption{Backend overview}
+ \caption{Main module internals}
\end{figure}
-\section{Internals of the crawler generation module}
-Data marked by the user is in principle just raw html data that contains a
-table with for every RSS feed entry a table row. Within the html the code
-several markers are placed to make the parsing more easy and removing the need
-of an advanced HTML parser. When the user presses submit a http POST request is
-prepared to send all the gathered data to the backend to get processed. The
-sending does not happen asynchronously, so the user has to wait for the data to
-be processed. Because of the sychronous data transmission the user is notified
-immediatly when the crawler is succesfully added. The data preparation that
-makes the backend able to process it is all done on the user side using
-Javascript.
-
-When the backend receives the http POST data it saves all the raw data and the
-data from the information fields. The raw data is processed line by line to
-extract the entries that contain user markings. The entries containing user
-markings are stripped from all html while creating an data structure that
-contains the locations of the markings and the original text. All data is
-stored, for example also the entries without user data, but not all data is
-processed. The storage of, at first sight, useless data is done because later
-when the crawlers needs editing the old data can be used to update the crawler.
-
-When the entries are isolated and processed they are converted to node-lists. A
-node-list is a literal list of words where a word is interpreted in the
-broadest sense, meaning that a word can be a character, a single byte, a string
-or basically anything. These node-lists are in this case the original entry
-string where all the user markers are replaced by single nodes. As an example
-lets take the following entry and its corresponding node-list assuming that the
-user marked the time, title and date correctly. Markers are visualized by
-enclosing the name of the marker in angle brackets.
-\begin{flushleft}
- Entry: \texttt{19:00, 2014-11-12 - Foobar}\\
- Node-list: \texttt{['<time>', ',', ' ', '<date>', ' ', '-', ' ', '<title>']}
-\end{flushleft}
-
-The last step is when the entries with markers are then processed to build
-node-lists. Node-lists are basically lists of words that, when concatenated,
-form the original entry. A word isn't a word in the linguistic sense. A word
-can be one letter or a category. The node-list is generated by putting all the
-separate characters one by one in the list and when a user marking is
-encountered, this marking is translated to the category code and that code is
-then added as a word. The nodelists are then sent to the actual algorithm to be
-converted to a graph representation.
-
-\section{Minimizing DAWGs}
+\section{HTML data}
+Data marked by the user will be returned as a \texttt{POST} http request and is
+basically the information in the description fields accompanied by the raw html
+of the table with the patterns. For every RSS feed entry there is a row in the
+table and every marker is a \texttt{SPAN} element containing the color of the
+text. Within the HTML code of the table markers are placed to make parsing more
+easy and remove the need of an advanced HTML parser. The \texttt{POST} request
+that will be send when the user presses the submit button will not be sent
+asynchronously, the user has to wait for the processing to finish. Because of
+this the user will be immediately notified when the processing has finished
+successfully. The preparation of the data for the \texttt{POST} request is all
+done in Javascript on the user side, the processing of the data in the backend
+is all done server side.
+
+\section{Table rows}
+When the backend receives the HTTP POST data it saves all the table HTML data and the
+data from the description fields. The program first extracts the table rows
+using simple pattern matching on the added markers. The table rows are
+processed one at the time, the processing mainly consists of finding the
+\texttt{SPAN} elements, extract the color, remove the elements and save the
+plain text and the location and type of the marking. When the table rows are
+processed a datastructure with all the entries accompanied by their markers
+will go to the next step. All intermediate data will be saved for later use.
+
+\section{Node lists}
+Every entry from the datastructure is processed to convert them to node-lists.
+A node-list can be seen as a path graph of every character and marking. A path
+graph $G$ is defined as $G=(V,n_1,E,n_i)$ where $V=\{n_1, n_2, \cdots, n_{i-1}, n_i\}$
+and $E=\{(n_1, n_2), (n_2, n_3), ... (n_{i-1}, n_{i})\}$. A path graph is
+basically a graph that is a path of nodes where every node is connected to the
+next on and the last on being final. The transitions between two nodes is
+either a character or a marking. As an example we take the entry
+\texttt{19:00, 2014-11-12 - Foobar} and create the corresponding node-lists and
+make it visible in Figure~\ref{nodelistexample}. Characters are denoted with
+single quotes, spaces with an underscore and markers with angle brackets.
+\begin{figure}[H]
+ \label{nodelistexample}
+ \centering
+ \includegraphics[width=\linewidth]{nodelistexample.eps}
+ \caption{Node list example}
+\end{figure}
+
+
+\section{DAWGs}
+\subsection{Terminology}
+\textbf{Parent nodes} are nodes that have an arrow to the child.\\
+\textbf{Confluence nodes} are nodes that have multiple parents.
+
\subsection{Datastructure}
We represent the user generated patterns as DAWGs by converting the node-lists
-to DAWGS. Normally DAWGs have as edgelabels letters from an alphabet, in our
-case the DAWGs alphabet contains all letters, whitespace and punctuation but
-also the specified user markers which can be multiple characters in length.
-
-DAWGs are graphs but due to the constraints we can check if a match occurs by
-checking if a path exists that creates the word by concatenating all the edge
-labels. The first algorithm to
-generate DAWGs from words was proposed by Hopcroft et
-al\cite{Hopcroft1971}. It is an incremental approach in generating the graph.
-Meaning that entry by entry the graph will be expanded. Hopcrofts algorithm has
-the constraint of lexicographical ordering. Later on Daciuk et
-al.\cite{Daciuk2000} improved on the original algorithm and their algorithm is
-the algorithm we used to minimize our DAWGs. A minimal graph is a graph $G$ for
-which there is no graph $G'$ that has less paths and $\mathcal{L}(G)=\mathcal{L
-}(G')$ where $\mathcal{L}(G)$ is the set of all words present in the DAWG.
+to DAWGS. Normally DAWGs have single letters from an alphabet as edgelabel but
+in our implementation the DAWGs alphabet contains all letters, whitespace and
+punctuation but also the specified user markers which can be multiple
+characters in length.
+
+DAWGs are graphs but due to the constraints we can use the DAWG to check if a
+match occurs by checking if a path exists that creates the word by
+concatenating all the edge labels. The first algorithm to generate DAWGs from
+words was proposed by Hopcroft et al\cite{Hopcroft1971}. It is an incremental
+approach in generating the graph. Meaning that entry by entry the graph will
+be expanded. Hopcrofts algorithm has the constraint of lexicographical
+ordering. Later on Daciuk et al.\cite{Daciuk2000} improved on the original
+algorithm and their algorithm is the algorithm we used to minimize our DAWGs. A
+minimal graph is a graph $G$ for which there is no graph $G'$ that has less
+paths and $\mathcal{L}(G)=\mathcal{L }(G')$ where $\mathcal{L}(G)$ is the set
+of all words present in the DAWG.
\subsection{Algorithm}
The algorithm of building DAWGs is an iterative process that goes roughly in
added in a naive way. We just create a new node for every transition of
character and we mark the last node as final. From then on all words are added
using a stepwise approach.
-\begin{itemize}
+\begin{enumerate}
\item
Step one is finding the common prefix of the word in the graph and we chop
it of the word. When the suffix that remains is empty and the last state of
Finally if in the suffix propagating from the end to the beginning there
are equal states we merge the state from the suffix to the equal state in
the graph.
-\end{itemize}
+\end{enumerate}
Pseudocode for the algorithm can be found in Listing~\ref{pseudodawg}.
--- /dev/null
+%!PS-Adobe-3.0 EPSF-3.0
+%%Creator: graphviz version 2.36.0 (20140111.2315)
+%%Title: %3
+%%Pages: 1
+%%BoundingBox: 36 36 988 139
+%%EndComments
+save
+%%BeginProlog
+/DotDict 200 dict def
+DotDict begin
+
+/setupLatin1 {
+mark
+/EncodingVector 256 array def
+ EncodingVector 0
+
+ISOLatin1Encoding 0 255 getinterval putinterval
+EncodingVector 45 /hyphen put
+
+% Set up ISO Latin 1 character encoding
+/starnetISO {
+ dup dup findfont dup length dict begin
+ { 1 index /FID ne { def }{ pop pop } ifelse
+ } forall
+ /Encoding EncodingVector def
+ currentdict end definefont
+} def
+/Times-Roman starnetISO def
+/Times-Italic starnetISO def
+/Times-Bold starnetISO def
+/Times-BoldItalic starnetISO def
+/Helvetica starnetISO def
+/Helvetica-Oblique starnetISO def
+/Helvetica-Bold starnetISO def
+/Helvetica-BoldOblique starnetISO def
+/Courier starnetISO def
+/Courier-Oblique starnetISO def
+/Courier-Bold starnetISO def
+/Courier-BoldOblique starnetISO def
+cleartomark
+} bind def
+
+%%BeginResource: procset graphviz 0 0
+/coord-font-family /Times-Roman def
+/default-font-family /Times-Roman def
+/coordfont coord-font-family findfont 8 scalefont def
+
+/InvScaleFactor 1.0 def
+/set_scale {
+ dup 1 exch div /InvScaleFactor exch def
+ scale
+} bind def
+
+% styles
+/solid { [] 0 setdash } bind def
+/dashed { [9 InvScaleFactor mul dup ] 0 setdash } bind def
+/dotted { [1 InvScaleFactor mul 6 InvScaleFactor mul] 0 setdash } bind def
+/invis {/fill {newpath} def /stroke {newpath} def /show {pop newpath} def} bind def
+/bold { 2 setlinewidth } bind def
+/filled { } bind def
+/unfilled { } bind def
+/rounded { } bind def
+/diagonals { } bind def
+/tapered { } bind def
+
+% hooks for setting color
+/nodecolor { sethsbcolor } bind def
+/edgecolor { sethsbcolor } bind def
+/graphcolor { sethsbcolor } bind def
+/nopcolor {pop pop pop} bind def
+
+/beginpage { % i j npages
+ /npages exch def
+ /j exch def
+ /i exch def
+ /str 10 string def
+ npages 1 gt {
+ gsave
+ coordfont setfont
+ 0 0 moveto
+ (\() show i str cvs show (,) show j str cvs show (\)) show
+ grestore
+ } if
+} bind def
+
+/set_font {
+ findfont exch
+ scalefont setfont
+} def
+
+% draw text fitted to its expected width
+/alignedtext { % width text
+ /text exch def
+ /width exch def
+ gsave
+ width 0 gt {
+ [] 0 setdash
+ text stringwidth pop width exch sub text length div 0 text ashow
+ } if
+ grestore
+} def
+
+/boxprim { % xcorner ycorner xsize ysize
+ 4 2 roll
+ moveto
+ 2 copy
+ exch 0 rlineto
+ 0 exch rlineto
+ pop neg 0 rlineto
+ closepath
+} bind def
+
+/ellipse_path {
+ /ry exch def
+ /rx exch def
+ /y exch def
+ /x exch def
+ matrix currentmatrix
+ newpath
+ x y translate
+ rx ry scale
+ 0 0 1 0 360 arc
+ setmatrix
+} bind def
+
+/endpage { showpage } bind def
+/showpage { } def
+
+/layercolorseq
+ [ % layer color sequence - darkest to lightest
+ [0 0 0]
+ [.2 .8 .8]
+ [.4 .8 .8]
+ [.6 .8 .8]
+ [.8 .8 .8]
+ ]
+def
+
+/layerlen layercolorseq length def
+
+/setlayer {/maxlayer exch def /curlayer exch def
+ layercolorseq curlayer 1 sub layerlen mod get
+ aload pop sethsbcolor
+ /nodecolor {nopcolor} def
+ /edgecolor {nopcolor} def
+ /graphcolor {nopcolor} def
+} bind def
+
+/onlayer { curlayer ne {invis} if } def
+
+/onlayers {
+ /myupper exch def
+ /mylower exch def
+ curlayer mylower lt
+ curlayer myupper gt
+ or
+ {invis} if
+} def
+
+/curlayer 0 def
+
+%%EndResource
+%%EndProlog
+%%BeginSetup
+14 default-font-family set_font
+1 setmiterlimit
+% /arrowlength 10 def
+% /arrowwidth 5 def
+
+% make sure pdfmark is harmless for PS-interpreters other than Distiller
+/pdfmark where {pop} {userdict /pdfmark /cleartomark load put} ifelse
+% make '<<' and '>>' safe on PS Level 1 devices
+/languagelevel where {pop languagelevel}{1} ifelse
+2 lt {
+ userdict (<<) cvn ([) cvn load put
+ userdict (>>) cvn ([) cvn load put
+} if
+
+%%EndSetup
+setupLatin1
+%%Page: 1 1
+%%PageBoundingBox: 36 36 988 139
+%%PageOrientation: Portrait
+0 0 1 beginpage
+gsave
+36 36 952 103 boxprim clip newpath
+1 1 set_scale 0 rotate 40 40 translate
+% n0
+% n9
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+920 72 19.5 19.5 ellipse_path stroke
+1 setlinewidth
+0 0 0 nodecolor
+920 72 23.5 23.5 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+913 68.3 moveto 14 (n9) alignedtext
+grestore
+% n1
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+27 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+20 68.3 moveto 14 (n1) alignedtext
+grestore
+% n2
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+159 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+152 68.3 moveto 14 (n2) alignedtext
+grestore
+% n1->n2
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 54.31 72 moveto
+73.69 72 100.29 72 121.7 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 121.73 75.5 moveto
+131.73 72 lineto
+121.73 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 121.73 75.5 moveto
+131.73 72 lineto
+121.73 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+72 75.8 moveto 42 (<time>) alignedtext
+grestore
+% n3
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+259 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+252 68.3 moveto 14 (n3) alignedtext
+grestore
+% n2->n3
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 186 72 moveto
+196.97 72 209.92 72 221.79 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 221.87 75.5 moveto
+231.87 72 lineto
+221.87 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 221.87 75.5 moveto
+231.87 72 lineto
+221.87 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+204.5 75.8 moveto 9 (',') alignedtext
+grestore
+% n4
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+361 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+354 68.3 moveto 14 (n4) alignedtext
+grestore
+% n3->n4
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 286.01 72 moveto
+297.5 72 311.19 72 323.66 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 323.77 75.5 moveto
+333.77 72 lineto
+323.77 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 323.77 75.5 moveto
+333.77 72 lineto
+323.77 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+304 75.8 moveto 12 ('_') alignedtext
+grestore
+% n5
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+491 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+484 68.3 moveto 14 (n5) alignedtext
+grestore
+% n4->n5
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 388.21 72 moveto
+407.11 72 432.85 72 453.72 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 453.82 75.5 moveto
+463.82 72 lineto
+453.82 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 453.82 75.5 moveto
+463.82 72 lineto
+453.82 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+406 75.8 moveto 40 (<date>) alignedtext
+grestore
+% n6
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+593 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+586 68.3 moveto 14 (n6) alignedtext
+grestore
+% n5->n6
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 518.01 72 moveto
+529.5 72 543.19 72 555.66 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 555.77 75.5 moveto
+565.77 72 lineto
+555.77 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 555.77 75.5 moveto
+565.77 72 lineto
+555.77 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+536 75.8 moveto 12 ('_') alignedtext
+grestore
+% n7
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+693 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+686 68.3 moveto 14 (n7) alignedtext
+grestore
+% n6->n7
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 620 72 moveto
+630.97 72 643.92 72 655.79 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 655.87 75.5 moveto
+665.87 72 lineto
+655.87 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 655.87 75.5 moveto
+665.87 72 lineto
+655.87 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+638 75.8 moveto 10 ('-') alignedtext
+grestore
+% n8
+gsave
+1 setlinewidth
+0 0 0 nodecolor
+795 72 27 18 ellipse_path stroke
+0 0 0 nodecolor
+14 /Times-Roman set_font
+788 68.3 moveto 14 (n8) alignedtext
+grestore
+% n7->n8
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 720.01 72 moveto
+731.5 72 745.19 72 757.66 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 757.77 75.5 moveto
+767.77 72 lineto
+757.77 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 757.77 75.5 moveto
+767.77 72 lineto
+757.77 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+738 75.8 moveto 12 ('_') alignedtext
+grestore
+% n8->n9
+gsave
+1 setlinewidth
+0 0 0 edgecolor
+newpath 822.09 72 moveto
+840.74 72 865.98 72 886.02 72 curveto
+stroke
+0 0 0 edgecolor
+newpath 886.26 75.5 moveto
+896.26 72 lineto
+886.26 68.5 lineto
+closepath fill
+1 setlinewidth
+solid
+0 0 0 edgecolor
+newpath 886.26 75.5 moveto
+896.26 72 lineto
+886.26 68.5 lineto
+closepath stroke
+0 0 0 edgecolor
+14 /Times-Roman set_font
+840 75.8 moveto 38 (<title>) alignedtext
+grestore
+endpage
+showpage
+grestore
+%%PageTrailer
+%%EndPage: 1
+%%Trailer
+end
+restore
+%%EOF
repositories / bsc-thesis1415.git / commitdiff