Sweeping through the archg

[msc-thesis1617.git] / results.arch.tex

The goal of the total system is to facilitate an ecosystem in which an
\gls{iTasks}-system can add, change and remove devices at runtime. Moreover,
the \gls{iTasks}-system can send \glspl{mTask} --- compiled at runtime to
bytecode by the \gls{mTask}-view --- to the device. The device runs an
interpreter which can execute the \gls{Task}'s bytecode. Device profiles should
be persistent during reboots of the \gls{iTasks}-system. The methods of
interacting with \glspl{mTask} should be analogous to interacting with
\gls{iTasks}-\glspl{Task}. Meaning that programmers can access the \glspl{SDS}
made for a device in the same way as a regular \gls{SDS} and they can execute
\glspl{mTask} as if it where a normal \gls{iTasks}-\gls{Task}.

The following terms will be used throughout the following chapter:
\begin{itemize}
        \item Device, Client

                This denotes the actual device connected to the system. This can be a
                real device such as a microcontroller but it can also just be a program
                on the same machine as the server functioning as a client.
        \item Server, \gls{iTasks}-System

                This is the actual executable serving the \gls{iTasks} application. The
                system contains \glspl{Task} taking care of the communication with the
                clients.
        \item System

                The system describes the complete ecosystem, containing both the server
                and the clients including the communication between them.
        \item Engine

                The runtime system of the client is called the engine. This program
                handles communicating with the server and runs the interpreter for the
                \glspl{Task} on the client.
\end{itemize}

\section{Devices}
A device is suitable for the system if it can run the engine.
The engine is compiled from one codebase and devices implement (part of) the
device specific interface. The shared codebase only uses standard \gls{C} and
no special libraries or tricks are used. Therefore, the code is compilable for
almost any device or system. Note that it is not needed to implement a full
interface. The full interface --- excluding the device specific settings --- is
listed in Appendix~\ref{app:device-interface}.  The interface works in a
similar fashion as the \gls{EDSL}. Devices do not have to implement all
functionality, this is analogous to the fact that views do not have to
implement all type classes in the \gls{EDSL}.  When the device connects for the
first time with a server the specifications of what is implemented is
communicated.

At the time of writing the following device families are supported and can run
the device software.
\begin{itemize}
        \item \texttt{POSIX} compatible systems connected via \gls{TCP}.

                This includes systems running \emph{Linux} and \emph{MacOS}.
        \item \texttt{STM32} family microcontrollers supported by \texttt{ChibiOS}
                connected via serial communication.

                This is tested in particular on the \texttt{STM32f7x} series \gls{ARM}
                development board.
        \item Microcontrollers who are programmable in the \gls{Arduino} \gls{IDE}
                connected via serial communication or via \gls{TCP} over WiFi.

                This does not only include \gls{Arduino} compatible boards but also
                other boards capable of running \gls{Arduino} code. A port of the
                client has been made for the \texttt{ESP8266} powered \emph{NodeMCU}
                that is connected via \gls{TCP} over WiFi. A port also has been made
                for the regular \gls{Arduino} \emph{UNO} board which only boasts a
                meager \emph{2K} of \emph{RAM}.  The stack size and storage for such
                small amount of \emph{RAM} have to be smaller than default but it still
                suitable to hold a hand full of \glspl{Task}.
\end{itemize}

\subsection{Client}
\subsubsection*{Engine}
The client is in a constant loop listening for input and waiting to execute
\gls{Task}. The pseudocode for this is shown in Algorithm~\ref{alg:client}.

\todo{make algorithm}
\begin{algorithm}[H]
        \KwData{\textbf{stack} stack, \textbf{time} $t, t_p$}

        $t\leftarrow\text{now}()$\;
        \Begin{
                \While{true}{
                        $t_p\leftarrow t$\;
                        $t\leftarrow\text{now}()$\;
                        \If{notEmpty$($queue$)$}{
                                $task\leftarrow \text{queue.pop}()$\;
                                $task$.wait $\leftarrow task$.wait $-(t-t_p)$\;
                                \eIf{$task.wait>t_0$}{
                                        queue.append$(task)$\;
                                }{
                                        run\_task$(task)$\;
                                }
                        }
                }
        }
        \caption{Engine pseudocode}\label{alg:client}
\end{algorithm}

\subsubsection*{Storage}
\glspl{Task} and \glspl{SDS} are stored on the client in one big memory space
that is fully allocated at the start of the program. The space could also have
been dynamically allocated but that would require using the heap which is
unwanted in small memory environments. \Glspl{Task} grow from the bottom up
and \glspl{SDS} grow from the top down. When a \gls{Task} or \gls{SDS} is
removed, all \glspl{Task} are relocated in the memory space to not leave
holes. Both \glspl{Task} and \glspl{SDS} are stored as structs and helper
functions are available to loop through them without having to fiddle in the
memory space. The instance for \glspl{Task} and \glspl{SDS} are shown in
Listing~\ref{lst:structs} accompanied by the helper functions for \glspl{Task}.
\Glspl{Task} consist the length, interval, last run time, id and the bytecode.
\Glspl{SDS} consist just of an id, value and type. The pointer to the bytecode
of the \gls{Task} always points to the location in the memory space.

\begin{lstlisting}[language=C,label={lst:structs},%
        caption={The data type storing the \glspl{Task}}]
struct task {
        uint16_t tasklength;
        uint16_t interval;
        unsigned long lastrun;
        uint8_t taskid;
        uint8_t *bc;
};

struct task *task_head(void);
struct task *task_next(struct task *t);

struct sds {
        int id;
        int value;
        char type;
};

struct sds *sds_head(void);
struct sds *sds_next(struct sds *s);
\end{lstlisting}

\subsubsection*{Interpretation}
Execution of a \gls{Task} always start with prepared the stack and the program
counter and stack pointer are set to zero and the bottom respectively. When
finished, the interpreter executes one step at the time while the program
counter is smaller than the program length. The code for this is listed in
Listing~\ref{lst:interpr}. One execution step is basically a big switch
statement going over all possible bytecode instructions. Some instructions are
detailed upon in the listing. The \CI{BCPush} instruction is a little more
complicated in the real code because some decoding will take place as not all
\CI{BCValue}'s are of the same length and are encoded.

\begin{lstlisting}[language=C,label={lst:interpr},%
        caption={Rough code outline for interpretation}]
#define f16(p) program[pc]*265+program[pc+1]

void run_task(struct task *t){
        uint8_t *program = t->bc;
        int plen = t->tasklength;
        int pc = 0;
        int sp = 0;
        while(pc < plen){
                switch(program[pc++]){
                case BCNOP:
                        break;
                case BCPUSH:
                        stack[sp++] = pc++ //Simplified
                        break;
                case BCPOP:
                        sp--;
                        break;
                case BCSDSSTORE:
                        sds_store(f16(pc), stack[--sp]);
                        pc+=2;
                        break;
                // ...
                case BCADD: trace("add");
                        stack[sp-2] = stack[sp-2] + stack[sp-1];
                        sp -= 1;
                        break;
                // ...
                case BCJMPT: trace("jmpt to %d", program[pc]);
                        pc = stack[--sp] ? program[pc]-1 : pc+1;
                        break;
}
\end{lstlisting}

\subsection{Specification}
The server stores a description for every device available in a record type
which are stored in a \gls{SDS}. From the macro settings in
the interface file, a profile is created for the device that describes the
specification. When a connection between the server and a client is established
the server will send a request for specification. The client will serialize his
specification and send it to the server so that the server knows what the
client is capable of. The exact specification is shown in
Listing~\ref{lst:devicespec} and stores the peripheral availability, the memory
available for storing \glspl{Task} and \glspl{SDS} and the size of the stack.

\begin{lstlisting}[label={lst:devicespec},
        caption={Device specification for \glspl{mTask}}]
:: MTaskDeviceSpec =
        { haveLed     :: Bool
        , haveLcd     :: Bool
        , have...
        , bytesMemory :: Int
        , stackSize   :: Int
        , aPins       :: Int
        , dPins       :: Int
        }
\end{lstlisting}

\subsection{Device Storage}
All devices available in the system are stored in a big \gls{SDS} that contains
a list of \CI{MTaskDevice}s. The exact specification is defined as in
Listing~\ref{lst:mtaskdevice} accompanied with the used classes and types.

The \CI{deviceResource} component of the record must implement the
\CI{MTaskDuplex} interface that provides a function that launches a \gls{Task}
used for synchronizing the channels.  The \CI{deviceTask} stores the
\gls{Task}-id for this \gls{Task} when active so that it can be checked upon.
This top-level task has the duty to report exceptions and errors as they are
thrown by setting the \CI{deviceError} field. All communication goes via these
channels. If the system wants to send a message to the device, it just puts it
in the channels.  Messages sent from the client to the server are also placed
in there. In the case of the \gls{TCP} device type, the \gls{Task} is just a
simple wrapper around the existing \CI{tcpconnect} function in \gls{iTasks}. In
case of a device connected by a serial connection, it uses the newly developed
serial port library of \gls{Clean}\footnote{\url{%
https://gitlab.science.ru.nl/mlubbers/CleanSerial}}.

Besides all the communication information, the record also keeps track of the
\glspl{Task} currently on the device, the compiler state (see
Section~\ref{sec:compiler}) and the according \glspl{SDS}. Finally, it stores
the specification of the device that is received when connecting. All of this
is given in Listing~\ref{lst:mtaskdevice}. The definitions of the message
format are explained in the following section. Specialized shares are available
per device. The internal mechanism for this is given in
Chapter~\ref{chp:itasksint}.

\begin{lstlisting}[caption={Device type},label={lst:mtaskdevice}]
deviceStoreNP :: Shared [MTaskDevice]
deviceShare :: MTaskDevice -> Shared MTaskDevice

:: Channels :== ([MTaskMSGRecv], [MTaskMSGSend], Bool)
:: BCState = ... // Compiler state, explained in later sections
:: MTaskResource 
        = TCPDevice TCPSettings
        | SerialDevice TTYSettings
:: MTaskDevice =
                { deviceTask :: Maybe TaskId
                , deviceError :: Maybe String
                , deviceChannels :: String
                , deviceName :: String
                , deviceState :: BCState
                , deviceTasks :: [MTaskTask]
                , deviceData :: MTaskResource
                , deviceSpec :: Maybe MTaskDeviceSpec
                , deviceShares :: [MTaskShare]
        }

channels :: MTaskDevice -> Shared Channels

class MTaskDuplex a where
        synFun :: a (Shared Channels) -> Task ()
\end{lstlisting}

\section{iTasks}
The server part of the system is written in \gls{iTasks}. Functions for
managing devices, \glspl{Task} and \glspl{SDS} have been created to support the
functionality. An interactive application has been created that allows an
interactive management console for the \gls{mTask} system. This interface
provides functionality to list \glspl{SDS}, add \glspl{Task}, remove
\glspl{Task}, administrate devices and view the state of the system.

\subsection{Integration}
When the system starts up the devices from the previous execution still
residing in the \gls{SDS} must be cleaned up. It might be the case that they
contain \glspl{Task}, \glspl{SDS} or errors that are no longer applicable in
this run. A user or programmer can later choose to reconnect to some devices.

\begin{lstlisting}[caption={Starting up the devices},%
        label={lst:startupdevs}]
startupDevices :: Task [MTaskDevice]
startupDevices = upd (map reset) deviceStoreNP
        where reset d = {d & deviceTask=Nothing, deviceTasks=[], deviceError=Nothing}
\end{lstlisting}

An image of the management interface is shown in Figure~\ref{lst:manage}.
The system management is done by a single \gls{Task} called \CI{mTaskManager}.
To manage the system, a couple of different functionalities are needed and
are launched. The left sidebar of the interface shows the list of example
\glspl{Task} that are present in the system. When clicking a \gls{Task}, a
dialog opens in which you can select the device to send the \gls{Task} to. The
dialog might contain user specified variables. All example \glspl{mTask} are of
the type \CI{Task (Main (ByteCode () Stmt))} and can thus ask for user input
first.

The bottom panel shows the device information. In this panel, the devices can
be created and modified. Moreover, this panel allows the user to reconnect with
a device after a restart of the server application.

\begin{figure}[H]
        \centering
        \includegraphics[width=\linewidth]{manage}
        \caption{The device management interface}\label{lst:manage}
\end{figure}

\subsection{Shares}
The architecture of the system needs to keep track of the \glspl{SDS} stored on
the client. \glspl{SDS} can be stored on only one device at the same time.
that also stores the of devices. This means that if a \gls{SDS} updates,
everyone watching it will be notified. This would result in to a lot of
notifications that are not ment to be for the listener. Moreover, when a client
updates the \gls{SDS} this is processed by the connection handler and results
in an update of the real \gls{SDS}.
Finally, the \gls{SDS} of a client must be synchronized with the actual device.
There are several ways of tackling this problem each with their own pros and
cons and their own level of abstraction.

\begin{itemize}
        \item Instantiate an actual \gls{iTasks}-\gls{SDS} for every \gls{SDS} used
                in a client.

        \item Instantiate a \gls{iTasks}-\gls{SDS} for every device that stores all
                their \glspl{SDS}.

        \item Use only one \gls{iTasks}-\gls{SDS} for all devices.
\end{itemize}

\begin{lstlisting}[label={lst:actualdev},%
        caption={Real types for the device \gls{SDS}}]
deviceStoreNP :: Shared [MTaskDevice]
deviceStore :: RWShared (Maybe (MTaskDevice, Int)) [MTaskDevice] [MTaskDevice]
\end{lstlisting}

\subsection{Parametric Lenses}
The type of the parametric lens is \CI{Maybe (MTaskDevice, Int)}. The \gls{SDS}
can be responsible for the entire list of devices, from now on global.
Moreover, the \gls{SDS} can focus on a single device, from now on local. A
local \gls{SDS} can also specifically focus on a single \gls{SDS} on a single
device, from now on called local-share.

\paragraph{Global \glspl{SDS}: }
Accessing the global \gls{SDS} is just a matter of focussing the
\CI{deviceStore} with the \CI{Nothing} parameter. The signature for
\CI{deviceStore} was given in Chapter~\ref{chp:arch}. The actual implementation
is as in Listing~\ref{lst:global}

\begin{lstlisting}[label={lst:shareimpl},%
        caption={Base share implementation}]
deviceStoreNP :: RWShared (Maybe (MTaskDevice, Int)) [MTaskDevice] [MTaskDevice]
deviceStoreNP = sdsFocus Nothing deviceStore
\end{lstlisting}




\paragraph{Local \glspl{SDS}: }
\paragraph{Local-share specific \glspl{SDS}: }

The implementation for the share is shown in Listing~\ref{lst:shareimpl}. The
\CI{realDeviceStore} \gls{SDS} is not exported through the header files. This
\gls{SDS} contains the actual \gls{SDS} that writes to disk or memory.
\CI{Int} is the identifier of the \gls{SDS}. The \gls{iTasks} way of applying
lenses is through the \CI{sdsFocus} function and through the \CI{sdsLens}
functions. \CI{sdsFocus} allows the programmer to fix the parameter.
\CI{sdsLens} is basically a \CI{mapReadWrite} that has access to the parameter.
This allows the programmer to create filters and lenses. Both of the methods
are not good enough for the device \gls{SDS} because they do not achieve the
writing to the actual device. Writing to a device requires being able to write
to \glspl{SDS}. To solve this problem, a real base \gls{SDS} is created. All
the details are visible in Listing~\ref{lst:shareimpl}.

\section{Communication}
The communication from the server to the client and vice versa is just a
character stream containing encoded \gls{mTask} messages. The specific encoding
is visible in Appendix~\ref{app:communication-protocol}. The type holding the
messages in Listing~\ref{lst:avmsg}. Detailed explanation about the message
types will be given in the following subsections.

\begin{lstlisting}[label={lst:avmsg},caption={Available messages}]
:: MTaskId :== Int
:: MSDSId :== Int
:: MTaskFreeBytes :== Int
:: MTaskMSGRecv
        = MTTaskAck MTaskId MTaskFreeBytes | MTTaskDelAck MTaskId
        | MTSDSAck MSDSId                  | MTSDSDelAck MSDSId
        | MTPub MSDSId BCValue             | MTMessage String
        | MTDevSpec MTaskDeviceSpec        | MTEmpty

:: MTaskMSGSend
        = MTTask MTaskInterval String | MTTaskDel MTaskId
        | MTShutdown                  | MTSds MSDSId BCValue
        | MTUpd MSDSId BCValue        | MTSpec

:: MTaskInterval = OneShot | OnInterval Int | OnInterrupt Int
\end{lstlisting}

\subsection{Add a device}
A device can be added by filling in the \CI{MTaskDevice} record as much as
possible and running the \CI{connectDevice} function. This function grabs the
channels, starts the synchronization \gls{Task}, makes sure the errors are
handled when needed and runs a processing function in parallel to react on the
incoming messages. Moreover, it sends a specification request to the device in
question to determine the details of the device and updates the record to
contain the top-level \gls{Task}-id. All the device functionality heavily
depends on the specific \CI{deviceShare} function that applies a function a device in
the \gls{SDS} when they are equal. Device equality is defined as equality on
their channels. This allows you to give an old device record to the function
and still update the latest instance. Listing~\ref{lst:connectDevice} shows the
connection function.

\begin{lstlisting}[label={lst:connectDevice},%
        caption={Connect a device}]
withDevices :: MTaskDevice (MTaskDevice -> MTaskDevice) -> Task [MTaskDevice]

connectDevice :: (MTaskDevice (Shared Channels) -> Task ()) MTaskDevice -> Task Channels
connectDevice procFun device = let ch = channels device
        in appendTopLevelTask 'DM'.newMap True
                (procFun device ch -||- catchAll (getSynFun d.deviceData ch) errHdl)
                >>= \tid->withDevices device (\d->{d&deviceTask=Just tid,deviceError=Nothing})
                >>| upd (\(r,s,ss)->(r,s++[MTSpec],ss)) ch
        where
                errHdl e = withDevices device (\d->{d & deviceTask=Nothing, deviceError=Just e}) @! ()
\end{lstlisting}

Figure~\ref{fig:handshake} shows the connection diagram. The client responds to
the server with their device specification. This is detected by the processing
function and the record is updated accordingly.

\begin{figure}[H]
        \centering
        \begin{sequencediagram}
                \newthread{s}{Server}
                \newinst[4]{c}{Client}
                \begin{call}{s}{MTSpec}{c}{MTDevSpec}
                \end{call}
        \end{sequencediagram}
        \caption{Connect a device}\label{fig:handshake}
\end{figure}

\subsection{\glspl{Task} \& \glspl{SDS}}
When a \gls{Task} is sent to the device it is added to the device record
without an identifier. The actual identifier is added to the record when the
acknowledgement of the \gls{Task} by the device is received. The connection
diagram is shown in Figure~\ref{fig:tasksend}.

\begin{figure}[H]
        \centering
        \begin{sequencediagram}
                \newthread{s}{Server}
                \newinst[4]{c}{Client}
                \begin{call}{s}{MTSDS}{c}{MTSDSAck}
                \end{call}
                \begin{call}{s}{MTTask}{c}{MTTaskAck}
                \end{call}
        \end{sequencediagram}
        \caption{Sending a \gls{Task} to a device}\label{fig:tasksend}
\end{figure}

The function for sending a \gls{Task} to the device is shown in
Listing~\ref{lst:sendtask}. First the \gls{Task} is compiled into messages. The
details of the compilation process are given in Section~\ref{sec:compiler}.
The new \glspl{SDS} that were made during compilation are added to the
deviceshares that were made during the compilation are merged with the existing
shares on the device. Furthermore the messages are placed in the channel share
of the device. This will result in sending the actual \gls{SDS} specification
and \gls{Task} specifications to the device. A \gls{Task} record is created
with the identifier $-1$ to denote a \gls{Task} not yet acknowledged. Finally
the device itself is updated with the new state and with the new \gls{Task}.
When the device returns an acknowledgement the \gls{Task} is updated
accordingly.

\begin{lstlisting}[label={lst:sendtask},%
        caption={Sending a \gls{Task} to a device}]
makeTask :: String Int -> Task MTaskTask
makeTask name ident = get currentDateTime @ \dt->{MTaskTask | name=name, ident=ident, dateAdded=dt}

makeShare :: String Int BCValue -> MTaskShare
makeShare withTask identifier value = {MTaskShare | withTask=[withTask], identifier=identifier, value=value}

sendTaskToDevice :: String (Main (ByteCode a Stmt)) (MTaskDevice, MTaskInterval) -> Task [MTaskDevice]
sendTaskToDevice wta mTask (device, timeout)
# (msgs, newState) = toMessages timeout mTask device.deviceState
# shares = [makeShare wta sdsi sdsval\\{sdsi,sdsval}<-newState.sdss, (MTSds sdsi` _)<-msgs | sdsi == sdsi`] 
= updateShares device ((++) shares)
        >>| sendMessages msgs device
        >>| makeTask wta -1
        >>= withDevices device o addTaskUpState newState
        where
                addTaskUpState :: BCState MTaskTask MTaskDevice -> MTaskDevice
                addTaskUpState st task device = {MTaskDevice | device &
                        deviceState=st, deviceTasks=[task:device.deviceTasks]}
\end{lstlisting}

\subsection{Miscellaneous Messages}
There exists one special type of message that is sent to the device only when
it needs to reboot. When the server wants to stop the bond with the device it
sends the \CI{MTShutdown} message. The device will then clear his memory, thus
losing all the \glspl{SDS} and \glspl{Task} that were stored and reset itself.
Shortly after the shutdown message a new server can connect to the device
because the device is back in listening mode.

\section{Lifting mTask to iTasks}
\todo{task lifting}

\section{Examples}
\todo{example program (demo)}