Shallow embedding on the other hand models the language constructs as functions with the semantics embedded.
Consequently, adding a construct is easy, i.e.\ it only entails adding another function.
Contrarily, adding semantics requires adapting all language constructs.
-Lifting the functions to type classes, i.e.\ parametrising the constructs over the semantics, allows extension of the language both in constructs and in semantics orthogonally. This advanced style of embedding is called tagless-final or class-based shallow embedding~\cite{kiselyov_typed_2012}.
+Lifting the functions to type classes, i.e.\ parametrising the constructs over the semantics, allows extension of the language both in constructs and in semantics orthogonally. This advanced style of embedding is called tagless-final or class-based shallow embedding~\citep{kiselyov_typed_2012}.
While it is often possible to lift values of a user-defined data type to a value in the \gls{DSL}, it is not possible to interact with it using \gls{DSL} constructs, they are not first-class citizens.
\end{enumerate*}
The functions for this are simply not available automatically in the embedded language.
For some semantics---such as an interpreter---it is possible to directly lift the functions from the host language to the \gls{DSL}.
-In other cases---e.g.\ \emph{compiling} \glspl{DSL} such as a compiler or a printer---this is not possible~\cite{elliott_compiling_2003}. %the torget this is not possible. cannot just be lifted from the host language to the \gls{DSL} so it requires a lot of boilerplate to define and implement them.
+In other cases---e.g.\ \emph{compiling} \glspl{DSL} such as a compiler or a printer---this is not possible~\citep{elliott_compiling_2003}. %the torget this is not possible. cannot just be lifted from the host language to the \gls{DSL} so it requires a lot of boilerplate to define and implement them.
Thus, all of the operations on the data type have to be defined by hand requiring a lot of plumbing and resulting in a lot of boilerplate code.
To relieve the burden of adding all these functions, metaprogramming\nobreak---\nobreak\hskip0pt and custom quasiquoters---can be used.
As a result, views on the \gls{DSL} are data types implementing these classes.
To illustrate the technique, a simple \gls{DSL}, a language consisting of literals and addition, is outlined.
-This language, implemented according to the tagless-final style~\cite{carette_finally_2009} in \gls{HASKELL}~\cite{peyton_jones_haskell_2003} consists initially only of one type class containing two functions.
+This language, implemented according to the tagless-final style~\citep{carette_finally_2009} in \gls{HASKELL}~\citep{peyton_jones_haskell_2003} consists initially only of one type class containing two functions.
The \haskellinline{lit} function lifts values from the host language to the \gls{DSL} domain.
The class constraint \haskellinline{Show} is enforced on the type variable \haskellinline{a} to make sure that the value can be printed.
The infix function \haskellinline{+.} represents the addition of two expressions in the \gls{DSL}.
\haskellinline{newtype}s are special data types only consisting a single constructor with one field to which the type is isomorphic.
During compilation the constructor is completely removed resulting in no overhead~\cite[Sec.~4.2.3]{peyton_jones_haskell_2003}.
}
-Since the language is typed, the printer data type has to have a type variable but it is only used during typing---i.e.\ a phantom type~\cite{leijen_domain_2000}:
+Since the language is typed, the printer data type has to have a type variable but it is only used during typing---i.e.\ a phantom type~\citep{leijen_domain_2000}:
\begin{lstHaskell}
newtype Printer a = P { runPrinter :: String }
Adding functions to the language is achieved by adding a multi-parameter class to the \gls{DSL}.
The type of the class function allows for the implementation to only allow first order function by supplying the arguments in a tuple.
Furthermore, by defining the \haskellinline{Main} type, the \gls{DSL} forces all functions to be defined at the top level and with the \haskellinline{:-} operator the syntax becomes usable.
-Finally, by defining the functions as a \gls{HOAS} type safety is achieved~\cite{pfenning_higher-order_1988,chlipala_parametric_2008}.
+Finally, by defining the functions as a \gls{HOAS} type safety is achieved~\citep{pfenning_higher-order_1988,chlipala_parametric_2008}.
The complete definition looks as follows:
\begin{lstHaskell}
}
Adding these classes and their corresponding instances is tedious and results in boilerplate code.
-We therefore resort to metaprogramming, and in particular \gls{TH}~\cite{sheard_template_2002} to alleviate this burden.
+We therefore resort to metaprogramming, and in particular \gls{TH}~\citep{sheard_template_2002} to alleviate this burden.
\section{Template metaprogramming}
Metaprogramming is a special flavour of programming where programs have the ability to treat and manipulate programs or program fragments as data.
-There are several techniques to facilitate metaprogramming, moreover it has been around for many years now~\cite{lilis_survey_2019}.
-Even though it has been around for many years, it is considered complex~\cite{sheard_accomplishments_2001}.
+There are several techniques to facilitate metaprogramming, moreover it has been around for many years now~\citep{lilis_survey_2019}.
+Even though it has been around for many years, it is considered complex~\citep{sheard_accomplishments_2001}.
-\gls{TH} is GHC's de facto metaprogramming system, implemented as a compiler extension together with a library~\cite{sheard_template_2002}\cite[Sec.~6.13.1]{ghc_team_ghc_2021}.
+\gls{TH} is GHC's de facto metaprogramming system, implemented as a compiler extension together with a library~\citep{sheard_template_2002}\cite[Sec.~6.13.1]{ghc_team_ghc_2021}.
Readers already familiar with \gls{TH} can safely skip this section.
\gls{TH} adds four main concepts to the language, na\-me\-ly AST data types, splicing, quasiquotation and reification.
With this machinery, regular \gls{HASKELL} functions can be defined that are called at compile time, inserting generated code into the {AST}.
These functions are monadic functions operating in the \haskellinline{Q} monad.
-The \haskellinline{Q} monad facilitates failure, reification and fresh identifier generation for hygienic macros~\cite{kohlbecker_hygienic_1986}.
+The \haskellinline{Q} monad facilitates failure, reification and fresh identifier generation for hygienic macros~\citep{kohlbecker_hygienic_1986}.
Within the \haskellinline{Q} monad, capturable and non-capturable identifiers can be generated using the \haskellinline{mkName} and \haskellinline{newName} functions respectively.
The \emph{Peter Parker principle}\footnote{With great power comes great responsibility.} holds for the \haskellinline{Q} monad as well because it executes at compile time and is very powerful.
-For example it can subvert module boundaries, thus accessing constructors that were hidden; access the structure of abstract types; and it may cause side effects during compilation because it is possible to call \haskellinline{IO} operations~\cite{terei_safe_2012}.
+For example it can subvert module boundaries, thus accessing constructors that were hidden; access the structure of abstract types; and it may cause side effects during compilation because it is possible to call \haskellinline{IO} operations~\citep{terei_safe_2012}.
To achieve the goal of embedding data types in a \gls{DSL} we refrain from using these \emph{unsafe} features.
\subsubsection{Data types}
\end{lstHaskell}
\subsubsection{Quasiquotation}
-Another key concept of \gls{TH} is Quasiquotation, the dual of splicing~\cite{bawden_quasiquotation_1999}.
+Another key concept of \gls{TH} is Quasiquotation, the dual of splicing~\citep{bawden_quasiquotation_1999}.
While it is possible to construct entire programs using the provided data types, it is a little cumbersome.
Using \emph{Oxford brackets} (\verb#[|# \ldots\verb#|]#) or single or double apostrophes, verbatim \gls{HASKELL} code can be entered that is converted automatically to the corresponding AST nodes easing the creation of language constructs.
Depending on the context, different quasiquotes are used:
\subsection{Custom quasiquoters}
The syntax burden of \glspl{EDSL} can be reduced using quasiquotation.
In \gls{TH}, quasiquotation is a convenient way to create \gls{HASKELL} language constructs by entering them verbatim using Oxford brackets.
-However, it is also possible to create so-called custom quasiquoters~\cite{mainland_why_2007}.
+However, it is also possible to create so-called custom quasiquoters~\citep{mainland_why_2007}.
If the programmer writes down a fragment of code between tagged \emph{Oxford brackets}, the compiler executes the associated quasiquoter functions at compile time.
A quasiquoter is a value of the following data type:
They correspond to a multi-way conditional expressions and can thus be converted to \gls{DSL} constructs straightforwardly~\cite[Chp.~4.4]{peyton_jones_implementation_1987}.
In contrast to the binary literal quasiquoter example, we do not create the parser by hand.
-The parser combinator library \emph{parsec} is used instead to ease the creation of the parser~\cite{leijen_parsec_2001}.
+The parser combinator library \emph{parsec} is used instead to ease the creation of the parser~\citep{leijen_parsec_2001}.
First the location of the quasiquoted code is retrieved using the \haskellinline{location} function that operates in the \haskellinline{Q} monad.
This location is inserted in the parsec parser so that errors are localised in the source code.
Then, the \haskellinline{expr} parser is called that returns an \haskellinline{Exp} in the \haskellinline{Q} monad.
\end{lstHaskell}
\section{Related work}
-Generic or polytypic programming is a promising technique at first glance for automating the generation of function implementations~\cite{lammel_scrap_2003}.
+Generic or polytypic programming is a promising technique at first glance for automating the generation of function implementations~\citep{lammel_scrap_2003}.
However, while it is possible to define a function that works on all first-order types, adding a new function with a new name to the language is not possible.
This does not mean that generic programming is not useable for embedding pattern matches.
In generic programming, types are represented as sums of products and using this representation it is possible to define pattern matching functions.
-For example, Rhiger showed a method for expressing statically typed pattern matching using typed higher-order functions~\cite{rhiger_type-safe_2009}.
+For example, Rhiger showed a method for expressing statically typed pattern matching using typed higher-order functions~\citep{rhiger_type-safe_2009}.
If not the host language but the \gls{DSL} contains higher order functions, the same technique could be applied to port pattern matching to \glspl{DSL} though using an explicit sums of products representation.
Atkey et al.\ describe embedding pattern matching in a \gls{DSL} by giving patterns an explicit representation in the \gls{DSL} by using pairs, sums and injections~\cite[Section~3.3]{atkey_unembedding_2009}.
-McDonell et al.\ extends on this idea, resulting in a very similar but different solution to ours~\cite{mcdonell_embedded_2021}.
+McDonell et al.\ extends on this idea, resulting in a very similar but different solution to ours~\citep{mcdonell_embedded_2021}.
They used the technique that Atkey et al.\ showed and applied it to deep embedding using the concrete syntax of the host language.
The scaffolding---e.g.\ generating the pairs, sums and injections---for embedding is automated using generics but the required pattern synonyms are generated using \gls{TH}.
The key difference to our approach is that we specialise the implementation for each of the backends instead of providing a general implementation of data type handling operations.
Furthermore, our implementation does not require a generic function to trace all constructors, resulting in problems with (mutual) recursion.
-Young et al.\ added pattern matching to a deeply \gls{EDSL} using a compiler plugin~\cite{young_adding_2021}.
+Young et al.\ added pattern matching to a deeply \gls{EDSL} using a compiler plugin~\citep{young_adding_2021}.
This plugin implements an \haskellinline{externalise :: a -> E a} function that allows lifting all machinery required for pattern matching automatically from the host language to the \gls{DSL}.
Under the hood, this function translates the pattern match to constructors, deconstructors, and constructor predicates.
The main difference with this work is that it requires a compiler plugin while our metaprogramming approach works on any compiler supporting a metaprogramming system similar to \gls{TH}.
We therefore do not claim an exhaustive overview of related work on all aspects of metaprogramming.
However, we have have tried to present most research on metaprogramming in \gls{TH}.
Czarnecki et al.\ provide a more detailed comparison of different metaprogramming techniques.
-They compare staged interpreters, metaprogramming and templating by comparing MetaOCaml, \gls{TH} and C++ templates~\cite{czarnecki_dsl_2004}.
+They compare staged interpreters, metaprogramming and templating by comparing MetaOCaml, \gls{TH} and C++ templates~\citep{czarnecki_dsl_2004}.
\gls{TH} has been used to implement related work.
They all differ slightly in functionality from our domain and can be divided into several categories.
\subsubsection{Generating extra code}
Using \gls{TH} or other metaprogramming systems it is possible to add extra code to your program.
-The original \gls{TH} paper showed that it is possible to create variadic functions such as \haskellinline{printf} using \gls{TH} that would be almost impossible to define without~\cite{sheard_template_2002}.
-Hammond et al.\ used \gls{TH} to generate parallel programming skeletons~\cite{hammond_automatic_2003}.
+The original \gls{TH} paper showed that it is possible to create variadic functions such as \haskellinline{printf} using \gls{TH} that would be almost impossible to define without~\citep{sheard_template_2002}.
+Hammond et al.\ used \gls{TH} to generate parallel programming skeletons~\citep{hammond_automatic_2003}.
In practise, this means that the programmer selects a skeleton and, at compile time, the code is massaged to suit the pattern and information about the environment is inlined for optimisation.
-Polak et al.\ implemented automatic GUI generation using \gls{TH}~\cite{polak_automatic_2006}.
-Dureg\aa{}rd et al.\ wrote a parser generator using \gls{TH} and the custom quasiquoting facilities~\cite{duregard_embedded_2011}.
+Polak et al.\ implemented automatic GUI generation using \gls{TH}~\citep{polak_automatic_2006}.
+Dureg\aa{}rd et al.\ wrote a parser generator using \gls{TH} and the custom quasiquoting facilities~\citep{duregard_embedded_2011}.
From a specification of the grammar, given in verbatim using a custom quasiquoter, a parser is generated at compile time.
-Shioda et al.\ used metaprogramming in the D programming language to create a \gls{DSL} toolkit~\cite{shioda_libdsl_2014}.
+Shioda et al.\ used metaprogramming in the D programming language to create a \gls{DSL} toolkit~\citep{shioda_libdsl_2014}.
They also programmatically generate parsers and a backend for either compiling or interpreting the \gls{IR}.
-Blanchette et al.\ use \gls{TH} to simplify the development of Liquid \gls{HASKELL} proofs~\cite{blanchette_liquid_2022}.
-Folmer et al.\ used \gls{TH} to synthesize C$\lambda$aSH~\cite{baaij_digital_2015} abstract syntax trees to be processed~\cite{folmer_high-level_2022}.
-In similar fashion, Materzok used \gls{TH} to translate YieldFSM programs to C$\lambda$aSH~\cite{materzok_generating_2022}.
+Blanchette et al.\ use \gls{TH} to simplify the development of Liquid \gls{HASKELL} proofs~\citep{blanchette_liquid_2022}.
+Folmer et al.\ used \gls{TH} to synthesize C$\lambda$aSH~\citep{baaij_digital_2015} abstract syntax trees to be processed~\citep{folmer_high-level_2022}.
+In similar fashion, Materzok used \gls{TH} to translate YieldFSM programs to C$\lambda$aSH~\citep{materzok_generating_2022}.
\subsubsection{Optimisation}
Besides generating code, it is also possible to analyse existing code and perform optimisations.
Yet, this is dangerous territory because unwantedly the semantics of the optimised program may be slightly different from the original program.
-For example, Lynagh implemented various optimisations in \gls{TH} such as automatic loop unrolling~\cite{lynagh_unrolling_2003}.
+For example, Lynagh implemented various optimisations in \gls{TH} such as automatic loop unrolling~\citep{lynagh_unrolling_2003}.
The compile-time executed functions analyse the recursive function and unroll the recursion to a fixed depth to trade execution speed for program space.
-Also, O'Donnell embedded Hydra, a hardware description language, in \gls{HASKELL} utilising \gls{TH}~\cite{odonnell_embedding_2004}.
+Also, O'Donnell embedded Hydra, a hardware description language, in \gls{HASKELL} utilising \gls{TH}~\citep{odonnell_embedding_2004}.
Using intensional analysis of the AST, it detects cycles by labelling nodes automatically so that it can generate \emph{netlists}.
The authors mention that alternatively this could have be done using a monad but this hampers equational reasoning greatly, which is a key property of Hydra.
-Finally, Viera et al.\ present a way of embedding attribute grammars in \gls{HASKELL} in a staged fashion~\cite{viera_staged_2018}.
+Finally, Viera et al.\ present a way of embedding attribute grammars in \gls{HASKELL} in a staged fashion~\citep{viera_staged_2018}.
Checking several aspects of the grammar is done at compile time using \gls{TH} while other safety checks are performed at runtime.
\subsubsection{Compiler extension}
Sometimes, expressing certain functionalities in the host languages requires a lot of boilerplate, syntax wrestling, or other pains.
Metaprogramming can relieve some of this stress by performing this translation to core constructs automatically.
For example, implementing generic---or polytypic--- functions in the compiler is a major effort.
-Norell et al.\ used \gls{TH} to implement the machinery required to implement generic functions at compile time~\cite{norell_prototyping_2004}.
-Adams et al.\ also explores implementing generic programming using \gls{TH} to speed things up considerably compared to regular generic programming~\cite{adams_template_2012}.
-Clifton et al.\ use \gls{TH} with a custom quasiquoter to offer skeletons for workflows and embed foreign function interfaces in a \gls{DSL}~\cite{clifton-everest_embedding_2014}.
-Eisenberg et al.\ showed that it is possible to programmatically lift some functions from the function domain to the type domain at compile time, i.e.\ type families~\cite{eisenberg_promoting_2014}.
-Furthermore, Seefried et al.\ argued that it is difficult to do some optimisations in \glspl{EDSL} and that metaprogramming can be of use there~\cite{seefried_optimising_2004}.
+Norell et al.\ used \gls{TH} to implement the machinery required to implement generic functions at compile time~\citep{norell_prototyping_2004}.
+Adams et al.\ also explores implementing generic programming using \gls{TH} to speed things up considerably compared to regular generic programming~\citep{adams_template_2012}.
+Clifton et al.\ use \gls{TH} with a custom quasiquoter to offer skeletons for workflows and embed foreign function interfaces in a \gls{DSL}~\citep{clifton-everest_embedding_2014}.
+Eisenberg et al.\ showed that it is possible to programmatically lift some functions from the function domain to the type domain at compile time, i.e.\ type families~\citep{eisenberg_promoting_2014}.
+Furthermore, Seefried et al.\ argued that it is difficult to do some optimisations in \glspl{EDSL} and that metaprogramming can be of use there~\citep{seefried_optimising_2004}.
They use \gls{TH} to change all types to unboxed types, unroll loops to a certain depth and replace some expressions by equivalent more efficient ones.
-Torrano et al.\ showed that it is possible to use \gls{TH} to perform a strictness analysis and perform let-to-case translation~\cite{torrano_strictness_2005}.
+Torrano et al.\ showed that it is possible to use \gls{TH} to perform a strictness analysis and perform let-to-case translation~\citep{torrano_strictness_2005}.
Both applications are examples of compiler extensions that can be implemented using \gls{TH}.
-Another example of such a compiler extension is shown by Gill et al.~\cite{gill_haskell_2009}.
+Another example of such a compiler extension is shown by Gill et al.~\citep{gill_haskell_2009}.
They created a meta level \gls{DSL} to describe rewrite rules on \gls{HASKELL} syntax that are applied on the source code at compile time.
\subsubsection{Quasiquotation}
By means of quasiquotation, the host language syntax that usually seeps through the embedding can be hidden.
-The original \gls{TH} quasiquotation paper~\cite{mainland_why_2007} shows how this can be done for regular expressions, not only resulting in a nicer syntax but syntax errors are also lifted to compile time instead of run time.
-Also, Kariotis et al.\ used \gls{TH} to automatically construct monad stacks without having to resort to the monad transformers library which requires advanced type system extensions~\cite{kariotis_making_2008}.
+The original \gls{TH} quasiquotation paper~\citep{mainland_why_2007} shows how this can be done for regular expressions, not only resulting in a nicer syntax but syntax errors are also lifted to compile time instead of run time.
+Also, Kariotis et al.\ used \gls{TH} to automatically construct monad stacks without having to resort to the monad transformers library which requires advanced type system extensions~\citep{kariotis_making_2008}.
-Najd uses the compile time to be able to do normalisation for a \gls{DSL}, dubbing it \glspl{QDSL}~\cite{najd_everything_2016}.
+Najd uses the compile time to be able to do normalisation for a \gls{DSL}, dubbing it \glspl{QDSL}~\citep{najd_everything_2016}.
They utilise the quasiquation facilities of \gls{TH} to convert \gls{HASKELL} \gls{DSL} code to constructs in the \gls{DSL}, applying optimisations such as eliminating lambda abstractions and function applications along the way.
-Egi et al.\ extended \gls{HASKELL} to support non-free data type pattern matching---i.e.\ data type with no standard form, e.g.\ sets, graphs---using \gls{TH}~\cite{egi_embedding_2022}.
+Egi et al.\ extended \gls{HASKELL} to support non-free data type pattern matching---i.e.\ data type with no standard form, e.g.\ sets, graphs---using \gls{TH}~\citep{egi_embedding_2022}.
Using quasiquotation, they make a complicated embedding of non-linear pattern matching available through a simple lens.
\subsubsection{\gls{TTH}}\label{ssec_fcd:typed_template_haskell}
-\gls{TTH} is a very recent extension/alternative to normal \gls{TH}~\cite{pickering_multi-stage_2019,xie_staging_2022}.
+\gls{TTH} is a very recent extension/alternative to normal \gls{TH}~\citep{pickering_multi-stage_2019,xie_staging_2022}.
Where in \gls{TH} you can manipulate arbitrary parts of the syntax tree, add top-level splices of data types, definitions and functions, in \gls{TTH} the programmer can only splice expressions but the abstract syntax tree fragments representing the expressions are well-typed by construction instead of untyped.
-Pickering et al.\ implemented staged compilation for the \emph{generics-sop}~\cite{de_vries_true_2014} generics library to improve the efficiency of the code using \gls{TTH}~\cite{pickering_staged_2020}.
-Willis et al.\ used \gls{TTH} to remove the overhead of parsing combinators~\cite{willis_staged_2020}.
+Pickering et al.\ implemented staged compilation for the \emph{generics-sop}~\citep{de_vries_true_2014} generics library to improve the efficiency of the code using \gls{TTH}~\citep{pickering_staged_2020}.
+Willis et al.\ used \gls{TTH} to remove the overhead of parsing combinators~\citep{willis_staged_2020}.
\section{Discussion}
Functional programming languages are especially suitable for embedding \glspl{DSL} but adding user-defined data types is still an issue.
For future work, it would be interesting to see how generating boilerplate for user-defined data types translates from shallow embedding to deep embedding.
In deep embedding, the language constructs are expressed as data types in the host language.
Adding new constructs, e.g.\ constructors, deconstructors, and constructor tests, for the user-defined data type therefore requires extending the data type.
-Techniques such as data types \`a la carte~\cite{swierstra_data_2008} and open data types~\cite{loh_open_2006} show that it is possible to extend data types orthogonally but whether metaprogramming can still readily be used is something that needs to be researched.
+Techniques such as data types \`a la carte~\citep{swierstra_data_2008} and open data types~\citep{loh_open_2006} show that it is possible to extend data types orthogonally but whether metaprogramming can still readily be used is something that needs to be researched.
It may also be possible to implemented (parts) of the boilerplate generation using \gls{TTH} (see \cref{ssec_fcd:typed_template_haskell}) to achieve more confidence in the type correctness of the implementation.
Another venue of research is to try to find the limits of this technique regarding richer data type definitions.
-It would be interesting to see whether it is possible to apply the technique on data types with existentially quantified type variables or full-fledged generalised \glspl{ADT}~\cite{hinze_fun_2003}.
+It would be interesting to see whether it is possible to apply the technique on data types with existentially quantified type variables or full-fledged generalised \glspl{ADT}~\citep{hinze_fun_2003}.
It is not possible to straightforwardly lift the deconstructors to type classes because existentially quantified type variables will escape.
Rank-2 polymorphism offers tools to define the types in such a way that this is not the case anymore.
However, implementing compiling views on the \gls{DSL} is complicated because it would require inventing values of an existentially quantified type variable to satisfy the type system which is difficult.
repositories / phd-thesis.git / blobdiff