[asr1617.git] / methods.tex

%Methodology

%Experiment(s) (set-up, data, results, discussion)
\section{Data \& Preprocessing}
Several experiments have been performed to gain insight on the research
question.  To run the experiments data has been collected from several \gls{dm}
albums. The exact data used is available in Appendix~\ref{app:data}. The
albums are extracted from the audio CD and converted to a mono channel waveform
with the correct samplerate utilizing \emph{SoX}%
\footnote{\url{http://sox.sourceforge.net/}}. Every file is annotated using
Praat~\cite{boersma_praat_2002} where the lyrics are manually aligned to the
audio. Examples of utterances are shown in Figure~\ref{fig:bloodstained} and
Figure~\ref{fig:abominations} where the waveform, $1-8000$Hz spectrals and
annotations are shown. It is clearly visible that within the genre of death
metal there are different spectral patterns visible over time.

\begin{figure}[ht]
        \centering
        \includegraphics[width=.7\linewidth]{cement}
        \caption{A vocal segment of the \acrlong{CC} song
                \emph{Bloodstained Cement}}\label{fig:bloodstained}
\end{figure}

\begin{figure}[ht]
        \centering
        \includegraphics[width=.7\linewidth]{abominations}
        \caption{A vocal segment of the \acrlong{DG} song
                \emph{Enthroned Abominations}}\label{fig:abominations}
\end{figure}

The data is collected from three studio albums. The first band is called
\gls{CC} and has been producing \gls{dm} for almost 25 years and has been
creating albums with a consistent style. The singer of \gls{CC} has a very
raspy growl and the lyrics are quite comprehensible. The vocals produced by
\gls{CC} are very close to regular shouting.

The second band is called \gls{DG} and makes even more violently
sounding music. The growls of the lead singer sound like a coffee grinder and
are sound less full. In the spectrals it is clearly visible that there are
overtones produced during some parts of the growling. The lyrics are completely
incomprehensible and therefore some parts were not annotated with the actual
lyrics because it was impossible to hear what was being sung.

The third band --- originating from Moscow --- is chosen bearing the name
\gls{WDISS}. This band is a little odd compared to the previous \gls{dm} bands
because they create \gls{dom}. \gls{dom} is characterized by the very slow
tempo and low tuned guitars. The vocalist has a very characteristic growl and
performs in several Muscovite bands. This band also stands out because it uses
piano's and synthesizers. The droning synthesizers often operate in the same
frequency as the vocals.

The data is labeled as singing and instrumental and labeled per band. The
distribution for this is shown in Table~\ref{tbl:distribution}. A random $10\%$
of the data is extracted for a held out test set.
\begin{table}[H]
        \centering
        \begin{tabular}{lcc}
                \toprule
                Instrumental & Singing\\
                \midrule
                0.59 & 0.41\\
                \bottomrule
        \end{tabular}
        \quad
        \begin{tabular}{lcccc}
                \toprule
                Instrumental & \gls{CC} & \gls{DG} & \gls{WDISS}\\
                \midrule
                0.59 & 0.16 & 0.19 & 0.06\\
                \bottomrule
        \end{tabular}
        \caption{Data distribution}\label{tbl:distribution}
\end{table}

\section{\acrlong{MFCC} Features}
The waveforms in itself are not very suitable to be used as features due to the
high dimensionality and correlation. Therefore we use the often used
\glspl{MFCC} feature vectors which have shown to be suitable~%
\cite{rocamora_comparing_2007}. It has also been found that altering the mel
scale to better suit singing does not yield a better
performance~\cite{you_comparative_2015}. The actual conversion is done using the
\emph{python\_speech\_features}%
\footnote{\url{https://github.com/jameslyons/python_speech_features}} package.

\gls{MFCC} features are inspired by human auditory processing inspired and are
created from a waveform incrementally using several steps:
\begin{enumerate}
        \item The first step in the process is converting the time representation
                of the signal to a spectral representation using a sliding window with
                overlap. The width of the window and the step size are two important
                parameters in the system. In classical phonetic analysis window sizes
                of $25ms$ with a step of $10ms$ are often chosen because they are small
                enough to only contain subphone entities. Singing for $25ms$ is
                impossible so it is arguable that the window size is very small.
        \item The standard \gls{FT} gives a spectral representation that has
                linearly scaled frequencies. This scale is converted to the \gls{MS}
                using triangular overlapping windows to get a more tonotopic
                representation trying to match the actual representation in the cochlea
                of the human ear.
        \item The \emph{Weber-Fechner} law describes how humans perceive physical
                magnitudes\footnote{Fechner, Gustav Theodor (1860). Elemente der
                Psychophysik}. They found that energy is perceived in logarithmic
                increments. This means that twice the amount of decibels does not mean
                twice the amount of perceived loudness. Therefore we take the log of
                the energy or amplitude of the \gls{MS} spectrum to closer match the
                human hearing.
        \item The amplitudes of the spectrum are highly correlated and therefore
                the last step is a decorrelation step. \Gls{DCT} is applied on the
                amplitudes interpreted as a signal. \Gls{DCT} is a technique of
                describing a signal as a combination of several primitive cosine
                functions.
\end{enumerate}

The default number of \gls{MFCC} parameters is twelve. However, often a
thirteenth value is added that represents the energy in the data.

\section{Experimental setup}
\subsection{Features}
The thirteen \gls{MFCC} features are chosen to feed to the classifier. The
parameters of the \gls{MFCC} features are varied in window step and window
length. The default speech processing parameters are tested but also bigger
window sizes since arguably the minimal size of a singing voice segment is a
lot bigger than the minimal size of a subphone component on which the
parameters are tuned.  The parameters chosen are as follows:

\begin{table}[H]
        \centering
        \begin{tabular}{lll}
                \toprule
                step (ms) & length (ms) & notes\\
                \midrule
                10 & 25 & Standard speech processing\\
                40 & 100 &\\ 
                80 & 200 &\\
                \bottomrule
        \end{tabular}
        \caption{\Gls{MFCC} parameter settings}
\end{table}

\subsection{\emph{Singing} voice detection}
The first type of experiment conducted is \emph{Singing} voice detection. This
is the act of segmenting an audio signal into segments that are labeled either
as \emph{Singing} or as \emph{Instrumental}. The input of the classifier is a
feature vector and the output is the probability that singing is happening in
the sample. This results in an \gls{ANN} of the shape described in
Figure~\ref{fig:bcann}. The input dimension is thirteen and the output is one.

\subsection{\emph{Singer} voice detection}
The second type of experiment conducted is \emph{Singer} voice detection. This
is the act of segmenting an audio signal into segments that are labeled either
with the name of the singer or as \emph{Instrumental}. The input of the
classifier is a feature vector and the outputs are probabilities for each of
the singers and a probability for the instrumental label. This results in an
\gls{ANN} of the shape described in Figure~\ref{fig:mcann}. The input dimension
is yet again thirteen and the output dimension is the number of categories. The
output is encoded in one-hot encoding. This means that the categories are
labeled as \texttt{1000, 0100, 0010, 0001}.

\subsection{\acrlong{ANN}}
The data is classified using standard \gls{ANN} techniques, namely \glspl{MLP}.
The classification problems are only binary and four-class so it is
interesting to see where the bottleneck lies; how abstract can the abstraction
be made. The \gls{ANN} is built with the Keras\footnote{\url{https://keras.io}}
using the TensorFlow\footnote{\url{https://github.com/tensorflow/tensorflow}}
backend that provides a high-level interface to the highly technical networks.

The general architecture of the networks is show in Figure~\ref{fig:bcann} and
Figure~\ref{fig:mcann} for respectively the binary classification and
multiclass classification. The inputs are fully connected to the hidden layer
which is fully connected too the output layer. The activation function used is
a \gls{RELU}. The \gls{RELU} function is a monotonic symmetric one-sided
function that is also known as the ramp function. The definition is given in
Equation~\ref{eq:relu}. \gls{RELU} was chosen because of its symmetry and
efficient computation. The activation function between the hidden layer and the
output layer is the sigmoid function in the case of binary classification. Of
which the definition is shown in Equation~\ref{eq:sigmoid}. The sigmoid is a
monotonic function that is differentiable on all values of $x$ and always
yields a non-negative derivative. For the multiclass classification the softmax
function is used between the hidden layer and the output layer. Softmax is an
activation function suitable for multiple output nodes. The definition is given
in Equation~\ref{eq:softmax}.

The data is shuffled before fed to the network to mitigate the risk of
overfitting on one album. Every model was trained using $10$ epochs and a
batch size of $32$. The training set and test set are separated by taking a
$90\%$ slice of all the data.

\begin{equation}\label{eq:relu}
        f(x) = \left\{\begin{array}{rcl}
                0 & \text{for} & x<0\\
                x & \text{for} & x \geq 0\\
        \end{array}\right.
\end{equation}

\begin{equation}\label{eq:sigmoid}
        f(x) = \frac{1}{1+e^{-x}}
\end{equation}

\begin{equation}\label{eq:softmax}
        \delta{(\boldsymbol{z})}_j = \frac{e^{z_j}}{\sum\limits^{K}_{k=1}e^{z_k}}
\end{equation}

\begin{figure}[H]
        \begin{subfigure}{.5\textwidth}
                \centering
                \includegraphics[width=.8\linewidth]{bcann}
                \caption{Binary classifier network architecture}\label{fig:bcann}
        \end{subfigure}%
%
        \begin{subfigure}{.5\textwidth}
                \centering
                \includegraphics[width=.8\linewidth]{mcann}
                \caption{Multiclass classifier network architecture}\label{fig:mcann}
        \end{subfigure}
        \caption{\acrlong{ANN} architectures.}
\end{figure}

\section{Results}
\subsection{\emph{Singing} voice detection}
Table~\ref{tbl:singing} shows the results for the singing-voice detection.
Figure~\ref{fig:bclass} shows an example of a segment of a song with the
classifier plotted underneath to illustrate the performance.

\begin{table}[H]
        \centering
        \begin{tabular}{rccc}
                \toprule
                   & \multicolumn{3}{c}{Parameters (step/length)}\\
                    & 10/25 & 40/100 & 80/200\\
                \midrule
                3h  & 0.86 (0.34) & 0.87 (0.32) & 0.85 (0.35)\\
                5h  & 0.87 (0.31) & 0.88 (0.30) & 0.87 (0.32)\\
                8h  & 0.88 (0.30) & 0.88 (0.31) & 0.88 (0.29)\\
                13h & 0.89 (0.28) & 0.89 (0.29) & 0.88 (0.30)\\
                \bottomrule
        \end{tabular}
        \caption{Binary classification results (accuracy
                (loss))}\label{tbl:singing}
\end{table}

\begin{figure}[H]
        \centering
        \includegraphics[width=.7\linewidth]{bclass}.
        \caption{Plotting the classifier under the audio signal}\label{fig:bclass}
\end{figure}

\subsection{\emph{Singer} voice detection}
Table~\ref{tbl:singer} shows the results for the singer-voice detection.

\begin{table}[H]
        \centering
        \begin{tabular}{rccc}
                \toprule
                   & \multicolumn{3}{c}{Parameters (step/length)}\\
                    & 10/25 & 40/100 & 80/200\\
                \midrule
                3h  & 0.83 (0.48) & 0.82 (0.48) & 0.82 (0.48)\\
                5h  & 0.85 (0.43) & 0.84 (0.44) & 0.84 (0.44)\\
                8h  & 0.86 (0.41) & 0.86 (0.39) & 0.86 (0.40)\\
                13h & 0.87 (0.37) & 0.87 (0.38) & 0.86 (0.39)\\
                \bottomrule
        \end{tabular}
        \caption{Multiclass classification results (accuracy
                (loss))}\label{tbl:singer}
\end{table}

\subsection{Alien data}
To test the generalizability of the models the system is tested on alien data.
The data was retrieved from the album \emph{The Desperation} by \emph{Godless
Truth}. \emph{Godless Truth} is a so called old-school \gls{dm} band that has
very raspy vocals and the vocals are very up front in the mastering. This means
that the vocals are very prevalent in the recording and therefore no difficulty
is expected for the classifier. Figure~\ref{fig:alien1} shows that indeed the
classifier scores very accurately. Note that the spectogram settings have been
adapted a little bit to make the picture more clear. The spectogram shows the
frequency range from $0$ to $3000Hz$.

\begin{figure}[H]
        \centering
        \includegraphics[width=.7\linewidth]{alien1}.
        \caption{Plotting the classifier under similar alien data}\label{fig:alien1}
\end{figure}

To really test the limits, a song from the highly atmospheric doom metal band
called \emph{Catacombs} has been tested on the system. The album \emph{Echoes
Through the Catacombs} is an album that has a lot of synthesizers, heavy
droning guitars and bass lines. The vocals are not mixed in a way that makes
them stand out. The models have never seen trainingsdata that is even remotely
similar to this type of metal. Figure~\ref{fig:alien2} shows a segment of the
data. Here it is clearly visible that the classifier can not distinguish
singing from non singing.

\begin{figure}[H]
        \centering
        \includegraphics[width=.7\linewidth]{alien2}.
        \caption{Plotting the classifier under different alien data}\label{fig:alien2}
\end{figure}