Computational Audio Content Analysis

Variability

The variable conditions are characterized by the properties of acoustic environment (e.g. size and the shape of acoustic space, type of reflective surfaces), the capturing microphone and device, the relative placement of the sound source and the microphone, and interfering noise sources present. In realistic usage scenarios, all of the condition variations cannot be taken into account explicitly in the data collection. If the collected material represents only a subset of the possible conditions, this can cause a mismatch between the material used to develop the system and the material encountered in the real usage stage, which eventually leads to poor performance. Hence in the data collection it is advisable to make extra effort to minimize this data mismatch by capturing as representative set of data as possible, under all identified conditions. When material captured under variable conditions is used in the learning stage, the system is said to use a multi-condition training approach [Li2016, p. 116]. Collected audio data can be diversified for the multi-condition training by adding artificially different impulse responses to it in the training stage [Zhao2014], since many of the variable conditions are reflected in the impulse response which characterizes the overall acoustic characteristics of the captured audio signal [Li2016, p. 206]. This approach requires obtaining recordings of the target source with as little external effects as possible and then convolving the audio signals with measured impulse responses from various real acoustic environments. The room impulse responses can be generated also with room simulation techniques [Zolzer2008, p. 191].

Interfering Noise Sources

Interfering noise sources can be handled similarly to acoustic conditions. In usage cases where potential noise sources are known and stationary, the data can be easily collected under similar conditions. However, in cases where the types of noise sources are varying or the relative position of the noise source with respect to the capturing microphone varies, data collection under all matched conditions is impractical. Depending on the level of variability, one can be still successful by collecting material as diversely as possible and using a multi-condition training approach. Another feasible approach is to obtain recordings of target sound sources without any interfering noise and recordings with the noise sources alone, and simulate noisy signals by artificially mixing these with various signal-to-noise ratios (SNR) [Mesaros2019a]. The artificial mixing approach can potentially produce larger quantities of relevant training material than a direct recording approach. On the other hand, the diversity of the available recordings influences and limits the variability of the artificially produced material.

Acoustic Scenes

Audio material for acoustic scene classification is often captured in a fixed position to ensure that the scene category stays the same throughout the recording [Mesaros2016, Mesaros2018b]. This simplifies the annotation process, as the category labels can be assigned for full signals or very long time-segments in it. Category labels should be clearly defined to minimize the subjectivity of the label selection in the annotation process. Examples of scene labels are busy street, office, and traveling by bus.

Sound events

Labeling sound events is a highly subjective process where perception and personal life experience of the annotator have an important role [Gygi2007]. Subjectivity can be controlled to some extent by defining the textual labels for the sound categories before the annotation process and forcing the label selection among these pre-defined labels. This is advisable in applications where the number of target sound categories is low and they are well-defined. In research where the aim is to recognize all sounds in the acoustic scene, or the target application is not defined before the data collection, the labels cannot be defined before actually annotating the audio material. In these cases, the most advisable approach is to allow free label selection during the annotation, i.e., each sound instance will be annotated with a descriptive and possibly a new label, based on the annotator‘s opinion; afterwards labels describing the same sound category can be manually grouped after all material is annotated [Mesaros2010, Heittola2013a, Mesaros2016, Fonseca2018]. The label post-processing stage is essential to make the material usable for supervised machine learning, as freely selected labels often contain typos, different wording (people talking versus people speaking, or synonyms (car versus automobile) for sounds clearly belonging to the same category.

Temporal Segmentation Process

Annotating sound events with full temporal information requires marking the time instance when the sound event is first perceived, and the time instance when the sound event is not anymore perceived. This temporal segmentation process will introduce a varying level of subjectivity to the annotations, depending on the sound event type [Mesaros2018, p. 157]. For sound events which have rapidly increasing and decreasing amplitude envelope, such as car horn, onsets and offsets can be pinpointed reliably with acceptable time resolution (e.g. 100 ms). On the other hand, for sound events which have slowly increasing and decreasing amplitude envelope, onsets and offsets can be sometimes very difficult to pinpoint reliably, especially when interfering background noise is present in the acoustic signal. Two such examples are illustrated in Figure 3.4.

Sound Source Separation

Audio captured in our everyday environments consists of sounds produced by various sound sources having distinctive structure in time and frequency. Sound sources can be considered to correspond to sound events in the acoustic scene, sometimes multiple sound sources belonging to the same event. The aim of sound source separation is ideally to decompose a given audio signal, mixture signal, with multiple simultaneous active sound sources into individual sound sources.

Non-Negative Matrix Factorization

One commonly used method for sound source separation is based on non-negative matrix factorization (NMF) [Virtanen2007]. NMF models the structure of the sound by representing the spectra of the mixture signal as a sum of components, each having a fixed magnitude spectrum and a time-varying gain. The assumption in NMF-based source separation is that each sound source has a characteristic spectral structure that differs from other sources present in the mixture signal, and ideally each source can be modeled using a distinct set of fixed magnitude spectra. In the signal model, the magnitude spectrum vector \(\boldsymbol{x}_{t}\) in frame \(t\) is defined as a linear combination of basis spectra \(\boldsymbol{b}_{k}\) (fixed spectrum) and corresponding \(h_{k,t}\) activation coefficients (gain). This can be expressed as:

$$ \boldsymbol{x}_{t}\approx\sum_{k=1}^{K}\boldsymbol{b}_{k}{h}_{k,t} \tag{3.1} $$

where \(K\) is the number of components, and \(h_{k,t}\) is the activation coefficient of the \(k\) th basis spectrum in frame \(t\). One basis spectrum and its activation coefficients are referred to as a component. Often NMF is used in an unsupervised manner without any prior knowledge of which components represent the given sound source. Ideally sound sources in the mixture signal become represented as a sum of one or more components, however, it is possible that the resulting components contain parts from multiple sound sources. This is considered as learning-free sound separation, and it gives a good separation performance in cases where the characteristics of the sound sources are distinctive. Figure 3.5 shows the spectrogram of an audio signal captured during a basketball game and the results of factorization into four components. In this example, the first component captures the squeaking sounds of basketball players’ shoes and residual audience sounds, the second component captures shouting from the audience, the third component captures applause, and the fourth component captures the whistle sound of the basketball referee.

The component-wise audio streams can be reconstructed by generating a time-frequency mask \(\boldsymbol{w}_{k}\) from basis spectra and activation coefficients, and filtering the mixture signal with it. The time-frequency soft mask for component :\(j\) is defined as

$$ \boldsymbol{w}_{j}=\frac{\boldsymbol{b}_{j}\boldsymbol{h}_{j,t}}{\sum_{k=1}^{K}\boldsymbol{b}_{k}\boldsymbol{h}_{k,t}}. \tag{3.2} $$

The mask can be considered as a time-varying Wiener filter which separates the signal into a stream containing approximately homogeneous spectral content that differs significantly from the other streams. The outputted streams do not represent individual sound sources, but they are a combination of the sources present in the original mixture signal.

Processing stages

Audio signals can generally be assumed to be non-stationary because of their rapidly changing signal statistics (e.g. magnitudes of the frequency components). This requires acoustic feature extraction in short time segments, analysis frames, which contain signal in a quasi-stationary state. The basic stages in acoustic feature extraction are frame blocking, windowing, spectrum calculation, and subsequent analysis, as illustrated in Figure 3.6.

In the frame blocking stage the audio signal is split into fixed-length analysis frames, which are shifted with a fixed time step (feature hop length). When using Fourier transform, the length of the analysis frame is related to frequency resolution: longer frames will give better frequency resolution than shorter frames, but at the same time the temporal resolution of the analysis is lower with longer frames. Usually for environmental audio analysis, the frame length is set between 20 and 100 ms with 25 - 50% overlapping frames. Sound event detection systems use shorter analysis frames (e.g. 20 ms in ) than acoustic scene classification systems, as spectral characteristics of sound events are changing more rapidly than the general characteristics of acoustic scenes, and good time resolution is required for detecting event onsets and offsets accurately. In order to avoid abrupt changes at the frame boundaries causing distortions in the spectrum, the analysis frames are smoothed with a windowing function such as Hamming or Hann function. The windowed analysis frames are transformed into a spectrum, forming a time-frequency representation, and acoustic features are extracted from it.

Until recently, the most common approach to develop acoustic features have been carefully engineering features from the time-frequency representation and using expert knowledge about acoustics, sound perception, sound classes, and their differences while developing features. These types of features are often called hand-crafted features. Recently, automatic feature learning techniques have also been used with increased dataset sizes [Salamon2015, Cakir2018]. These techniques produce high-level feature representations given the data and specified task, and have shown impressive performance compared to hand-crafted features. The main advantage of feature learning over feature engineering is that no specific knowledge about the target task is required. This thesis only discusses hand-crafted acoustic features, and the most common hand-crafted features extracted from the spectrum, mel-band energies and decorrelated mel-band energies called mel-frequency cepstral coefficients (MFCCs) [Davis1980]. These features are illustrated in Figure 3.7 for an audio example.

Mel-band energies

Mel-band energies are a perceptually motivated representation based on mel-scaled frequency bands. The aim of the mel-scale is to mimic the non-linearity of human auditory perception, by having narrower bands at lower frequencies than at higher frequencies. The scale has been created through listening experiments, having listeners listen to two alternating sinusoids and adjusting one of them to have a perceived pitch half to the other one [Stevens1937]. The resulting mel-scale is approximately linear up to 1 kHz, after which it is approximately logarithmic. The relation between mel and linear frequency can be approximated as [OShaughnessy2000, p. 128]

$$ F_{mel}=2595log_{10}(1+\frac{F_{Hz}}{700}). \tag{3.3} $$

A mel-scale filterbank consists of overlapping band-pass filters (typically 20, 40, 64, or 128 filters in total) having triangular frequency response and with filters’ center frequencies linearly spaced on the mel scale. Figure 3.8 illustrates the process of constructing such a filterbank; the relation between center frequencies of the filters in hertz and mels is shown in the top panel, and the triangular filters are shown in the bottom panel. Many alternative filterbank implementations have been proposed in the literature throughout the years, mostly varying in how the nonlinear pitch perception of humans is approximated in the filterbank design [Davis1980, Slaney1998, Young2006].

The mel-scale filterbank is applied on the spectrum (either magnitude or power spectrum) to obtain energy per mel-band. The resulting representation is called a mel spectrogram. Following humans’ logarithmic perception of sound loudness, the dynamic range of the energy values per band is compressed by taking the logarithm of these values. The resulting features are called log mel energies in the literature. This acoustic representation retains the coarse shape of the spectrum, while the fine structure related to the harmonic structure of the signal is smoothed out. This is beneficial because the identity of everyday sounds is not determined based on the exact perceived pitch, and some of the sources do not even have any harmonic structure. The process is shown in Figure 3.7; panel b shows the spectrogram of the signal, and panel c shows corresponding mel-band energies.

The mel-scale filterbank was originally designed for speech analysis, and to be used as a processing block for MFCCs features. Later, as MFCCs were shown to perform robustly in more general sound classification tasks such as speaker recognition [Reynolds1995], music genre recognition [Tzanetakis2002], and musical instrument classification [Heittola2009], they gained popularity also as a standard feature for acoustic scene classification and sound event detection tasks. MFCCs are calculated by decorrelating the outputs of the mel-scale filterbank with a linear transform, the discrete cosine transform (DCT) to allow efficient data modeling in Gaussian mixture models and hidden Markov models by enabling the usage of a diagonal covariance matrix in Gaussian distributions. Along with the emergence of deep learning based approaches the decorrelation step has been dropped out because modern deep learning techniques can efficiently take advantage of correlated information in the data during the learning process. Currently, log mel energies are the most commonly used acoustic features in deep learning based approaches for the content analysis of environmental audio [Cakir2017, Mesaros2018b, Mesaros2019a].

Mel-frequency cepstral coefficients

Mel-frequency cepstral coefficients (MFCC) represent the output of the mel-scale filterbank in the cepstral domain. MFCCs are obtained from the previously described log mel energies by computing type-II discrete cosine transform (DCT). An example of MFCCs is shown in Figure 3.7, panel c.

The role of this added processing step is two-fold. Firstly, the DCT is used to decorrelate feature values, since the filterbank outputs are heavily correlated due to neighboring filters overlapping in frequency. Decorrelated values enable usage of a diagonal covariance matrix in Gaussian distribution based acoustic models. Secondly, by keeping only the first few values of DCT (coefficients), the spectral representation is smoothed and the dimension of the feature vector is reduced. The first coefficients contain information about the overall shape of the spectrum, while higher coefficients contain information about the fine structure of the spectrum. The amount of coefficients retained for analysis varies between 12-20 depending on the target application requirements; in speech recognition 8-12 coefficients are sufficient to represent the coarse shape of the spectrum; in musical instrument recognition usually a higher number of coefficients (e.g. 20) are used to capture fine details of the spectrum [Heittola2009], while in environmental audio usually 16-20 coefficients are used . The first MFCC (zeroth coefficient) is related to signal energy (log energy), and depending on the application this information is either retained or omitted from the feature vector. Signal energy is closely related to acoustic scene class, e.g. park is quieter than a street with cars, and because of this, the signal energy information is retained in acoustic scene classification applications. Sound events can be regarded as having the same source regardless of loudness, and thus the zeroth coefficient is often omitted from the feature vector in sound event detection applications.

Dynamic features

The audio is a time-variant signal and one of the main characteristics for sound identification is its dynamic change over time. However, many acoustic features such as mel energies and MFCCs, estimate only the instantaneous spectral shape. The temporal evolution of the acoustic features can be dealt at the feature level by adding dynamic features to the final feature vector or by modeling the temporal aspect in the acoustic modeling stage (e.g. using recurrent layers in neural networks) [Cakir2017].

To incorporate the temporal evolution of the features into the acoustic feature vector, one can use estimates of the local time derivatives of the features by approximating with a first-order orthogonal polynomial fit [Furui1981] as

$$ \Delta c\left(i,u\right)=\frac{\sum_{k=-K}^{K}k\cdot c\left(i,u+k\right)}{\sum_{k=-K}^{K}k^{2}} \tag{3.4} $$

where \(c\left(i,u\right)\) denotes the \(i^{th}\) feature value in a time frame \(u\) [Rabiner1993, p. 116–117]. Computation is performed over \((2K+1)\) frames, and \(K\) is typically set to either three or four. The resulting dynamic features are commonly referred to as delta features, opposite to the static features. In addition to the delta features, the second derivatives can be computed by applying the same polynomial fit to the already computed delta features, resulting in a parabolic fit. These features are referred to as delta-delta or acceleration features. Delta and acceleration features are used together with static features to integrate the dynamic aspect of the spectrum to the feature set . An example of the static and dynamic features is shown in Figure 3.9.

Another method to incorporate temporal information into the feature vector is to concatenate consecutive frames within a window into a single vector, a supervector, [Gencoglu2014, Cakir2015a, Mesaros2019a]. The idea is to provide contextual information by stacking together \(N_{win}\) frames before and after the current frame. The length of the newly constructed feature vector is defined as \((2\times N_{win} + 1) \times N_{feat}\), where \(N_{feat}\) denotes the length of the original feature vector. This method is often used together with fully connected feed-forward neural networks.

Other features

In addition to the previously discussed features, numerous other features derived from the time-frequency representation of a signal have been proposed for computational audio content analysis.

Low-level features describing specific aspects of the spectral shape of a signal are traditionally used as part of a larger feature set together with MFCCs as they are not powerful enough to be used alone. Common low-level features that describe spectral shape include signal energy, spectral envelope, spectral moments (e.g. spectral centroid and flatness), spectral slope, spectral roll-off, spectral flux, and spectral irregularity [Eronen2006, Geiger2013].

Spectral descriptors adapted from image processing have shown competitive performance compared with traditional features, especially in the case of acoustic scene classification. These descriptors are treating the spectrogram as an image and use techniques adapted from computer vision to characterize the shape, texture, and evolution of the content in it. To detect different shapes in the spectrogram, one can use a histogram of oriented gradients (HOG) features [Rakotomamonjy2015], and to characterize textures in the spectrogram one can use local binary pattern (LBP) features [Kobayashi2014, Battaglino2015]. Subband power distribution (SPD) transforms the spectrogram into representation characterizing the spectral power distribution over time at frequency subbands [Dennis2013].

Automatic representation and feature learning techniques have become recently increasingly popular for acoustic scene classification and sound event detection facilitated by the availability of larger high-quality datasets. Sounds occurring in everyday environments have substantial diversity and this results in a widely varying set of time-frequency structures, however, only a subset of the information in the spectrogram is relevant for actual classification. The aim of feature learning is to learn a representation that reflects this relevant information, and is accomplished using different techniques, including deep learning [Hershey2017]. Features created through this process are commonly referred to as embeddings. The input for the feature learning network can be some established time-frequency representation or even raw acoustic waveform [Kim2019].

Classification

In sound classification, class-presence information of multiple short analysis frames from an audio segment to be classified are combined into a single classification output. In a single-label classification task, a single class label is assigned to an item, whereas in multi-label classification, multiple category labels are assigned to a single item.

Frame-level class-presence probabilities can be processed into a single-label classification output either by taking classification decisions at frame-level and combining decisions (hard voting) or by combining probabilities and making the decision at segment-level (soft voting). In the first approach, classification decisions are done at the frame-level by selecting the class with the highest probability, and performing majority voting over these frame-level estimates to get the class with the highest number of occurrences [Mesaros2019a]. In the latter approach, the class-presence probabilities are combined either by summing or by averaging, then the class with the highest probability is selected as the classification result [Mesaros2018c]. This approach often results in a higher classification accuracy than hard voting at frame-level, because it gives more weight to more confident estimates.

In multi-label classification, the number of active classes is usually unknown, and one cannot do classification just by selecting the class with the highest probability. The single-label classification scheme can be extended into a multi-label classification either by modifying how the system output is interpreted into activity or by adding extra classes to capture cases when target sound classes are not active. In the first approach, class presence probabilities from frames are summed or averaged similarly to the soft voting scheme, and a probability threshold is used to select the active classes. Probability thresholding to get activity estimates is called binarization and it is a common procedure with neural networks [Cakir2017, Mesaros2019a]. In the second approach, the non-activity of the class is explicitly modeled by introducing extra classes. One extra class, universal background model (UBM), can be added to represent the case when no target sound is active [Heittola2013a]. Another way to model non-activity with extra classes is to create class-wise classifiers such that each of these classifiers is able to recognize only the activity of that particular sound class [Cakir2015b]. Binary classifiers are trained with two classes: a positive class, when the specified sound is active, and a negative class when the specified sound is not active.

Detection

In sound event detection, class presence probabilities of consecutive analysis frames are converted into sound activity information, onset and offset timestamps of the active sound events. The process is a multi-class multi-label classification task at frame-level, and the aim is to process the classification output into estimates of sound event activity. The process is illustrated in Figure 3.12.

Generally, sound events are much longer than the used analysis window (typical length 20-100 ms) making classification outputs of consecutive analysis frames correlated. Furthermore, everyday sounds have a characteristic temporal structure which is important for the recognition, and this structure spans several analysis frames. The temporal structure is taken into account in the detection stage by using temporal post-processing applied to the frame-wise class presence probabilities or the binary class presence estimates. The classification at frame level produces noisy results because feature distributions of sound classes overlap each other and short frames do not have enough information for robust classification. Median filtering using a sliding window can be used to filter out this classification noise and to smooth the results from consecutive frames [Mesaros2018c, Mesaros2019a]. In this process, the median of binary sound activity estimates within the processing window (e.g. one second worth of analysis frames) is outputted, and the window is shifted one frame forward. The temporal structure of a sound can be addressed in the acoustic model as well by incorporating contextual information in the classification, e.g. using recurrent layers in neural networks [Cakir2017], which makes post-processing in such approaches less important.

Gaussian Mixture Model

The Gaussian mixture model is a probabilistic model that can be used to estimate normally distributed sub-populations within the training data in an unsupervised manner. It can be extended into a multi-class supervised classifier by fitting a separate GMM model for training data coming from a specific sound category. In the test stage, class-wise likelihoods (posterior probability \(p(y|x)\) are evaluated for an observation, and the class giving maximum likelihood is selected. In sound classification, sub-populations modeled with the mixture model can be seen as variations in acoustic characteristics within a sound category. For example, in the case of a dog barking sound class, the sub-populations can be produced by small, medium, and large dogs with various types of barking as well as different portions of the barking sequence itself.

Before the recent emergence of deep learning, GMM was one of the most commonly used classification methods in sound classification tasks because of its good generalization properties. In automatic speech recognition, it has been used to model individual phonemes, while the transitions from one phoneme to another were modeled with hidden Markov models [Rabiner1993]. In speaker verification, a GMM representing all speakers, a universal background model, was adapted to model the target speaker, with the actual speaker verification done based on the likelihood ratio between the target model and UBM [Reynolds1995]. In music information retrieval, GMMs have been used to classify, for example, musical genre [Tzanetakis2002] and musical instruments [Heittola2009].

A GMM estimates the underlying probability density function (pdf) of the observations (acoustic features), and it is capable of representing arbitrarily shaped densities through a weighted mixture of \(N\) multivariate Gaussian distributions (also called normal distributions) [Pearson1893, Duda1973]. The model sums together multiple Gaussian distributions (components), and the whole model is parameterized by the mixture component weights, and the mean and variance of the individual component distributions. The model can be seen as a clustering algorithm with soft assignments, where each modeled data point could have been generated by any of the component distributions used in the model with a corresponding probability, i.e., each mixture component has some responsibility for generating a data point. The probability density for an observation \(\boldsymbol{x}\) from a class \(k\) is computed as

$$ \label{ch3:eq:GMM:mixture} p_{k}(\boldsymbol{x}) = \sum_{n=1}^{N}\omega_{n}\mathcal{N}(\boldsymbol{x} ; \boldsymbol{\mu}_{n},\boldsymbol{\Sigma}_{n}) \tag{3.5} $$

where \(\omega_{n}\) is the positive weight of the component \(n\) and the normal distribution \(\mathcal{N}(\boldsymbol{x} | \boldsymbol{\mu}_{n},\boldsymbol{\Sigma}_{n})\) is defined by the the mean vector \(\boldsymbol{\mu}_{n}\) and the covariance matrix \(\boldsymbol{\Sigma}_{n}\). Component weights \(\omega_{n}\) sum to unity. Figure 3.13 illustrates the basic principle of the mixture model in the one-variate case. The multivariate Gaussian density function is defined as

$$ %\mathcal{N}(\vect{x} | \mu_{n},\Sigma_{n}) %= \frac{1}{\sqrt{(2\pi)^{d}|\Sigma_{n}}|}e^{-\frac{1}{2}(\vect{x}-\mu_{n})^{T}\sum_{n}^{-1}(\vect{x}-\mu_{n})} \mathcal{N}(\boldsymbol{x} ; \boldsymbol{\mu},\boldsymbol{\Sigma}) = \frac{1}{\sqrt{(2\pi)^{d}|\boldsymbol{\Sigma}}|}e^{-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^{T}\sum^{-1}(\boldsymbol{x}-\boldsymbol{\mu})} \tag{3.6} $$

where \(d\) corresponds to the length of the feature vector.

Model parameters \(\omega_{n}\), \(\boldsymbol{\mu}_{n}\), \(\boldsymbol{\Sigma}_{n}\) in Eq 3.5 are learned based on the training material for each class separately to allow multi-class classification. The training is implemented using the expectation-maximization (EM) algorithm where optimal model parameters are iteratively estimated [Dempster1977, McLachlan2008]. A latent variable \(\gamma\) is introduced for each data point to indicate the responsibility of each mixture component for generating a particular data point. The EM algorithm alternates between an expectation step and a maximization step: the first estimating the latent variable, the second updating the model parameters to maximize the likelihood based on the estimated latent variable. The process is often initialized by clustering the data with the k-means algorithm. The number of mixture components \(N\) can be optimized as a hyper-parameter using cross-validation, or it can be optimized during the training process as one of the model parameters using information criteria [Figueiredo1999, Heittola2014]. Depending on the level of intra-class variation and the amount of training data, the best performing \(N\) can also vary across classes.

The parameter optimization process is commonly simplified by using diagonal covariance matrices \(\boldsymbol{\Sigma}_{n}\) to restrict the parameter space. This simplifies the calculations, but at the same time prevents the model from capturing correlations between variables. However, the models trained on decorrelated features perform well in practice. One can use decorrelated acoustic features such as MFCC or use e.g. principal component analysis to decorrelate other acoustic features before using them with a classifier.

In the inference stage, the likelihood of an observation to be generated from each class-wise model is estimated using Eq 3.5. The predicted class is selected based on the maximum likelihood classification principle. As discussed earlier, one can use either hard voting or soft voting to get a single-label classification output for an audio segment containing many analysis frames. With GMMs the usual approach is based on the soft voting scheme, where frame-level likelihoods are accumulated class-wise over the whole segment and the predicted class is selected based on these accumulated likelihoods. The aim is to find class \(k\) which has highest likelihood \(L\) for the set of observations \(X=\boldsymbol{x_1},\boldsymbol{x_2},\cdots,\boldsymbol{x_M}\):

$$ \label{ch3:eq:GMM:classification} L(X;\lambda_{k})=\prod_{m=1}^{M}p_{k}(\boldsymbol{x_{m}}) \tag{3.7} $$

where \(\lambda_{k}\) denotes GMM for class \(k\) and \(p_{k}(\boldsymbol{x_{m}})\) is the probability density function value for observation \(\boldsymbol{x_{m}}\). The general assumption in this approach is that consecutive analysis frames are statistically independent.

Hidden Markov Model

The hidden Markov model is a probabilistic temporal model used to model sequential data such as a sequence of acoustic features [Rabiner1993, p. 321]. HMM relies on a few assumptions: data can be divided into a sequence of stationary time segments called states, transitions between the states depend only on the origin and destination (Markov property), and the probability of an observation to be produced by a state does not depend on previous observations.

An HMM is composed of states that respect the previous assumptions. Information on which state produced the output is not observable directly, i.e., the state information is hidden. The HMM is defined by the initial state probability distribution \(\Pi\), state transition probabilities, and output distributions of the states. The probability of being in state \(i\) at the beginning of the process is defined by the initial state probability distribution, \(\boldsymbol{\Pi}=\left [ \pi_1, \dotsc, \pi_i, \dotsc, \pi_N \right ]\). Transition probability from state :\(i\) to state \(j\) is defined by \(a_{ij}\), where \(i,j=1,\dotsc,N\) and \(N\) is the number of states in the HMM. Transition probabilities are presented as \(N \times N\) matrix \(\boldsymbol{A}\). The observations are the outputs of the states, and the probability of observation \(x\) in state \(j\) is defined by \(b_j\left ( x \right )\), with parameters of the state distributions collected in \(N \times M\) matrix \(\boldsymbol{B}\), where \(M\) is the number of possible observation symbols. The state output probabilities are based on GMMs, and determined by static and dynamic features [Rabiner1993]. Dynamic features are used to represent the short-time context in HMM, and this can be seen as a heuristic approach to compensate for the assumption of observations being independent [Gales2008].

Before the era of deep learning, the standard approach for automatic speech recognition was to model individual acoustic units in speech (phones) with GMMs and the temporal sequence of these units with HMMs [Rabiner1993]. Following the success with speech recognition, HMMs have been successfully applied to many classification and detection tasks in the field of music information retrieval [Casey2008] and content analysis of everyday environments. As the time-varying properties of sound are important information for sound identification, the HMMs with temporal modeling abilities have produced good results in many works related to general sound recognition [Casey2001, Eronen2006, Weninger2011, Butko2011b] and sound event detection [Mesaros2010, Heittola2013a, Heittola2011, Heittola2013b].

The topology of the HMM model is defined by the non-zero values in the state transition probability matrix \(\boldsymbol{A}\) and it can be represented as a graph having connections \(a_{ij}\). The default topology, a fully-connected topology, has all transitions enabled, however, domain knowledge can be used to select the optimal topology for a task by setting some state transitions to zero before model training. The often-used left-to-right topology allows only transitions from one state to itself or the next state, the topology being well-suited to model sequential processes that have a clear start, steady part, and end. Such topology is used to model acoustic units in speech, notes played with musical instruments, and everyday sounds. In the case of everyday sounds, hidden states can be seen as modeling different stages of an individual sound event. For example, the sound of a glass cup smashing onto the floor and breaking could be divided into the following three parts suitable for a left-to-right model topology: transient-like sound from the impact, sound produced by a high amount of debris moving on the floor for a short distance, and lastly sound produced by a lower amount of debris moving longer distances on the floor surface. An example of the HMM model represented as a graph can be seen in Figure 3.14.

The HMM model is trained using a special case of the EM algorithm, the Baum-Welch re-estimation algorithm [Rabiner1989]. During the training procedure, the model parameters are iteratively re-estimated while computing the probability that the observed sequence (training data) was produced by the current model. When using HMM for a classification task, each class is represented by a separate HMM, and classification is done with the maximum-likelihood method by finding the model having maximum aposteriori probability for the given observation sequence. Aposteriori probability can be computed using the Viterbi algorithm to find the optimal state sequence, a single best path through the model which provides the highest total likelihood of specific observation sequence being produced by this particular state sequence [Viterbi1967]. In the case of a detection task, like sound event detection, transitions between classes are also important for robust detection. In order to do detection, class-wise HMM models can be merged into a single large HMM and allowing transitions between classes only from certain states. Viterbi algorithm is then used to find the optimal path through this large HMM, and the detection output is produced based on the found optimal state sequence and the sequence of classes these states belong to.

Deep learning

The state-of-the-art computational audio content analysis systems are nowadays almost without exception based on deep learning approaches, and they are showing exceptional performance over classical machine learning approaches. The work included in this thesis was carried out before the era of deep learning and does not utilize any deep learning. However, a brief introduction to deep learning is given here to provide the reader with a modern perspective.

The data processing in deep learning is inspired by information processing in the human brain during the process of acquiring new knowledge and the process of recalling stored knowledge. The main idea of the data processing in deep learning is to model complex concepts, such as sound event classes, by combining simpler and more abstract concepts learned automatically from data. The modeling is done using an artificial neural network, a deep multilayered structure, where each layer is constructed from multiple artificial neurons, and neurons between layers are connected. Deep learning is used for classification as a discriminative learning approach, and robust modelling is ensured by using a large amount of learning examples.

An artificial neuron is an elementary unit of artificial neural networks. A neuron has \(n\) inputs and each input has a weight parameter \(w_i\). The weighted sum of the inputs summed with a constant bias factor \(b\) is passed through an activation function to produce the neuron’s output. By using a non-linear activation function, the neural network can model highly complex relationships in the data. A classical example of an artificial neural network is the feedforward neural network (FNN) where information between layers flows only from the input towards the output without feedback connections [Goodfellow2016]. The basic inner-structure of an artificial neuron and the feedforward network structure is shown in Figure 3.15.

A network consists of an input layer, an output layer, and a number of intermediate layers called hidden layers for which the training data does not provide desired outputs. During the learning process, parameters \(w_i\) and \(b\) for each neuron in the network are iteratively optimized to produce the desired target outputs (reference labels) at the output layer given the input data (acoustic features) to the network. The learning process uses the back-propagation algorithm; at each learning iteration, a loss function is calculated between the output of the network and the target output for the learning examples and this loss is fed back through the network and the network parameters are adjusted to lower the loss for the next iteration. The non-linear optimization method called gradient descent is used to update the network parameters in the opposite direction of the gradient of the loss function after a small set of learning examples (a batch). The activation function for neurons in the output layer is selected based on the classification task; for a single-label classification, softmax activation is commonly used, whereas for a multi-label classification sigmoid activation is used. The hidden layers often use rectified linear units (ReLU) [Glorot2011] as activation function.

The feedforward neural network was the first deep learning approach successfully used for audio content analysis applications, and it showed clear improvement in performance over the established GMM and HMM-based approaches, for example, in acoustic scene classification [Mesaros2019b] and sound event detection [Gencoglu2014, Cakir2015a, Mesaros2019b]. Short temporal context can be taken into account in FNN-based systems by using context windowing at network input (see Section Dynamic Features). Two main problems with FNNs for audio content analysis are their sensitivity to variations in time and frequency due to fixed connections between the input and the hidden layers of the network, and their inability to model long temporal structures essential for the recognition of some sound events.

Convolutional neural networks (CNN) are nowadays chosen over FNNs as they generally produce more robust models [Mesaros2018b, Mesaros2019b]. CNN is a time and frequency shift-invariant model designed to take advantage of the 2D structure of the input [Lecun1998]. This is achieved by utilizing the local connectivity inside the network to allow parts of the network to specialize in different high-level features of the data during the learning process. The final classification output is produced based on the learned high-level features using a few feedforward layers before the output layer. Convolutional recurrent neural networks (CRNN) are often used for sound event detection [Cakir2017, Adavanne2017, Xu2018]. These networks resemble CNNs, but they add layers containing feedback connections that capture long temporal structures in the data.