MSc in Data Science & Artificial Intelligence.

The session introduced calculus as a foundation for optimisation, machine learning and statistical modelling. It defined functions, linear and affine functions, and common nonlinear functions including powers, exponentials and logarithms. Derivatives were explained as local slopes, with rules for sums, constants, powers, products, quotients and composite functions, alongside links to neural-network differentiation. The class then extended derivatives to several variables through partial derivatives, higher-order and mixed partial derivatives, and their use in describing slopes in different directions. Taylor series and Taylor polynomials were presented as local approximations, with applications to numerical computation, optimisation methods and interpolation.

The session began with a recap of the product rule for derivatives, using a rectangle-area argument and the limit definition to show why the small product of two variations vanishes. It then introduced vectors as one-dimensional arrays, covering vector addition, scalar multiplication, linear combinations, and the importance of matching dimensions in mathematical and programming operations. Linear independence, dependence, collinearity and bases were explained as ways to identify redundant variables and reduce the dimensionality of data. The dot product, orthogonality, vector norms, normalisation and orthonormal bases were developed, with links to correlation, variance, regression and numerical scaling. The class also introduced changes of basis, including their role in principal component analysis (PCA), compression and viewing data from more informative directions.

The session examined matrix inverses, showing that a square matrix is invertible exactly when its columns are linearly independent, equivalently when it has full rank. It introduced matrix rank, numerical sensitivity near singular matrices, and the special simplicity of diagonal matrices, whose inverses and determinants can be computed element by element. Determinants were defined and calculated for two-by-two and three-by-three matrices using minors and cofactors, with discussion of why this recursive method is impractical for large matrices. The class then introduced trace, eigenvalues and eigenvectors for symmetric matrices, deriving the characteristic equation and using orthonormal eigenvectors to diagonalise a matrix through a change of basis.

The class examined how eigenvalues and eigenvectors in data analysis can reveal approximate linear relationships and support linear regression, including the effect of scaling variables and interpreting small eigenvalues as residual error. It covered adding an intercept through centring variables or including a column of ones, comparing resulting coefficients with a statistical regression model, and avoiding overfitting by selecting simpler models and validating them. The session also explained how changing to eigenvector coordinates can reveal trends in data that are not visible when plotting individual variables, linking this to principal component analysis. It then introduced positive-definite matrices and quadratic forms through the signs of eigenvalues, with applications to convexity and optimisation. Finally, it introduced complex eigenvalues of non-symmetric matrices, complex-number arithmetic, Cartesian and polar forms, and the use of complex exponentials in trigonometry and Fourier transforms.

The session introduced the course’s focus on probability, statistics and their role as foundations for machine learning, with R and RStudio as supporting tools. It reviewed mathematical notation for sums, products and sets, including unions, intersections and complements. Descriptive statistics were covered through frequency tables, bar charts, histograms, density plots and empirical cumulative distribution functions, with emphasis on interpreting distribution shape. The class defined quantiles, the median, mean, variance, standard deviation, interquartile range, skewness and kurtosis, and used box plots to examine spread and potential outliers. It then introduced multivariate descriptive analysis using contingency tables, scatter plots, covariance and correlation, stressing that correlation does not establish causation.

The session completed an introduction to probability distributions for finite, countably infinite and continuous sample spaces. It covered conditional probability, Bayes’ theorem and base rates through diagnostic-test, screening, production-defect and transmission examples, as well as the Monty Hall problem. Independence, incompatibility and mutual independence were distinguished using coin-toss and permutation examples. Students then completed practice multiple-choice questions on descriptive statistics, including means, medians, empirical cumulative distribution functions and probability calculations. The session ended by introducing random variables as functions mapping outcomes to real-number values and motivating probability distributions for them.

The session reviewed random variables through uniform, binomial, Poisson and normal distributions, focusing on their parameters, probability mass or density functions, cumulative probabilities, quantiles, means and variances. Simulated samples were compared with theoretical distributions to show how observed data can differ from ideal probability models. The class then introduced multivariate random variables, joint and marginal distributions, conditional distributions, independence, covariance and correlation, including a bivariate normal simulation and covariance matrix. It concluded with an introduction to convergence, random samples that are independent and identically distributed, parameter estimation, and the motivation for the law of large numbers and central limit theorem.

The session reviewed descriptive statistics, probability theory, random variables, probability distributions, convergence, the law of large numbers and the central limit theorem. It introduced estimators and estimations, including bias, variance, mean squared error and the use of large samples to improve estimates. Confidence intervals were explained as ranges around an estimate associated with a chosen risk level, and a practical activity used one-sample t-tests in R to calculate 95% and 99% confidence intervals for mean height. Statistical hypothesis testing was then introduced through null and alternative hypotheses, rejection regions, p-values, significance levels, and Type I and Type II errors, with examples involving coin tosses and comparisons of group means. The session concluded by outlining common tests for one mean, two means and independence, in preparation for further study.

The session introduced the simple linear model as a way to model one numerical response using one numerical explanatory variable, and explained why estimation theory, confidence intervals and statistical tests are needed to assess such models. It covered continuous random variables and density functions, including positivity, normalisation, support, probabilities as areas under a density, expectations, variances, moments and distribution functions. Worked examples showed how to verify a density, calculate expectation and variance, and derive a distribution function. The class also demonstrated how to find the distribution and density of a transformed random variable using distribution functions and differentiation. Finally, it introduced uniform, exponential and Gaussian random variables, focusing on the uniform distribution, its properties, transformations from a uniform distribution on [0,1], and inverse-distribution simulation.

The session explained how to use standard Gaussian distribution tables to calculate probabilities, including interval probabilities, complementary events and symmetry for negative values. It then showed how to standardise a non-standard Gaussian variable using its mean and standard deviation before applying the standard table, and compared table-based approximations with calculations in R. R functions for densities, distribution functions, quantiles and random simulation were introduced, including the use of a seed to reproduce simulated results. The class then began statistical representation of data, covering independent and identically distributed samples, population representativeness and sampling bias, and distinguishing qualitative, discrete quantitative and continuous quantitative observations. Pie charts, bar plots and cumulative proportion curves were introduced, with proportions presented as estimates of probabilities and simulations used to show that larger samples give more accurate estimates.

The session introduced point estimators as functions of a random sample, distinguishing an estimator (a random variable) from its numerical estimate based on observed data. It explained unbiased and asymptotically unbiased estimators through examples for a uniform distribution, including the sample mean and sample maximum, and showed how to adjust a biased maximum-based estimator. The law of large numbers and convergence in probability were introduced to explain why empirical averages approach expectations and why larger samples generally produce more accurate estimates. Simulations and box plots were used to illustrate estimator variability, the effect of sample size, and the interpretation of potential outliers. Consistency and quadratic error were defined as criteria for assessing and comparing estimators.

The class reviewed point estimation using the method of moments and maximum likelihood, including the maximum likelihood estimator for the upper bound of a uniform distribution. It compared estimators using mean squared error, variance and bias, and summarised common estimators for uniform, exponential and Bernoulli models. The session introduced confidence intervals as random intervals with a specified confidence level, explaining why they provide information that point estimates alone cannot. A confidence interval for a Gaussian mean with known variance was derived by standardising the sample mean and using standard normal quantiles, including symmetric and one-sided intervals. Simulations illustrated coverage, the possibility that an interval misses the true parameter, and how interval width depends on sample size, variability and confidence level; sample-size planning and limitations of large datasets were also discussed.

The session developed asymptotic confidence intervals for a population mean when neither the distribution nor the variance is known, using the central limit theorem, consistency of the sample variance and Slutsky’s lemma. It then applied these ideas to confidence intervals for a Bernoulli proportion, including a simpler conservative interval based on the bound p(1 − p) ≤ 1/4. Confidence intervals for the parameter of an exponential distribution were derived by first estimating its mean, with the delta method introduced for functions of an expectation. The class also derived an exact confidence interval for a Gaussian variance using the chi-squared distribution, and introduced the Fisher distribution to compare the variances of two independent Gaussian samples. Examples illustrated how confidence intervals can be used to assess equality of variances and how sample size affects conclusions.

The session developed the simple linear regression model with a numerical response and explanatory variable, using a fixed-design approach and assumptions on the error terms. It explained least-squares estimation of the intercept and slope, deriving the fitted coefficients and relating the slope to empirical covariance and variance. Covariance, random vectors, independence, marginal distributions and Gaussian random vectors were introduced to support the regression theory. Maximum likelihood estimation under Gaussian errors was used to estimate the error variance, leading to residuals, fitted values and a Student-t confidence interval for the slope. Software was used to fit models, inspect coefficients, residuals and fitted values, and calculate confidence intervals.

The session introduced time series as regularly spaced observations indexed by time, with the aim of forecasting future values. It used plotted examples to distinguish seasonal patterns with fixed frequency, cycles with variable frequency, trends, changing seasonal amplitude, and the role of external information in forecasting. Students were shown how to create and plot time-series objects in R, including specifying frequency and start dates, producing seasonal plots, and treating missing values through interpolation. The class covered descriptive statistics, autocorrelation and partial autocorrelation plots, how these can indicate trends or seasonality, and tests for whether autocorrelation is present. Students practised loading a dataset, constructing a time series, identifying its patterns, and aggregating monthly observations into annual totals.

The session reviewed exponential smoothing for forecasting time series with trend and seasonality, and time-series cross-validation for comparing forecasting methods using rolling training periods. It introduced additive decomposition into trend, seasonal pattern and residual noise, noting that multiplicative series can be made additive through a logarithmic transformation. Moving averages and the decomposition function were used to estimate and remove trend and seasonality for exploratory analysis, while their limitations for extrapolating trends were explained. Differencing operators were introduced to remove polynomial trends and seasonal patterns, with seasonal differencing followed by first differencing used to obtain approximately stationary series. Students assessed residual autocorrelation using autocorrelation plots and Box tests, then practised this on air-passenger and Google stock-price data.

The session introduced backshift notation and used it to express autoregressive integrated moving average (ARIMA) models, including how ordinary differencing converts a non-stationary series into one that can be modelled with autoregressive and moving-average terms. Seasonal ARIMA (SARIMA) models were explained as combining regular and seasonal dependence, seasonal differencing, and lagged relationships at multiples of the seasonal period. Students practised identifying candidate SARIMA orders from autocorrelation and partial autocorrelation plots, fitting models, checking residual independence, comparing models using the Akaike information criterion (AIC), and producing forecasts for retail-index and San Francisco precipitation data. The class also covered heteroscedasticity and the Box–Cox transformation, including estimating the transformation using training data and applying the resulting model to forecasts on the original scale. A final exercise asked students to select a SARIMA model for airline-passenger data using Box–Cox transformation, differencing, autocorrelation diagnostics, and train-test evaluation.

The session reviewed forecasting methods for univariate time series, including exponential smoothing, ARIMA models and machine-learning approaches using lagged observations. It stressed that methods should be compared through time-series cross-validation rather than a single training and test split, with the test horizon matching the required forecasting horizon. The class introduced multivariate time series forecasting using known future covariates, fitting time-series linear models with trend and seasonal effects and then modelling correlated residuals with ARIMA to form dynamic regression models. An electricity-demand example used temperature, squared temperature and working-day status, showing how nonlinear covariate effects and seasonal information can improve forecasts. Students practised splitting the data, fitting models, diagnosing residual autocorrelation, and comparing dynamic regression, neural-network and random-forest forecasts.

The session introduced SAS Base Programming, its role in accessing, managing, analysing and presenting data, and its relevance to certification and data preparation. It explained that SAS programs consist mainly of DATA steps, which typically create data sets, and PROC steps, which process data or produce output. Students practised setting up course data, using SAS interfaces, submitting code, and checking the log and results to identify errors. The class covered SAS syntax, including semicolons, keywords, formatting, comments, quotation marks and common syntax errors. It also introduced SAS data sets, libraries, metadata, PROC CONTENTS, PROC PRINT, PROC MEANS, variable types, missing values and SAS date values.

The session reviewed PROC PRINT, including selecting variables with a VAR statement and adding, checking and clearing titles and footnotes. It introduced SAS formats as a way to alter the display rather than the stored value, covering numeric, currency and date formats, widths, decimal places, and the importance of periods in FORMAT statements. Students created user-defined character and numeric formats with PROC FORMAT to translate codes, group salary ranges, handle other values and identify invalid or missing data. The class then began creating customised SAS data sets using DATA and SET statements, WHERE and subsetting IF conditions, assignment statements for calculated variables, DROP and KEEP statements, and permanent labels and formats. Exercises practised filtering observations, creating variables such as bonuses and discounts, and producing formatted reports.

The session covered importing delimited raw files into SAS, including delimiters, date informats, labels, formats, and the DSD and MISSOVER options for handling missing values. SAS functions were used to create and manipulate variables, including SUM, MONTH, YEAR, QUARTER, WEEKDAY, TODAY and MDY, alongside an explanation of the program data vector (PDV) and the effect of DROP, KEEP and LENGTH statements. Conditional processing with IF–THEN–ELSE and DO–END groups was practised to assign values based on job titles, countries, order types, birth quarters and ages. The class also covered concatenating data sets with SET, renaming variables to align structures, introductory one-to-one merging, and using PROC FREQ to inspect data, produce one- and two-way frequency tables, identify invalid, missing and duplicate values, and group values through formats.

The session reviewed solutions for creating and filtering SAS data sets, using WHERE statements, conditional OUTPUT statements, KEEP statements and PROC PRINT or PROC FREQ reports. It introduced accumulating totals in the PDV through RETAIN statements and sum statements, including the treatment of missing values, and showed how to summarise sorted data by BY groups using FIRST. and LAST. variables. Fixed-column raw files were read with formatted input, column pointers, informats, multiple input records and trailing @ controls. Character manipulation covered variable lists and functions including SUBSTR, LENGTH, PROPCASE, SCAN, CATX, FIND, TRANWRD, COMPRESS and LEFT. Numeric functions for rounding, truncation, descriptive statistics, medians and largest values were demonstrated through exercises.

The session introduced continuous optimisation as the minimisation of a cost function over decision or control variables, subject to geometric, equality, inequality and differential-equation constraints. It distinguished local from global minima, explaining why convex cost functions and convex feasible sets are particularly important. Examples included constrained linear optimisation for minimum-cost nutrition planning, least-squares solutions of inconsistent linear systems, and infinite-dimensional optimisation in the calculus of variations with regularisation to limit oscillations. The class then defined directional and Gateaux derivatives, compared Gateaux and Fréchet differentiability, related derivatives in finite-dimensional spaces to gradients, and derived the derivative of a quadratic cost function. Second-order directional derivatives and the Hessian were introduced, followed by the mean-value formula and the chain rule.

The session introduced necessary optimality conditions for differentiable cost functions, showing that an unconstrained local minimum in an open domain satisfies the Euler equation J′(u)=0. It explained open sets, local minima, critical points, and why boundary minima do not necessarily satisfy this equation. For optimisation over convex sets, it derived the variational inequality J′(u)(v−u)≥0 and applied it to vector and affine subspaces, projections onto convex sets, and least-squares problems. The convex case was then studied, showing that local minima are global, strictly convex functions have at most one minimum, and the Euler equation or variational inequality becomes sufficient for global optimality. Finally, quadratic cost functions and symmetric positive-definite linear systems were related to minimisation problems.

The session examined a non-convex two-variable function by finding its critical points, comparing their function values, and using Hessian eigenvalues to distinguish two minima from a saddle point. It explained first- and second-order optimality conditions, including Euler’s equation on open sets, Euler’s inequality on convex constraint sets, and how convexity makes first-order conditions sufficient as well as necessary. The class then introduced least-squares minimisation for inconsistent or non-unique linear systems, deriving the normal equations AᵀAx = Aᵀb from the gradient of the squared residual norm. It considered existence and uniqueness of least-squares and minimum-norm solutions, before introducing regularisation by adding a small squared-norm penalty to obtain a better-behaved, uniquely solvable optimisation problem.

The session defined norms on vector spaces and introduced saddle points of a Lagrangian as points that minimise with respect to the primal variable and maximise with respect to the multiplier variable. It established the min–max property of saddle points and linked saddle points of a Lagrangian to solutions of optimisation problems with inequality constraints. The class derived complementary slackness, explained active and inactive constraints, and interpreted Lagrange multipliers as constraint prices. It then presented the Karush–Kuhn–Tucker conditions for convex optimisation, including linear programming and problems with both equality and inequality constraints. An exercise began by computing the gradient and Hessian of a quadratic function in order to test convexity.

The session introduced unconstrained optimisation as an iterative process that seeks to reduce a cost function at every step. Gradient descent was derived using a first-order Taylor expansion, showing why the update direction is the negative gradient and why selecting an appropriate step size involves a trade-off between speed and reliability. Fixed-step and optimal-step gradient methods were discussed, including convergence conditions based on strong convexity and Lipschitz continuity of the gradient, and line-search approaches for choosing practical steps. Quadratic objectives were examined in detail, linking their minimisation to solving positive-definite linear systems and showing how eigenvalues and conditioning affect convergence. The class then formulated linear regression as a least-squares optimisation problem and began implementing its cost function and gradient in R using matrix operations.

The session introduced artificial intelligence as systems that perform tasks normally associated with human intelligence, distinguishing it from general artificial intelligence. It positioned machine learning as learning from data and deep learning as machine learning based on neural networks, while comparing neural networks with classical methods such as regression, random forests, boosting, clustering and support vector machines. Applications in computer vision, audio processing and natural language processing were explored, including image classification, object detection, segmentation, speech recognition, sentiment analysis and question answering. The machine-learning workflow was reviewed: defining objectives and success metrics, collecting and preparing data, selecting and evaluating models, deployment, monitoring and retraining in response to data drift. The class also covered labelled versus unlabelled data, the importance of balanced, shuffled and deduplicated datasets, and cautious use of sampling and data augmentation.

The session reviewed the relationship between artificial intelligence, machine learning, deep learning and neural networks, including the distinction between shallow and deep networks. It introduced PyTorch tensor handling, device selection, graphics processing unit (GPU) availability and moving data between the central processing unit (CPU) and GPU. Data preparation for neural networks was covered through numerical encoding, ordinal and one-hot encoding, and feature scaling using min-max scaling and standardisation. The training cycle was explained: feeding data in batches, forward propagation, loss calculation, backpropagation and parameter updates across epochs. The class examined mean squared error and binary cross-entropy losses, gradient descent, learning rates, vanishing gradients, and batch, stochastic and mini-batch strategies, before comparing momentum, Adagrad, RMSProp and Adam optimisers.

The session reviewed how neural-network models should generalise to unseen data and how datasets should be divided into training, validation and test sets. It covered epochs, mini-batches, loss functions, backpropagation, gradient descent, learning rates and optimisers, including momentum and Adam. The class examined underfitting, overfitting, the bias–variance trade-off, early stopping, model complexity, regularisation, and the importance of data quality and relevant features. It also discussed avoiding data leakage, maintaining comparable distributions across data splits through stratified sampling, and balancing groups where fairness requires it. Students then practised structuring a PyTorch workflow with datasets, data loaders, models, training loops and validation-loss logging in TensorBoard.

The session examined how training and validation curves can indicate overfitting, underfitting, unstable optimisation and the need for early stopping. It covered causes and remedies for poor generalisation, including improving data quality, increasing data or model capacity where appropriate, removing irrelevant features, and adjusting learning rate and batch size. Regularisation methods were explained, including L1 and L2 penalties, dropout, data augmentation and batch normalisation, alongside vanishing gradients and internal covariate shift. The class also discussed why deep networks learn hierarchical features, their advantages for complex unstructured data, and their limitations in computation, data requirements and interpretability. In practical work, students added batch normalisation, dropout and L2 weight decay to a PyTorch classifier, then used Optuna to tune hyperparameters and inspect the resulting trials.

The session introduced Rust as a systems programming language chosen for performance, memory safety, concurrency and use in data-intensive applications. It covered Rust tooling, including Cargo projects, crates, the Cargo.toml manifest, building and running programs, and configuring an editor with Rust analysis and debugging support. The class explained memory management through the stack and heap, including scope, allocation, fixed and dynamic data, strings and pointers. It also introduced immutable and mutable variables, ownership, moving values, cloning, borrowing through references, and basic control structures and functions.

The session examined Rust lifetimes, explaining how references must remain valid and how explicit lifetime annotations prevent dangling references at compile time. It covered Option and Result types for handling missing values and failures, including matching, unwrapping, default values and propagating errors with the question-mark operator. The class introduced structs for grouping related data, their instantiation and mutation, ownership implications when copying fields, and implementation blocks for adding methods and constructor-like functions. It then introduced traits as shared behaviour across types, using the Iterator trait and operations such as next, map, filter, take and collect in preparation for building a linked-list example.

The session reviewed Rust exercises on lifetimes and references, mutable borrowing in structs, generic functions, Option and Result types, iterators, collections, and filtering data. It compared core data structures such as vectors, double-ended queues, heaps, hash maps, hash sets, B-tree maps and sets, focusing on their operations, complexity and appropriate use cases. Advanced structures including graphs, trees and spatial trees were introduced, alongside graph traversal and shortest-path concepts. The class then covered modularity, separation of concerns, encapsulation, abstraction, decoupling, interfaces, reuse and testing, before beginning a modular Rust graph implementation using type aliases, structs and derived traits.

The session covered software testing in Rust, including the purposes and distinctions of unit, integration and property-based tests. It demonstrated how to write and run tests with Cargo, use assertions, test error conditions and private functions, mock external dependencies, organise tests, and keep tests independent and focused on behaviour. Practical examples applied unit and integration tests to graph structures and algorithms. The class also introduced benchmarking, code coverage, debugging and profiling, before reviewing core Rust principles such as ownership, borrowing, memory safety, modularity and error handling.

The session introduced C# as an object-oriented language used with the .NET platform and Common Language Runtime, including automatic memory management and just-in-time compilation. It covered Visual Studio project organisation, solutions, projects, namespaces, program entry points, primitive and reference types, operators, functions, conditions and loops. Practical examples demonstrated type conversion, integer division, user input validation with TryParse, and the differences between casting and parsing. The class also examined arrays, multidimensional and jagged arrays, and dynamic data structures including lists, linked lists, stacks and dictionaries. A merge sort example was introduced to illustrate divide-and-conquer sorting and its time complexity.

The session introduced object-oriented programming, defining objects as entities with state, behaviour and identity, and classes as blueprints containing attributes, methods and constructors. It covered encapsulation through public and private members, static members, the current-instance reference, properties, constructors and method overloading, using bank-account and fraction examples in C#. SOLID design principles were introduced, particularly single responsibility and open–closed principles, with discussion of maintainability, extensibility and testability. The class also introduced the Unified Modelling Language (UML), including use case diagrams for identifying actors and system requirements, and class diagrams for modelling attributes, methods, visibility and static members. Finally, it began explaining horizontal relationships between classes, including association and aggregation.

The session covered object-oriented relationships: inheritance through generalisation and specialisation, and interface realisation as a contract for shared capabilities. It explained polymorphism, abstract classes and methods, method overriding, virtual methods, protected access, base-class constructors, casting, and the difference between overriding at runtime and overloading at compile time. Examples modelled animals, vehicles, flying objects, database connectors, property ownership and transaction records. A practical C# example used abstract transaction classes, sales and refund specialisations, Language Integrated Query (LINQ)-based reporting, and interchangeable comma-separated values (CSV) and JavaScript Object Notation (JSON) data exporters through an interface. Students were set a guided UML-based exercise to build an Internet of Things (IoT) data pipeline using these concepts.

The session introduced model–view–controller (MVC) architecture, separating an application into models for data and logic, views for presentation, and controllers for handling input and coordinating the other components. A console-based user-management example was developed to demonstrate this separation, including input validation, menus, error handling and exception handling. The class then covered creational, structural and behavioural design patterns, focusing on singleton, factory, adapter and observer patterns, including their purposes and limitations. These ideas were applied to a sensor-data pipeline using a factory to convert database records into reading objects, a repository and Entity Framework with SQLite for data access, report generation, and CSV and JSON export. The example was extended with an observer-based event system to notify application observers when database, reporting and export actions occurred.

The session introduced the structure of the machine learning course, including its focus on exploratory data analysis, feature engineering and modelling within a complete machine learning pipeline. A group project was outlined in which students were asked to use book metadata to predict Goodreads ratings, clean and analyse imperfect data, compare at least two models, and deliver code, a simple application, a report, a video demonstration and a GitHub repository. The class emphasised that machine learning begins with a clearly defined problem, suitable data and an understanding of how a human would approach the task. It also reviewed Python foundations needed for machine learning, including built-in data structures, mutability and immutability, indexing, conditions, loops and functions.

The session reviewed pandas data frames, including their tabular structure, indexing, comma-separated values (CSV) import, and the use of head(), tail() and describe() for inspection. It introduced the Titanic data set as a continuing case study and defined the machine-learning task as predicting passenger survival from available passenger information. The class examined data quality issues such as missing ages, identified the target column and ground truth, and discussed class imbalance and its effect on model bias. It also interpreted summary statistics, distributions, standard deviation and outliers, considering when unusual cases should be retained, removed or contextualised through additional features.

The session reviewed exploratory data analysis and feature engineering using the Titanic data set, including handling missing values, removing unhelpful columns and considering bias when retaining scarce passenger profiles. It demonstrated how to extract structured titles from unstructured name text using string splitting, indexing, slicing, functions and DataFrame apply operations, then create and analyse a new title feature. The class examined how uneven category frequencies can limit the usefulness of a feature for machine-learning models. It also introduced data visualisation with Plotly, using wine-quality data to inspect distributions, relationships and outliers, housing data to map prices geographically, and stock and country data to show changes over time.

The session completed exploratory analysis and feature engineering for a Titanic survival prediction task. It examined missing data, gender, passenger class, age distributions, fares and outliers, using bar charts, pie charts, histograms, distribution plots and box plots to identify relationships with survival. The class discussed removing unhelpful columns, imputing missing ages using grouped averages, and preparing data so that it is numerical and suitable for machine learning. It introduced ordinal/label encoding and one-hot/vector encoding, including their trade-offs, the k−1 rule, and the encoding of binary and passenger-class variables. The session then introduced the distinction between training and testing data, explaining how models learn from labelled examples and are evaluated on unseen examples.

The session introduced machine learning operations (MLOps) as a culture of communication, automation and collaboration between development, data and operations teams. It compared waterfall and agile approaches to software delivery, emphasising frequent releases, feedback and the ability to respond to changing requirements. The class covered core development and operations (DevOps) concepts including service-level measures, semantic versioning, readable naming conventions and community coding standards such as PEP 8. It then introduced Git as a distributed version-control system, explaining repositories, commits, staging, pushing, pulling, branching, merging and merge conflicts through command-line demonstrations.

The session reviewed Agile and Waterfall project-management approaches, explaining why DevOps aligns with Agile through shorter development cycles, collaboration, adaptability and automation. Students practised the essential Git workflow for creating or cloning repositories, tracking changes, reviewing diffs, making conventional commits, and synchronising with remote repositories. The class introduced Python virtual environments and then focused on continuous testing, including the purpose of automated tests for reliable software delivery. It distinguished unit, functional, integration and end-to-end testing, emphasising high unit-test coverage, self-contained tests and avoiding slow or interdependent test cases. Test-driven development was presented as a cycle of writing a failing test, implementing the minimum code to pass it, then refactoring; students were assigned Python exercises using pytest to test input validation functions.

The session reviewed semantic versioning, explaining that patch releases fix bugs, minor releases add non-breaking changes, and major releases may include breaking changes. It also revisited Git commands for cloning an existing repository, initialising a new local project and pulling remote updates, alongside the distinctions between unit, functional, integration and end-to-end testing. Test-driven development was outlined as writing a failing test before implementing the function, then iterating through passing tests, new features and refactoring. The main topic was CI/CD: continuous integration frequently merges and tests changes, continuous delivery keeps software ready for release, and continuous deployment automatically makes delivered software available to users. Students were assigned a GitHub Actions activity to automate tests and linting across branches and pull-request events.

The session reviewed DevOps as a culture focused on automation, communication, agile delivery and shorter development cycles. It revisited semantic versioning, Git as a distributed version-control system, testing levels, test-driven development, CI/CD, and Docker containers versus virtual machines. The class then introduced container orchestration, explaining why Kubernetes is used to manage decoupled, scalable and resilient containerised services. Key Kubernetes concepts included clusters, control planes, nodes, pods, services, declarative versus imperative configuration, persistent storage and networking. Students were assigned a Minikube lab to practise deploying, exposing and scaling Kubernetes applications locally.

The session reviewed how MLOps differs from traditional DevOps, emphasising that changing data, alongside code, models and environments, must be managed throughout the machine-learning lifecycle. It covered data governance, security, privacy, versioning and the need to improve computational efficiency as model-training costs rise. Students continued practical work on data versioning and examined why large data files should not be versioned directly in Git, including repository-size limits, performance issues and repeated storage of binary-file changes. The class introduced end-to-end machine learning platforms, criteria for selecting them, and Kubeflow as a Kubernetes-based open-source platform supporting notebooks, pipelines, training, model serving and model registries. Students then began installing Kubeflow locally in preparation for platform exploration and machine learning pipeline labs.

The session introduced data and model testing within MLOps, showing how machine-learning systems require software, data, model and infrastructure tests across an end-to-end pipeline. It reviewed train, validation and test data splits, including cross-validation, and explained their role in avoiding data leakage, overfitting and misleading performance results. Model-testing approaches included baseline comparisons, invariance tests, directional-expectation tests and minimum-functionality tests. Data testing covered schema and null-value checks, outliers, text handling, feature-engineering validation and training-serving skew. Students completed labs using pytest, Great Expectations, data profiling, automated linear-regression testing and GitHub Actions-based continuous integration and delivery.

The session introduced experiment tracking in MLOps as a way to ensure reproducibility, support iteration and improve collaboration. It covered recording model parameters, training data, dataset splits, features, results, metrics, environments and dependency versions so that experiments can be compared and repeated reliably. The class considered the value of saving generated artefacts such as plots and visualisations, rather than recreating them, to reduce computational cost and simplify reporting. Students began a Kubeflow lab that involved creating a pod and pipeline to run and track machine-learning experiments, alongside discussion of choosing suitable open-source or proprietary tools.

The session covered monitoring in production MLOps systems, including standard operational measures such as service availability, resource use, network behaviour and hardware utilisation. It explained why machine learning pipelines also require monitoring of data quality, schema changes, missing or discontinued features, data outages, bias and performance across different user segments. The class distinguished data drift, where input-feature distributions change, from concept or target drift, where the relationship between inputs and outcomes or the business problem changes. It considered gradual and sudden causes of drift, how declining performance can trigger retraining or full model redevelopment, and the need for defined key performance indicators (KPIs) to balance remediation against computational and staffing costs. Students then worked on a practical Evidently notebook lab to detect data drift and deliberately induce model failure.

The session introduced relational databases, explaining how tabular data is organised into rows and columns and how tables are linked through primary and foreign keys. It contrasted relational and non-relational data models, using document and key-value structures to show why highly variable data may create sparse tables. The class explained client-server database architecture, including database servers, clients, web interfaces, back-end code, SQL queries, connection details and access permissions. It also compared proprietary and open-source database systems, and outlined the course case-study approach and SQL-based assessment.

The session covered restoring a SQL Server database from a backup file, including verifying the backup and using the database interface to inspect tables. It reviewed relational tables, primary and foreign keys, schemas, and the use of schemas for functional organisation and access permissions. Students wrote basic SQL queries using SELECT, FROM, USE and COUNT, and discussed why row counts should be checked before using SELECT * on an unfamiliar or very large table. The class also examined database collation, including case and accent sensitivity, and explored customer data to identify data-quality inconsistencies and infer how customer and billing identifiers were structured.

Self-joins compared product prices, including a query for non-USB products priced at least as highly as a USB product. Text filters, wildcard placement and case conversion affected which rows matched; DISTINCT addressed repeated results from multiple comparisons. An entity–relationship diagram then connected customers, orders and invoices through their keys. The class explored orders without invoices using an outer join with missing-value filtering and alternative subqueries. These examples led into invoice structure: identifiers, dates, line items, quantities, prices and totals. The queries were examined in their data context, without asserting that every alternative behaves identically for all missing values.

The session reviewed database normalisation, covering first, second, third and Boyce-Codd normal forms, functional dependencies, atomic values, composite keys and the risks of duplicated or dependent non-key data. Examples from a sample database were used to identify non-atomic JavaScript Object Notation (JSON)-style fields and third-normal-form breaches that could create inconsistent customer and invoice information. The class then restored and explored a second SQL Server database, checked table keys and case-sensitive collation, and used the graphical interface to add and edit a city column. Finally, it practised GROUP BY, COUNT and HAVING, then developed two approaches for finding customers who had bought every product, including a division query using nested NOT EXISTS conditions.

The session introduced data engineering on AWS, focusing on the five Vs of data: value, veracity, volume, velocity and variety. It explained data pipelines, including extract, transform and load (ETL) and extract, load and transform (ELT) approaches, structured and unstructured data, and the choice between batch and streaming processing. AWS ingestion services were covered, including AppFlow for software-as-a-service data, DataSync for file transfers, and Data Exchange for third-party data. The class examined AWS Glue for data catalogues, crawlers, ETL jobs, schema management, data quality and handling sensitive information. Streaming tools including Kinesis Data Firehose, Kinesis Data Streams, managed Apache Flink and Kafka services were compared, alongside data lakes, data warehouses, Lake Formation, Athena, Redshift and OpenSearch.

Disaster recovery planning connected acceptable data loss and recovery delay with storage, computing, databases and networks. Backups, snapshots and cross-region replication were compared, including why replication alone does not protect against corruption. The class reviewed storage migration, lifecycle management and shared file services, then considered machine recovery, automatic scaling, load balancing and regional failover. Managed database recovery and repeatable infrastructure deployment added further recovery options. A guided hybrid-storage lab covered configuring storage, moving files to Amazon Simple Storage Service (S3) and replicating data into another region, connecting architectural choices with the mechanics of protecting and moving data.

The session completed the disaster recovery module by reviewing recovery point objectives (RPOs), recovery time objectives (RTOs), and the relationship between disaster recovery and business continuity planning. It covered AWS services and approaches for resilience, including S3 replication, Storage Gateway, Amazon Machine Images, snapshots, CloudFormation infrastructure as code, Route 53 health checks and failover. Four disaster recovery patterns were compared: backup and restore, pilot light, warm standby, and multi-site active-active, with emphasis on their cost, recovery speed and suitable use cases. Students worked through knowledge-check and exam-style questions on selecting the appropriate recovery pattern and AWS service for different scenarios. The session also introduced the certification exam structure, exam guide and practice-question resources.

Certification preparation used a modular study guide and short quizzes to identify topics needing further revision. The material revisited computing, storage, networking, databases and security, alongside name resolution, logging, resilience, performance and cost. Students were shown how quiz results could guide their study rather than treating every topic as equally familiar. Additional resources included hands-on AWS practice environments, an architecture project and configurable practice tests. The session concerned ways to prepare and assess understanding; the source does not establish completion of the project or an eventual certification result.

The session reviewed commonly misunderstood AWS concepts in preparation for an assessment. It covered service-level agreements, Elastic Compute Cloud (EC2) Auto Scaling, S3 durability, encryption and access logs, virtual private cloud (VPC) networking, network address translation (NAT) gateways, VPC peering, security groups and availability zones. It also compared relational databases with DynamoDB, explained Identity and Access Management (IAM) permissions and roles, and distinguished CloudWatch, CloudTrail and AWS Config for monitoring and auditing. Further topics included messaging with SNS and SQS, CloudFormation, caching, VPC flow logs, cost-allocation tags, SSL offloading, and gateway versus interface VPC endpoints. Practice-question answers were used to reinforce how to select appropriate AWS services for performance, security, scalability and cost management.

The session introduced big data as data that cannot be stored or processed effectively by a single machine, focusing on volume, velocity and variety. It explained distributed systems, horizontal and vertical scaling, clusters and nodes, along with the benefits and challenges of distributing computation across networked machines. The consistency, availability and partition tolerance (CAP) theorem was used to compare consistency, availability and partition tolerance in distributed data storage. The Hadoop ecosystem was outlined, including Hadoop Distributed File System (HDFS) for storage, Yet Another Resource Negotiator (YARN) for resource management, and processing tools such as MapReduce and Spark. Students began a practical activity by connecting securely to a Hadoop edge node through Secure Shell (SSH) and practising basic Linux command-line use.

The session reviewed distributed systems, Hadoop HDFS and YARN, including master, worker and edge nodes, replication, fault tolerance and high availability. It distinguished online transaction processing (OLTP) from online analytical processing (OLAP) systems and compared databases, data lakes, data lakehouses and data warehouses, including atomicity, consistency, isolation and durability (ACID) properties. The class introduced Apache Hive as a data-warehouse tool built on HDFS, using HiveQL and execution engines to query stored data. It covered external versus managed Hive tables, row-based and columnar file formats, Hive partitioning, and the bronze, silver and gold stages of the medallion architecture. Students then began practical Hive exercises creating and querying external and managed tables, including use of ORC files.

The session reviewed core big-data concepts, including the three Vs, data types, horizontal versus vertical scaling, networking risks and the CAP theorem. It covered Hadoop’s main components: HDFS for distributed block storage, YARN for cluster resource management, and MapReduce for distributed processing. The class examined how HDFS splits and replicates files, why small files create metadata overhead, and why processing should be located with the relevant data blocks where possible. MapReduce was introduced through a word-count example, explaining map, shuffle and sort, reduce, network communication, and disk-based fault tolerance. Students then worked on Hadoop-based Python MapReduce word-count exercises and began designing a MapReduce job from scratch.

The session reviewed distributed-system reliability, including single points of failure, standby NameNodes, split-brain scenarios, leader election and ZooKeeper heartbeats. It compared MapReduce and Spark, covering disk-based versus memory-based fault tolerance, resilient distributed datasets, transformations and actions, lazy evaluation, and narrow versus wide transformations and shuffles. The teaching then introduced Spark DataFrames and Spark Structured Query Language (SQL) as higher-level, more readable alternatives to low-level resilient distributed dataset (RDD) programming, explaining schemas, the Catalyst Optimiser and the benefits of explicitly defining schemas for large production data sets. Students began a practical Docker-based Jupyter lab using New York City taxi data to explore DataFrame operations, SQL queries and joins, with exercises requiring work in both PySpark and Spark SQL.

The session introduced real-time event streaming, distinguishing unbounded streams from bounded datasets and comparing hard, soft, near-real-time and batch processing requirements. It reviewed stream-processing concepts including event time versus processing time, windows, triggers and watermarks, and strategies for refining aggregated results. The class then examined Kafka as an event-streaming platform, covering events, producers, consumers, brokers, topics, partitions, offsets and consumer groups. It explained partitioning by round-robin allocation or message keys, replication through leaders and followers, producer acknowledgements and delivery guarantees, and leader election using KRaft rather than ZooKeeper. Students were asked to deploy a Kafka cluster in containers, run a consumer, and optionally process streamed events with Python, Spark or an extract, transform and load (ETL) service.

The session reviewed OLAP and OLTP workloads, row-based and columnar file formats, the medallion architecture, Apache Spark, and Kafka’s publish-subscribe model. It introduced Apache NiFi as a tool for managing data flows and compared ETL, extract, load and transform (ELT) and light-transformation loading pipelines. The class covered NiFi’s security, scalability, delivery guarantees, data provenance and lineage, clustering, registry-based version control, and core components including flow files, processors, connections, funnels and process groups. It also considered why NiFi is best suited to moving data and carrying out light transformations rather than heavy processing. Students began a practical Docker-based lab to build, version and document a basic NiFi data flow.

The session reviewed Apache NiFi as a tool for building ETL and ELT data flows, contrasting transformation before loading with transformation after loading into a data lake. It covered NiFi architecture, including flow files, metadata attributes, content, provenance, repositories, processors, controller services and clustering. The class demonstrated the NiFi interface, process groups, ports, funnels, connections, relationships, queues, back-pressure thresholds, load balancing and debugging. It also explained scheduling, retry and penalty behaviour, parameter contexts, expression language, and version-control alternatives following the planned retirement of NiFi Registry. Students were assigned sequenced labs to build increasingly complete pipelines, culminating in consuming Kafka streams with NiFi, transforming the data, and loading cleaned results into PostgreSQL.

The session reviewed distributed cluster roles and introduced four security principles: identification, authentication, authorisation and encryption. It explained how LDAP, Kerberos, Apache Ranger, Knox and Atlas can support identity management, ticket-based access, role-based permissions, gateway access and governance in Hadoop ecosystems, with comparisons to cloud-native tools. The class then compared file, block and object storage, focusing on object storage features such as buckets, metadata, flat namespaces, immutability, S3-compatible application programming interfaces (APIs) and storage tiers. Software-defined storage, MinIO, Ceph and erasure coding were discussed as ways to provide scalable, resilient cloud storage. Students were assigned a lab to deploy MinIO in Docker and create and manipulate object-storage buckets.

The class introduced simple and multiple linear regression using R, including fitted lines, model output, confidence intervals for mean responses and prediction intervals for new observations. It explained variable selection in multiple regression, comparing exhaustive, stepwise, ridge and lasso approaches, and noted the effect of correlated explanatory variables. The session also outlined analysis of variance for qualitative variables, regression and classification trees, and random forests as a way to stabilise unstable trees. It then developed the simple linear regression model, least-squares estimators, their unbiasedness and variance properties, assumptions on errors, and the role of random vectors, covariance and Gaussian distributions. Model checking through residuals, standardised and studentised residuals, quantile–quantile (Q–Q) plots and goodness-of-fit testing was introduced.

The session covered validation of simple linear regression assumptions using residual diagnostics, focusing on homoscedasticity, zero-mean errors and Gaussian error distributions. Residual-versus-fitted plots were used to identify changing variance and systematic imbalance around zero, with simulations illustrating why heteroscedasticity invalidates several inferential quantities in model output. The distinction between unobservable errors and residuals was explained, including why residuals need not have identical variances even when the errors are homoscedastic. Standardised residuals, histograms and normal quantile–quantile (Q–Q) plots were introduced to assess normality, alongside an explanation of empirical distribution functions and quantiles. The session concluded by motivating studentised residuals and a future Kolmogorov–Smirnov test for quantifying departures from normality.

The session completed simple linear regression by distinguishing prediction intervals for a new observation from confidence intervals for its expected value, explaining why prediction intervals are wider and how both change away from the mean explanatory value. These intervals were calculated manually and checked against R output, with emphasis on the assumptions required for results to be meaningful. Multiple linear regression was then introduced, including data preprocessing, the distinction between numeric and qualitative explanatory variables, and how R treats factors within analysis of covariance. The multiple regression model was written in matrix form, and the least-squares estimator was derived using gradients and matrix operations. The class considered invertibility, linear dependence and multicollinearity among explanatory variables, showing that different equivalent model parameterisations can produce identical predictions. Finally, the session introduced testing overall model significance, multiple-testing corrections, and the motivation for variable selection.

The session developed multiple linear regression model assessment, focusing on the limitations of the multiple R-squared statistic and the use of adjusted R-squared to balance goodness of fit against model complexity. It used geometric projections and software demonstrations to show why ordinary R-squared increases as explanatory variables are added, even when those variables are irrelevant. The class considered diagnostic checks for residual behaviour, including plots for zero mean and constant variance, tests for heteroscedasticity and normality, and the identification of influential observations. It then derived prediction intervals for a new response under Gaussian assumptions, distinguishing intervals for a future observation from those for the expected response. The session also discussed the differing definition of R-squared when no intercept is included and introduced forthcoming approaches to variable selection, including exhaustive, stepwise, ridge and lasso methods.

The session reviewed variable selection for multiple linear regression using exhaustive search, forward selection, backward elimination and stepwise selection. These methods were illustrated in R, using adjusted R-squared and Fisher tests to choose or remove explanatory variables, and their computational costs were compared. Models selected by different methods were evaluated by splitting data into training and test sets and comparing prediction errors. Penalised regression was introduced through ridge regression and lasso, including L1 and L2 penalties, the role of the tuning parameter lambda, cross-validation and the one-standard-error rule; lasso was highlighted as performing variable selection. The session then introduced one-factor analysis of variance as a special case of multiple linear regression, using indicator variables for qualitative factor levels.

The class introduced CART (Classification and Regression Trees), including tree terminology, binary splits, node numbering, leaves and the recursive construction of trees. It explained how possible splits are defined for quantitative and qualitative predictors, and how regression trees maximise variance reduction while classification trees maximise impurity reduction using measures such as the Gini index and Shannon entropy. Examples in R used the Iris data to interpret tree outputs, posterior probabilities, predicted classes, losses and decision regions, alongside a regression-tree example. The session then covered maximal trees, overfitting, cost-complexity pruning, nested subtrees, cross-validation and the one-standard-error rule for selecting a final tree. It also introduced surrogate splits for handling missing predictor values and noted the instability of CART as motivation for later study of random forests.

The session introduced a methodology for exploring a new dataset, beginning with clarifying the business question, intended audience, reporting format, assumptions and practical decision-making context. Using a wine-quality dataset in RStudio and R Markdown, the class loaded and cleaned data, identified variables and outputs, produced descriptive statistics, and discussed the importance of domain knowledge when interpreting values and outliers. It covered the curse of dimensionality, correlations and scatterplot matrices, including Pearson, Spearman and Kendall correlations, and the limitations of using correlation alone for feature selection because of variable interactions. The class then distinguished prediction from interpretation, and explained model selection using training, validation and isolated test sets, cross-validation, bagging, random versus stratified splitting, and the risks of overfitting and test-set leakage.

The session introduced statistical learning within the wider field of artificial intelligence, emphasising the trade-off between model complexity, usability and interpretability. It discussed challenges in artificial intelligence, including reliable decision-making, heterogeneous data, limited supervision, and small high-dimensional datasets, using medical screening, metabolomics and electricity-consumption analysis as examples. Different data types were distinguished, including continuous, count, ordinal, categorical, image, functional, network and text data, alongside the practical stages of a data-science project and the treatment of missing, abnormal and extreme values. Classical descriptive statistics for univariate and multivariate data were reviewed, including covariance and correlation matrices, box plots, scatter plots and pair plots. The session then defined statistical learning, machine learning, supervised and unsupervised learning, and explained regression, classification, clustering and dimensionality reduction, ending with the need to use separate learning and validation data when evaluating supervised models.

The session reviewed validation and cross-validation for comparing classifiers on flower measurements, including linear discriminant analysis (LDA), k-nearest neighbours (KNN) and quadratic discriminant analysis (QDA). It showed how to tune the number of neighbours in KNN, explained why nested cross-validation is needed to evaluate a tuned method fairly, and compared average errors and their standard deviations. Classification boundaries were visualised in two dimensions to contrast the linear behaviour of LDA, quadratic boundaries from QDA, and data-dependent boundaries from KNN. Logistic regression was introduced as a maximum-likelihood binary classifier based on modelling log-odds, and was applied to a binary version of the flower problem. Support vector machines were then introduced through nonlinear feature mappings, maximum-margin separating hyperplanes, support vectors and kernel functions.

The session compared linear and quadratic discriminant analysis, then used support vector machines with linear, polynomial and radial basis function kernels for multiclass classification. Classification boundaries were visualised, and cross-validation was used to tune polynomial degree and the radial basis function kernel parameter gamma, before double cross-validation compared LDA, QDA, KNN and support vector machine methods on the Iris data. The importance of retaining reference methods and selecting models according to validation error was emphasised. The class then introduced unsupervised learning, distinguishing dimensionality reduction from clustering, and feature extraction from variable selection. Principal component analysis, probabilistic PCA and sparse PCA were explained, including dimension selection, correlation circles for interpretation, and an R exercise using Swiss socioeconomic data.

The session reviewed clustering as unsupervised classification and introduced mixture models, particularly Gaussian mixture models, in which latent variables represent cluster membership. It explained the EM algorithm as an iterative expectation and maximisation procedure for estimating mixture parameters, assigning observations through posterior probabilities, and addressing local optima through multiple initialisations. Model constraints and selection were discussed, including the relationship between K-means and a restricted Gaussian mixture model, and use of the Bayesian information criterion (BIC) to balance fit against complexity and choose both a model and the number of clusters. Practical R demonstrations applied K-means, hierarchical clustering and Gaussian-mixture clustering to Swiss village data, comparing cluster interpretations, dendrogram linkage methods, uncertainty and BIC values. The session also considered whether dimensionality reduction should occur before or after clustering, showing that the order can change the resulting partition.

The session introduced the structure of the survival analysis course, including duration data, non-parametric survival methods, Cox proportional hazards models, model checking and advanced machine-learning approaches. It refreshed statistical inference, covering populations and samples, sampling methods, uncertainty, estimation, confidence intervals and hypothesis testing. The class distinguished statistical significance from practical significance, explained p-values and Type I and Type II errors, and compared statistics with machine learning. It reviewed the normal distribution, standardisation, probability functions in R and simulation of data. Students also began working with R and RStudio, including linear regression, model interpretation, prediction and diagnostic checks of model assumptions.

The session introduced survival analysis as the study of time to an event, using examples from medical, industrial, behavioural and customer-retention settings. It defined survival, cumulative distribution, density, hazard and cumulative hazard functions, including median survival time and the relationship between survival and hazard. Right censoring, left censoring and the assumption of non-informative censoring were explained, alongside the effect of shrinking numbers at risk on confidence intervals. Students learned the Kaplan–Meier estimator and the Nelson–Aalen alternative, then used R to estimate and plot survival curves, censored observations and cumulative survival. The session also introduced comparisons between group survival curves using the log-rank test and illustrated these methods with animal-behaviour, cancer and smoking-cessation data.

The session introduced Cox proportional hazards regression as a semi-parametric model for assessing the effects of categorical and continuous covariates on hazard rates while accounting for right censoring through partial likelihood. Hazard ratios were interpreted for binary, continuous and multi-level categorical predictors, including the importance of reference categories, confidence intervals and distinguishing hazard from probability. Practical R examples fitted Cox models to rat, lung cancer, smoking and smoking-cessation data, including changing reference groups, estimating predicted survival curves and examining multivariable effects. The class also covered comparing nested models with likelihood-ratio tests and analysis of variance, comparing models with the Akaike information criterion (AIC), automated model selection, and the concordance index as an indicator of predictive performance.

The session reviewed Cox model diagnostics, including Martingale and DFBETA measures of coefficient influence, tests and plots for the proportional hazards assumption, and approaches for violations such as stratification, truncation and time interactions. Practical R examples showed how time-varying covariates require start–stop data structures and can substantially alter estimated hazard ratios, using transplant, employment and treatment data. The class introduced accelerated failure time models, contrasting their parametric assumptions and time-ratio interpretation with the hazard-ratio interpretation of Cox models. It also introduced ridge regression, lasso and elastic net methods for high-dimensional survival modelling, explaining regularisation penalties and cross-validation for selecting tuning parameters.

Direct problems mapped known inputs to outputs; inverse problems sought unknown inputs or parameters from observations. Differentiation, image deblurring and tomography illustrated questions of existence, uniqueness and sensitivity to data. Regularisation introduced a way to stabilise a solution, with parameter choices affecting conditioning and optimisation. Data assimilation then combined physical models with sparse, noisy measurements. Weather and ocean examples introduced dynamical behaviour, observation models and heterogeneous measurements, making uncertainty part of the formulation rather than an afterthought.

Four-dimensional variational assimilation sought an initial state that balanced a background estimate with observation mismatches over time. Model dynamics entered as constraints, leading to direct, adjoint and optimality equations. Solving the adjoint backwards provided a gradient for optimisation. The class examined why discretising a continuous adjoint may not produce the correct discrete gradient. Calibration extended the approach to model parameters as well as initial conditions, connecting the derivation with forecasting and digital-twin applications without removing that consistency requirement.

Reduced-rank filtering addressed the cost of storing and propagating large covariance matrices. Leading eigenvalues and eigenvectors represented important uncertainty directions in a smaller form. Comparisons with standard and extended Kalman filters used a non-linear transport model to examine linearisation and propagation choices. Ensembles introduced empirical covariance estimates from several model states. Nudging offered a lower-cost approach that relaxes a model towards observations, with stability conditions considered alongside its relationship to filtering and variational methods in the session.

The discrete Lorenz adjoint was developed alongside a constructed reference trajectory and noisy, intermittent observations. A cost function combined background information with observation errors. Gradient checks examined whether the adjoint matched the discrete forward scheme, rather than assuming that a continuous derivation could simply be discretised. Optimisation methods were compared; a quasi-Newton method recovered a trajectory close to the reference in the experiment. Model coupling also allowed an unobserved variable to be estimated within that particular setup.

The session introduced Neo4j as a native, schema-optional graph database and compared its relationship-focused storage with relational databases. It covered graph theory fundamentals, including nodes, relationships, directed and weighted graphs, labels, properties and shortest-path traversal. Students explored common graph-database applications such as fraud detection, recommendation systems and network analysis. The class introduced Cypher syntax for matching, filtering and returning nodes and relationships, using a movie graph dataset for practical queries. Students also practised creating nodes and relationships with MERGE, updating or removing properties, and using conditional creation and matching behaviour.

The session demonstrated how to import comma-separated values (CSV) data into Neo4j, create uniqueness constraints, load person, location and visit nodes, and create relationships between them. It then introduced intermediate Cypher querying, including case-insensitive string filtering, checking query plans with EXPLAIN and PROFILE, and writing efficient graph traversal patterns. Further activities covered OPTIONAL MATCH, ordering, filtering null values, limiting and paginating results, and removing duplicates with DISTINCT. The class also practised map projections and CASE expressions to customise returned JSON-style objects and categorise movie runtimes.

The session introduced cloud computing models, service types and deployment approaches, including the benefits and limitations of public, private, hybrid and multi-cloud environments. It compared using Neo4j through Amazon Web Services (AWS), Google Cloud and Azure marketplaces, including single instances, clustered deployments, containers and relevant certification routes. The class then practised intermediate Cypher queries using aggregation functions such as COUNT, COLLECT, SUM, AVG, MIN and MAX, alongside DISTINCT, UNWIND, list indexing and list slicing. It also covered date, time and duration values in Cypher, including calculating intervals and converting timestamps for comparisons. Students continued the COVID graph mini-project exercises.

The session introduced NoSQL databases, explaining how document, graph, wide-column and key-value databases differ from relational databases and suit different use cases. It compared SQL and NoSQL in terms of schema flexibility, scalability, transaction guarantees, querying, performance, security and vendor lock-in. MongoDB was presented as a document-oriented database using Binary JSON (BSON), with databases, collections, documents, fields, embedded documents, arrays and object IDs. Students installed MongoDB and Compass, created databases and collections, inserted, deleted and updated documents, and imported a restaurant dataset. The class then explored filtering, projections, sorting, pagination, logical operators, comparisons, field existence, regular expressions and array queries, before introducing equivalent operations through the PyMongo Python library.

The session introduced MongoDB aggregation pipelines, in which documents pass through ordered stages such as $match, $project, $group, $unwind, $sort and $lookup. It demonstrated accumulator operations including counts, sums, averages, minima, maxima, and first or last values, alongside reshaping, cleaning and constructing fields within pipelines. Examples used restaurant data to calculate statistics by borough and cuisine, flatten arrays of inspection grades, create map-ready outputs, and obtain random samples. The class then covered indexes, explaining collection scans versus index scans and the use of single-field, compound, text, unique, time-to-live (TTL), wildcard, collation and geospatial indexes. Practical work was assigned to create indexes and write queries and aggregations that generate a restaurant report.

Computer vision tasks ranged from classification and detection to segmentation, pose and depth estimation. The class examined why flattening an image into a general multilayer network can lose useful spatial structure and require many parameters. Convolutional networks introduced filters, receptive fields, padding, stride and pooling as alternatives. A PyTorch classification lab using the CIFAR-10 image dataset was introduced, with normalisation and batching discussed before later fine-tuning work. Historical architectures provided context for the development of these approaches.

Detection and segmentation introduced bounding boxes, overlapping-region measures and architectures that identify or label image regions. Practical guidance covered pretrained weights, evaluation mode, input transformations and confidence thresholds, with training data informing whether adaptation was needed. Transformers then introduced encoder and decoder arrangements, self-attention and parallel processing. Embeddings, tokenisation and vector similarity connected these models with the representation of sequences, extending the discussion from spatial information in images to the structure used for language tasks.

Different transformer architectures were examined through masked, span-reconstruction and next-token training objectives. General pretraining, domain adaptation and task-specific fine-tuning were connected with the need for curated data. Evaluation covered classification, text-comparison measures, error rates, benchmarks and human judgement, including their limitations. Practical activities used language models for prediction, sentiment, similarity, search and extractive questions. Retrieval-augmented generation connected embeddings with retrieved information. An introduction to Whisper then began the transition to speech recognition, transcription and translation.

Project guidance emphasised reproducible experiments, exploratory analysis, controlled comparisons and careful use of generated code. Students practised pretrained audio classification and speech transcription, interpreting scores, labels and language outputs. Waveforms, sampling and spectrograms provided the signal-processing context. Large language models introduced a distinction between base and instruction-tuned systems. Generation strategies and settings were compared through their effects on coherence and diversity, including deterministic selection, sampling and repetition controls rather than a single preferred setting for every task.

The session introduced modelling as a purposeful simplification of reality used to understand relationships and make predictions. It compared statistical models, Markov models, system dynamics, microsimulations and agent-based models, emphasising heterogeneity, interactions between agents, feedback loops and unintended consequences. Predictive modelling was discussed through regression, uncertainty, extrapolation beyond observed data, parameter estimation and Monte Carlo simulation. Markov models were introduced through state transitions, transition matrices, steady states and the distinction between discrete and continuous time. Students were also introduced to NetLogo, opened a segregation model from its library, and explored the software’s interface, documentation, code tabs and support resources.

The session compared population-averaged system dynamics models with agent-based models, using alcohol use and disease transmission to show how individual histories, heterogeneity and network structure can affect outcomes. It explained Jensen’s inequality and why averaging individual characteristics before modelling non-linear processes can give different results from modelling individuals first and then averaging. Agents were defined through states, transitions, behavioural rules, interactions, networks and environmental resources, with discussion of deterministic and stochastic rules and multiple linked state diagrams. The class introduced decision modelling for selecting between actions under uncertainty, including predictive and prescriptive analysis, decision trees, expected values, utilities, Monte Carlo simulation and Bayesian use of broad and local data. Examples considered clinical treatment choices and emergency responses to wildfire risk, emphasising problem formulation, stakeholder criteria, uncertainty and the costs of possible outcomes.

The class introduced the ODD protocol for documenting agent-based models: Overview, Design Concepts and Details. It examined how to define a model’s purpose, entities, variables, rules, population, boundaries, and temporal and spatial scales. The session compared discrete and continuous time, and explained why update order, synchronisation and sequential processing can affect model outcomes. Design concepts including emergence, agent objectives, adaptation, learning, prediction, sensing, interaction and stochasticity were illustrated through predator–prey and disease-spread models. Students also considered how randomness can produce small or large epidemics and why models should include only the complexity needed for their purpose.

The session examined verification, calibration and validation of agent-based models, distinguishing internal testing, fitting models to observed data, and testing predictions or comparisons with other models. It covered calibration strategies for uncertain numerical and categorical parameters, steady states and transitions, time-series fitting, uncertainty bands, and the importance of choosing criteria that match a model’s purpose. Sensitivity, uncertainty and robustness were differentiated, with experimental designs, Monte Carlo simulation, regression and decision trees introduced as ways to assess parameter effects and interactions. The class also considered sources of uncertainty, including parameter estimates, model structure, output interpretation and unpredictable future events. Finally, it used simple epidemic models to explain the susceptible–infectious–recovered (SIR) framework, reproduction numbers, herd immunity and the value and limitations of theoretical models.

The Web was distinguished from the Internet as a distributed hypermedia application, built around Uniform Resource Locators (URLs), the Hypertext Transfer Protocol (HTTP) and Hypertext Markup Language (HTML). The session traced a path through structured documents using Extensible Markup Language (XML) to linked data and Resource Description Framework (RDF) graphs. Knowledge graphs connected named entities and relationships across datasets, supporting search, recommendation, integration, validation and reasoning. Retrieval-augmented generation was another application considered. The discussion also examined data collection, tracking, algorithmic influence and biased datasets, bringing privacy, accessibility and ethics into the technical picture.

RDF statements were studied as triples forming directed, labelled graphs. Uniform Resource Identifiers (URIs), literals and blank nodes supplied the building blocks for linking data across datasets. Students practised graph completion and learnt N-Triples, Turtle and RDF/XML syntax, including prefixes, namespaces, typed values and language tags. Activities included validating, converting and visualising data, creating a personal RDF profile and examining an external library dataset. The practical guidance was to test incrementally and check generated RDF rather than copying the output of artificial intelligence without verification.

SPARQL, the query language for RDF, introduced a way to ask questions by matching patterns in a graph. Students constructed patterns with triples, variables, prefixes and filters, then practised SELECT queries, distinct results, ordering and pagination. Remote endpoints such as DBpedia and Wikidata provided datasets to query. Further exercises used optional patterns, alternatives, exclusions and supplied values, alongside type and language tests, casting and string functions. Translating natural-language requests into queries connected the syntax to a precise account of which graph matches should be returned.

Ontologies were introduced as shared, formal vocabularies that give data meaning and support logical inference. Classes, properties and their definitions were distinguished from taxonomies, thesauri and validation schemas such as Shapes Constraint Language (SHACL). A museum modelling exercise exposed ambiguity, user needs, domain expertise and cultural assumptions. RDF Schema (RDFS) then provided classes, subclass and subproperty hierarchies, domains and ranges. Examples showed how these declarations allow additional types and relationships to be inferred, making the consequences of a model’s definitions visible in the graph.

An RDFS lab showed how loading a schema and enabling reasoning changes the available triples and query results. The Web Ontology Language (OWL) introduced richer class definitions and property relationships, including intersections, disjointness, equivalence, restrictions, inverse relations and property chains. Students interpreted axioms and predicted inferred facts from small graphs, using biological and organisational examples. Restrictions on values and cardinalities extended the modelling possibilities. Open-world reasoning and the absence of a unique-name assumption were central cautions: incomplete information and different names cannot simply be treated as proof of absence or difference.

Media Fragments and a media-resources ontology introduced descriptions of image regions, audiovisual segments and tracks. Students considered how to assess existing vocabularies through their definitions, scope, reuse and expressivity, and how validation supports data quality. The session surveyed mappings from relational databases, HTML, JavaScript Object Notation (JSON) and comma-separated values (CSV) into RDF. JSON for Linking Data (JSON-LD) contexts and other mapping standards connected familiar formats to graphs. Practical work included extracting embedded triples, interpreting mappings and converting profiles to JSON-LD. The Linked Data Platform introduced managing RDF resources and containers through HTTP operations.

The session introduced the legal, ethical and geopolitical issues raised by data and artificial intelligence, emphasising that regulation varies across jurisdictions and societal contexts. It explored why artificial intelligence (AI) and data engage areas including data protection, liability, intellectual property, constitutional law, international law and discrimination. Students considered the limits of using AI in legal decision-making, including predictive systems for human-rights cases, balancing legal certainty and efficiency against human interpretation and the evolving nature of law. The class also examined definitions of AI, the distinction between narrow and general AI, calls to regulate future technological risks, and the possible legal personhood of robots. A case discussion considered arguments for and against a conversational robot inheriting an owner’s assets.

The session examined the development and purpose of the European Union (EU) AI Act, including tensions between innovation, competitiveness, sovereignty, public regulation and private ethical codes. It distinguished legal obligations from voluntary ethical commitments, explaining why enforceable rules and sanctions are significant. The class explored the Act’s risk-based model, covering prohibited, high-risk, limited-risk and minimal-risk AI systems, alongside the duties imposed on providers and other actors. It also considered the Act’s extraterritorial reach, exclusions such as military and research uses, enforcement mechanisms, regulatory sandboxes, and the particular rules for general-purpose AI models. Students discussed practical difficulties in assessing risks to fundamental rights, democracy and human dignity, as well as potential conflicts between European and international regulatory approaches.

The session introduced data protection law through discussion of children’s data, profiling, algorithmic bias and the limits of consent in digital services. It examined why data protection is important for AI, comparing the European rights-based approach with the more sector-specific approach in the United States. The class covered privacy as a broader concept than information control, including dignity, intimacy, home and communications, and considered the EU Charter rights to privacy and personal-data protection. It then introduced the General Data Protection Regulation (GDPR), including personal data, identifiable natural persons, processing activities, material and territorial scope, and the basic obligations on organisations processing data. Students discussed examples involving health data, policing, profiling, data transfers and cross-border services.

The session examined GDPR data-subject rights in AI systems, including access, rectification, erasure, restriction of processing, portability and objection, and explained privacy by design across training and deployment. It considered how biased datasets and algorithmic design can create or reinforce discrimination, and discussed fairness, auditing, impact assessments and ethics by design. The class then introduced the EU Data Act, its application to data from connected products and related services, its relationship with the GDPR, and rules on user-led data sharing, competition, gatekeepers, contracts, compensation and dispute resolution. The final section used neurotechnology to explore mental privacy, brain data, cognitive liberty, and the need for ethical and legal safeguards against misuse by companies or governments.

The session examined the ethical, legal and professional responsibilities of scientists and data specialists in relation to autonomous lethal weapons and AI used in conflict. It considered the limits of existing regulation, particularly the military exemption in the AI Act, and debated whether emerging technologies should be regulated or prohibited before they are fully developed. The class compared arguments for and against autonomous weapons, including meaningful human control, bias, accountability, international consensus and historical preventive bans on certain weapons. It also discussed whether advanced AI systems merely imitate reasoning or might develop forms of understanding or consciousness, and considered the implications for law and responsibility. Finally, the session explored corporate due diligence, value-chain oversight and contractual controls through examples of technology companies supplying tools that may be used for immigration enforcement, surveillance or defence.

The session introduced the course structure and its aim of preparing students to participate in or lead data science projects. It contrasted traditional, waterfall-style project management with agile approaches, explaining time-boxed sprints, sprint reviews, retrospectives, planning and visual task boards. A project was defined as a temporary undertaking with a unique objective, a defined timescale and required resources, distinguishing it from routine operations. The class also examined product and sprint backlogs, prioritisation, team commitment, transparency and measuring progress. Business cases, break-even points, return on investment and the need to reassess a project when business conditions change were introduced.

The session began with sprint planning and reflection, using completed work and team capacity to agree a realistic commitment for the next sprint. It covered project vision statements, the limitations of AI-generated content, and the need to understand project-management concepts well enough to assess artificial intelligence (AI) outputs critically. Key traditional project-management concepts included the triple constraint of scope, time and cost; project life-cycle phases; uncertainty, risk and the rising cost of change; and the Project Management Institute (PMI) waterfall framework, including project charters, planning, resources, communication, procurement and stakeholder management. Students practised preparing elements of a project charter and assigning stakeholder roles through a responsible, accountable, consulted and informed (RACI) matrix, then discussed stakeholder power and influence. The class also covered Gantt charts, task estimation, S-curves, earned value, cost and schedule variance, key performance indicators (KPIs), and the Plan-Do-Check-Act cycle.

The session used a simulated sprint review, retrospective and planning exercise to demonstrate approval of completed work, velocity tracking, forecasting and commitment to a minimum sprint scope. It introduced Agile values from the Agile Manifesto, stressing individuals and interactions, working software, customer collaboration and responding to change whilst recognising the continuing value of plans, documentation and contracts. Agile and waterfall approaches were compared in terms of fixed constraints, planning, suitability for different levels of uncertainty, stakeholder involvement and regulated environments. The class examined timeboxing, incremental delivery, technical debt, minimum viable products, proof of concepts, product backlogs, prioritisation and Agile contract models. It also covered the roles of the product owner, Scrum Master and self-managing cross-functional teams, followed by a scenario activity, quiz and video on Agile product ownership.

The session showed how an agile backlog can be adapted as new information emerges, using prioritised user stories and changing investigation scenarios to illustrate this process. It emphasised the importance of direct, timely communication for agile teams, particularly where teams are distributed. Agile software development practices included automated testing, continuous integration and delivery, code standards, collective code ownership, technical-debt reduction, and responsible use of AI-assisted programming. The class then introduced scaled agile working through Scaled Agile Framework (SAFe), including Agile Release Trains, programme increments, programme increment (PI) planning, cross-team dependencies, and the roles of product manager, Release Train Engineer and system architect. It also considered how agile delivery can be connected with wider business strategy and product value.