MSc in Data Engineering for Artificial Intelligence.

The session introduced data engineering on AWS, focusing on the five Vs of data: value, veracity, volume, velocity and variety. It explained data pipelines, including extract, transform and load (ETL) and extract, load and transform (ELT) approaches, structured and unstructured data, and the choice between batch and streaming processing. AWS ingestion services were covered, including AppFlow for software-as-a-service data, DataSync for file transfers, and Data Exchange for third-party data. The class examined AWS Glue for data catalogues, crawlers, ETL jobs, schema management, data quality and handling sensitive information. Streaming tools including Kinesis Data Firehose, Kinesis Data Streams, managed Apache Flink and Kafka services were compared, alongside data lakes, data warehouses, Lake Formation, Athena, Redshift and OpenSearch.

Disaster recovery planning connected acceptable data loss and recovery delay with storage, computing, databases and networks. Backups, snapshots and cross-region replication were compared, including why replication alone does not protect against corruption. The class reviewed storage migration, lifecycle management and shared file services, then considered machine recovery, automatic scaling, load balancing and regional failover. Managed database recovery and repeatable infrastructure deployment added further recovery options. A guided hybrid-storage lab covered configuring storage, moving files to Amazon Simple Storage Service (S3) and replicating data into another region, connecting architectural choices with the mechanics of protecting and moving data.

The session completed the disaster recovery module by reviewing recovery point objectives (RPOs), recovery time objectives (RTOs), and the relationship between disaster recovery and business continuity planning. It covered AWS services and approaches for resilience, including S3 replication, Storage Gateway, Amazon Machine Images, snapshots, CloudFormation infrastructure as code, Route 53 health checks and failover. Four disaster recovery patterns were compared: backup and restore, pilot light, warm standby, and multi-site active-active, with emphasis on their cost, recovery speed and suitable use cases. Students worked through knowledge-check and exam-style questions on selecting the appropriate recovery pattern and AWS service for different scenarios. The session also introduced the certification exam structure, exam guide and practice-question resources.

Certification preparation used a modular study guide and short quizzes to identify topics needing further revision. The material revisited computing, storage, networking, databases and security, alongside name resolution, logging, resilience, performance and cost. Students were shown how quiz results could guide their study rather than treating every topic as equally familiar. Additional resources included hands-on AWS practice environments, an architecture project and configurable practice tests. The session concerned ways to prepare and assess understanding; the source does not establish completion of the project or an eventual certification result.

The session reviewed commonly misunderstood AWS concepts in preparation for an assessment. It covered service-level agreements, Elastic Compute Cloud (EC2) Auto Scaling, S3 durability, encryption and access logs, virtual private cloud (VPC) networking, network address translation (NAT) gateways, VPC peering, security groups and availability zones. It also compared relational databases with DynamoDB, explained Identity and Access Management (IAM) permissions and roles, and distinguished CloudWatch, CloudTrail and AWS Config for monitoring and auditing. Further topics included messaging with SNS and SQS, CloudFormation, caching, VPC flow logs, cost-allocation tags, SSL offloading, and gateway versus interface VPC endpoints. Practice-question answers were used to reinforce how to select appropriate AWS services for performance, security, scalability and cost management.

The session introduced cloud computing as the delivery of on-demand computing, storage, networking and higher-level services, with emphasis on service-level agreements, scalability, resource pooling, cost optimisation and global availability. It compared cloud services with renting dedicated hardware, and considered benefits and limitations including resilience, security and compliance, bandwidth, provider outages and vendor lock-in. The class outlined major Azure service areas, including virtual machines, networking, data services and artificial intelligence services, and discussed how to select a cloud provider based on requirements, cost and staff expertise. Students were shown the Azure portal, subscriptions, resources and resource groups, then followed a demonstration of configuring an Ubuntu virtual machine with networking, public access, storage, security rules, monitoring, backups and tagging. The afternoon practical introduced load balancing for web applications, using multiple virtual machines to provide high availability and redirect traffic when an instance is unavailable.

The session introduced Azure Data Factory as a cloud service for building, scheduling and monitoring data integration and extract, transform and load (ETL) pipelines. It covered data movement, transformation, orchestration, event-driven and batch processing, monitoring, security, access control, cost considerations, vendor lock-in and limitations in debugging or complex transformations. Azure Data Factory Studio was demonstrated, including linked services, datasets, copy activities, mappings, triggers, pipeline dependencies and monitoring. A practical example transferred customer data from a comma-separated values (CSV) file in Azure Blob Storage to an Azure SQL Database, using scheduled upsert operations. Students were then assigned practical exercises on loading data into Azure Synapse Analytics and retrieving weather data from an application programming interface (API) into a database.

The session introduced Azure Databricks as a managed, collaborative platform built around Apache Spark for big-data processing, analytics, machine learning and business intelligence. It covered scalability through clusters and serverless compute, integrations with Azure storage and analytics services, identity-based access control, encryption, monitoring, and cost management. Examples showed how Databricks can support ETL pipelines, batch and real-time processing, predictive maintenance, anomaly detection, dashboards and machine-learning workflows. Practical demonstrations created a Databricks workspace, explored its catalogue, notebooks, jobs, compute resources and SQL tools, then imported CSV and API weather data into Delta tables using scheduled jobs. A simple weather-prediction model was prepared and registered for reuse within the Databricks environment.

The session introduced Azure Machine Learning Studio as a no-code environment for building, training, evaluating and deploying machine-learning models. A car-price prediction example used data selection, missing-data cleaning, a 70/30 train-test split, regression models, scoring and evaluation metrics to compare model performance and avoid overfitting. It also covered deploying a selected model through a real-time inference pipeline so that predictions could be requested via an API, although the live deployment did not complete. Azure Cosmos DB was presented as a globally distributed NoSQL document database, including partition keys, SQL-style queries, global distribution, security and MongoDB-compatible access. Practical work was assigned to practise machine-learning pipelines and build an end-to-end data project using an API, virtual machine, data lake layers, Data Factory and Cosmos DB.

The Web was distinguished from the Internet as a distributed hypermedia application, built around Uniform Resource Locators (URLs), the Hypertext Transfer Protocol (HTTP) and Hypertext Markup Language (HTML). The session traced a path through structured documents using Extensible Markup Language (XML) to linked data and Resource Description Framework (RDF) graphs. Knowledge graphs connected named entities and relationships across datasets, supporting search, recommendation, integration, validation and reasoning. Retrieval-augmented generation was another application considered. The discussion also examined data collection, tracking, algorithmic influence and biased datasets, bringing privacy, accessibility and ethics into the technical picture.

RDF statements were studied as triples forming directed, labelled graphs. Uniform Resource Identifiers (URIs), literals and blank nodes supplied the building blocks for linking data across datasets. Students practised graph completion and learnt N-Triples, Turtle and RDF/XML syntax, including prefixes, namespaces, typed values and language tags. Activities included validating, converting and visualising data, creating a personal RDF profile and examining an external library dataset. The practical guidance was to test incrementally and check generated RDF rather than copying the output of artificial intelligence without verification.

SPARQL, the query language for RDF, introduced a way to ask questions by matching patterns in a graph. Students constructed patterns with triples, variables, prefixes and filters, then practised SELECT queries, distinct results, ordering and pagination. Remote endpoints such as DBpedia and Wikidata provided datasets to query. Further exercises used optional patterns, alternatives, exclusions and supplied values, alongside type and language tests, casting and string functions. Translating natural-language requests into queries connected the syntax to a precise account of which graph matches should be returned.

Ontologies were introduced as shared, formal vocabularies that give data meaning and support logical inference. Classes, properties and their definitions were distinguished from taxonomies, thesauri and validation schemas such as Shapes Constraint Language (SHACL). A museum modelling exercise exposed ambiguity, user needs, domain expertise and cultural assumptions. RDF Schema (RDFS) then provided classes, subclass and subproperty hierarchies, domains and ranges. Examples showed how these declarations allow additional types and relationships to be inferred, making the consequences of a model’s definitions visible in the graph.

An RDFS lab showed how loading a schema and enabling reasoning changes the available triples and query results. The Web Ontology Language (OWL) introduced richer class definitions and property relationships, including intersections, disjointness, equivalence, restrictions, inverse relations and property chains. Students interpreted axioms and predicted inferred facts from small graphs, using biological and organisational examples. Restrictions on values and cardinalities extended the modelling possibilities. Open-world reasoning and the absence of a unique-name assumption were central cautions: incomplete information and different names cannot simply be treated as proof of absence or difference.

Media Fragments and a media-resources ontology introduced descriptions of image regions, audiovisual segments and tracks. Students considered how to assess existing vocabularies through their definitions, scope, reuse and expressivity, and how validation supports data quality. The session surveyed mappings from relational databases, HTML, JavaScript Object Notation (JSON) and comma-separated values (CSV) into RDF. JSON for Linking Data (JSON-LD) contexts and other mapping standards connected familiar formats to graphs. Practical work included extracting embedded triples, interpreting mappings and converting profiles to JSON-LD. The Linked Data Platform introduced managing RDF resources and containers through HTTP operations.

The session introduced Rust as a systems programming language chosen for performance, memory safety, concurrency and use in data-intensive applications. It covered Rust tooling, including Cargo projects, crates, the Cargo.toml manifest, building and running programs, and configuring an editor with Rust analysis and debugging support. The class explained memory management through the stack and heap, including scope, allocation, fixed and dynamic data, strings and pointers. It also introduced immutable and mutable variables, ownership, moving values, cloning, borrowing through references, and basic control structures and functions.

The session examined Rust lifetimes, explaining how references must remain valid and how explicit lifetime annotations prevent dangling references at compile time. It covered Option and Result types for handling missing values and failures, including matching, unwrapping, default values and propagating errors with the question-mark operator. The class introduced structs for grouping related data, their instantiation and mutation, ownership implications when copying fields, and implementation blocks for adding methods and constructor-like functions. It then introduced traits as shared behaviour across types, using the Iterator trait and operations such as next, map, filter, take and collect in preparation for building a linked-list example.

The session reviewed Rust exercises on lifetimes and references, mutable borrowing in structs, generic functions, Option and Result types, iterators, collections, and filtering data. It compared core data structures such as vectors, double-ended queues, heaps, hash maps, hash sets, B-tree maps and sets, focusing on their operations, complexity and appropriate use cases. Advanced structures including graphs, trees and spatial trees were introduced, alongside graph traversal and shortest-path concepts. The class then covered modularity, separation of concerns, encapsulation, abstraction, decoupling, interfaces, reuse and testing, before beginning a modular Rust graph implementation using type aliases, structs and derived traits.

The session covered software testing in Rust, including the purposes and distinctions of unit, integration and property-based tests. It demonstrated how to write and run tests with Cargo, use assertions, test error conditions and private functions, mock external dependencies, organise tests, and keep tests independent and focused on behaviour. Practical examples applied unit and integration tests to graph structures and algorithms. The class also introduced benchmarking, code coverage, debugging and profiling, before reviewing core Rust principles such as ownership, borrowing, memory safety, modularity and error handling.

The session introduced C# as an object-oriented language used with the .NET platform and Common Language Runtime, including automatic memory management and just-in-time compilation. It covered Visual Studio project organisation, solutions, projects, namespaces, program entry points, primitive and reference types, operators, functions, conditions and loops. Practical examples demonstrated type conversion, integer division, user input validation with TryParse, and the differences between casting and parsing. The class also examined arrays, multidimensional and jagged arrays, and dynamic data structures including lists, linked lists, stacks and dictionaries. A merge sort example was introduced to illustrate divide-and-conquer sorting and its time complexity.

The session introduced object-oriented programming, defining objects as entities with state, behaviour and identity, and classes as blueprints containing attributes, methods and constructors. It covered encapsulation through public and private members, static members, the current-instance reference, properties, constructors and method overloading, using bank-account and fraction examples in C#. SOLID design principles were introduced, particularly single responsibility and open–closed principles, with discussion of maintainability, extensibility and testability. The class also introduced the Unified Modelling Language (UML), including use case diagrams for identifying actors and system requirements, and class diagrams for modelling attributes, methods, visibility and static members. Finally, it began explaining horizontal relationships between classes, including association and aggregation.

The session covered object-oriented relationships: inheritance through generalisation and specialisation, and interface realisation as a contract for shared capabilities. It explained polymorphism, abstract classes and methods, method overriding, virtual methods, protected access, base-class constructors, casting, and the difference between overriding at runtime and overloading at compile time. Examples modelled animals, vehicles, flying objects, database connectors, property ownership and transaction records. A practical C# example used abstract transaction classes, sales and refund specialisations, Language Integrated Query (LINQ)-based reporting, and interchangeable comma-separated values (CSV) and JavaScript Object Notation (JSON) data exporters through an interface. Students were set a guided UML-based exercise to build an Internet of Things (IoT) data pipeline using these concepts.

The session introduced model–view–controller (MVC) architecture, separating an application into models for data and logic, views for presentation, and controllers for handling input and coordinating the other components. A console-based user-management example was developed to demonstrate this separation, including input validation, menus, error handling and exception handling. The class then covered creational, structural and behavioural design patterns, focusing on singleton, factory, adapter and observer patterns, including their purposes and limitations. These ideas were applied to a sensor-data pipeline using a factory to convert database records into reading objects, a repository and Entity Framework with SQLite for data access, report generation, and CSV and JSON export. The example was extended with an observer-based event system to notify application observers when database, reporting and export actions occurred.

The session introduced the structure of the machine learning course, including its focus on exploratory data analysis, feature engineering and modelling within a complete machine learning pipeline. A group project was outlined in which students were asked to use book metadata to predict Goodreads ratings, clean and analyse imperfect data, compare at least two models, and deliver code, a simple application, a report, a video demonstration and a GitHub repository. The class emphasised that machine learning begins with a clearly defined problem, suitable data and an understanding of how a human would approach the task. It also reviewed Python foundations needed for machine learning, including built-in data structures, mutability and immutability, indexing, conditions, loops and functions.

The session reviewed pandas data frames, including their tabular structure, indexing, comma-separated values (CSV) import, and the use of head(), tail() and describe() for inspection. It introduced the Titanic data set as a continuing case study and defined the machine-learning task as predicting passenger survival from available passenger information. The class examined data quality issues such as missing ages, identified the target column and ground truth, and discussed class imbalance and its effect on model bias. It also interpreted summary statistics, distributions, standard deviation and outliers, considering when unusual cases should be retained, removed or contextualised through additional features.

The session reviewed exploratory data analysis and feature engineering using the Titanic data set, including handling missing values, removing unhelpful columns and considering bias when retaining scarce passenger profiles. It demonstrated how to extract structured titles from unstructured name text using string splitting, indexing, slicing, functions and DataFrame apply operations, then create and analyse a new title feature. The class examined how uneven category frequencies can limit the usefulness of a feature for machine-learning models. It also introduced data visualisation with Plotly, using wine-quality data to inspect distributions, relationships and outliers, housing data to map prices geographically, and stock and country data to show changes over time.

The session completed exploratory analysis and feature engineering for a Titanic survival prediction task. It examined missing data, gender, passenger class, age distributions, fares and outliers, using bar charts, pie charts, histograms, distribution plots and box plots to identify relationships with survival. The class discussed removing unhelpful columns, imputing missing ages using grouped averages, and preparing data so that it is numerical and suitable for machine learning. It introduced ordinal/label encoding and one-hot/vector encoding, including their trade-offs, the k−1 rule, and the encoding of binary and passenger-class variables. The session then introduced the distinction between training and testing data, explaining how models learn from labelled examples and are evaluated on unseen examples.

The session introduced front-end and back-end web development, including Hypertext Transfer Protocol (HTTP) requests and responses, and the roles of browsers, web servers and web applications. It covered Hypertext Markup Language (HTML) structure and semantics, including headings, paragraphs, images, links, sections, tables, lists and forms, with attention to accessibility attributes and search-engine visibility. Cascading Style Sheets (CSS) were then used to style pages through inline, internal and external stylesheets, selectors, specificity, colours, sizing, positioning, the box model, display types and Flexbox layouts. A practical exercise began building and styling a Kanban-style to-do application that was intended to be made interactive with JavaScript.

The session introduced client-side JavaScript as a dynamically and weakly typed language run by the browser, covering variable scope with var, let and const, loops, conditionals, functions, asynchronous functions, promises and await. It covered arrays, including mixed and jagged arrays, and creating and accessing objects. JavaScript inclusion in HTML through script tags was explained, including the effects of placing scripts in the head or at the end of the body. Practical Document Object Model (DOM) work included selecting elements, changing text, HTML, styles and CSS classes, creating animations, and responding to events with event listeners. A to-do list exercise began, using DOM content loading, form submission handling, input validation, dynamic task creation and appending task elements to a list.

The session introduced backend web development with ASP.NET Core, Razor templates, Entity Framework and SQL Server. It explained HTTP request–response communication, URL parameters and the intended use of GET, POST, PUT and DELETE methods, including idempotency. Model–View–Controller architecture was reviewed, alongside routing, controllers, views, layouts, static assets and transferring data from controllers to Razor views. A blog application was built to retrieve posts from a database, display individual posts and their related comments using Entity Framework, and submit new comments through a form followed by redirection. The session then began adding user registration, covering user models, validation annotations, password hashing, anti-forgery tokens and data transfer objects for registration forms.

The session introduced application programming interfaces, focusing on web APIs, Representational State Transfer (REST) principles and the need for predictable, scalable, well-documented and non-breaking interfaces. It compared REST with GraphQL as a layer that can unify queries across multiple services and manage resource costs. The main HTTP methods were explained: GET for retrieval, POST for processing or creation, PUT for idempotent replacement or update, and DELETE for removing a resource state. A bookstore API was designed by identifying resources, permitted actions and routes, then implemented in ASP.NET using controllers, models, database context and HTTP status codes. The session also began covering API authentication using authorisation headers, bearer tokens and JSON Web Tokens, including expiry and permission claims.

The session introduced relational databases, explaining how tabular data is organised into rows and columns and how tables are linked through primary and foreign keys. It contrasted relational and non-relational data models, using document and key-value structures to show why highly variable data may create sparse tables. The class explained client-server database architecture, including database servers, clients, web interfaces, back-end code, SQL queries, connection details and access permissions. It also compared proprietary and open-source database systems, and outlined the course case-study approach and SQL-based assessment.

The session covered restoring a SQL Server database from a backup file, including verifying the backup and using the database interface to inspect tables. It reviewed relational tables, primary and foreign keys, schemas, and the use of schemas for functional organisation and access permissions. Students wrote basic SQL queries using SELECT, FROM, USE and COUNT, and discussed why row counts should be checked before using SELECT * on an unfamiliar or very large table. The class also examined database collation, including case and accent sensitivity, and explored customer data to identify data-quality inconsistencies and infer how customer and billing identifiers were structured.

Self-joins compared product prices, including a query for non-USB products priced at least as highly as a USB product. Text filters, wildcard placement and case conversion affected which rows matched; DISTINCT addressed repeated results from multiple comparisons. An entity–relationship diagram then connected customers, orders and invoices through their keys. The class explored orders without invoices using an outer join with missing-value filtering and alternative subqueries. These examples led into invoice structure: identifiers, dates, line items, quantities, prices and totals. The queries were examined in their data context, without asserting that every alternative behaves identically for all missing values.

The session reviewed database normalisation, covering first, second, third and Boyce-Codd normal forms, functional dependencies, atomic values, composite keys and the risks of duplicated or dependent non-key data. Examples from a sample database were used to identify non-atomic JavaScript Object Notation (JSON)-style fields and third-normal-form breaches that could create inconsistent customer and invoice information. The class then restored and explored a second SQL Server database, checked table keys and case-sensitive collation, and used the graphical interface to add and edit a city column. Finally, it practised GROUP BY, COUNT and HAVING, then developed two approaches for finding customers who had bought every product, including a division query using nested NOT EXISTS conditions.

The session introduced the purpose of data warehousing and extract, transform and load (ETL) within business intelligence, focusing on collecting, integrating, modelling and delivering data for decision-making. It explained ETL and extract, load and transform (ELT) processes, data sources such as files, databases and enterprise resource planning systems, and the importance of working from stakeholder requirements and intended reporting outcomes. The class covered staging, operational data storage and data warehouse layers, including bronze, silver and gold data, data marts, and central versus departmental warehouse architectures. It distinguished transactional online transaction processing (OLTP) databases from analytical online analytical processing (OLAP) databases, and stressed trusted, consistent key performance indicators (KPIs) and reusable data models. Students installed and configured Visual Studio, SQL Server Integration Services (SSIS) and SQL Server databases, then explored the SSIS project interface and its control-flow and data-flow tasks.

The session reviewed the role of a staging area in an ETL pipeline and demonstrated loading a sales comma-separated values (CSV) file into a staging database using SSIS flat-file sources, OLE DB destinations and table truncation. It then considered how to design an operational data store to analyse sales by employee, customer and geography, including incorporating geographical information into employee and customer data. A practical SSIS flow was built for employee data, using derived columns to create email addresses and clean salary values, data conversions for salary and birth dates, and a lookup to enrich records with country and continent data. The class also covered column sizing, destination mappings, data viewers for checking transformations, and identifying invalid date values with SQL TRY_CONVERT.

The session developed an operational data store (ODS) sales ETL flow by selecting sales fields, cleaning decimal separators, converting monetary and date values, resizing text columns, and loading the resulting data into an ODS sales table. Technical error handling was configured to redirect rejected rows, record metadata about conversion failures, combine reject streams, and review the resulting errors. The class then introduced data warehouse modelling, distinguishing fact tables containing analysable events and measures from dimension tables containing descriptive attributes. It compared star, snowflake and constellation schemas, and explained primary and foreign keys alongside natural and surrogate keys. Finally, it covered slowly changing dimensions, including types 0, 1, 2 and 3, and how each approach handles changes and historical dimension data.

The session covered the implementation of Slowly Changing Dimensions (SCD) types 2 and 3 in SSIS for employee data. SCD type 2 was explained as retaining historical versions through separate rows, start dates and end dates, while type 3 stores a previous value in an additional column. A customer dimension was then configured using the SSIS Slowly Changing Dimension wizard, including fixed, changing and historical attributes and the use of business keys. The class also began building a sales fact-table ETL flow, looking up surrogate keys for customer, employee and date dimensions, handling missing references with an unknown key value of -1, and logging functional rejects. Data-type alignment between source dates and the date dimension was addressed before mapping the final fact-sales fields.

The session introduced Neo4j as a native, schema-optional graph database and compared its relationship-focused storage with relational databases. It covered graph theory fundamentals, including nodes, relationships, directed and weighted graphs, labels, properties and shortest-path traversal. Students explored common graph-database applications such as fraud detection, recommendation systems and network analysis. The class introduced Cypher syntax for matching, filtering and returning nodes and relationships, using a movie graph dataset for practical queries. Students also practised creating nodes and relationships with MERGE, updating or removing properties, and using conditional creation and matching behaviour.

The session demonstrated how to import comma-separated values (CSV) data into Neo4j, create uniqueness constraints, load person, location and visit nodes, and create relationships between them. It then introduced intermediate Cypher querying, including case-insensitive string filtering, checking query plans with EXPLAIN and PROFILE, and writing efficient graph traversal patterns. Further activities covered OPTIONAL MATCH, ordering, filtering null values, limiting and paginating results, and removing duplicates with DISTINCT. The class also practised map projections and CASE expressions to customise returned JSON-style objects and categorise movie runtimes.

The session introduced cloud computing models, service types and deployment approaches, including the benefits and limitations of public, private, hybrid and multi-cloud environments. It compared using Neo4j through Amazon Web Services (AWS), Google Cloud and Azure marketplaces, including single instances, clustered deployments, containers and relevant certification routes. The class then practised intermediate Cypher queries using aggregation functions such as COUNT, COLLECT, SUM, AVG, MIN and MAX, alongside DISTINCT, UNWIND, list indexing and list slicing. It also covered date, time and duration values in Cypher, including calculating intervals and converting timestamps for comparisons. Students continued the COVID graph mini-project exercises.

The session introduced NoSQL databases, explaining how document, graph, wide-column and key-value databases differ from relational databases and suit different use cases. It compared SQL and NoSQL in terms of schema flexibility, scalability, transaction guarantees, querying, performance, security and vendor lock-in. MongoDB was presented as a document-oriented database using Binary JSON (BSON), with databases, collections, documents, fields, embedded documents, arrays and object IDs. Students installed MongoDB and Compass, created databases and collections, inserted, deleted and updated documents, and imported a restaurant dataset. The class then explored filtering, projections, sorting, pagination, logical operators, comparisons, field existence, regular expressions and array queries, before introducing equivalent operations through the PyMongo Python library.

The session introduced MongoDB aggregation pipelines, in which documents pass through ordered stages such as $match, $project, $group, $unwind, $sort and $lookup. It demonstrated accumulator operations including counts, sums, averages, minima, maxima, and first or last values, alongside reshaping, cleaning and constructing fields within pipelines. Examples used restaurant data to calculate statistics by borough and cuisine, flatten arrays of inspection grades, create map-ready outputs, and obtain random samples. The class then covered indexes, explaining collection scans versus index scans and the use of single-field, compound, text, unique, time-to-live (TTL), wildcard, collation and geospatial indexes. Practical work was assigned to create indexes and write queries and aggregations that generate a restaurant report.

The session introduced big data as data that cannot be stored or processed effectively by a single machine, focusing on volume, velocity and variety. It explained distributed systems, horizontal and vertical scaling, clusters and nodes, along with the benefits and challenges of distributing computation across networked machines. The consistency, availability and partition tolerance (CAP) theorem was used to compare consistency, availability and partition tolerance in distributed data storage. The Hadoop ecosystem was outlined, including Hadoop Distributed File System (HDFS) for storage, Yet Another Resource Negotiator (YARN) for resource management, and processing tools such as MapReduce and Spark. Students began a practical activity by connecting securely to a Hadoop edge node through Secure Shell (SSH) and practising basic Linux command-line use.

The session reviewed distributed systems, Hadoop HDFS and YARN, including master, worker and edge nodes, replication, fault tolerance and high availability. It distinguished online transaction processing (OLTP) from online analytical processing (OLAP) systems and compared databases, data lakes, data lakehouses and data warehouses, including atomicity, consistency, isolation and durability (ACID) properties. The class introduced Apache Hive as a data-warehouse tool built on HDFS, using HiveQL and execution engines to query stored data. It covered external versus managed Hive tables, row-based and columnar file formats, Hive partitioning, and the bronze, silver and gold stages of the medallion architecture. Students then began practical Hive exercises creating and querying external and managed tables, including use of ORC files.

The session reviewed core big-data concepts, including the three Vs, data types, horizontal versus vertical scaling, networking risks and the CAP theorem. It covered Hadoop’s main components: HDFS for distributed block storage, YARN for cluster resource management, and MapReduce for distributed processing. The class examined how HDFS splits and replicates files, why small files create metadata overhead, and why processing should be located with the relevant data blocks where possible. MapReduce was introduced through a word-count example, explaining map, shuffle and sort, reduce, network communication, and disk-based fault tolerance. Students then worked on Hadoop-based Python MapReduce word-count exercises and began designing a MapReduce job from scratch.

The session reviewed distributed-system reliability, including single points of failure, standby NameNodes, split-brain scenarios, leader election and ZooKeeper heartbeats. It compared MapReduce and Spark, covering disk-based versus memory-based fault tolerance, resilient distributed datasets, transformations and actions, lazy evaluation, and narrow versus wide transformations and shuffles. The teaching then introduced Spark DataFrames and Spark Structured Query Language (SQL) as higher-level, more readable alternatives to low-level resilient distributed dataset (RDD) programming, explaining schemas, the Catalyst Optimiser and the benefits of explicitly defining schemas for large production data sets. Students began a practical Docker-based Jupyter lab using New York City taxi data to explore DataFrame operations, SQL queries and joins, with exercises requiring work in both PySpark and Spark SQL.

The session introduced real-time event streaming, distinguishing unbounded streams from bounded datasets and comparing hard, soft, near-real-time and batch processing requirements. It reviewed stream-processing concepts including event time versus processing time, windows, triggers and watermarks, and strategies for refining aggregated results. The class then examined Kafka as an event-streaming platform, covering events, producers, consumers, brokers, topics, partitions, offsets and consumer groups. It explained partitioning by round-robin allocation or message keys, replication through leaders and followers, producer acknowledgements and delivery guarantees, and leader election using KRaft rather than ZooKeeper. Students were asked to deploy a Kafka cluster in containers, run a consumer, and optionally process streamed events with Python, Spark or an extract, transform and load (ETL) service.

The session reviewed OLAP and OLTP workloads, row-based and columnar file formats, the medallion architecture, Apache Spark, and Kafka’s publish-subscribe model. It introduced Apache NiFi as a tool for managing data flows and compared ETL, extract, load and transform (ELT) and light-transformation loading pipelines. The class covered NiFi’s security, scalability, delivery guarantees, data provenance and lineage, clustering, registry-based version control, and core components including flow files, processors, connections, funnels and process groups. It also considered why NiFi is best suited to moving data and carrying out light transformations rather than heavy processing. Students began a practical Docker-based lab to build, version and document a basic NiFi data flow.

The session reviewed Apache NiFi as a tool for building ETL and ELT data flows, contrasting transformation before loading with transformation after loading into a data lake. It covered NiFi architecture, including flow files, metadata attributes, content, provenance, repositories, processors, controller services and clustering. The class demonstrated the NiFi interface, process groups, ports, funnels, connections, relationships, queues, back-pressure thresholds, load balancing and debugging. It also explained scheduling, retry and penalty behaviour, parameter contexts, expression language, and version-control alternatives following the planned retirement of NiFi Registry. Students were assigned sequenced labs to build increasingly complete pipelines, culminating in consuming Kafka streams with NiFi, transforming the data, and loading cleaned results into PostgreSQL.

The session reviewed distributed cluster roles and introduced four security principles: identification, authentication, authorisation and encryption. It explained how LDAP, Kerberos, Apache Ranger, Knox and Atlas can support identity management, ticket-based access, role-based permissions, gateway access and governance in Hadoop ecosystems, with comparisons to cloud-native tools. The class then compared file, block and object storage, focusing on object storage features such as buckets, metadata, flat namespaces, immutability, S3-compatible application programming interfaces (APIs) and storage tiers. Software-defined storage, MinIO, Ceph and erasure coding were discussed as ways to provide scalable, resilient cloud storage. Students were assigned a lab to deploy MinIO in Docker and create and manipulate object-storage buckets.

Data formats were distinguished from web programming languages and human-readable presentation. XML introduced nested elements, attributes, declarations and a tree structure, alongside the distinction between well-formedness and validation. Namespaces, resource identifiers and language tags addressed how names and context are expressed. Students created, formatted and viewed a simple document in an editor and browser. Comparison with JSON and graph-based semantic data placed the exercise within a wider discussion of representations used to exchange information between systems.

XML Schema introduced simple and complex types, element and attribute declarations, and rules governing how often information can occur. Ranges, patterns and enumerations constrained values, while type extension offered a way to build on definitions. Students created and validated schemas for bibliography and film data, deliberately adding or removing information to test the rules. Namespace and schema-location declarations connected documents to their definitions. Work then began on more precise types for a curriculum example within the session.

XPath expressions introduced paths, tests and conditions for selecting nodes in an XML tree. Extensible Stylesheet Language Transformations (XSLT) then used matching templates to generate other representations. The class examined template application, default rules, specificity, extracted values and dynamically constructed output. Catalogue and film examples supported activities in filtering, counting, sorting and formatting results for web presentation. The focus was on transforming existing structured information through declared rules rather than manually reproducing each output record by hand.

JSON objects, arrays and primitive values were compared with XML structures. Ordered collections and repeated items led into JSON Schema rules for properties, required fields, types and constraints. Students created representations and schemas for bibliographies, catalogues and film data, then changed the data to test validation. Restrictions on additional information and dependencies extended the contract beyond basic types. The class also prepared to transform XML film records into JSON using XSLT, carrying the transformation theme across formats.

The session introduced the aims of production-grade data engineering, including end-to-end pipelines, common engineering patterns, infrastructure as code, continuous integration and delivery (CI/CD) and the role of data engineers in connecting transactional and analytical systems. It compared online transaction processing (OLTP) and online analytical processing (OLAP) workloads, discussed data lakes and lakehouses, and explained why SQL and self-service data solutions are widely used. The class covered conceptual, logical and physical data modelling; normalisation through the first, second and third normal forms; denormalisation; star and snowflake schemas; slowly changing dimensions; fact tables and dimension tables. It also examined data retention, sampling, bucketing, query plans, pre-computation and data-granularity trade-offs, including bit-based date-list encoding for efficient activity and churn queries. The final section introduced functional programming principles for pipelines, especially atomicity, idempotency and avoiding side effects, alongside factory and strategy design patterns in Python ETL code.

The session explained Python context managers, data classes and Pydantic models as ways to manage resources safely, represent structured data clearly and validate incoming JavaScript Object Notation (JSON) data early. It covered pytest features including parametrised tests, fixtures, exception testing, test-driven development, mocking and dependency injection, with emphasis on testing transformation logic and Spark applications. Further Python topics included caching, decorators, object references versus copies in pandas, type hints, protocols, singleton design and avoiding magic strings and excessive coupling. The class then introduced Hadoop-related technologies, Parquet compression and Hive metadata, alongside Ranger-based data-access controls. It also covered distributed processing and Spark concepts, including MapReduce, lazy evaluation, partitioning, joins, testing, and the write-audit-publish pattern for data quality.

The session introduced Amazon Web Services (AWS) Glue as a serverless way to run Spark workloads and use a central data catalogue, alongside other cloud and orchestration options. It explained lakehouse architecture and Apache Iceberg table formats, focusing on metadata, snapshots, manifests, partition pruning, time travel, schema and partition evolution, upserts, and write-audit-publish workflows. Practical exercises were outlined for using Spark, Iceberg and Docker to create tables, merge data, inspect metadata, and validate data before publication. The session also covered infrastructure as code with Terraform, including providers, resources, variables, modules, dependencies, plans, state, identity and access management (IAM) permissions and scheduled AWS Lambda deployments. Finally, it refreshed CI/CD concepts such as pipelines, runners, artefacts, reusable container images, formatting checks and security scanning for Terraform.

The session developed an AWS data pipeline that moves change-data-capture files from a transactional system into Iceberg tables for analytical use. It demonstrated creating Glue catalogue databases and Iceberg tables, then using Spark SQL MERGE operations to apply inserts, updates and deletions to users and orders data. A country-level analytics table was built by joining the source tables, deriving order years and aggregating order values. The prototype was then refactored towards a production-oriented design using modular Python code, typed configuration, unit tests, Terraform-managed infrastructure, least-privilege identity and access management (IAM) permissions and deployment of code artefacts to S3. The session also discussed how orchestration tools could detect arriving files and trigger Glue jobs with runtime parameters.

The session introduced the legal, ethical and geopolitical issues raised by data and artificial intelligence, emphasising that regulation varies across jurisdictions and societal contexts. It explored why artificial intelligence (AI) and data engage areas including data protection, liability, intellectual property, constitutional law, international law and discrimination. Students considered the limits of using AI in legal decision-making, including predictive systems for human-rights cases, balancing legal certainty and efficiency against human interpretation and the evolving nature of law. The class also examined definitions of AI, the distinction between narrow and general AI, calls to regulate future technological risks, and the possible legal personhood of robots. A case discussion considered arguments for and against a conversational robot inheriting an owner’s assets.

The session examined the development and purpose of the European Union (EU) AI Act, including tensions between innovation, competitiveness, sovereignty, public regulation and private ethical codes. It distinguished legal obligations from voluntary ethical commitments, explaining why enforceable rules and sanctions are significant. The class explored the Act’s risk-based model, covering prohibited, high-risk, limited-risk and minimal-risk AI systems, alongside the duties imposed on providers and other actors. It also considered the Act’s extraterritorial reach, exclusions such as military and research uses, enforcement mechanisms, regulatory sandboxes, and the particular rules for general-purpose AI models. Students discussed practical difficulties in assessing risks to fundamental rights, democracy and human dignity, as well as potential conflicts between European and international regulatory approaches.

The session introduced data protection law through discussion of children’s data, profiling, algorithmic bias and the limits of consent in digital services. It examined why data protection is important for AI, comparing the European rights-based approach with the more sector-specific approach in the United States. The class covered privacy as a broader concept than information control, including dignity, intimacy, home and communications, and considered the EU Charter rights to privacy and personal-data protection. It then introduced the General Data Protection Regulation (GDPR), including personal data, identifiable natural persons, processing activities, material and territorial scope, and the basic obligations on organisations processing data. Students discussed examples involving health data, policing, profiling, data transfers and cross-border services.

The session examined GDPR data-subject rights in AI systems, including access, rectification, erasure, restriction of processing, portability and objection, and explained privacy by design across training and deployment. It considered how biased datasets and algorithmic design can create or reinforce discrimination, and discussed fairness, auditing, impact assessments and ethics by design. The class then introduced the EU Data Act, its application to data from connected products and related services, its relationship with the GDPR, and rules on user-led data sharing, competition, gatekeepers, contracts, compensation and dispute resolution. The final section used neurotechnology to explore mental privacy, brain data, cognitive liberty, and the need for ethical and legal safeguards against misuse by companies or governments.

The session examined the ethical, legal and professional responsibilities of scientists and data specialists in relation to autonomous lethal weapons and AI used in conflict. It considered the limits of existing regulation, particularly the military exemption in the AI Act, and debated whether emerging technologies should be regulated or prohibited before they are fully developed. The class compared arguments for and against autonomous weapons, including meaningful human control, bias, accountability, international consensus and historical preventive bans on certain weapons. It also discussed whether advanced AI systems merely imitate reasoning or might develop forms of understanding or consciousness, and considered the implications for law and responsibility. Finally, the session explored corporate due diligence, value-chain oversight and contractual controls through examples of technology companies supplying tools that may be used for immigration enforcement, surveillance or defence.

The session introduced the course structure and its aim of preparing students to participate in or lead data science projects. It contrasted traditional, waterfall-style project management with agile approaches, explaining time-boxed sprints, sprint reviews, retrospectives, planning and visual task boards. A project was defined as a temporary undertaking with a unique objective, a defined timescale and required resources, distinguishing it from routine operations. The class also examined product and sprint backlogs, prioritisation, team commitment, transparency and measuring progress. Business cases, break-even points, return on investment and the need to reassess a project when business conditions change were introduced.

The session began with sprint planning and reflection, using completed work and team capacity to agree a realistic commitment for the next sprint. It covered project vision statements, the limitations of AI-generated content, and the need to understand project-management concepts well enough to assess artificial intelligence (AI) outputs critically. Key traditional project-management concepts included the triple constraint of scope, time and cost; project life-cycle phases; uncertainty, risk and the rising cost of change; and the Project Management Institute (PMI) waterfall framework, including project charters, planning, resources, communication, procurement and stakeholder management. Students practised preparing elements of a project charter and assigning stakeholder roles through a responsible, accountable, consulted and informed (RACI) matrix, then discussed stakeholder power and influence. The class also covered Gantt charts, task estimation, S-curves, earned value, cost and schedule variance, key performance indicators (KPIs), and the Plan-Do-Check-Act cycle.

The session used a simulated sprint review, retrospective and planning exercise to demonstrate approval of completed work, velocity tracking, forecasting and commitment to a minimum sprint scope. It introduced Agile values from the Agile Manifesto, stressing individuals and interactions, working software, customer collaboration and responding to change whilst recognising the continuing value of plans, documentation and contracts. Agile and waterfall approaches were compared in terms of fixed constraints, planning, suitability for different levels of uncertainty, stakeholder involvement and regulated environments. The class examined timeboxing, incremental delivery, technical debt, minimum viable products, proof of concepts, product backlogs, prioritisation and Agile contract models. It also covered the roles of the product owner, Scrum Master and self-managing cross-functional teams, followed by a scenario activity, quiz and video on Agile product ownership.

The session showed how an agile backlog can be adapted as new information emerges, using prioritised user stories and changing investigation scenarios to illustrate this process. It emphasised the importance of direct, timely communication for agile teams, particularly where teams are distributed. Agile software development practices included automated testing, continuous integration and delivery, code standards, collective code ownership, technical-debt reduction, and responsible use of AI-assisted programming. The class then introduced scaled agile working through Scaled Agile Framework (SAFe), including Agile Release Trains, programme increments, programme increment (PI) planning, cross-team dependencies, and the roles of product manager, Release Train Engineer and system architect. It also considered how agile delivery can be connected with wider business strategy and product value.

The session introduced customer relationship management (CRM) data modelling in Dataverse, including tables, primary and foreign keys, and one-to-many, many-to-one and many-to-many relationships. It compared on-premises and cloud CRM deployments, and covered Dataverse features such as application programming interfaces (APIs), security, auditing, business rules, workflows, duplicate detection, reporting and integrations. Students learnt how development, test and production environments are managed, including environment types, access groups, backup, copying and solution deployment. Practical activities involved accessing a shared CRM environment, creating account and contact records, creating a solution with a publisher and prefix, and creating, publishing and testing a custom table. The session also explained table ownership, activity tables, virtual tables, auditing, quick create forms and record management.

The session began with a practical exercise in exporting account data to Excel, amending a phone number, importing the file back into the CRM system, and checking that the change had been applied. It then covered Dataverse security, including business units, security roles, record privileges, access levels, teams, column-level security and hierarchy security. Further topics included environment analytics, storage capacity, auditing, duplicate detection, bulk deletion and long-term data retention. The class also introduced model-driven, canvas and Power Pages applications, before practising the creation of a model-driven app with grouped tables for customer and sales functions.

The session covered testing and troubleshooting Power Automate flows using Flow Checker, test runs, run history, action statuses and error details. It also explained administrative tasks such as monitoring flow activity, reviewing run histories and sharing flows with appropriate permissions. Business rules were introduced as a no-code way to apply conditions and actions, including showing, hiding, enabling, disabling or requiring form fields, with a practical exercise to control four contact fields according to whether an account had been selected. The class then introduced business process flows as stage-based guides for users, including their steps, limits, associated tables, conditional branches and use in opportunity management.

The session explained the CRM sales pipeline, from creating leads through imports, website forms or direct contact, to recording follow-up activities such as calls, emails, meetings and tasks. Leads can be qualified to create account, contact and opportunity records, or disqualified with a recorded reason while remaining available for later reactivation. Opportunities use price lists and product line items to calculate prices and totals, then progress through quotes, orders, fulfilment, payment and invoicing. The class also distinguished system-generated record IDs from customer-facing quote and order numbers, discussing uniqueness and restrictions on changes after creation. A practical activity asked students to carry out both a successful sales-pipeline case and a disqualified-lead case in the Sales Hub application.

DevOps was introduced as collaboration between development and operations, supported by communication and automation. Waterfall and agile approaches were compared through iteration, stakeholder feedback and sustainable working practices. Reliability discussions covered monitoring, service objectives and the expectation that failures will occur. Semantic versioning and naming conventions introduced ways to communicate changes clearly. Students began a small project lab with documentation and a change log, preparing to use Git and Node.js as part of the practical work.

The Git workflow covered repositories, staging, commits, remote updates and inspection of differences. Automated testing was discussed through unit, functional, integration and end-to-end checks, including browser automation. Test-driven development introduced a cycle of writing a failing test, implementing the behaviour and refactoring while keeping the checks passing. Students began a JavaScript lab for user-related application programming interface functionality. Self-contained tests, setup, validation and cleanup connected the method with tests that are fast enough to use repeatedly.

Continuous integration and continuous delivery or deployment connected version control, tests and releases. Infrastructure as code then introduced declarative and imperative approaches, repeatable application of configuration, and mutable or immutable infrastructure. Vagrant provided a setting for managing virtual machines, while Ansible introduced declarative provisioning. Students worked on labs using shell provisioning and Ansible to deploy and check a GitLab environment. Earlier discussion of small changes and testing remained relevant as the subject moved from application code to its infrastructure.

Containers were compared with virtual machines through isolation, shared operating-system resources and portability. Docker demonstrations covered obtaining images, running containers and managing their lifecycle. Dockerfiles introduced custom builds through base images, copied files and build commands, including tagging and troubleshooting. Storage volumes addressed data that must persist, while Docker Compose described applications with several containers. A WordPress site and MySQL database illustrated this arrangement, connecting application packaging with the services and data it needs to run.

Kubernetes introduced deployment, networking, scaling and recovery across a cluster. The class examined control-plane and worker components, then used manifests and commands to inspect and change resources in Minikube. A demonstration showed a deployment replacing a deleted managed pod, unlike a standalone pod. Namespaces, labels and services helped organise the workloads. Configuration, secrets and persistent storage were introduced before students were directed to practical labs, extending the discussion from individual containers to the system that manages them.

The class traced a pod request through authentication, scheduling and container creation before examining storage choices. Temporary volumes, host-mounted data, configuration and secrets raised different persistence and security questions. Demonstrations covered static and dynamic provisioning through persistent volumes, claims and storage classes. Binding, reclaim policies and pending workloads were investigated, including checking data after pod replacement. Students were directed to practise temporary and host-path volumes, with further persistent-volume work offered as an optional exploration rather than assumed completed work.

Cloud-native architecture linked loosely coupled services with containers and delivery practices. Kubernetes networking was examined through pod communication and stable service addresses, with demonstrations of different service types and troubleshooting. Load balancing and ingress routing introduced distinctions between transport and application-level traffic handling. Service meshes then brought together observability, encrypted service communication and traffic control, including canary releases. The discussion connected these mechanisms with the architecture around them rather than presenting orchestration as simply another way to start a container.

Grafana introduced dashboards over metrics queried from Prometheus. The class examined connections, local access, query-based panels and imported dashboard definitions, then discussed related tools for logs, traces and metric storage. GitOps connected declarative infrastructure with Git as the record of intended configuration. Push and pull approaches were compared, including Argo CD's reconciliation of Kubernetes applications. Students were asked to complete labs using the monitoring and reconciliation tools, bringing observation and configuration management into the same operational discussion.

The session introduced cybersecurity through physical security, secure system design and the importance of planning security controls early. It examined software-update lifecycles, vulnerabilities, exploit value, mass surveillance risks and the security implications of physical access to devices. Students used a virtual machine to create a memory dump after a failed web login, extract readable strings, locate credentials in memory, and use regular expressions and CyberChef to transform data. The class also covered symmetric and asymmetric encryption, public and private keys, certificate authorities, and the role of trust in encrypted web connections. Further examples illustrated how memory modification, malicious USB devices and keystroke injection can compromise systems.

The session examined social engineering and phishing, including how attackers impersonate brands through misleading URLs, fake applications, adverts and artificial-intelligence-generated content. Students learned to identify the genuine domain within a URL, recognise typosquatting and subdomain deception, and understand how phishing pages can evade automated detection through obfuscation. The class compared authentication methods, covering strong unique passwords, password managers, multi-factor authentication, passkeys and hardware security keys, with emphasis on phishing-resistant FIDO2 (Fast Identity Online) authentication. It also covered wireless threats such as evil-twin access points, physical attacks on devices and firmware, data-at-rest encryption, and the use of BitLocker or FileVault to protect stored data. Practical activities included analysing password recovery from video, considering cloned-audio scams, and resetting a local Windows password in a virtual machine to demonstrate the importance of disk encryption.

The session introduced FTP as a file-transfer protocol and compared command-line FTP with the FileZilla graphical client. Students used Wireshark to capture network traffic, inspect packets, and observe that unencrypted FTP credentials can appear in clear text, whereas Transport Layer Security (TLS) protects data in transit. The class examined denial-of-service concepts, including bandwidth saturation, packet flooding and socket exhaustion, through controlled demonstrations using ping, hping3 and Slowloris. It also covered the MITRE ATT&CK and D3FEND frameworks, Common Vulnerabilities and Exposures (CVE) and Common Weakness Enumeration (CWE) identifiers, and Common Vulnerability Scoring System (CVSS) scores for assessing vulnerabilities and their impact.

The session reviewed network reconnaissance: identifying an Internet Protocol (IP) address range, finding active hosts with ping sweeps, scanning exposed services with Nmap, and selecting appropriate tools to interact with those services. It demonstrated how anonymous FTP access, weak passwords and excessive permissions can expose files and enable unauthorised access, alongside the principle of least privilege and password-strength considerations. The class explored credential attacks and defences, including word lists, online and offline password cracking, stronger authentication, virtual private networks (VPNs), port knocking and canary accounts. It also covered local-network interception techniques, including name-resolution poisoning, Address Resolution Protocol (ARP) cache poisoning and man-in-the-middle attacks, plus the use of Wireshark and defensive monitoring. Finally, it introduced open-source intelligence, advanced search modifiers, subdomain discovery, cloud-hosted services and Shodan for identifying internet-exposed assets.

The session examined why some software vulnerabilities are difficult to eliminate at scale, including memory-corruption flaws, Rowhammer and the move towards memory-safe languages such as Rust. It discussed ransomware as a criminal business model, including encryption, data theft, double extortion, negotiation and money laundering. Students used CyberChef to practise data extraction with regular expressions and to explore Base64 encoding, Advanced Encryption Standard (AES) encryption, entropy and frequency analysis. The class also considered how phishing sites imitate legitimate services and how users can identify them by checking context, urgency and domain names.

The session introduced analysis and design as the work required before implementation, stressing the need to clarify requirements and plan solutions before writing code. It examined software complexity through the large number of possible system states, using general-purpose tools such as spreadsheets to show why complete testing is often impossible. The class compared information technology challenges with mathematically modelled engineering problems, and discussed the limitations of generative artificial intelligence and machine learning as tools for reasoning and software development. It defined information systems as organised processes for transforming data into information and potentially knowledge, with data models enabling useful access to raw data. The session also used the house-building analogy to explain requirements analysis, design, implementation, compliance, integration, security, maintenance, liability and software quality.

The session introduced information-systems analysis as the phase for understanding business needs before making design or technology choices. It covered requirements, stakeholders, business rules, events, data attributes and data dictionaries, emphasising the need to distinguish what a system must achieve from how it will be implemented. The class compared models and methods, explaining that models represent reality while methods provide steps for operating on a model, and discussed why analysis relies on clear, simple models rather than a guaranteed method. Examples involving JavaScript Object Notation (JSON), Extensible Markup Language (XML) and user-interface prototyping illustrated the importance of precise terminology, data representation, validation, and documenting operational requirements.

The session introduced context diagrams for analysing an information system as a black box, identifying external actors, events and time-driven triggers. It explained the distinction between data-carrying events and signals, and compared pulling data from external sources with receiving pushed data through gateways, queues and agreed protocols. The class examined application programming interfaces (APIs) as interfaces exposing reusable software functions, particularly for exchanging data between systems. It then modelled responses to events as action flowcharts, using checks for mandatory attributes and duplicates before creating a customer, and distinguished programming from coding and software engineering. The session also began data-dictionary analysis, discussing attributes, identifiers, universally unique identifiers (UUIDs), relational versus graph databases, scalability, transaction management and data-privacy implications of universal identifiers.

The session examined the relational data model, including relations, attributes, tuples, primary keys and foreign keys. It explained how functional dependencies are identified through business context and used to derive tables, keys and associations, with one-to-one relationships considered before many-to-many relationships. Relational algebra was introduced as the basis of Structured Query Language (SQL), covering selection, projection and joins, alongside query planning and optimisation using database statistics. The class began normalisation, focusing on first normal form and atomic values, then second normal form and the problems caused by partial dependencies in composite keys. A human-resources allocation example was used to show data inconsistency, redundant storage and the need for programmed checks to enforce dynamic constraints such as overlapping employment periods.

The session introduced calculus as a foundation for optimisation, machine learning and statistical modelling. It defined functions, linear and affine functions, and common nonlinear functions including powers, exponentials and logarithms. Derivatives were explained as local slopes, with rules for sums, constants, powers, products, quotients and composite functions, alongside links to neural-network differentiation. The class then extended derivatives to several variables through partial derivatives, higher-order and mixed partial derivatives, and their use in describing slopes in different directions. Taylor series and Taylor polynomials were presented as local approximations, with applications to numerical computation, optimisation methods and interpolation.

The session began with a recap of the product rule for derivatives, using a rectangle-area argument and the limit definition to show why the small product of two variations vanishes. It then introduced vectors as one-dimensional arrays, covering vector addition, scalar multiplication, linear combinations, and the importance of matching dimensions in mathematical and programming operations. Linear independence, dependence, collinearity and bases were explained as ways to identify redundant variables and reduce the dimensionality of data. The dot product, orthogonality, vector norms, normalisation and orthonormal bases were developed, with links to correlation, variance, regression and numerical scaling. The class also introduced changes of basis, including their role in principal component analysis (PCA), compression and viewing data from more informative directions.

The session examined matrix inverses, showing that a square matrix is invertible exactly when its columns are linearly independent, equivalently when it has full rank. It introduced matrix rank, numerical sensitivity near singular matrices, and the special simplicity of diagonal matrices, whose inverses and determinants can be computed element by element. Determinants were defined and calculated for two-by-two and three-by-three matrices using minors and cofactors, with discussion of why this recursive method is impractical for large matrices. The class then introduced trace, eigenvalues and eigenvectors for symmetric matrices, deriving the characteristic equation and using orthonormal eigenvectors to diagonalise a matrix through a change of basis.

The class examined how eigenvalues and eigenvectors in data analysis can reveal approximate linear relationships and support linear regression, including the effect of scaling variables and interpreting small eigenvalues as residual error. It covered adding an intercept through centring variables or including a column of ones, comparing resulting coefficients with a statistical regression model, and avoiding overfitting by selecting simpler models and validating them. The session also explained how changing to eigenvector coordinates can reveal trends in data that are not visible when plotting individual variables, linking this to principal component analysis. It then introduced positive-definite matrices and quadratic forms through the signs of eigenvalues, with applications to convexity and optimisation. Finally, it introduced complex eigenvalues of non-symmetric matrices, complex-number arithmetic, Cartesian and polar forms, and the use of complex exponentials in trigonometry and Fourier transforms.

The session introduced the course’s focus on probability, statistics and their role as foundations for machine learning, with R and RStudio as supporting tools. It reviewed mathematical notation for sums, products and sets, including unions, intersections and complements. Descriptive statistics were covered through frequency tables, bar charts, histograms, density plots and empirical cumulative distribution functions, with emphasis on interpreting distribution shape. The class defined quantiles, the median, mean, variance, standard deviation, interquartile range, skewness and kurtosis, and used box plots to examine spread and potential outliers. It then introduced multivariate descriptive analysis using contingency tables, scatter plots, covariance and correlation, stressing that correlation does not establish causation.

The session completed an introduction to probability distributions for finite, countably infinite and continuous sample spaces. It covered conditional probability, Bayes’ theorem and base rates through diagnostic-test, screening, production-defect and transmission examples, as well as the Monty Hall problem. Independence, incompatibility and mutual independence were distinguished using coin-toss and permutation examples. Students then completed practice multiple-choice questions on descriptive statistics, including means, medians, empirical cumulative distribution functions and probability calculations. The session ended by introducing random variables as functions mapping outcomes to real-number values and motivating probability distributions for them.

The session reviewed random variables through uniform, binomial, Poisson and normal distributions, focusing on their parameters, probability mass or density functions, cumulative probabilities, quantiles, means and variances. Simulated samples were compared with theoretical distributions to show how observed data can differ from ideal probability models. The class then introduced multivariate random variables, joint and marginal distributions, conditional distributions, independence, covariance and correlation, including a bivariate normal simulation and covariance matrix. It concluded with an introduction to convergence, random samples that are independent and identically distributed, parameter estimation, and the motivation for the law of large numbers and central limit theorem.

The session reviewed descriptive statistics, probability theory, random variables, probability distributions, convergence, the law of large numbers and the central limit theorem. It introduced estimators and estimations, including bias, variance, mean squared error and the use of large samples to improve estimates. Confidence intervals were explained as ranges around an estimate associated with a chosen risk level, and a practical activity used one-sample t-tests in R to calculate 95% and 99% confidence intervals for mean height. Statistical hypothesis testing was then introduced through null and alternative hypotheses, rejection regions, p-values, significance levels, and Type I and Type II errors, with examples involving coin tosses and comparisons of group means. The session concluded by outlining common tests for one mean, two means and independence, in preparation for further study.

The session introduced R as free, cross-platform software and explained its role alongside Python in professional data science workflows. Students were shown how to install and use R with RStudio, including arranging the interface panels for code, console output, plots and files. The class covered calculations, vectorised operations, built-in functions, variable assignment, and the creation of vectors using concatenation, sequences and repetition. It then introduced matrices, data frames and lists, including their structures, data types and methods for accessing and modifying values. Practical exercises used indexing, missing values and conditional selection to extract and update data, including calculating summaries for selected groups in a data frame.

The session reviewed how to replace missing values in data-frame variables with their non-missing mean values in R. It introduced function calls, argument ordering and naming, return values, for and while loops, conditional statements, and the use of switch. Students practised writing custom functions, including a general function to impute missing values, and considered scope, function masking, package-qualified calls, and passing additional arguments with .... The class then covered base R graphics, including plot types, axis limits, adding lines, points, titles, labels, legends, text, reference lines and shaded regions. It also introduced histograms, the importance of choosing an appropriate number of bins, and comparing a histogram with an estimated density curve.

The session reviewed histograms, density estimates and box plots, including how to write functions that combine these displays and pass optional arguments to sub-functions. It covered visualising multivariate data with box plots, scatter plots, pairs plots, categorical summaries and legends, using the iris data set as an example. The class introduced ggplot2, including installation and loading of packages, and created histograms, density curves, box plots, bar charts, pie charts, scatter plots and fitted regression lines using vehicle data. It also explored grouping, colouring and faceting plots by categorical variables to assess relationships within subgroups. Finally, it demonstrated how to create an R package containing functions and data, write documentation, build it, install it locally and load it for use.

The class introduced the dplyr package for manipulating large datasets in R, using New York City flight data as an example. Students practised filtering rows with conditions, compared dplyr filtering with base R, and considered the different treatment of missing values. They also used arrange to sort records, select to retain relevant variables, and mutate to create calculated variables such as delay changes and distances in kilometres. The session covered summarise and group_by for calculating and comparing delay statistics by airport, month and airline, with visualisations produced using ggplot. Finally, the pipe operator was introduced for chaining data-processing steps without storing unnecessary intermediate datasets.

The session introduced artificial intelligence as systems that perform tasks normally associated with human intelligence, distinguishing it from general artificial intelligence. It positioned machine learning as learning from data and deep learning as machine learning based on neural networks, while comparing neural networks with classical methods such as regression, random forests, boosting, clustering and support vector machines. Applications in computer vision, audio processing and natural language processing were explored, including image classification, object detection, segmentation, speech recognition, sentiment analysis and question answering. The machine-learning workflow was reviewed: defining objectives and success metrics, collecting and preparing data, selecting and evaluating models, deployment, monitoring and retraining in response to data drift. The class also covered labelled versus unlabelled data, the importance of balanced, shuffled and deduplicated datasets, and cautious use of sampling and data augmentation.

The session reviewed the relationship between artificial intelligence, machine learning, deep learning and neural networks, including the distinction between shallow and deep networks. It introduced PyTorch tensor handling, device selection, graphics processing unit (GPU) availability and moving data between the central processing unit (CPU) and GPU. Data preparation for neural networks was covered through numerical encoding, ordinal and one-hot encoding, and feature scaling using min-max scaling and standardisation. The training cycle was explained: feeding data in batches, forward propagation, loss calculation, backpropagation and parameter updates across epochs. The class examined mean squared error and binary cross-entropy losses, gradient descent, learning rates, vanishing gradients, and batch, stochastic and mini-batch strategies, before comparing momentum, Adagrad, RMSProp and Adam optimisers.

The session reviewed how neural-network models should generalise to unseen data and how datasets should be divided into training, validation and test sets. It covered epochs, mini-batches, loss functions, backpropagation, gradient descent, learning rates and optimisers, including momentum and Adam. The class examined underfitting, overfitting, the bias–variance trade-off, early stopping, model complexity, regularisation, and the importance of data quality and relevant features. It also discussed avoiding data leakage, maintaining comparable distributions across data splits through stratified sampling, and balancing groups where fairness requires it. Students then practised structuring a PyTorch workflow with datasets, data loaders, models, training loops and validation-loss logging in TensorBoard.

The session examined how training and validation curves can indicate overfitting, underfitting, unstable optimisation and the need for early stopping. It covered causes and remedies for poor generalisation, including improving data quality, increasing data or model capacity where appropriate, removing irrelevant features, and adjusting learning rate and batch size. Regularisation methods were explained, including L1 and L2 penalties, dropout, data augmentation and batch normalisation, alongside vanishing gradients and internal covariate shift. The class also discussed why deep networks learn hierarchical features, their advantages for complex unstructured data, and their limitations in computation, data requirements and interpretability. In practical work, students added batch normalisation, dropout and L2 weight decay to a PyTorch classifier, then used Optuna to tune hyperparameters and inspect the resulting trials.

Computer vision tasks ranged from classification and detection to segmentation, pose and depth estimation. The class examined why flattening an image into a general multilayer network can lose useful spatial structure and require many parameters. Convolutional networks introduced filters, receptive fields, padding, stride and pooling as alternatives. A PyTorch classification lab using the CIFAR-10 image dataset was introduced, with normalisation and batching discussed before later fine-tuning work. Historical architectures provided context for the development of these approaches.

Detection and segmentation introduced bounding boxes, overlapping-region measures and architectures that identify or label image regions. Practical guidance covered pretrained weights, evaluation mode, input transformations and confidence thresholds, with training data informing whether adaptation was needed. Transformers then introduced encoder and decoder arrangements, self-attention and parallel processing. Embeddings, tokenisation and vector similarity connected these models with the representation of sequences, extending the discussion from spatial information in images to the structure used for language tasks.

Different transformer architectures were examined through masked, span-reconstruction and next-token training objectives. General pretraining, domain adaptation and task-specific fine-tuning were connected with the need for curated data. Evaluation covered classification, text-comparison measures, error rates, benchmarks and human judgement, including their limitations. Practical activities used language models for prediction, sentiment, similarity, search and extractive questions. Retrieval-augmented generation connected embeddings with retrieved information. An introduction to Whisper then began the transition to speech recognition, transcription and translation.

Project guidance emphasised reproducible experiments, exploratory analysis, controlled comparisons and careful use of generated code. Students practised pretrained audio classification and speech transcription, interpreting scores, labels and language outputs. Waveforms, sampling and spectrograms provided the signal-processing context. Large language models introduced a distinction between base and instruction-tuned systems. Generation strategies and settings were compared through their effects on coherence and diversity, including deterministic selection, sampling and repetition controls rather than a single preferred setting for every task.