Teaching / Course 01
LLM Engineering
From prompting and statistical inference to reliable RAG systems
How to build an assistant that answers from your own documents, cites where each claim came from, and ships with a test suite that catches it inventing things.
A practical course for understanding how large language models behave, designing robust instruction and context layers, and building an evaluated, evidence-grounded application.
- 3 days · 21 contact hours
- Engineers, data scientists, technical leads and advanced AI practitioners.
- One shared codebase developed through guided experiments, retrieval labs and evaluation.
- An evidence-grounded assistant with hybrid retrieval, citations, tests and safety controls.
Feasible as a working-proficiency course, provided GraphRAG indexing and prompt-program optimization are demonstrations rather than separate full projects.
Outcomes
What a participant can do afterwards.
- Explain tokenization, embeddings, attention, next-token prediction, context windows, training and inference at an engineering level.
- Interpret logits, softmax probabilities, entropy, temperature, top-p sampling and the practical limits of probabilistic generation.
- Design system, developer, user and tool instruction layers, including typed and schema-constrained outputs.
- Build a conventional RAG pipeline with ingestion, chunking, dense and lexical retrieval, metadata filtering, reranking and citations.
- Decide when GraphRAG, long-context prompting, database queries or fine-tuning are more appropriate than basic RAG.
- Create an evaluation dataset and measure retrieval quality, groundedness, citation validity, latency and cost.
- Recognize direct and indirect prompt injection, retrieval poisoning, sensitive-data leakage and unsupported claims.
Depth
Realistic depth within three days.
The difference between these three columns is the difference between a course description and a course. Everything in the third column is real content — it is simply not something anyone builds from scratch in 21 hours.
Implemented
Written by participants, from scratch or near it.
- Prompt contract with typed output
- Baseline and hybrid RAG pipeline
- Small automated evaluation suite
Practised
Run, tuned and measured against prepared scaffolding.
- Sampling and failure-mode experiments
- Chunking, retrieval and reranking
- Prompt-injection and grounding tests
Surveyed
Explained and demonstrated. Not built in three days.
- Full transformer mathematics and training
- GraphRAG indexing at scale
- DSPy optimization and fine-tuning workflows
Day 1
How LLMs work and how to instruct them
Build an accurate mental model before learning prompt patterns.
-
Orientation and baseline
Course architecture, diagnostic task and the distinction between a model, an application and an agent.
-
Model mechanics
Tokens, embeddings, transformer blocks, self-attention, context windows, layers, logits, training versus inference.
-
Statistical inference lab
Conditional probability, softmax, entropy, temperature, top-p and sampling. Participants inspect distributions and output variance.
-
Capabilities and limitations
In-context learning, reasoning, memorization, uncertainty, hallucination, long context and sensitivity to wording.
-
Instruction hierarchy
System, developer, user, assistant and tool messages; conflicts, delimiters, trust boundaries and structured outputs.
-
Prompt engineering
Objectives, constraints, examples, rubrics, decomposition, few-shot patterns, output schemas and error recovery.
-
Lab 1 — prompt contract
Create a typed prompt contract, test it against adversarial and ambiguous examples, and record failure categories.
A reusable prompt specification, a 20-case test set and a short catalogue of observed failure modes.
Day 2
Retrieval, RAG and graph-enhanced context
Move from isolated prompting to evidence-grounded applications.
-
Recap and retrieval diagnostic
Demonstrate why an LLM without retrieval cannot reliably answer corpus-specific questions.
-
Search foundations
Embeddings, cosine similarity, BM25, dense versus sparse search, metadata filters and hybrid retrieval.
-
RAG pipeline
Parsing, chunking, overlap, metadata, indexing, query rewriting, candidate retrieval and reranking. Guided implementation.
-
Grounded generation
Context construction, citation design, abstention, source trust, context budgets and answer synthesis.
-
GraphRAG
Entities, relations, communities, global and local questions, indexing cost and decision criteria for graph-based retrieval.
-
RAG security
Indirect prompt injection, poisoned documents, access control, sensitive data, malicious metadata and trust separation.
-
Lab 2 — retrieval comparison
Build baseline dense retrieval, add lexical fusion and reranking, then compare with a prepared GraphRAG index.
A working RAG pipeline and a retrieval comparison report covering recall, precision, latency and answer grounding.
Day 3
Evaluation, security and production readiness
Turn a convincing demonstration into a testable engineering system.
-
Evaluation design
Golden datasets, deterministic assertions, semantic grading, human review, model judges, pairwise comparison and leakage risks.
-
RAG metrics
Retrieval recall, context precision, groundedness, completeness, citation validity, latency and cost.
-
Lab 3 — evaluation pipeline
Implement automated checks, run regressions and analyse failure clusters across prompt and retrieval versions.
-
Production architecture
Model selection, routing, caching, fallbacks, rate limits, privacy, long context versus RAG and fine-tuning decisions.
-
Observability and operations
Prompt and model versions, traces, retrieval diagnostics, feedback, drift, cost budgets and release gates.
-
Advanced optimization demo
DSPy signatures, modules, metrics and optimization as a transition from hand-edited prompts to measurable programs.
-
Capstone hardening
Add injection tests, uncertainty behaviour, citation checks, monitoring fields and an architecture decision record.
-
Demonstrations and review
Short team demonstrations, trade-off discussion and next-step recommendations.
An evaluated, red-teamed domain assistant with a concise architecture and operations report.
Labs
Four deliverables, each one the input to the next.
- Prompt laboratory Typed prompt contract, structured output schema, test cases and failure-mode log.
- Evidence-grounded retrieval Dense baseline, hybrid search, reranking, citations and a prepared GraphRAG comparison.
- Evaluation and red team Automated regression report, grounding checks, injection cases, latency and cost measurements.
- Capstone integration A coherent assistant combining the previous labs with documentation and a release decision.
Capstone specification
- Separate developer instructions from untrusted user and retrieved content.
- Typed input and output models with validation and graceful error handling.
- At least two retrieval signals, such as dense and lexical retrieval, followed by reranking.
- Source citations, uncertainty or abstention behaviour, and unsupported-claim detection.
- A minimum 20-case evaluation set including normal, ambiguous and adversarial cases.
- Measured retrieval quality, groundedness, latency and approximate cost.
- A one-page architecture decision record explaining why conventional RAG, GraphRAG or another approach was selected.
Stack
The tools, and why each one is in the room.
- One provider SDK Keeps attention on model behaviour and application design rather than cross-provider syntax.
- Pydantic Typed inputs and structured outputs make prompt contracts testable.
- Haystack Explicit modular pipelines suit teaching ingestion, retrieval, reranking and generation.
- Qdrant or PostgreSQL + pgvector Qdrant is convenient for hybrid-search labs; pgvector keeps relational data in PostgreSQL.
- Microsoft GraphRAG A prepared comparison for entity, relationship and corpus-level questions.
- DSPy Demonstrates metric-driven optimization beyond manually editing prompt strings.
- pytest + task-specific evaluators Combines deterministic assertions with model- and retrieval-level evaluation.
Discussed, not taught hands-on
- LlamaIndex can replace Haystack when data connectors and indexing abstractions are the primary concern; teaching both would dilute the workshop.
- Managed vector databases may replace Qdrant or pgvector in production, but the workshop keeps infrastructure transparent.
- Long-context prompting, SQL or graph queries and fine-tuning are treated as alternative architectural choices, not automatic upgrades to RAG.
Logistics
Who it is for, and what it needs.
- Software engineers, data scientists, ML practitioners, technical product managers, researchers and advanced business users who need to design or assess LLM applications.
- Basic Python, JSON and API familiarity. No prior deep-learning coursework is required, but participants should be comfortable reading short code examples and simple probability notation.
- 12–20 participants per instructor; a teaching assistant above 16 keeps the labs moving.
-
- One primary provider API and one prepared Python environment; provider comparisons are conceptual rather than repeated implementations.
- A curated document corpus, a starter ingestion pipeline and an initial question set are supplied before the workshop.
- The GraphRAG index is pre-built; participants query and compare it instead of spending hours on extraction and indexing.
- The capstone is a small domain assistant, not a complete production platform.
- Roughly 9.5 hours of guided coding and evaluation, 9 hours of concepts and demonstrations, and 2.5 hours of review and presentations.
The course teaches working proficiency. It does not claim that participants can train a foundation model, build a large enterprise knowledge graph or certify a production system after three days.
Book a training
Tell me which course and roughly when. You get a reply from me, not a sales sequence — dates, a quote, and an honest answer if the course is a poor fit for what your team actually needs.