Parameter choice and neuro-symbolic approaches for deep domain-invariant learning

My doctoral thesis, submitted to the Institute for Machine Learning at Johannes Kepler University Linz in July 2024, supervised by Sepp Hochreiter with Werner Zellinger as co-supervisor and Gary Marcus as second evaluator. It is a cumulative thesis: three published papers reprinted in full, plus a chapter arguing why they belong together. Posted to arXiv on 8 October 2024 as 2410.06235.

The problem

A model trained on one distribution and deployed on another degrades, and the gap is rarely something you can fix by collecting more of the same data. The automotive case is the standard illustration: train segmentation in simulation where labels are free, deploy on real footage where they are not.

Unsupervised domain adaptation formalises this. You have labelled data from a source distribution and unlabelled data from a target distribution, and you want low error on the target. The usual approach is domain-invariant learning: train a feature extractor whose output distribution looks the same whether the input came from the source or the target, then classify on top of that shared representation.

This runs into a problem that is embarrassing once you see it. Domain-invariant methods are regularised — you trade source accuracy against a distance penalty between the two feature distributions — and the regularisation strength has to be chosen. But choosing it means evaluating on the target, and the target has no labels. The field’s answer was to train a sequence of models across settings and then apply a model-selection method to pick one.

Two things are wrong with that. The selected model can never be better than the best model in the sequence, so selection caps your result at whatever you happened to sweep. And every other model you trained is discarded, along with the compute that produced it.

The idea, part one: aggregate instead of selecting

Rather than picking one model from the sequence, combine them. Form a weighted sum of all the trained models and choose the weights to minimise target risk.

This is free of downside by construction. The best single model corresponds to putting all the weight on it, so it is inside the space being searched. The optimal combination is therefore never worse than the best selection, and usually better.

The catch is computing the weights. Written out with vector-valued least squares, the optimal coefficients depend on the target-domain Bayes predictor, which is precisely what you cannot access without target labels. The thesis removes the dependency under two assumptions: covariate shift, and a bounded density ratio between the source and target input distributions. Under those, the unknown target quantity rewrites as an expectation over the source data, reweighted by the density ratio — the factor telling you how much more or less likely each input is under the target than under the source. Density-ratio estimation is a solved problem with a literature of its own, so this converts an impossible quantity into an estimable one.

What remains is mechanical: estimate the ratio, build the empirical matrix of inner products between model predictions, solve for the coefficients. The evaluation covers a text domain (AmazonReviews), an image domain (DomainNet) and four time-series datasets spanning activity recognition, sleep-stage EEG and biometrics.

The idea, part two: when you cannot train at all

The second half asks what happens when gradient updates are off the table — the model is too large, or behind an API, or fine-tuning risks catastrophic forgetting.

The thesis argues that in-context learning is already doing domain adaptation, and can be analysed as such. Treat a prompt as a triple of instruction, context and query, with source and target distributions defined over that space. Adaptation then means the network’s transformations map source and target components of the prompt into nearby regions of representation space, without a single weight changing.

The supporting evidence is illustrative rather than conclusive, and the thesis presents it that way. A UMAP projection of last-layer embeddings from GPT-Neo 1.3B shows mathematical, programming and natural-language expressions of the same idea — a = b, a == b, “a equals b” — clustering together, while a control domain of random strings sits far away. Models also resolve arithmetic written in substituted Unicode glyphs they cannot have memorised in that form. The claim is that pretraining builds cross-domain semantic correspondences, and that a prompt interpolates between them.

This is where SymbolicAI enters: treat the model as a semantic parser, compose its operations into computational graphs, and evaluate those multi-step processes with the VERTEX score, a quality measure built on the Fréchet distance.

Why it matters, and what it does not settle

The aggregation result is the solid contribution: a method with a guarantee, benchmarked broadly. The in-context-learning argument is a perspective, supported by visualisations and worked examples rather than by theorems. It reframes a phenomenon usefully; it does not prove a theory of it, and the thesis does not claim otherwise.

Where it applies

The aggregation method is immediately usable and costs nothing extra. Anyone doing unsupervised domain adaptation already trains a sweep of regularisation strengths and then throws most of it away. Keeping those checkpoints and combining them, with weights estimated from unlabelled target data, uses compute you have already spent to get an error no worse than the model you would otherwise have picked.