Dinu Marius-Constantin

Ensemble Learning for Domain Adaptation by Importance Weighted Least Squares

13. August 2022 0 comments General

Abstract

We study ensemble learning for unsupervised domain adaptation, i.e., with labeled data in a source domain and unlabeled data in a target domain, drawn from a different input distribution. An open problem is to find an optimal aggregation of given models without making strong assumptions on the model classes. While several heuristics exist, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends the theory of weighted least squares to linear aggregations and vector-valued functions. Our method is asymptotically error-rate-optimal, in the sense that the error of the computed aggregation is asymptotically not worse than twice the error of the unknown optimal aggregation. In experiments, we compare our method to (1) classical ensemble learning on source data only, (2) majority voting on target predictions, (3) ensemble learning based on pseudo-labels, (4) importance weighted validation, and, (5) deep embedded validation; on several datasets including language, images and time-series. As a result, our method sets a new state-of-the-art performance for ensemble learning in unsupervised domain adaptation under theoretical error guarantees.

Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning

Marius-Constantin Dinu

1. August 2022 0 comments General

Abstract

In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react to continual domain shifts in lifelong reinforcement learning, and to update the policy correspondingly. To this end, we conduct experiments in order to investigate different exploration strategies. We empirically show that representatives of the policy-gradient family are better suited for lifelong learning, as they adapt more quickly to distribution shifts than Q-learning. Thereby, policy-gradient methods profit the most from Reactive Exploration and show good results in lifelong learning with continual domain shifts. Our code is available at: this https URL.

A Dataset Perspective on Offline Reinforcement Learning

Marius-Constantin Dinu

1. August 2022 0 comments General

Abstract

The application of Reinforcement Learning (RL) in real world environments can be expensive or risky due to sub-optimal policies during training. In Offline RL, this problem is avoided since interactions with an environment are prohibited. Policies are learned from a given dataset, which solely determines their performance. Despite this fact, how dataset characteristics influence Offline RL algorithms is still hardly investigated. The dataset characteristics are determined by the behavioral policy that samples this dataset. Therefore, we define characteristics of behavioral policies as exploratory for yielding high expected information in their interaction with the Markov Decision Process (MDP) and as exploitative for having high expected return. We implement two corresponding empirical measures for the datasets sampled by the behavioral policy in deterministic MDPs. The first empirical measure SACo is defined by the normalized unique state-action pairs and captures exploration. The second empirical measure TQ is defined by the normalized average trajectory return and captures exploitation. Empirical evaluations show the effectiveness of TQ and SACo. In large-scale experiments using our proposed measures, we show that the unconstrained off-policy Deep Q-Network family requires datasets with high SACo to find a good policy. Furthermore, experiments show that policy constraint algorithms perform well on datasets with high TQ and SACo. Finally, the experiments show, that purely dataset-constrained Behavioral Cloning performs competitively to the best Offline RL algorithms for datasets with high TQ.

Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Marius-Constantin Dinu

10. March 2022 0 comments General

Abstract

Reinforcement learning algorithms require many samples when solving complex hierarchical tasks with sparse and delayed rewards. For such complex tasks, the recently proposed RUDDER uses reward redistribution to leverage steps in the Q-function that are associated with accomplishing sub-tasks. However, often only few episodes with high rewards are available as demonstrations since current exploration strategies cannot discover them in reasonable time. In this work, we introduce Align-RUDDER, which utilizes a profile model for reward redistribution that is obtained from multiple sequence alignment of demonstrations. Consequently, Align-RUDDER employs reward redistribution effectively and, thereby, drastically improves learning on few demonstrations. Align-RUDDER outperforms competitors on complex artificial tasks with delayed rewards and few demonstrations. On the Minecraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is available at this https URL.

The balancing principle for parameter choice in distance-regularized domain adaptation

Marius-Constantin Dinu

24. September 2021 0 comments General

Abstract

We address the unsolved algorithm design problem of choosing a justified regularization parameter in unsupervised domain adaptation. This problem is intriguing as no labels are available in the target domain. Our approach starts with the observation that the widely-used method of minimizing the source error, penalized by a distance measure between source and target feature representations, shares characteristics with regularized ill-posed inverse problems. Regularization parameters in inverse problems are optimally chosen by the fundamental principle of balancing approximation and sampling errors. We use this principle to balance learning errors and domain distance in a target error bound. As a result, we obtain a theoretically justified rule for the choice of the regularization parameter. In contrast to the state of the art, our approach allows source and target distributions with disjoint supports. An empirical comparative study on benchmark datasets underpins the performance of our approach.

Lighter: Dependency Injection for PyTorch

Marius-Constantin Dinu

9. June 2021 0 comments General

Python & Machine Learning: If you ever thought that handling all those kwargs, dictionaries of config properties, or huge method parameter listings feels tedious and somewhat like a step back in the evolution of programming language, well... then this is because you are completely right! It is a step back. Prominent languages such as Java and C# have matured in the practices of decoupling components of interest through the application of Aspect-oriented programming schemes. With the Python programming language this trend somewhat got overlooked, since it seems to contradict with the pythonic way of coding. One of the most crucial components to offer well-structured project management, especially when transitioning from prototyping to large scale project structures, has gotten forgotten - namely dependency injection. With dependency injection one can ignore specifying the concrete object when connecting components in a larger scale project, but rather focus on specific parts and merging those parts with flexible and lightweight configs. Lighter offers a glimpse in this direction and shows how one can easily patch together a protype project in a well separated and manageable way. For more details and examples follow this link.

#JKU #AI #Awesome

Marius-Constantin Dinu

18. February 2021 0 comments General

JKU AI Overview Video

This video gives a cool overview about the possibilities AI enables us.

XAI and Strategy Extraction via Reward Redistribution

Marius-Constantin Dinu

17. October 2020 0 comments General

Abstract

In reinforcement learning, an agent interacts with an environment from which it receives rewards, that are then used to learn a task. However, it is often unclear what strategies or concepts the agent has learned to solve the task. Thus, interpretability of the agent’s behavior is an important aspect in practical applications, next to the agent’s performance at the task itself. However, with the increasing complexity of both tasks and agents, interpreting the agent’s behavior becomes much more difficult. Therefore, developing new interpretable RL agents is of high importance. To this end, we propose to use Align-RUDDER as an interpretability method for reinforcement learning. Align-RUDDER is a method based on the recently introduced RUDDER framework, which relies on contribution analysis of an LSTM model, to redistribute rewards to key events. From these key events a strategy can be derived, guiding the agent’s decisions in order to solve a certain task. More importantly, the key events are in general interpretable by humans, and are often sub-tasks; where solving these sub-tasks is crucial for solving the main task. Align-RUDDER enhances the RUDDER framework with methods from multiple sequence alignment (MSA) to identify key events from demonstration trajectories. MSA needs only a few trajectories in order to perform well, and is much better understood than deep learning models such as LSTMs. Consequently, strategies and concepts can be learned from a few expert demonstrations, where the expert can be a human or an agent trained by reinforcement learning. By substituting RUDDER’s LSTM with a profile model that is obtained from MSA of demonstration trajectories, we are able to interpret an agent at three stages: First, by extracting common strategies from demonstration trajectories with MSA. Second, by encoding the most prevalent strategy via the MSA profile model and therefore explaining the expert’s behavior. And third, by allowing the interpretation of an arbitrary agent’s behavior based on its demonstration trajectories.

Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Marius-Constantin Dinu

30. September 2020 0 comments General Credit Assignment Problem, Learning from Demonstrations, Reinforcement Learning, Reward Redistribution, RUDDER

Reinforcement Learning algorithms require a large number of samples to solve complex tasks with sparse and delayed rewards. Complex tasks can often be hierarchically decomposed into sub-tasks. A step in the Q-function can be associated with solving a sub-task, where the expectation of the return increases. RUDDER has been introduced to identify these steps and then redistribute reward to them, thus immediately giving reward if sub-tasks are solved. Since the problem of delayed rewards is mitigated, learning is considerably sped up. However, for complex tasks, current exploration strategies as deployed in RUDDER struggle with discovering episodes with high rewards. Therefore, we assume that episodes with high rewards are given as demonstrations and do not have to be discovered by exploration. Typically the number of demonstrations is small and RUDDER's LSTM model as a deep learning method does not learn well. Hence, we introduce Align-RUDDER, which is RUDDER with two major modifications. First, Align-RUDDER assumes that episodes with high rewards are given as demonstrations, replacing RUDDER's safe exploration and lessons replay buffer. Second, we replace RUDDER's LSTM model by a profile model that is obtained from multiple sequence alignment of demonstrations. Profile models can be constructed from as few as two demonstrations as known from bioinformatics. Align-RUDDER inherits the concept of reward redistribution, which considerably reduces the delay of rewards, thus speeding up learning. Align-RUDDER outperforms competitors on complex artificial tasks with delayed reward and few demonstrations. On the MineCraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is published on GitHub

Overcoming Catastrophic Forgetting with Context-Dependent Activations (XdA) and Synaptic Stabilization

Marius-Constantin Dinu

25. November 2019 0 comments General

Abstract Overcoming Catastrophic Forgetting in neural networks is crucial to solving continuous learning problems. Deep Reinforcement Learning uses neural networks to make predictions of actions according to the current state space of an environment. In a dynamic environment, robust and adaptive life-long learning algorithms mark the cornerstone of their success. In this thesis we will examine an elaborate subset of algorithms countering catastrophic forgetting in neural networks and reflect on their weaknesses and strengths. Furthermore, we present an enhanced alternative to promising synaptic stabilization methods, such as Elastic Weight Consolidation or Synaptic Intelligence. Our method uses context-based information to switch between different pathways throughout the neural network, reducing destructive activation interference during the forward pass and destructive weight updates during the backward pass. We call this method Context-Dependent Activations (XdA). We show that XdA enhanced methods outperform basic synaptic stabilization methods and are a better choice for long task sequences.

Thesis link
GitHub link

1 2