Dinu Marius-Constantin
Website
Menu
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Home
  • Resume
  • Newsletter
  • Login
  • About

A Neuro-Symbolic Perspective on Large Language Models (LLMs)

Marius-Constantin Dinu

22. January 2023 0 comments General

We are excited to present our work, combining the power of a symbolic approach and Large Language Models (LLMs). Our Symbolic API bridges the gap between classical programming (Software 1.0) and differentiable programming (Software 2.0). Conceptually, our framework uses neural networks - specifically LLMs - at its core, and composes operations based on task-specific zero-shot or few-shot prompting. We adopt a divide and conquer approach to decompose a complex problem into smaller problems. Therefore, each operation solves a simple task. By re-combining these operations, we can solve the complex problem. Furthermore, we demonstrate how to combine the strengths of both neural networks and symbolic reasoning to create AI systems that can solve a wide range of hard tasks. This includes fact-based generation of text, flow control of a generative process towards a desired outcome, and interpretability within generative processes.

ensemble 2

Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation

Marius-Constantin Dinu

13. August 2022 0 comments General

Abstract

We study the problem of choosing algorithm hyper-parameters in unsupervised domain adaptation, i.e., with labeled data in a source domain and unlabeled data in a target domain, drawn from a different input distribution. We follow the strategy to compute several models using different hyper-parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform a large scale empirical comparative study on several datasets, including text, images, electroencephalogram, body sensor signals and signals from mobile phones. Our method outperforms deep embedded validation (DEV) and importance weighted validation (IWV) on all datasets, setting a new state-of-the-art performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.

reactexp 1

Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning

Marius-Constantin Dinu

1. August 2022 0 comments General

Abstract

In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react to continual domain shifts in lifelong reinforcement learning, and to update the policy correspondingly. To this end, we conduct experiments in order to investigate different exploration strategies. We empirically show that representatives of the policy-gradient family are better suited for lifelong learning, as they adapt more quickly to distribution shifts than Q-learning. Thereby, policy-gradient methods profit the most from Reactive Exploration and show good results in lifelong learning with continual domain shifts. Our code is available at: this https URL.

datasetpers 2

A Dataset Perspective on Offline Reinforcement Learning

Marius-Constantin Dinu

1. August 2022 0 comments General

Abstract

The application of Reinforcement Learning (RL) in real world environments can be expensive or risky due to sub-optimal policies during training. In Offline RL, this problem is avoided since interactions with an environment are prohibited. Policies are learned from a given dataset, which solely determines their performance. Despite this fact, how dataset characteristics influence Offline RL algorithms is still hardly investigated. The dataset characteristics are determined by the behavioral policy that samples this dataset. Therefore, we define characteristics of behavioral policies as exploratory for yielding high expected information in their interaction with the Markov Decision Process (MDP) and as exploitative for having high expected return. We implement two corresponding empirical measures for the datasets sampled by the behavioral policy in deterministic MDPs. The first empirical measure SACo is defined by the normalized unique state-action pairs and captures exploration. The second empirical measure TQ is defined by the normalized average trajectory return and captures exploitation. Empirical evaluations show the effectiveness of TQ and SACo. In large-scale experiments using our proposed measures, we show that the unconstrained off-policy Deep Q-Network family requires datasets with high SACo to find a good policy. Furthermore, experiments show that policy constraint algorithms perform well on datasets with high TQ and SACo. Finally, the experiments show, that purely dataset-constrained Behavioral Cloning performs competitively to the best Offline RL algorithms for datasets with high TQ.

Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Marius-Constantin Dinu

10. March 2022 0 comments General

Abstract

Reinforcement learning algorithms require many samples when solving complex hierarchical tasks with sparse and delayed rewards. For such complex tasks, the recently proposed RUDDER uses reward redistribution to leverage steps in the Q-function that are associated with accomplishing sub-tasks. However, often only few episodes with high rewards are available as demonstrations since current exploration strategies cannot discover them in reasonable time. In this work, we introduce Align-RUDDER, which utilizes a profile model for reward redistribution that is obtained from multiple sequence alignment of demonstrations. Consequently, Align-RUDDER employs reward redistribution effectively and, thereby, drastically improves learning on few demonstrations. Align-RUDDER outperforms competitors on complex artificial tasks with delayed rewards and few demonstrations. On the Minecraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is available at this https URL.

bpda 1

The balancing principle for parameter choice in distance-regularized domain adaptation

Marius-Constantin Dinu

24. September 2021 0 comments General

Abstract

We address the unsolved algorithm design problem of choosing a justified regularization parameter in unsupervised domain adaptation. This problem is intriguing as no labels are available in the target domain. Our approach starts with the observation that the widely-used method of minimizing the source error, penalized by a distance measure between source and target feature representations, shares characteristics with regularized ill-posed inverse problems. Regularization parameters in inverse problems are optimally chosen by the fundamental principle of balancing approximation and sampling errors. We use this principle to balance learning errors and domain distance in a target error bound. As a result, we obtain a theoretically justified rule for the choice of the regularization parameter. In contrast to the state of the art, our approach allows source and target distributions with disjoint supports. An empirical comparative study on benchmark datasets underpins the performance of our approach.

Lighter: Dependency Injection for PyTorch

Marius-Constantin Dinu

9. June 2021 0 comments General
Python & Machine Learning: If you ever thought that handling all those kwargs, dictionaries of config properties, or huge method parameter listings feels tedious and somewhat like a step back in the evolution of programming language, well... then this is because you are completely right! It is a step back. Prominent languages such as Java and C# have matured in the practices of decoupling components of interest through the application of Aspect-oriented programming schemes. With the Python programming language this trend somewhat got overlooked, since it seems to contradict with the pythonic way of coding. One of the most crucial components to offer well-structured project management, especially when transitioning from prototyping to large scale project structures, has gotten forgotten - namely dependency injection. With dependency injection one can ignore specifying the concrete object when connecting components in a larger scale project, but rather focus on specific parts and merging those parts with flexible and lightweight configs. Lighter offers a glimpse in this direction and shows how one can easily patch together a protype project in a well separated and manageable way. For more details and examples follow this link.
lighter

#JKU #AI #Awesome

Marius-Constantin Dinu

18. February 2021 0 comments General

JKU AI Overview Video

This video gives a cool overview about the possibilities AI enables us.

XAI and Strategy Extraction via Reward Redistribution

Marius-Constantin Dinu

17. October 2020 0 comments General

Abstract

In reinforcement learning, an agent interacts with an environment from which it receives rewards, that are then used to learn a task. However, it is often unclear what strategies or concepts the agent has learned to solve the task. Thus, interpretability of the agent’s behavior is an important aspect in practical applications, next to the agent’s performance at the task itself. However, with the increasing complexity of both tasks and agents, interpreting the agent’s behavior becomes much more difficult. Therefore, developing new interpretable RL agents is of high importance. To this end, we propose to use Align-RUDDER as an interpretability method for reinforcement learning. Align-RUDDER is a method based on the recently introduced RUDDER framework, which relies on contribution analysis of an LSTM model, to redistribute rewards to key events. From these key events a strategy can be derived, guiding the agent’s decisions in order to solve a certain task. More importantly, the key events are in general interpretable by humans, and are often sub-tasks; where solving these sub-tasks is crucial for solving the main task. Align-RUDDER enhances the RUDDER framework with methods from multiple sequence alignment (MSA) to identify key events from demonstration trajectories. MSA needs only a few trajectories in order to perform well, and is much better understood than deep learning models such as LSTMs. Consequently, strategies and concepts can be learned from a few expert demonstrations, where the expert can be a human or an agent trained by reinforcement learning. By substituting RUDDER’s LSTM with a profile model that is obtained from MSA of demonstration trajectories, we are able to interpret an agent at three stages: First, by extracting common strategies from demonstration trajectories with MSA. Second, by encoding the most prevalent strategy via the MSA profile model and therefore explaining the expert’s behavior. And third, by allowing the interpretation of an arbitrary agent’s behavior based on its demonstration trajectories.

Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Marius-Constantin Dinu

30. September 2020 0 comments General Credit Assignment Problem, Learning from Demonstrations, Reinforcement Learning, Reward Redistribution, RUDDER
Reinforcement Learning algorithms require a large number of samples to solve complex tasks with sparse and delayed rewards. Complex tasks can often be hierarchically decomposed into sub-tasks. A step in the Q-function can be associated with solving a sub-task, where the expectation of the return increases. RUDDER has been introduced to identify these steps and then redistribute reward to them, thus immediately giving reward if sub-tasks are solved. Since the problem of delayed rewards is mitigated, learning is considerably sped up. However, for complex tasks, current exploration strategies as deployed in RUDDER struggle with discovering episodes with high rewards. Therefore, we assume that episodes with high rewards are given as demonstrations and do not have to be discovered by exploration. Typically the number of demonstrations is small and RUDDER's LSTM model as a deep learning method does not learn well. Hence, we introduce Align-RUDDER, which is RUDDER with two major modifications. First, Align-RUDDER assumes that episodes with high rewards are given as demonstrations, replacing RUDDER's safe exploration and lessons replay buffer. Second, we replace RUDDER's LSTM model by a profile model that is obtained from multiple sequence alignment of demonstrations. Profile models can be constructed from as few as two demonstrations as known from bioinformatics. Align-RUDDER inherits the concept of reward redistribution, which considerably reduces the delay of rewards, thus speeding up learning. Align-RUDDER outperforms competitors on complex artificial tasks with delayed reward and few demonstrations. On the MineCraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is published on GitHub
1 2

Latest Tweets

  • RT @DrJimFan: 10x engineer is a myth. 100x AI-powered engineer is more real than ever. As OpenAI winds down Codex, Microsoft announces GitH…
    4 days ago
  • RT @GaryMarcus: Speaking of bullshit, anyone remember this? pic.twitter.com/ekDvjPmoPR
    6 days ago
  • RT @DrJimFan: Let's talk about the elephant in the room - will LLM take your job? OpenAI & UPenn conclude that ~80% of the U.S. workforce…
    7 days ago

Recent Posts

  • A Neuro-Symbolic Perspective on Large Language Models (LLMs)A Neuro-Symbolic Perspective on Large Language Models (LLMs)
    22. January 2023
  • ensemble 2Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation
    13. August 2022
  • reactexp 1Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning
    1. August 2022

Feedback

Leave a comment.

Find me on →

Please, feel free to leave a comment and give me feedback about my webpage.
You can also contact me via email.

© Dinu Marius-Constantin 2023
Powered by WordPress • Themify WordPress Themes

[ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin

 

Loading Comments...