Unknown Speaker 0:03
Hello everyone, my name is Zhaoxing Li, I am a first year PhD student at Durham University in Department of Computer Science. Today I'd like to present our paper a survey of collaborative reinforcement learning: interactive methods and design patterns before the start, I would like to introduce myself.
Unknown Speaker 0:28
My main research interests lie in machine learning, HCI, human AI co-decision making and explainable artificial intelligence.
Unknown Speaker 0:39
Lets move to the introduction, with repaired development of AI in recent years. So mainstream media has two opposing view. first AI is described as Savior who frees human from labor. And it is also described as the devil who takes away workers job. So it's a savior or a destroyer?
Unknown Speaker 1:09
Due to a limitation of current technology, in current stage, AI often performance much worse than humans in highly complex decision making task.
Unknown Speaker 1:21
But much better than AI in a task with well specific feedback and the large scale data. Most computers science researchers believes that human and AI will be in a long term collaborative relationship. So AI will be a good partner of us.
Unknown Speaker 1:44
Recently, so methods enablling human and AI agent to collaborate towards improving efficiency of reinforcement learning, also called collaborative reinforcement learning.
Unknown Speaker 2:00
As a human AI collaborative model usually involves two roles. So, human and AI, and collaboration patterns bettern them. so we divided the articles collected into three categories. from a human perspective, we focus on how humans interact with a AI agent, is part is described described as interactive Methods section. From AI agent perspective, we focus on how agents accept human instructions or suggestions in algorithm implementation. This part is described as section four. From the perspective of collaboration patterns, we focus on how human AI can collaborate, this part is described in Section six.
Unknown Speaker 3:03
This taxonomy could be a modeling tool for HCI researchers and designers from the DIS community to select and improve the CRL design.
Unknown Speaker 3:17
system researchers and designers have two main tasks in recent years. First is developing new interactive technologies. And the second is applying this technology to the specific scenarios. When A researcher start to design an AI system, they could first selected collaborative model from a macro perspective in design pattern category, then they could seeclt suitable interactive methods and algorithm models from the specific task requirement categories of interactive methods and algorithm models.
Unknown Speaker 4:02
So we defined this three axes and populates with Representative works from the literature.
Unknown Speaker 4:13
So, the main contribution of our Paper lie in four perspective. First, we take stock and summarise the latest collaborative reinforcement learning algorithms, defining the state-of-the- art at the start of this new decade. Second, we summarise the most important human-machine collaborative patterns from an engineering perspective, which will provide better guidance for Human-Computer Interaction (HCI) researchers. Third, we have provided a new CRL classification and CRL taxonomy as a modelling tool to help researchers select and improve new CRL designs.
Unknown Speaker 5:00
We propose concrete challenges for future researcher in this area as a guide towards effective human AI collaboration.
Unknown Speaker 5:12
Next, we move into the interactive methods.
Unknown Speaker 5:17
it can be divided into three types. The first is explicit method, second is implicit method, third is a multi modal method.
Unknown Speaker 5:30
explicit methods means that human provides agent with clear numerical feedback, this method is better for the agent, which can process feedback more easily, but it is likely to cause human fatigue and thus lead to inefficiency in a long-term training process.
Unknown Speaker 5:54
For the implicit method, humans give feedback to AI agents through natural interactions such as posture and gaze, this method makes higher demands for the agent, but it could improve the fatigue resistance of human trainers, thereby achieving long-term and stable cooperation. And as a multimodal method is a method that combines a different type of interactive methods.
Unknown Speaker 6:26
And the next is algorithm models. When a classifiers algorithmics models based on how agent receives humans feedback, it can be divided into four types. First is reward based methods, second is policy based methods and so value function based methods and exploration process methods.
Unknown Speaker 6:53
Next, we move through the human AI cooperation design patterns. There are many important design patterns in the engineering area for example, the CSE patterns and Bosch and Bronkhorst’s Pattern and collaborative design patterns and Schmidt’ s patterns.
Unknown Speaker 7:20
The Schmidt’ s pattern is the most widely used pattern in the engineering area. Schmidt’ s model is can be divided into three levels, the argumentative level, and integrative level and the debative level.
Unknown Speaker 7:40
Next is a future work. So there's a lot of work here needs to beexplored. For example, how to combine different interactive Methods and how to build user modeling, how to develop a safe interactive reinforcement learning and how to development and dynamic Mental models.
Unknown Speaker 8:04
To conclusion, we have present a survey of collaborative reinforced learning
Unknown Speaker 8:10
to empower the research into human interaction and collaborative design. We also have provides new CRL classification method sees the table one and the new CRL textdomain see figure two as a modeling tool for selecting and improving new CRLdesigns. Despite a great process with a in the field of machine learning, there are still many challenges and are needed to be addressed before the generically cooperative model is proposed.
Unknown Speaker 8:45
So that's all. if you have any questions, please feel free to contact with me. Thanks!
Transcribed by https://otter.ai