COHABIT — Collaborative Human-AI Annotation for Behavioural Insight and Tracking

Noldus IT (an innovative company producing solutions for researchers) and Utrecht University (Department of Information and Computing Sciences) will work together to create new tools for behavioural researchers such as psychologists. The tools will use AI to analyse videos of children and adults interacting with each other to semi-automatically label what they are doing. In addition, we will develop techniques based on large language models which will enable researchers to query their data naturally.

Labelling videos is currently done manually and is very time-consuming. The new technique will enable scientists to carry out this work at least ten times faster. The new querying technique will enable researchers to gain deeper insights into their data. The research carried out in the project will directly benefit scientists working in this field, as they will be able to use the tools developed to work more efficiently, which means that science will progress faster. In addition, the project’s results will be brought onto the market by the Dutch company Noldus. That will economically benefit Noldus. Noldus will also put marketing effort into making sure that the project’s results reach as many scientists as possible.

The faster analysis will be based on an AI technique called active learning. The computer will be trained to detect a basic set of behaviours in the videos, e.g. standing up. When the AI sees a section of video where it is not sure what the behaviour is, the researcher labels that behaviour. In that way, the computer learns the new behaviour, and the researcher only has to train it on the unknown behaviours.

The project’s deliverables will be requirements and then several prototypes of the active learning model and the new data query method. There will also be several scientific papers leading to a PhD thesis.