Student assignments

Here are some available student assignments – note that these are general descriptions and can be adjusted such that they fit a certain type of assignment (bachelor graduation project / master graduation project / capita selecta / advanced research project / i-tech project) and that they fit the interests and skills of the student.

Analysis of depression in speech

The NESDO research (Nederlandse Studie naar Depressie bij Ouderen) was a large longitudinal Dutch study into depression in older adults (>60yr old). Older adults with and without depression were followed over a period of six years. Measurements included questionnaires, a medical examination, cognitive tests and information was gathered about mental health outcomes, demographic, psychosocial and cognitive determinants. Some of these measurements were taken in face-to-face assessments. After baseline measurement, face-to-face assessments were held after 2 and 6 years.

Currently, we have a few audio recordings available from the 6-yrs measurement from depressed and non-depressed older persons. We are looking for a student (who preferably has knowledge of Dutch) who is interested in performing speech analyses on these recordings with the eventual goal to detect depression automatically in speech.

This work will be carried out in collaboration with Dr. Paul Naarding, GGNet Apeldoorn.

Reading material:

  • https://nesdo.onderzoek.io/
  • https://nesdo.onderzoek.io/wp-content/uploads/2016/08/Comijs-et-al-2011_design-NESDO_incl-erratum.pdf
  • Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., & Quatieri, T. F. (2015). A review of depression and suicide risk assessment using speech analysis. Speech Communication, 71, 10-49.
  • Low, D. M., Bentley, K. H., & Ghosh, S. S. (2020). Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investigative Otolaryngology, 5(1), 96-116.
  • Cummins, N., Matcham, F., Klapper, J., & Schuller, B. (2020). Artificial intelligence to aid the detection of mood disorders. In Artificial Intelligence in Precision Health (pp. 231-255). Academic Press.

Automatic Laughter analysis in human-Computer Interaction

Laughter analysis is currently a hot topic in Human-Computer Interaction. Computer scientists generally study how humans communicate through laughter and how this can be implemented in Automatic Laughter Detection and Automatic Laughter Synthesis. Development of such tools would be very helpful in fields like Human-Computer/Robot Interaction, where voice assistants like Alexa and Google assistant might understand more complex natural communication through the interpretation of social signals such as laughter and generating well timed and appropriate realistic laughter responses. Another application could be multi-media laughter retrieval, automatically extracting laughter occurrences from large amounts of video and audio data, opening the way for retrieving large laughter datasets. As a final example, laughter detection could also possibly be used to study group behavior or automatic person identification.

However there are several challenges in laughter research that need to be considered when aiming for automatic laughter analysis. For one, annotating laughter is a much discussed challenge for the field, there are debates on how laughter should be segmented and labeled. Are there different kinds of laughs for different situations and how do we label these? Do people have specific laughter profiles? How does context play a role in laughter detection? How could a real-time implementation in a conversational agent look like and for what purpose? Students can choose to go for a more human-centered or technology oriented direction.

This makes achieving automatic laugher analysis an interesting goal. Students are invited to explore the topic of Automatic Laughter analysis and come up with an interesting question or challenge they want to address. You will be supervised by assistant professor Khiet P. Truong, who is an expert in laughter research and SSP and PhD student Michel-Pierre Jansen whose PhD work evolves around human laughter recognition and Social Signal Processing.

Spoken Interaction with Conversational Agents and Robots

Speech technology for conversational agents and robots has taken a flight (e.g., Siri, Alexa), but we are not quite there yet. While there are technical challenges to address (e.g., how can an agent display listening behavior such as backchannels “uh-uhm”, how can we recognize a user’s stance/attitude/intent, how can we express intent without using words, how can an agent build rapport with a user), there are also more human-centered questions such as how to design such a spoken conversational interaction, how do people actually talk to an agent or robot, what effect does a certain agent/robot behavior (e.g., robot voice, appearance) have on a child’s perception and behavior in a specific context?

These are some examples of research questions we are interested in. Much more is possible. Are you also interested? Khiet Truong and Ella Velner can tell you more.

Supervision bachelor and master students


Have supervised and am supervising numerous AI/CS/CreaTe bachelor and AI/HMI master students on various topics in affective computing and social signal processing.

A selection of MSc graduation projects that I have supervised / am currently supervising:

  • Carmen Mijnders (Language and Speech Pathology RU, 2022) – together with Esther Janse (RU) and Paul Naarding (GGNet): Analysis of depression in older adults’ speech
  • Katariina Martikainen (EIT HCID, 2020) – together with Jussi Karlgren: Podcast content modelling – building a stylistic framework for podcast recommendation and search
  • Nynke Zwart (Interaction Technology, 2020) – together with Thomas Beelen, Ella Velner: A stranger presence within an embodied conversation agent during privacy permission requests: effects on information disclosure and privacy awareness of children
  • Ellen Tournier (Language and Speech Pathology RU, 2019) – together with Esther Janse (RU), Deniece Nazareth: Valence of Emotional Events: A study of linguistic and nonlinguistic speech characteristics in affective speech production
  • Jiska Koemans (Language and Communication RU, 2019) – together with Odette Scharenborg (TUD): Comparing human emotion recognition to automatic emotion recognition in speech
  • Judith Zissoldt (Interaction Technology, 2019) – together with Gwenn Englebienne, Deniece Nazareth: Topic modelling in life memories of older adults
  • Meike Berkhoff (Interaction Technology, 2019) – together with Deniece Nazareth: VR in dementia
  • Pim Willemse (HMI, 2017) – together with Randy Klaassen: Mood recognition and empathic lighting for elderly
  • Sanne van Waveren (HMI, 2017) – together with Willem Paul Brinkman (TUDelft): Negotiations in virtual reality
  • Anne-Greeth van Herwijnen (HMI, 2016): The influence of inner-state displays on human-human interaction
  • Cristina Zaga (HMI, 2014): The effect of a robot’s social character on children’s task engagement: peer versus tutor

A selection of MSc internship projects that I have co-supervised:

  • Laduona Lai (CS) – Automatic pain detection in facial expression (together with Joost Broekens at TUDelft and Interactive Robotics)
  • Emiel Harmsen (HMI, 2016) – Developing a wearable device for emotion detection (together with Kristin Neidlinger at Sensoree).

A selection of BSc graduation projects that I have supervised:

  • Joop Arts (ATLAS, 2022): Exploring the effects of perceived gender on the perceived social function of laughter
  • Quirien Hover (ATLAS, 2020): Uncanny, Sexy and Threatening robots – The online community’s attitude to and perceptions of social robots varying in humanlikeness and gender
  • Tenzing Dolmans (ATLAS, 2019): Associating children’s response with behavioural valence of robots using frontal asymmetry
  • Maaike Slot (Create, 2017): Sociometric badges
  • Mikael Pratama (Create, 2017): Sociometric badges
  • Lyubomir Andreev (Create, 2016): Sociometric badges

Also supervising research topics, capita selecta, and Human Media Interaction project (developing and evaluating interactive systems).