OptimHum aims to support the development of human-machine integration in a citizen science classification project called Koster Seafloor Observatory, part of the Ocean Data Factory Sweden, which aims to study how climate change and human activities influence Sweden’s marine ecosystems. We aim to contribute to optimizing the overall experience of marine citizen scientists, not just the efficiency and speed of classifications.
We combine qualitative and quantitative analytical approaches, as we are using different types of data. Examples of data can include forum posts (qualitative) and accompanying meta-data (quantitative); information from documents (e.g., websites, reports, etc.) (qualitative), and interviews (qualitative). The analytical approaches we will be using include trace ethnography and document analysis.
The research will last three years (September 2019 – August 2022), organized into the following phases:
- State-of-the-Art Analysis – Citizen Science Classification Projects
- Data Collection and Analysis
- Dissemination
Project development activities
Activity 1
The main goals of this activity were (i) to describe the distribution of work between citizens and researchers and between humans and algorithms, as well as configurations of human-in-the-loop, (ii) to identify the forms of human-machine learning integration in citizen science projects. Through this activity, we aimed to answer the following questions:
- when designing a citizen science classification project involving humans and algorithms, which tasks or functions are allocated between citizens and researchers and between humans and algorithms?
- what citizen scientists, experts, and algorithms did – or were expected to do – respectively, with the data in CS projects using ML, to achieve the goals of projects? What was the stage of inquiry at which citizen scientists and algorithms, respectively, performed their tasks?
Deliverables:
Results of a Systematic Literature Review
An article containing the results of a systematic review of the literature on citizen science projects combining human and machine learning efforts to achieve projects’ goals. Preprint.
Results of a Study Based on Document Analysis
An article based on document analysis of 12 citizen science projects deploying ML techniques to optimize classification tasks, where we describe the distribution of work between citizens and researchers and between humans and algorithms, as well as configurations of human-in-the-loop. Article under review.
Activity 2
The main goal of this activity was to examine how serendipity can be fostered. Through this activity, we aimed to answer the following question:
how can we use AI techniques to facilitate opportunities for citizen scientists’ creative exploration and chance encounters?
Deliverable:
Manuscript in progress
Activity 3
During this activity, we (i) conduct Koster participants’ performance data analysis such as frequency, productivity of volunteers’ engagement, differences between the frequency of annotation of videos (object identification) and object location in still images (drawing). A goal of this activity is to estimate the abilities and performance measures of citizen scientists in this ecological setting, to investigate how the combination of citizen scientists – to detect objects – and machine learning – to classify megafauna – could be used to automate annotation of underwater images. Our work will be used to help build a classification ML algorithm using input training data to predict the likelihood that subsequent data will fall into one of the predetermined categories.
We use the spreadsheets with the log of collected data provided by Zooniverse, and (ii) to conduct corpus analysis of comments entered by participants in the “Talk” section of the Koster Project. The corpus includes 6419 comments and 83341 annotations.