Released Open-Source Speaker Diarization Model for Two English Speakers

mocomoco Inc. has released an open-source Speaker Diarization model tailored for two English speakers. This model is available in the following repositories:

We will present our work on this model at YANS (Spoken Language Information Processing Research Group). We will discuss the model's design background, evaluation results, and future prospects, aiming for advancement in both academic and industrial fields. For presentation details, please click here.

Model Features and Applications

This model was developed with the goal of automatically separating two speakers in English audio. Unlike conventional models, it processes everything consistently end-to-end, from feature extraction to diarization. Furthermore, by continuously referencing previous frame information, it maintains speaker ID consistency and can be used for long-duration audio without causing memory overflows.

It is anticipated to have a wide range of applications, including meeting transcription, subtitle generation, and call center analysis. Additionally, we have published comparative verification results with Pyannote 3.1, making it a highly transparent OSS that researchers and developers can use with confidence.

Specifications

Supported Language: English
Number of Speakers: Fixed at 2
Base Model: Built on BW-EDA-EEND
Feature Extraction: CPC (Contrastive Predictive Coding)
Acoustic Modeling: Adopted Conformer
Inference Mode: Offline (Online processing is theoretically possible)

We have prepared a simple execution method, allowing it to be loaded directly from the Hugging Face Hub for immediate use.

Architecture

This model performs feature extraction using CPC and acquires embedding representations with a powerful Conformer-based encoder. These are fed into a Transformer decoder, and by combining the attractor calculation proposed in previous EEND research, speaker diarization is achieved. Moreover, by carrying over information from past frames to maintain speaker ID consistency, stable processing is possible even for long-duration audio.

Challenges

The model released this time was trained on a relatively small amount of data, and challenges remain regarding its generalization performance. While there are audio sources that can be diarized with high accuracy, cases where performance degrades have also been observed. As a result of visualizing the embedding space, we found that there are limits to diarization between speakers with similar voice qualities.

The research team will continue to tackle this challenge and plans to release a higher-performance model. Please stay tuned for future updates.

(1) Visualization of an audio source with good performance

(2) Visualization of an audio source with degraded performance

Future Prospects

Moving forward, we will proceed with research and development in the following directions:

Expansion to a multi-speaker supported model
Multilingual support, including Japanese
Further improvement of real-time processing capabilities
Application of speaker diarization specialized for specific domains such as IR and medicine

As an OSS open to researchers and developers, mocomoco will continue to contribute to the advancement and social implementation of speaker diarization technology.

For inquiries, please use our Contact Form.

Our company employs researchers with strong expertise in the field of spoken dialogue, providing AI solutions that combine practicality and cutting-edge technology.

▼ Try our state-of-the-art transcription and meeting minutes AI https://cloud.mocomoco.ai/sign-up