How Hark recognises voices: speaker recognition in plain English

How Hark works out who spoke in a meeting: finding speech, making voiceprints, grouping similar voices, and naming each person once so Hark knows them next time.

A transcript that says what was said but not who said it is only half useful. Working out who spoke is called speaker diarization, and recognising those speakers again later is speaker recognition. Hark does both, on your own computer. This post explains how, without the maths, and also where it can go wrong and how you fix it.

The short version

Hark finds the parts of the recording where someone is talking.
It turns short slices of that speech into voiceprints: lists of numbers that describe what a voice sounds like.
It groups slices with similar voiceprints. Each group is one voice: “Speaker 1”, “Speaker 2” and so on.
You listen to each voice and name it once. Hark saves that person’s voiceprint.
In later meetings, Hark compares new voices with the people you’ve named and suggests matches for you to confirm.
If it gets something wrong, you mark who spoke on the timeline and Hark redoes the transcript using your marks.

Here is each step in a bit more detail.

Step 1: Finding the speech

A meeting recording is full of things that aren’t speech: silence, typing, breathing, a dog barking. Before Hark can tell voices apart, it needs to know where the talking is. It uses a small model called Silero VAD (voice activity detection) whose only job is to say, moment by moment, “speech” or “not speech”.

This also matters for the transcript itself. Hark cuts audio at natural pauses so it never chops a word in half, and during a live meeting it uses the same detection to send each spoken phrase to the transcriber a few seconds after it’s said.

Step 2: Making voiceprints

Next, Hark cuts the speech into overlapping windows two seconds long, starting one second apart. Each window goes through a model called NVIDIA TitaNet, which was trained to tell speakers apart. Out of it comes a voiceprint, also called an embedding: a list of numbers that captures the character of the voice (pitch, timbre, the way someone shapes sounds) while mostly ignoring the words being said.

The useful property of a voiceprint is that two slices of the same person’s voice produce similar lists of numbers, and slices from different people produce different ones. “Similar” here is measured as the angle between the two lists (cosine similarity): the smaller the angle, the more alike the voices.

Very short bits of speech, under a second, don’t carry enough voice to be trusted, so they don’t shape the voices. They’re given to whichever voice is closest afterwards.

Step 3: Grouping voices

Now Hark has a pile of voiceprints and needs to sort them into people, without being told how many people were there.

It uses a standard technique called average-linkage clustering. Start with every slice in its own group, then keep merging the two groups whose voiceprints are most alike on average, until everything is in one group. That produces a family tree of possible groupings, from “everyone is a different person” to “everyone is the same person”.

To choose how many voices there really were, Hark scores each possible split with a measure called the silhouette score, which asks: is each slice much closer to its own group than to the next nearest group? The split with the best score wins. If even the best split scores poorly, Hark concludes there was only one voice. That stops it from inventing a second speaker when one person simply changed their tone.

Each group becomes “Speaker 1”, “Speaker 2” and so on, and every word of the transcript is given to whoever was speaking at that moment.

Step 4: Naming people once

For each voice, Hark picks clips you can listen to and shows them as cards on the meeting’s Speakers tab. You listen, then name the voice: pick an existing person or create a new one.

When you name a voice, Hark saves a voiceprint for that person, made from up to 60 seconds of their speech in that meeting. It then goes back through every meeting you have and re-checks any unconfirmed speakers against the people you’ve named, so naming someone today can label them in last week’s meeting as well.

If you name two voices in one meeting as the same person (it happens when someone’s voice changes a lot between talking and presenting), Hark merges them: their lines, timeline marks and speaking time move to one voice, and both voiceprints stay with the person, which helps future matching.

Step 5: Recognising people in future meetings

After every new meeting, Hark compares each voice’s voiceprint with the voiceprints of the people you’ve named, using the best match for each person. If a voice is close enough to someone, Hark suggests that name.

A few rules keep suggestions sensible:

Suggestions are not decisions. A match stays a suggestion until you confirm it.
Only confirmed voices teach Hark. A voiceprint is added to a person only when you’ve confirmed it, so one wrong guess doesn’t spread to other meetings.
One person, one voice per meeting. Within a meeting, a person can be suggested for at most one voice.

During a live meeting, Hark does a quicker, looser version of this so the live transcript can show names as people talk. Live labels are provisional. The full pass after the meeting replaces them.

Each person also has a card on the People page, with their own color (the same everywhere they appear: timeline, transcript, speaker cards), a photo, and contact details if you want them.

Step 6: Fixing mistakes on the timeline

Speaker recognition is good, not perfect. It struggles most with:

Similar voices, for example two people of a similar age and accent on the same call.
Overlapping speech, when people talk over each other.
Short interjections like “yes” or “right”, which carry very little voice.
Poor audio, such as a distant laptop mic or heavy compression on a call.

When Hark gets voices wrong, you don’t edit line by line. The meeting player shows the waveform with speakers along a timeline. You mark stretches of the meeting as belonging to a particular person, then press Update transcript.

Hark then reprocesses the meeting with your marks as the ground truth:

Marked stretches belong to the person you marked, and that person’s voiceprint for the meeting comes from your marks.
Any automatically found voice that touches one of your marks is treated as suspect. Its unmarked speech is cut into pieces about three seconds long, and each piece goes to whichever marked voice it sounds most like.
Voices that touch none of your marks are left as they were.

In practice, marking a few clear stretches for each person is usually enough to sort out a meeting where two voices were mixed up. Your marks, and the names you’ve given, survive reprocessing.

What stays on your computer

All of this happens locally: finding speech, making voiceprints, grouping, matching and reprocessing. Voiceprints are stored on your computer with your meetings and are never uploaded unless you turn on sync, which encrypts them before upload. There is more on that choice in why Hark transcribes on your computer.