Speaker labels in transcription, explained

Why speaker labels matter

Without labels, a meeting transcript is one long block of text. With them, you can see who agreed to what, quote an interviewee accurately, and skim a call for what one person said. Labels are what turn a transcript into a record of a conversation.

There are two separate jobs here, and it helps to keep them apart:

• Speaker diarization answers "who spoke when?" It tells voices apart and labels them Speaker 1, 2, 3.
• Speaker identification answers "who is Speaker 2?" It matches a voice to a known person. Most tools leave this to you.

How speaker diarization works

Different tools use different methods, but most modern systems follow the same broad steps:

1. Find the speech. A voice activity detector marks where someone is talking and ignores silence, music and noise.
2. Cut it into short windows. Speech is split into pieces a second or two long, short enough that each usually holds one voice.
3. Make a voiceprint of each piece. A neural network turns each window into an "embedding": a list of numbers that captures what the voice sounds like (pitch, timbre, how it resonates) rather than the words.
4. Group similar voiceprints. A clustering algorithm puts windows that sound alike in the same group. Deciding how many groups there are is the hard part: some tools ask you for the number of speakers, others estimate it.
5. Label and line up with the words. Each group becomes "Speaker N", and every transcribed word is given to whoever was speaking at that moment.

Notice what's missing: the system never understands the conversation. It's matching sounds, which is why it can be very good on clean audio and confused by situations people find easy.

Why speaker labels go wrong

Crosstalk

When two people talk at once, a window holds both voices, and its voiceprint matches neither well. The overlapping words often go to the louder person.

Short turns

"Yes." "Right." "Mm-hm." A half-second reply doesn't give the system enough sound to judge. These get pulled into the surrounding speaker's turn, which is why a transcript can show someone agreeing with themselves.

Similar voices

Two people of similar age and accent, or relatives, can produce voiceprints close enough to merge into one speaker.

One person, two "speakers"

The opposite also happens: someone who moves away from the mic, switches from laptop speakers to a headset, or laughs and shouts can get split into two labels.

Poor or mixed audio

A single phone in the middle of a big table, echoey rooms, background noise and compressed call audio all blur the differences between voices. Recording each side of a call separately gives the system a head start.

How to get better speaker labels

• Improve the audio. Mics close to people, quiet rooms, headphones on calls.
• Let people finish. Fewer interruptions means fewer mixed windows.
• Tell the tool the number of speakers if it asks; it removes the hardest guess.
• Record separate tracks for your mic and for the computer's audio on calls.

How to fix speaker labels after the fact

1. Rename each speaker once. Most tools (Word's Transcribe included) let you rename "Speaker 1" and apply it to every line.
2. Merge duplicates. If one person got two labels, give both the same name, or use a merge option if the tool has one.
3. Scan quick exchanges. Questions followed by one-word answers are where most errors hide.
4. Reassign individual lines where the tool allows it, checking against the audio.
5. Check before you quote. Never attribute a quote from labels alone; listen to it.

How Hark handles speaker labels

Hark is a free desktop app for Windows, Mac and Linux that records meetings and transcribes them on your computer. Speaker recognition is the center of it:

• Diarization on your computer. Hark works out how many people spoke and who said what, using voiceprints from NVIDIA's TitaNet model. Nothing is uploaded to do it.
• Name voices by ear. Each speaker comes with a short clip. Listen, type a name, done.
• Recognition in future meetings. When you name a voice, Hark keeps a voiceprint for that person. In later meetings it suggests their name automatically, and you confirm it. Suggestions only become certain once you say so.
• Fix mistakes on a timeline. If Hark split or mixed up voices, mark who was speaking on the meeting timeline and choose to update the transcript. Hark reassigns the speech around your marks.
• People, not just labels. Each person you name gets an entry on the People page, with a photo, color, job title, company and your own notes.

It works with any meeting app, with no bot joining the call. On Windows and Linux, Hark records your mic and the computer's audio as separate tracks. On a Mac it records the microphone only. Transcription is English only.

You can also import existing recordings (voice memos, MP3s, videos) and Hark will label and recognize the speakers in those too. See transcribing an audio recording to text.

Speaker labels across tools

Online meeting tools that join as a bot can often use the meeting app's own information about who was talking, which is a real advantage for remote calls. Desktop tools that record the computer's audio, like Hark, rely on the voices themselves, which also works for in-person conversations and imported files.

Related: how to transcribe an interview and AI note takers without a bot.