Advanced Transcription Features: Segments, Prompts, and Multilingual Support

Welcome back! So far, you have learned how to capture audio from a microphone and set up an audio analysis pipeline using the Web Audio API. Now, you are ready to take your transcription skills to the next level.

In this lesson, you will learn about advanced features that make your transcriptions more useful and accurate. These features include:

• Segments: Breaking the transcription into smaller, time-stamped parts.
• Prompts: Giving the model extra context to improve accuracy.
• Multilingual Support: Transcribing audio in different languages or letting the model detect the language automatically.

These tools are especially helpful when working with long audio files, specific topics, or audio in multiple languages. By the end of this lesson, you will know how to use these features to get more detailed and helpful transcriptions.

Before we dive into advanced features, let’s quickly remind ourselves how to set up the transcription client and prepare an audio stream:

import { OpenAI } from 'openai';

const client = new OpenAI({ apiKey: 'YOUR_API_KEY' });
const stream = /* your audio stream here */;

This setup allows you to send audio data to the Whisper model for transcription. In this lesson, we will build on this foundation to use advanced options.

When you transcribe long audio files, it is helpful to break the text into smaller parts, each with its own start and end time. These are called segments. Segments make it easier to follow along with the audio, find specific parts, or display subtitles.

const response = await client.audio.transcriptions.create({
  model: 'whisper-1',
  file: stream,
  response_format: 'verbose_json'
});

console.log(response);
/*
{
  segments: [
    { start: 0.0, end: 3.0, text: "Welcome to the AI podcast." },
    { start: 3.0, end: 6.0, text: "Today we're exploring transformers." }
  ],
  language: 'en',
  ...
}
*/

Explanation:

• The verbose_json format includes detailed segments with timestamps.
• Now we're getting structured segments that include start, end, and text. This format unlocks features like timestamped captions and more granular control over UI rendering.