Should you use it: real-time interpretation vs offline transcribe-then-translate
Qwen3.8-LiveTranslate is Alibaba Qwen's real-time simultaneous interpretation model, exposed as a WebSocket realtime API through the Qianwen AI platform and Alibaba Cloud Model Studio (Bailian). It turns speech into another language while the speaker is still talking, and reads it back in a voice close to the original speaker's.
Two metrics must be stated correctly. First, 2.3 seconds is the official LAAL (Length-Adaptive Average Lagging) figure, the average lag of translation behind the source speech. It is not an end-to-end first-packet latency, and you must not reframe it as "2.3 seconds after the first word, sound comes out". Second, 60 and 29 are different scopes: 60 is the count of recognizable input languages, 29 is the count it can speak, and the other 31 return text only. When marketing says "supports 60 languages", ask whether your target is among the 29 that can speak.
Real-time interpretation and offline transcribe-then-translate differ in latency tolerance. Offline transcribes then translates the whole clip, allowing post-processing and batch savings at the cost of waiting. Real-time emits translation while audio streams in, giving "listen and understand as you go" at the cost of a shorter per-sentence context and more engineering.
Use it for multilingual meetings, cross-border live subtitles, video localization, expos, overseas support and training, medical interpretation, and travel translation. Be cautious for contracts or legal text needing post-hoc consistency, literary translation needing finesse, and compliance cases where audio cannot leave a private cloud. For a GPT-family comparison see /en/posts/gpt-live-1-release-hotspot.
Credentials and environment setup
Step one is obtaining an API key the right way. After logging into the Qianwen AI platform or enabling the service on Alibaba Cloud Model Studio, create a key. The iron rule belongs on your monitor: the key lives only in environment variables, never in source, repository, or frontend bundles. Once committed, it lands in version history, logs, and frontend bundles where anyone can take it. The only response to a leak is to revoke the old key, create a new one, and rotate every service that used it.
Put the key into a shell variable or a secrets manager (KMS, Vault), and read it only through os.getenv("DASHSCOPE_API_KEY"). Never hardcode the string, never commit .env to Git, and exclude local secret files in .gitignore.
Environment: Python 3.10 or newer plus a WebSocket client (the official sample uses websocket-client; the DashScope SDK also works, version per docs). The protocol is the same event-driven WebSocket: open a connection, receive session.created, stream audio, receive events.
One pitfall early: a browser cannot set the Authorization header during the WebSocket handshake, so you cannot place the key in the frontend for a direct connection. The correct architecture is your own backend holding the key and opening the connection, while the browser exchanges audio through your own signaling channel. The frontend page is only a shell; authentication lives on your server.
Run the minimal real-time stream
The minimal goal: audio in, bilingual text and translated speech out. The skeleton below follows the official docs; if your console fields differ, trust the official docs.
# dependency: pip install websocket-client
import json
import base64
import os
import websocket # from websocket-client
API_KEY = os.getenv("DASHSCOPE_API_KEY") # read only from env, never hardcode
if not API_KEY:
raise SystemExit("DASHSCOPE_API_KEY environment variable is not set")
# endpoint per official docs; this is the Qwen Cloud public endpoint example
WS_URL = "wss://maas.qwencloudapi.com/api-ws/v1/realtime?model=qwen3.8-livetranslate-flash-realtime"
# a browser cannot set Authorization on the WS handshake; in production your
# own server holds the key and relays audio
HEADERS = [f"Authorization: Bearer {API_KEY}"]
def on_open(ws):
# configure the session right after connect, then start pushing audio
cfg = {
"type": "session.update",
"session": {
"output_modalities": ["text", "audio"], # ["text"] for text only
"translation": {"language": "en"}, # target language, required
"input_audio_transcription": {"language": "zh"}, # source language, auto if omitted
# optional capabilities below; field names per official docs
"enable_voice_clone": False, # voice cloning switch
"voice_clone_options": {"frequency": "once"},
"corpus": {"phrases": {"Qianwen": "Qwen"}} # hotwords, cap per official docs
}
}
ws.send(json.dumps(cfg))
print("session configured, start pushing audio")
def on_message(ws, message):
evt = json.loads(message)
t = evt.get("type")
if t == "conversation.item.input_audio_transcription.delta":
print("[src]", evt.get("delta", ""), end="", flush=True) # source ASR text
elif t == "response.audio_transcript.delta":
print("[out]", evt.get("delta", ""), end="", flush=True) # translation text
elif t == "response.audio.delta":
audio = base64.b64decode(evt.get("delta", "")) # 24kHz PCM chunks
# play(audio)
elif t == "response.done":
print("\n[segment done]")
elif t == "session.finished":
ws.close()
def on_error(ws, error):
print("error:", error)
def on_close(ws, code, msg):
print("connection closed", code, msg)
ws = websocket.WebSocketApp(WS_URL, header=HEADERS,
on_open=on_open, on_message=on_message,
on_error=on_error, on_close=on_close)
# ws.run_forever() # driven by the capture loop below; send session.finish before closingSession configuration is the most important frame. The session.update fields are shown separately for console comparison. The qwen3.8 generation differs from the older one, so verify every field against the docs.
{
"type": "session.update",
"session": {
"output_modalities": ["text", "audio"],
"translation": { "language": "en" },
"input_audio_transcription": { "language": "zh" },
"enable_voice_clone": true,
"voice_clone_options": { "frequency": "always" },
"corpus": { "phrases": { "Qianwen": "Qwen", "Tongyi": "Tongyi" } }
}
}Field meanings (per official docs): output_modalities chooses text only or text plus audio; translation.language is the required target with no implicit default; input_audio_transcription.language is the source and auto-detects when omitted; enable_voice_clone makes the translation use the original speaker's voice, after which preset voices are ignored and voice must be default or a cloned id; voice_clone_options.frequency is never (pre-cloned profile), once (clone at start), or always (re-clone before each response, good for multiple speakers); corpus.phrases is a hotword map for proper nouns.
Audio is sent continuously with input_audio_buffer.append, base64-encoded PCM. The spec is 16kHz, 16-bit, mono input and 24kHz PCM output. The local capture and push loop is shown below.
import pyaudio, base64, json
pa = pyaudio.PyAudio()
stream = pa.open(format=pyaudio.paInt16, channels=1,
rate=16000, input=True, frames_per_buffer=2048)
print("capturing, press Ctrl+C to stop")
try:
while True:
chunk = stream.read(2048)
ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(chunk).decode()
}))
except KeyboardInterrupt:
ws.send(json.dumps({"type": "session.finish"})) # wait for session.finished before close
finally:
stream.stop_stream(); stream.close(); pa.terminate()Closing discipline: always send session.finish before disconnecting, or the last segment is dropped. The server then returns conversation.item.input_audio_transcription.completed and session.finished when speech was present, and you close only after session.finished. Video or image input uses input_image_buffer.append (JPG/JPEG, under 500KB before base64, at most 2 per second, audio sent first).
Scenario 1: multilingual meeting interpretation and minutes
Multilingual meetings are the best fit because three capabilities combine: real-time speaker diarization, voice cloning, and source-translation same-frame output.
Speaker diarization is on by default with the turn configuration speaker_detection: you keep streaming audio, the server decides who is speaking and when a turn ends, then triggers translation. Downstream voice cloning stabilizes each speaker's voice because of it. Set voice_clone_options.frequency to always in turn-taking meetings to re-clone before each response and avoid one voice drifting onto another. For a voice-cloning tool comparison see /en/posts/ai-voice-cloning-tools-comparison-review.
Same-frame output is the second win. The server feeds two streams: conversation.item.input_audio_transcription.delta is the source transcript (ASR, free per docs), and response.audio_transcript.delta is the translation text, time-aligned and emitted together, so you do no alignment. For minutes this means storing source and translation side by side with timestamps, then retrieving either by source or target keyword to land on the same sentence. Long-context disambiguation keeps a name or term consistent across an hour.
Two tips: first, persist the timestamp or sequence of every delta; do not wait for completed, or a disconnect loses the middle. Second, do not hand meeting cleanup (summary, action items, attribution) to the interpretation model; it lacks function calling, structured output, batch inference, and fine-tuning. Those belong to a separate text model downstream.
Scenario 2: cross-border live streaming and video localization subtitles
Cross-border live streams and video localization use same-frame output as subtitles. The difference from meetings is that live streams are often one-way long flows and usually carry video.
Video input is supported: real-time frames go through input_image_buffer.append, and the model uses lip movement, gestures, and on-screen text as visual cues to disambiguate, helping in noisy rooms. Constraints are rate and size: at most 2 images per second, under 500KB each before base64, JPG/JPEG, audio sent first. Treat frames as a bonus disambiguation signal, not the main input.
Subtitle delivery has three jobs: chunking, timestamp alignment, and writing the translated subtitle. Chunking follows the server's delta granularity; you do not cut fixed durations on the client. Timestamp alignment uses the same-stream output where source and translation share one timeline; record each delta arrival time and you get a time-coded subtitle. Accumulate response.audio_transcript.delta into sentences, attach time codes, and write SRT or WebVTT; to keep the original, write conversation.item.input_audio_transcription.delta into a second track.
A mistake: wanting to "translate the whole video offline then publish" fits an offline pipeline better than real-time interpretation. Real-time fits live broadcasts and bilingual subtitles appearing as you stream. For cloud versus local voice trade-offs see /en/posts/cloud-vs-local-voice-ai-review. Two live tips: first, public-network latency tracks viewer geography, so pick a nearby regional endpoint and stabilize with your own server relay; second, translated audio is 24kHz PCM by default and must be decoded into a browser-playable format such as WAV/PCM or Opus before feeding <audio>.
Latency and cost optimization
Put 2.3 seconds back where it belongs: it is the official LAAL figure, the average lag of translation behind source speech, not an end-to-end first-packet latency, and do not extend it to "every sentence is only 2.3 seconds slow". The real perceived latency is the sum of four segments.
| Segment | Meaning | How much you can influence it |
|---|---|---|
| Audio capture | microphone sampling and local preprocessing | High: low-latency capture, smaller buffer, no extra resampling |
| Network transfer | audio up to server, translation back down | Medium: nearby region, server relay, stable link |
| Model inference | Interleave interpretation latency, official LAAL 2.3s | Low: decided by the model, soften wait with streaming |
| Playback | decode translated audio and push to speaker | Medium: play as it arrives, not after the whole segment |
Locate the bottleneck by timestamping each segment: record push-frame time at capture, first response.audio.delta as model sound-out, and actual speaker time as audible. If capture to sound-out is long, look at network and model; if sound-out to audible is long, the problem is your playback.
Chunking and buffering: do not batch audio into large blocks to "save traffic", because large blocks manufacture latency; push small frames at the suggested size and let speaker_detection segment naturally. On playback, play as received and decode delta into the queue immediately instead of waiting for response.done.
Concurrency and rate limits: the official docs give a default quota (for example a common per-minute request count and per-minute token count), but exact numbers change with region, account, and campaigns, so always follow the Qianwen AI platform and Alibaba Cloud Model Studio official docs and your console; do not hardcode from memory. Key reality: the default RPM is plenty for single-user trials, but a service for a whole tour group or live broadcast may exhaust its own concurrency on the first night. For multi-room scenarios, evaluate the quota early and request an increase from the official channel.
Cost: the official docs bill by tokens, with separate per-second consumption for audio in and out (commonly about 7 tokens per second in and 12.5 per second out); exact pricing and region (Beijing or Singapore) follow the official docs. It lacks function calling, structured output, batch inference, fine-tuning, and web search, so interpretation only produces translation text and audio; downstream summary and CRM write-back belong to another text model.
Pitfalls and pre-launch checklist
Six common pitfalls:
- Treating 2.3 seconds as first-packet latency and promising it, then the live experience mismatches. Remember it is LAAL.
- Connecting the browser directly and finding the
Authorizationheader cannot be set. Your server holds the key and connects; the frontend uses your own channel. - Forgetting
session.finishand closing the socket, so the last segment is lost. Sendsession.finishfirst and wait forsession.finished. - Reusing older LiveTranslate code by only changing the model name, and fields mismatch (such as
modalitiesversusoutput_modalities, orvoiceignored when cloning is on). Verify every field against the docs. - Assuming all 60 languages do speech translation, then the target is only among the 60 recognizable, not the 29 that speak. Check the table first.
- Batching audio into large blocks or adding local resampling, which raises latency. Push small frames continuously and shrink the local buffer.
Pre-launch checklist (confirm each item):
- API key only from environment variables or a secrets manager; not in source, repo, or frontend bundle
-
.envand secret files are in.gitignore, and CI never prints the full key - A leak plan exists: revoke old key, create new key, rotate services
- Your server holds the key and connects; the frontend uses your own channel, no browser direct auth header
- Target language confirmed among the 29 voice-output languages (not merely the 60 recognizable)
-
session.updatesent with target required, source set or left for auto-detect - Audio captured at 16kHz/16-bit/mono PCM, base64, then
input_audio_buffer.append - Video frames (if used) respect at most 2 per second, under 500KB each, audio sent first
-
session.finishsent before close andsession.finishedawaited - Latency timestamped across capture, network, model, playback to locate the bottleneck, and default quota assessed for concurrency
FAQ
Q1: Is 2.3 seconds the end-to-end first-packet latency? Can I assume every sentence is only 2.3 seconds slow?
A1: No. 2.3 seconds is the official LAAL (Length-Adaptive Average Lagging) figure, the average lag of translation behind the source speech. It is not a first-packet latency, and it should not be extended to a single sentence. The real audible latency adds audio capture, network transfer, model inference, and playback.
Q2: The model supports 60 languages, so can all 60 do speech translation?
A2: Different scopes. 60 is the count of recognizable input languages, of which only 29 support voice output; the other 31 return text only, no audio. Before integrating, confirm your target language is among the 29 that can speak.
Q3: Can a frontend web page connect to the interpretation service directly with WebSocket?
A3: No. A browser cannot set the Authorization header during the WebSocket handshake, so the key cannot sit in the frontend for a direct connection. The correct design is your own backend holding the key and connecting to the service, while the browser exchanges audio through your own signaling channel. This is also the security requirement that the key never enters the frontend.
Q4: How do I enable "the translation is read in the original speaker's voice" voice cloning?
A4: In session.update, set enable_voice_clone to true and set voice_clone_options.frequency as needed: never uses a pre-cloned profile, once clones once at session start, always re-clones before each response (recommended for multiple speakers). After enabling, preset system voices are ignored and voice must be default or a cloned voice id. Field names follow the official docs.
Q5: Is Qwen3.8-LiveTranslate open source? Can I download weights and deploy locally?
A5: It is not open source and has no public code repository or downloadable weights. It is an API service provided through the Qianwen AI platform and Alibaba Cloud Model Studio. All calls go through the officially hosted WebSocket realtime interface, and the model itself cannot be deployed locally.