LFM Omni Q8 + Q8 mmproj
Quality & speed (tokens/s) — LLM-judge scores 0–1; cells show mean ± standard error (SE)
Statistics: each cell is the MEAN (arithmetic average of scored samples) ± STANDARD ERROR of the mean (SE = sample standard deviation / √n); (n/N) = scored/total samples. Scores are LLM-judge quality 0–1 (higher = better); TPS = output tokens/second (higher = faster).
Quality / ratings
Image understanding
| Metric | Score (0–1) |
|---|
| Analysis | 0.75 ± 0.053 |
| Total question-answer | 0.40 ± 0.24 |
| Public transit timetable reading | 0.30 ± 0.20 |
Summarization
| Metric | Score (0–1) |
|---|
| English | 0.70 ± 0.042 |
Translation
Combined (both directions)
| Metric | Score (0–1) |
|---|
| English (all en→X) | 0.37 ± 0.027 |
| African / Middle eastern | 0.36 ± 0.063 |
| Western European | 0.48 ± 0.029 |
| Eastern European | 0.31 ± 0.041 |
| Asian | 0.32 ± 0.030 |
| All pairs | 0.40 |
To English (X→en)
| Metric | Score (0–1) |
|---|
| African / Middle eastern | 0.43 ± 0.071 |
| Western European | 0.49 ± 0.038 |
| Eastern European | 0.27 ± 0.060 |
| Asian | 0.38 ± 0.060 |
| All X→en | 0.43 ± 0.028 |
From English (en→X)
| Metric | Score (0–1) |
|---|
| African / Middle eastern | 0.22 ± 0.11 |
| Western European | 0.48 ± 0.044 |
| Eastern European | 0.36 ± 0.056 |
| Asian | 0.29 ± 0.035 |
| All en→X | 0.37 ± 0.027 |
Audio transcription
| Metric | Score (0–1) |
|---|
| Short (1 min) | 0.84 ± 0.093 |
| Medium (5 min) | 0.85 ± 0.065 |
| Long (30 min) | no data |
Tokens per second (TPS) — average: 32
Image understanding
| Metric | tokens/s |
|---|
| Analysis | 150 ± 3.8 |
| Total question-answer | 22 ± 9.4* |
| Public transit timetable reading | 48 ± 25* |
Summarization
| Metric | tokens/s |
|---|
| English | 130 ± 12 |
Translation
Combined (both directions)
| Metric | tokens/s |
|---|
| English (all en→X) | 23 ± 2.8 |
| African / Middle eastern | 18 ± 2.7 |
| Western European | 23 ± 2.8 |
| Eastern European | 15 ± 2.4 |
| Asian | 31 ± 4.2 |
| All pairs | 24 |
To English (X→en)
| Metric | tokens/s |
|---|
| African / Middle eastern | 20 ± 3.8 |
| Western European | 26 ± 3.9 |
| Eastern European | 18 ± 4.1 |
| Asian | 28 ± 3.6 |
| All X→en | 24 ± 2.3 |
From English (en→X)
| Metric | tokens/s |
|---|
| African / Middle eastern | 15 ± 2.9 |
| Western European | 20 ± 3.8 |
| Eastern European | 13 ± 2.3 |
| Asian | 32 ± 5.8 |
| All en→X | 23 ± 2.8 |
Audio transcription
| Metric | tokens/s |
|---|
| Short (1 min) | 110 ± 19 |
| Medium (5 min) | 130 ± 2.4 |
| Long (30 min) | no data |
Notes
- TPS for image question/answer and timetable reading may be inaccurate because the result text is very short.
Generated from results-*/results.json via html/build_report.py → html/aggregate.json on 2026-11-12.
This site does not use cookies.
