EverydayBench - Small LLM Test Results

Summary

Small, multimodal LLMs (< 10 GB) are poised for use in personal assistants. A variety of vendors have released open-weight models that can process text, images, and audio. These small models are expected to process any material that their operator comes across easily and simply, but not to write code or perform long-horizon tasks.

Small multimodal LLM evaluation goals are different from those associated with large LLMs. Instead of trying to identify the smartest model, we want a constellation of models that can perform a variety of simple functions quickly and reliably. Instead of "Einstein", we want a reliable clerk.

Once sufficient models of adequate quality are available, optimization should proceed along the axes of speed and size.

We developed a test suite which we call "EverydayBench". It is intended to sample everyday tasks that one might perform using a lightweight LLM. It includes translation, summarization, image analysis, and audio analysis. Notably, it does not include coding.

EverydayBench was applied to five small LLMs:

LFM 3 Omni BF16 with BF16 mmproj
LFM 3 Omni Q8 with Q8 mmproj
Gemma E4B Q8 (Unsloth) with BF16 mmproj
Gemma E4B IQ2M (Unsloth) with BF16 mmproj
Nvidia Nemotron 3.5 Bolt E4B Q4_K_M with Gemma E4B BF16 mmproj

The best results are achieved using Gemma E4B Q8 (9.3 GB, 150 TPS), but the difference between this and Nemotron E4B Q4KM (7.4 GB, 180 TPS) is insubstantial.

For those who have less VRAM to spare, we compare the two LFM models to a heavily quantized Gemma E4B IQ2M (4.5 GB, 190 TPS). The difference between LFM Q8 (2.3 GB, 32 TPS translation / >120 TPS other tasks) and LFM BF16 (4.3 GB, 93 TPS) is small enough to ignore. The LFM models provide image analysis and summarization quality that are similar to that of Gemma E4B IQ2M. Although the LFM audio scores are better than that of Gemma IQ2M, LFM is able to handle only 1 and 5 minute audio clips, whereas Gemma also processes 30 minute clips successfully. In addition, LFM is broadly worse on translation, across all regions of the world.

Statistics: each cell is the MEAN (arithmetic average of scored samples) ± STANDARD ERROR of the mean (SE = sample standard deviation / √n); (n/N) = scored/total samples. Scores are LLM-judge quality 0–1 (higher = better); TPS = output tokens/second (higher = faster). Grouped bars of quality means per model and category; each category combines its subcategories exactly from the per-leaf stats. LFM Q8 and LFM bf16 have no long-audio data, so their audio mean covers short/medium clips only. Exact values are in the page tables below.

Quality by category (graph data)

Model (size)	Image	Audio	Summarization	Translation
LFM Omni Q8 + Q8 mmproj (2.3 GB)	0.65 ± 0.086	0.84 ± 0.056	0.70 ± 0.042	0.40 ± 0.019
LFM Omni bf16 + bf16 mmproj (4.3 GB)	0.63 ± 0.081	0.83 ± 0.053	0.69 ± 0.038	0.40 ± 0.020
Gemma E4B UD-IQ2 + BF16 mmproj (4.5 GB)	0.64 ± 0.084	0.69 ± 0.068	0.60 ± 0.026	0.64 ± 0.013
Nemotron-3.5-Bolt + dflash + mmproj (7.4 GB)	0.76 ± 0.066	0.79 ± 0.071	0.73 ± 0.026	0.69 ± 0.013
Gemma E4B BF16 + BF16 mmproj (9.3 GB)	0.81 ± 0.062	0.82 ± 0.044	0.71 ± 0.028	0.74 ± 0.011

Model (size)	LFM Q8	LFM bf16	Gemma IQ2M	Nemotron	Gemma Q8
Image	0.65 ± 0.086	0.63 ± 0.081	0.64 ± 0.084	0.76 ± 0.066	0.81 ± 0.062
Audio	0.84 ± 0.056	0.83 ± 0.053	0.69 ± 0.068	0.79 ± 0.071	0.82 ± 0.044
Summarization	0.70 ± 0.042	0.69 ± 0.038	0.60 ± 0.026	0.73 ± 0.026	0.71 ± 0.028
Translation	0.40 ± 0.019	0.40 ± 0.020	0.64 ± 0.013	0.69 ± 0.013	0.74 ± 0.011

Exact values of the chart above, computed from the per-subset stats with the same variance decomposition as the error bars. LFM Q8 and LFM bf16 have no long-audio data, so their audio cells cover short/medium clips only.

Per-measurement-class views (models as rows)

Individual Model Pages

Generated from results-*/results.json via html/build_report.py → html/aggregate.json on 2026-11-12.