LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models
Runtime latency inside an on-device language model can be measured live, phase by phase and kernel by kernel, at under 3% throughput cost.
Abstract
Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose LM-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. LM-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement LM-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging LM-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities.
2.58%
Throughput reduction in prefill, under the most constrained governor
0.99%
Throughput reduction in decode
SEC ’25
ACM/IEEE Symposium on Edge Computing
The problem
Local inference is a black box
5
Inference phases profiled: embedding, prefill, decode, softmax, sampling
Kernel
Granularity, without auxiliary measurement hardware
Cite this work
Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, and Kyungtae Han (2025). LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models. The Tenth ACM/IEEE Symposium on Edge Computing (SEC ’25). https://doi.org/10.1145/3769102.3770614
@inproceedings{wang2025lmmeter,
title = {LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models},
author = {Haoxin Wang and Xiaolong Tu and Hongyu Ke and Huirong Chai and Dawei Chen and Kyungtae Han},
booktitle = {The Tenth ACM/IEEE Symposium on Edge Computing (SEC ’25)},
year = {2025},
doi = {10.1145/3769102.3770614},
url = {https://doi.org/10.1145/3769102.3770614}
}