LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models
Runtime latency inside an on-device language model can be measured live, phase by phase and kernel by kernel, at under 3% throughput cost.
Georgia State University · Toyota InfoTech Labs
The Tenth ACM/IEEE Symposium on Edge Computing (SEC ’25)
Running language models locally protects privacy and cuts cost, but the performance-efficiency trade-offs on constrained hardware are largely invisible. LM-Meter is a lightweight online profiler that exposes fine-grained latency during on-device inference without auxiliary equipment, making phase- and kernel-level bottlenecks measurable rather than assumed.
Abstract
Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose LM-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. LM-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement LM-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging LM-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities.
Contributions
The first lightweight, online latency profiler built specifically for on-device language model inference.
Fine-grained real-time latency at both the phase level and the kernel level, without any auxiliary measurement hardware.
Empirical studies on commercial mobile platforms that locate phase- and kernel-level bottlenecks, quantify accuracy-efficiency trade-offs, and identify systematic optimisation opportunities.
2.58%
Throughput reduction in prefill, under the most constrained governor
0.99%
Throughput reduction in decode
SEC ’25
ACM/IEEE Symposium on Edge Computing
The problem
Local inference is a black box
The phases a runtime profiler instruments during on-device generation. Segment widths illustrate pipeline shape rather than a measured run.
Throughput cost of profiling, by phase, under the most constrained Powersave governor — the worst case for overhead. Bars are shown against a 5% ceiling so the scale is not exaggerated.
5
Phases profiled: embedding, prefill, decode, softmax, sampling
Kernel
Granularity, without auxiliary measurement hardware
Findings
2.58%
Prefill throughput cost
Profiling prefill costs 2.58% of throughput under the most constrained Powersave governor.
Measured on commercial mobile hardware in the worst-case power configuration, so the overhead in normal operation is lower.
0.99%
Decode throughput cost
Profiling decode costs under 1% of throughput in the same configuration.
Decode dominates wall-clock time in generation, so keeping its instrumentation cost below one percent is what makes continuous profiling viable.
Kernel
Finest granularity reached
Bottlenecks are visible at the level of individual kernels, not just whole phases.
Phase-level timing shows which stage is slow; kernel-level timing shows which operation inside it is responsible.
How it was done
Approach
Lightweight online instrumentation during live inference
Apparatus
Commercial mobile platforms
Datasets
On-device LLM inference workloads
Measures
Phase and kernel latency, throughput overhead
The paper
BibTeX
Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, and Kyungtae Han (2025). LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models. The Tenth ACM/IEEE Symposium on Edge Computing (SEC ’25). https://doi.org/10.1145/3769102.3770614
@inproceedings{wang2025lmmeter,
title = {LM-Meter: Unveiling Runtime Inference Latency for On-Device Language Models},
author = {Haoxin Wang and Xiaolong Tu and Hongyu Ke and Huirong Chai and Dawei Chen and Kyungtae Han},
booktitle = {The Tenth ACM/IEEE Symposium on Edge Computing (SEC ’25)},
year = {2025},
doi = {10.1145/3769102.3770614},
url = {https://doi.org/10.1145/3769102.3770614}
}Publications