What’s the best balance between power and accuracy for on-device AI?

[Luke.C]

I’ve been prototyping a small device that needs to run a basic vision model locally, and I’m stuck trying to choose between a microcontroller with a hardware accelerator or a low-power single-board computer. The latency is okay, but I can’t seem to get the power draw low enough for the battery life I need without the model accuracy dropping off a cliff. Has anyone else hit this wall with on-device AI?

[Jack_R]

I’ve been there. I went with an MCU plus a tiny accelerator and watched the numbers. Idle current was sane, but once the model started pulling weights and feature maps, the draw spiked. I was getting roughly 20 hours on a small LiPo when I ran inferences continuously; when I cut the frame rate to once a second, it stretched to ~50 hours but the latency still felt acceptable for my use. The bottleneck turned out to be memory bandwidth and weight loading, not raw compute.

[Ella_L]

On the other path I tried a low power single board computer. Accuracy stayed nicer, but the power budget was the blocker. Even with DVFS, turning off radios, and a lean kernel, I saw something like 100–200 mW idle and a watt or more during inference. I pruned and quantized, but the accuracy dropped a few points and the power didn’t improve enough to matter. Ended up chasing a smaller model rather than squeezing more juice out of the same one.

[Joshua.J]

I did the event-driven thing too. Wake on trigger, do one fast pass, sleep. It helped the average draw but then you fight missed frames, jitter, and user perception. I logged the duty cycle and found the majority of energy was spent just waiting for the next wake, not doing math. Still not solved, but the numbers told a different story than the intuition. Is that really the bottleneck, or is there another constraint?

[AndrewTJ]

I keep circling back to the model shape. If you need several ops per pixel, the math isn't friendly to tiny hardware. Maybe a purpose-built tiny model with a couple of layers and a fixed camera resolution would stretch life more than chasing parity on a big model. I’ve seen folks pivot to different use cases rather than chasing precision. Not sure this helps your exact problem.