LabHub

Blog

Edge AI & TinyML 2026 — LiteRT / ExecuTorch / Edge Impulse / Jetson / Coral / Hailo / Sipeed K230 / llama.cpp / Phi-4 Deep-Dive Guide

2026-05-16 · 32 min read · #edge-ai#tinyml#tflite-micro#litert#executorch#edge-impulse#jetson-orin#coral-dev-board#hailo#sipeed-k230#rockchip-rk3588#maixpy#micropython-ml#onnx-runtime-mobile#core-ml#tensorrt#apache-tvm#mlc-llm#whisper-cpp#llama-cpp#gguf#phi-3#phi-4#gemma#llama-3-2#deep-dive#english

한국어English日本語

1. The 2026 Edge AI Map — Four Categories: MCU / SBC / Phone / Auto

Edge AI in 2026 is not a single category. The single word "edge" spans devices from 100 mW microcontrollers to autonomous-driving computers consuming over 100 W, and the models that run on them range from sub-1KB keyword spotting networks to 4-bit quantized 70B LLMs.

First, the four broad categories of 2026 Edge AI devices:

The biggest events of 2024 were two: First, Google rebranded the TensorFlow Lite mobile/embedded runtime as LiteRT — TFLite's official name is now LiteRT, and TFLite Micro is now LiteRT Micro. Second, Meta announced ExecuTorch as GA — the PyTorch camp's mobile/embedded runtime emerged as a direct alternative to TFLite/LiteRT.

Until then the conventional wisdom was "to run PyTorch on the edge, convert via ONNX to TFLite." Now there is a direct PyTorch → ExecuTorch path. So the first fork in 2026 Edge AI is: do you go with the LiteRT (Google) camp or the ExecuTorch (Meta/PyTorch) camp?

This article lays out all of those forks as a single map: from MCUs to phones, from Google to Meta, from ONNX Runtime to Core ML, from small models (Phi-3, Gemma 3, Llama 3.2) to large ones (70B GGUF), with Korean/Japanese case studies along the way.

2. TFLite Micro → LiteRT (the 2024 Rebrand)

Let us start with the story of TFLite Micro becoming LiteRT.

Ever since Google released TensorFlow Lite in 2017, TFLite has become the de facto standard for mobile/embedded ML. On top of that, TFLite Micro arrived in 2018 — a lighter runtime that runs even on MCUs with only tens of KB of RAM — and for almost seven years these two were the core of Google's edge ML strategy.

Then at Google I/O 2024 (May), Google announced two changes at once:

The rebrand reason is clear. The name "TFLite" felt too TensorFlow-bound, and by 2023-2024 the ML ecosystem was dominated by PyTorch. Google had to break the perception that "the TFLite runtime is good but it cannot run PyTorch models."

The key LiteRT changes:

LiteRT Micro (formerly TFLite Micro) follows the same flow. The C++ header-only runtime stays, but you can now build a model directly in PyTorch and send it to LiteRT Micro.

A simple PyTorch → LiteRT conversion example:

# PyTorch model -> LiteRT (old .tflite) conversion
import torch
import ai_edge_torch

class TinyClassifier(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(1, 8, 3)
        self.fc = torch.nn.Linear(8 * 26 * 26, 10)
    def forward(self, x):
        x = self.conv(x)
        x = torch.relu(x)
        x = x.flatten(1)
        return self.fc(x)

model = TinyClassifier().eval()
sample_input = (torch.randn(1, 1, 28, 28),)

# torch.export-based conversion
edge_model = ai_edge_torch.convert(model, sample_input)
edge_model.export("tiny_classifier.tflite")

That .tflite runs identically on Android, iOS, Raspberry Pi, Coral, and ESP32-S3.

The deeper significance is market competition with ExecuTorch. Had Google not embraced PyTorch compatibility, the PyTorch camp would have gone 100% with ExecuTorch. Now both standards coexist. From an edge-ML engineer's perspective, you can run the same model on both runtimes and pick whichever is faster.

3. ExecuTorch (PyTorch) GA — The Direct Alternative to LiteRT

ExecuTorch is the mobile/embedded PyTorch runtime that Meta (PyTorch) first announced at PyTorch Conference 2023. It hit 1.0 GA in 2024 and became a direct competitor to LiteRT.

Two key ideas:

Old PyTorch Mobile used a separate IR called TorchScript, which often failed to convert PyTorch's dynamic graphs cleanly. ExecuTorch adopts torch.export (PyTorch 2.x's new static graph API) as the standard, dramatically improving conversion success rates.

The ExecuTorch backend list shows how serious it is:

A single ExecuTorch graph can target iPhone Neural Engine, Snapdragon Hexagon, and Cortex-M Ethos-U with the same source.

A simple conversion example:

# PyTorch -> ExecuTorch conversion
import torch
from torch.export import export
from executorch.exir import to_edge

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = torch.nn.Linear(10, 1)
    def forward(self, x):
        return self.lin(x)

model = MyModel().eval()
example_args = (torch.randn(1, 10),)

# torch.export
exported = export(model, example_args)

# ExecuTorch conversion
edge_program = to_edge(exported)
et_program = edge_program.to_executorch()

# Save as .pte (PyTorch Edge format)
with open("my_model.pte", "wb") as f:
    f.write(et_program.buffer)

Loaded by the ExecuTorch SDK on Android/iOS, that .pte runs the same model with the same semantics as the original PyTorch dynamic graph.

LiteRT vs ExecuTorch comparison:

As of 2026 ExecuTorch is the official mobile execution path for Llama 3.2 1B/3B. It is natural for Meta to push its own LLM with its own runtime, and most Llama 3.2 mobile demos use ExecuTorch + iOS/Android.

4. Edge Impulse — The Largest TinyML Platform

Edge Impulse is a TinyML-focused startup founded in 2019. As of 2026 it is effectively the standard cloud platform for TinyML.

Its strength is handling the full stack — from data collection to deployment — in a single UI. A typical TinyML workflow:

  1. Collect sensor data — upload accelerometer, microphone, and camera data from Arduino / ESP32 / phones
  2. Labeling — label clips by class in the web UI
  3. Preprocessing — pick DSP blocks like FFT, spectrogram, MFCC
  4. Model training — Keras / scikit-learn / Edge Impulse's EON Tuner searches automatically
  5. Quantization + compilation — int8 quantization, EON Compiler generates a C++ library
  6. Deployment — Arduino IDE library, PlatformIO, or firmware OTA

The EON Compiler is Edge Impulse's secret weapon. While a generic TFLite Micro interpreter uses ~100 KB of RAM, EON compiles the model into static C++ code, cutting RAM usage by 30-50%. That is how it runs ML on Cortex-M0+ chips with only 64 KB of RAM.

Representative use cases:

Edge Impulse has official partnerships with virtually every major MCU vendor — Sony Spresense, Nordic nRF5340, Renesas RA, Silicon Labs xG24 — so SDK support is clean.

# Connect an Arduino Nano 33 BLE Sense via Edge Impulse CLI
npm install -g edge-impulse-cli

# Flash device firmware (Arduino Nano 33 BLE Sense)
edge-impulse-daemon --clean

# Export trained model as an Arduino library
edge-impulse-runner --download
# -> Import the downloaded .zip via Arduino IDE: Sketch > Include Library > Add .ZIP Library

From a company perspective, Edge Impulse's "data -> model -> firmware" full stack lowers the barrier dramatically. Firmware engineers do not need a PhD in ML, and ML engineers do not need to be firmware veterans — both sides meet inside Edge Impulse.

In 2026 LLM integration started landing on Edge Impulse Studio. A ChatGPT-style chat UI lets you say "analyze sensor data and propose a new model," and it suggests datasets, preprocessing, and candidate models.

5. NVIDIA Jetson Orin Nano / NX / Thor / AGX

NVIDIA Jetson is the standard in SBC / industrial embedded / robotics. The 2026 Jetson lineup is very strong.

Jetson Thor was unveiled at GTC 2025 and shipped in earnest in early 2026 as a computer for humanoid robots. A Blackwell-architecture GPU plus 128 GB LPDDR5X lets you run 70B-class LLMs locally and handle 14 concurrent camera/LiDAR streams. Standard usage is alongside NVIDIA Isaac Lab's robot learning environment and the Cosmos sim-to-real model.

Jetson's software stack is essentially aligned with NVIDIA desktop GPUs:

The standard way to run LLMs on Jetson is llama.cpp (GGUF) or TensorRT-LLM. On Orin Nano 8GB, Phi-3 mini (3.8B) runs at ~5-10 ms per token; on AGX Orin 64GB, Llama 3.1 70B (4-bit) hits ~30-50 ms per token. On Jetson Thor the same 70B drops below 5 ms per token, roughly matching desktop RTX 4090.

# Run llama.cpp + Phi-3 mini on Jetson Orin Nano
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_CUDA=1 -j

# Download Phi-3 mini 4-bit GGUF (example model name)
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf \
  Phi-3-mini-4k-instruct-q4.gguf --local-dir ./models

./llama-cli -m ./models/Phi-3-mini-4k-instruct-q4.gguf \
  -p "What is the capital of Korea?" -n 64 -ngl 32

Jetson's weakness is price and thermals. AGX Orin 64GB is nearly $3000, and 60 W TDP requires active cooling. People wanting lower power / lower cost look to Coral, Hailo, or Rockchip alternatives.

6. Coral Dev Board (Google TPU) — 4 TOPS, 2 W

Coral is Google's Edge TPU (Tensor Processing Unit) and the board series that ships it. One of NVIDIA Jetson's lowest-power alternatives.

The Edge TPU only runs int8 quantized models, specializing in light CNNs like MobileNet / EfficientNet-Lite / PoseNet. It cannot run large LLMs, but for "always-on 24/7 inference of a fixed small model" it is dramatically more efficient than Jetson.

Typical Coral use cases:

Coding for the Edge TPU on top of TFLite/LiteRT is straightforward:

# Object classification on Coral Edge TPU
from pycoral.utils.edgetpu import make_interpreter
from pycoral.adapters import classify, common
from PIL import Image

interpreter = make_interpreter('mobilenet_v2_quant_edgetpu.tflite')
interpreter.allocate_tensors()

image = Image.open('cat.jpg').convert('RGB')
size = common.input_size(interpreter)
common.set_input(interpreter, image.resize(size, Image.LANCZOS))

interpreter.invoke()
classes = classify.get_classes(interpreter, top_k=3)
for c in classes:
    print(f"class={c.id} score={c.score}")

Coral's 2024-2026 limitation is obvious. The Edge TPU silicon is a 2018 design, Google has not pushed a major update, and newer architectures (Transformer, ViT) are weakly accelerated. From 2024 onward, late entrants like Hailo / Sipeed / Rockchip have started taking market share.

Still, when you need a "proven, stable, low-power AI board with 4+ years of support," Coral remains a top pick.

7. Hailo-15 / Hailo-8 NPU — The Dark Horse from Israel

Hailo is an NPU (Neural Processing Unit) startup based in Tel Aviv, Israel. Founded in 2017, it became a unicorn after a $340M Series D in 2024.

The Hailo NPU lineup:

Hailo's key strength is TOPS per watt. Coral Edge TPU is ~2 TOPS/W; Hailo-8 is ~10 TOPS/W. A 5x gap.

Hailo-15 in particular is reshaping the IP camera market. Previously a camera streamed 1080p H.264 to an NVR (Network Video Recorder) that ran AI analytics. A Hailo-15-based camera runs object detection + person re-identification + pose estimation inside the camera and only transmits metadata. That is a triple win: 99% bandwidth reduction, stronger privacy, lower latency.

Hailo's SDK is its proprietary Dataflow Compiler:

# Download a pre-trained Hailo Model Zoo model and run it
pip install hailo-platform hailo-model-zoo

# Compile YOLOv8 (.hef = Hailo Executable Format)
hailomz compile yolov8s --ckpt yolov8s.pt --hw-arch hailo8

# Run inference
hailomz eval yolov8s --target hailo8 --data-zip-path coco_val.zip

Hailo's weakness is the ecosystem. The community / documentation / examples are not yet at the NVIDIA CUDA or Google TFLite level. Still, between 2025 and 2026 automotive Tier-1s like Bosch, Ficosa, and Continental adopted the Hailo-10H for ADAS, and in the automotive market Hailo now stands as one of the top three players alongside NVIDIA and Mobileye.

8. Sipeed K230 — The First Mainstream RISC-V + NPU

Sipeed is a Shenzhen-based embedded-ML board company. Famous for the MaixPy series, it began shipping the Sipeed K230 (RISC-V + NPU SoC) in earnest in 2024.

Sipeed K230 specs:

Packing 6 TOPS NPU + camera ISP + dual RISC-V cores at this price is a big deal. For comparison, Raspberry Pi 5 is $80 with no NPU (needs a separate accelerator module). Coral Dev Board is $130 at 4 TOPS. Jetson Orin Nano starts at $249.

RISC-V matters too. Unlike ARM Cortex, there is no licensing fee, and aligned with China's RISC-V self-sufficiency plan (2023-2030) the RISC-V infrastructure is maturing fast. MicroPython, OpenCV, and ONNX Runtime all officially ship RISC-V builds.

The Sipeed K230 development environment is MaixPy IDE or the raw SDK.

# YOLOv5 object detection on the K230 camera via MaixPy
from maix import camera, display, nn

# Load a YOLOv5 model onto the Kendryte KPU
model = nn.YOLOv5s(model="yolov5s_quant.kmodel")

cam = camera.Camera(640, 480)
disp = display.Display()

while True:
    img = cam.read()
    boxes = model.detect(img, conf_thres=0.5, iou_thres=0.45)
    for box in boxes:
        img.draw_rect(box.x, box.y, box.w, box.h, color="red")
        img.draw_string(box.x, box.y, box.class_name, color="green")
    disp.show(img)

The ".kmodel" format is Canaan's proprietary NPU format. A compiler called nncase converts ONNX / TFLite models to .kmodel.

# Convert ONNX -> .kmodel (Canaan NPU format)
pip install nncase

ncc compile yolov5s.onnx yolov5s.kmodel \
  --target k230 \
  --input-type uint8 \
  --output-type float32

Sipeed's 2026 hit product, MaixCAM (K230 + 5 MP camera + 2.3-inch display), runs full vision-AI demos out of the box at $65 and is selling explosively in education / maker markets.

9. Rockchip RK3588 — The De Facto Standard SBC NPU

Rockchip is an ARM-SoC design company in Fuzhou, China. The RK3588, released in 2022, has become the de facto standard SoC of the 2024-2026 SBC market.

RK3588 specs:

RK3588 boards have overwhelming bang per buck. Orange Pi 5 Plus 16GB is $130-150 and Radxa Rock 5B 16GB is $160-180 — more memory and faster CPU than Jetson Orin Nano 8GB ($249), though the NPU maturity (software + model compatibility) does not yet match NVIDIA TensorRT.

Rockchip RKNN-Toolkit is the SDK.

# Install RKNN-Toolkit2 (host PC, x86)
pip install rknn-toolkit2

# Convert ONNX -> .rknn (Rockchip NPU format)
python -c "
from rknn.api import RKNN
rknn = RKNN()
rknn.config(target_platform='rk3588')
rknn.load_onnx('yolov8n.onnx')
rknn.build(do_quantization=True, dataset='./dataset.txt')
rknn.export_rknn('./yolov8n.rknn')
"

# Run .rknn on an RK3588 board (rknnlite)
from rknnlite.api import RKNNLite
import cv2

rknn = RKNNLite()
rknn.load_rknn('./yolov8n.rknn')
rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_AUTO)

img = cv2.imread('test.jpg')
outputs = rknn.inference(inputs=[img])
print(outputs[0].shape)

The RK3588 appeal is NPU + 8K video + generous memory options in one chip. It has become standard in 4K/8K security cameras, IoT gateways, digital signage, and industrial HMI. The follow-ups RK3588S (lower-end) and RK3576 (mid) are also popular, and the RK3688 (next gen, with an expected 14 TOPS NPU) unveiled in late 2025 is on track to be the 2026-2027 standard.

10. MaixPy / Arduino Nano 33 BLE Sense / Seeed Wio AI

This section covers representative MCU / maker boards.

MaixPy (Sipeed)

MaixPy is Sipeed's embedded MicroPython environment. It runs on Maixduino, MaixCube, MaixCAM, and similar boards, integrating camera + NPU + display into a maker kit. The progression has been K210 (Gen 1, 2018), K510 (Gen 2, 2022), K230 (Gen 3, 2024).

MaixCube in particular packs LCD + camera + microphone + battery + gyro for about $30 and lets you run full AI demos — keyword spotting, face recognition, pose estimation — right out of the box.

Arduino Nano 33 BLE Sense

The Arduino Nano 33 BLE Sense (Rev2) is effectively the standard learning board for TinyML. Since its launch in 2019 it has been the official demo board for Edge Impulse and TensorFlow Lite Micro and appears in nearly every TinyML book and course.

Specs:

At that price you can run nearly every TinyML demo (keyword spotting, gesture, vibration, environmental monitoring), which is why it dominates the education market.

// Arduino Nano 33 BLE Sense + TFLite Micro keyword spotting (conceptual)
#include <TensorFlowLite.h>
#include <PDM.h>

#include "model_data.h"  // Trained model (generated by Edge Impulse, etc.)

const tflite::Model* model = tflite::GetModel(g_model);
static tflite::MicroInterpreter* interpreter;

constexpr int kTensorArenaSize = 80 * 1024;
alignas(16) uint8_t tensor_arena[kTensorArenaSize];

void setup() {
  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
      model, resolver, tensor_arena, kTensorArenaSize);
  interpreter = &static_interpreter;
  interpreter->AllocateTensors();
  PDM.begin(1, 16000);  // 1 channel, 16 kHz
}

void loop() {
  // Collect 1-second microphone clip
  // Extract MFCC features
  // Copy into the model input tensor
  // interpreter->Invoke();
  // Print result label ("yes", "no", "stop", ...)
}

Seeed Wio AI / XIAO ESP32-S3

Seeed Studio's (Shenzhen, China) Wio AI line and XIAO ESP32-S3 (Sense) are core to the maker market. XIAO ESP32-S3 Sense packs ESP32-S3 + camera + microphone + microSD onto a stamp-sized board (21x18 mm) for $10-15. It is an officially supported Edge Impulse board.

The ESP32-S3 also brings built-in Wi-Fi. Arduino Nano 33 only has BLE, but ESP32-S3 ships Wi-Fi + BLE, which makes it more suitable for IoT scenarios (uploading results to the cloud, OTA firmware updates).

MicroPython for ML

MicroPython is the embedded edition of Python. Between 2024 and 2026, running ML on top of MicroPython became more common.

The MicroPython appeal is rapid prototyping. With C++, every compile + flash cycle takes 30 seconds. With MicroPython you can execute via REPL on the device, which speeds up sensor data exploration.

11. ONNX Runtime Mobile / Core ML / TensorRT / Apache TVM

This section covers four mobile / edge inference runtimes.

ONNX Runtime Mobile

ONNX Runtime is Microsoft's multi-framework inference engine. It runs models in the ONNX (Open Neural Network Exchange) standard format and can convert from PyTorch, TF, JAX, and Keras.

ONNX Runtime Mobile is the slim mobile build.

The appeal of ONNX Runtime is camp neutrality. Between the PyTorch camp (ExecuTorch) and Google camp (LiteRT), ONNX is the safe "compatible with both" choice. The trade-off is that for quantization and NPU optimization, native runtimes (LiteRT / ExecuTorch) are usually one or two steps ahead.

Core ML (Apple)

Core ML is Apple's first-party ML runtime for its own devices (iPhone, iPad, Mac, Watch). Introduced in iOS 11 (2017), it has become the standard path for tapping the Neural Engine on A17 Pro / A18 Pro / M3 / M4 between 2024 and 2026.

Core ML's strength is Apple Silicon integration. It schedules across CPU / GPU / Neural Engine (ANE) automatically, and the M3/M4 ANE delivers 35-38 TOPS. Mobile Stable Diffusion, on-device Whisper, and all of Apple Intelligence's on-device LLMs (WWDC 2024) run on Core ML.

# PyTorch -> Core ML conversion (coremltools)
import torch
import coremltools as ct

class MyModel(torch.nn.Module):
    def forward(self, x):
        return torch.nn.functional.relu(x)

model = MyModel().eval()
traced = torch.jit.trace(model, torch.randn(1, 3, 224, 224))

mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(shape=(1, 3, 224, 224))],
    compute_units=ct.ComputeUnit.ALL,  # CPU + GPU + ANE
)
mlmodel.save("MyModel.mlpackage")

Apple Intelligence's on-device model is reported to be roughly 3B parameters (2-bit quantized) and runs at ~30 ms per token on the Neural Engine of iPhone 15 Pro and above.

TensorRT (NVIDIA)

TensorRT is NVIDIA's GPU-only inference accelerator. The same API spans desktop RTX, server H100/H200/B200, and edge Jetson.

# PyTorch -> ONNX -> TensorRT engine build
import torch
import tensorrt as trt

# 1. PyTorch -> ONNX
torch.onnx.export(model, dummy_input, "model.onnx", opset_version=17)

# 2. ONNX -> TensorRT engine
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
with open("model.onnx", "rb") as f:
    parser.parse(f.read())

config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16)
engine = builder.build_serialized_network(network, config)
with open("model.engine", "wb") as f:
    f.write(engine)

TensorRT-LLM is a dedicated LLM accelerator library that performs graph fusion + KV-cache optimization + quantization (FP8 / INT4) automatically for Llama / Mistral / Qwen. Llama 3.1 8B reaches ~5-7 ms per token on Jetson AGX Orin.

Apache TVM

Apache TVM is an ML compiler project led by OctoML. It takes PyTorch / TF / ONNX models and auto-generates code that runs on CPU / GPU / NPU / DSP.

MLC LLM (next section) is built on TVM. TVM itself has a steep learning curve, but via the MLC user-friendly wrapper it has become the key infrastructure for running LLMs on phones.

12. LLMs on Phones — MLC LLM / llama.cpp / Whisper.cpp / GGUF

The biggest change from 2024 to 2026 is that 1-8B LLMs run at practical speeds on phones. The key tools:

llama.cpp

A C++ LLM inference engine by ggerganov. Started in spring 2023, by 2026 it is effectively the standard local LLM runtime.

Its core values:

# Build llama.cpp on Android (Termux)
pkg install clang make git
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

# Download a Phi-3.5 mini GGUF (4-bit, example)
huggingface-cli download bartowski/Phi-3.5-mini-instruct-GGUF \
  Phi-3.5-mini-instruct-Q4_K_M.gguf --local-dir ./models

./llama-cli -m ./models/Phi-3.5-mini-instruct-Q4_K_M.gguf \
  -p "Explain attention." -n 128 -t 4

On phones like the Galaxy S24 Ultra or iPhone 15 Pro, Phi-3.5 mini (3.8B Q4_K_M, ~2.2 GB) runs at 30-50 ms per token (20-30 tok/s).

Whisper.cpp

A C++ port of OpenAI's Whisper speech recognition, also by ggerganov. Lets you run speech-to-text on phones / laptops without the cloud.

# Korean speech recognition with Whisper.cpp (CPU)
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
bash ./models/download-ggml-model.sh medium
make -j

./build/bin/whisper-cli -m models/ggml-medium.bin -l ko -f my_audio.wav

Whisper.cpp's Core ML build on iPhone processes a 30-minute medium model (769M) clip in ~5 minutes. The small model (244M) runs faster than real time, and base (74M) is essentially real time on a phone.

MLC LLM

MLC (Machine Learning Compilation) LLM is a phone / browser LLM engine from the Carnegie Mellon / Apache TVM camp.

The WebGPU backend is particularly interesting. As a user lands on a page, the model downloads, and Chrome / Edge / Safari runs the GPU-accelerated LLM right inside the browser. No server calls — fully local.

# MLC LLM Android demo build
git clone --recursive https://github.com/mlc-ai/mlc-llm
cd mlc-llm
python -m mlc_llm package --model "HF://mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC"
# Open the android/MLCChat project in Android Studio and build

With MLC LLM, a Galaxy S24 Ultra runs Llama 3.2 3B at ~25 ms per token (40 tok/s). On the same device the GPU backend is slightly faster than llama.cpp.

The GGUF Format

GGUF (Georgi Gerganov Unified Format) is llama.cpp's standard model file. A single file packs:

That means one .gguf can run identically across llama.cpp / Ollama / LM Studio / GPT4All.

By May 2026 Hugging Face hosts 50,000+ GGUF models, with "Q4_K_M" or "Q5_K_M" as the standard quantization. Q4_K_M is the recommended quality / size sweet spot.

13. Small Models — Phi-3 / 3.5 / 4 (MS) / Gemma 2 / 3 (Google) / Llama 3.2 1B / 3B

The biggest variable in edge LLM is model selection. Between 2024 and 2026 we saw an explosion of "1-4B parameter models that match GPT-3.5 quality." The three main families:

Microsoft Phi Series

Phi is Microsoft's small-LLM series. Building on the "Textbooks Are All You Need" paper, the goal is to approach the performance of much larger models using high-quality synthetic data with small models.

Phi-3 mini's popularity comes from being the first practical phone LLM. It reaches ~12-15 tok/s on iPhone 15 Pro and ~20-25 tok/s on Galaxy S24 Ultra, fast enough for real-time chat.

Google Gemma Series

Gemma is Google's open-model series, derived from the same research infrastructure as Gemini.

Gemma 3 27B punches above 9B, and the 4B Gemma 3n shows quality comparable to typical 8B models, optimized for mobile. PLE (Per-Layer Embeddings) distributes embeddings across layers for memory efficiency.

Meta Llama 3.2 1B / 3B

Llama 3.2, announced in September 2024, is Meta's small-model lineup. Effectively the mobile / edge line.

Llama 3.2 1B is the smallest practical LLM that still gives usable answers, running at ~50-80 tok/s on iPhone 15 / Galaxy S24. Sufficient for light scenarios — voice interfaces, chatbots, text classification.

Meta itself recommends ExecuTorch as the official mobile path for Llama 3.2 1B / 3B and ships demo apps on Android / iOS.

Model Selection Guide

For the fastest answers on a phone the order is: Llama 3.2 1B (50-80 tok/s) -> Phi-3 mini (20-25 tok/s) -> Gemma 3 4B (15-20 tok/s) -> Llama 3.2 3B (10-15 tok/s). For answer quality the order roughly reverses — Phi-3 mini / Gemma 3 4B / Llama 3.2 3B clearly outperform 1B.

14. Always-on AI — The Era of Sensor + ML

The real value of Edge AI is not one-shot inference but 24/7 always-on operation. That is Always-on AI.

Typical scenarios:

The technical core of Always-on AI is:

  1. Dual-core / dual-model — a tiny model (1-10 KB) runs constantly catching "candidates," then a larger model (100 KB - 1 MB) wakes up to verify. Keyword spotting is the canonical example. Apple Watch / Pixel Buds work this way.
  2. Quantization — int8 or below (4-bit, 2-bit) for 99% power savings. Edge TPU, Hexagon DSP, Cortex-M NPUs are all int8.
  3. Inference on NPU / DSP — the main CPU stays in deep sleep while the NPU does inference solo.
  4. Direct sensor -> ML path — camera ISP / microphone PDM share the same SoC as the NPU, so data bypasses CPU memory and goes straight to the NPU.
// Pseudocode: Cortex-M NPU always-on keyword spotting
void main(void) {
  while (1) {
    // 1. First-pass filter with a tiny model (10 KB)
    int trigger = run_tiny_kws_model(audio_buffer);

    if (trigger > THRESHOLD_LOW) {
      // 2. Wake the larger model (500 KB)
      int label = run_large_kws_model(audio_buffer);

      if (label == LABEL_HEY_SIRI) {
        // 3. Wake the application processor (UART / SPI / IPC)
        wake_application_processor();
      }
    }

    // Sleep until the next frame (DMA collects microphone data automatically)
    enter_deep_sleep();
  }
}

That pattern is why "Hey Siri" on Apple Watch runs 24 hours on almost no battery. A Cortex-M-class NPU (Apple's in-house design) listens on the microphone all day, and the main SoC only wakes when a keyword matches.

Industrial vibration anomaly detection follows the same pattern. STM32H7 + ST MEMS accelerometer + a 1 KB TFLite Micro autoencoder runs 24/7 to monitor bearing health, running for over six months on a single battery.

The 2026 trend is Visual Wake Words — the camera ISP stays on, but the main SoC only wakes when a "person is visible." The Visual Wake Words model is ~250 KB, an ultra-light MobileNet-V2 variant, and runs at the 1 mW level on Cortex-M55 + Ethos-U65 type integrated NPUs.

15. Korea / Japan — ETRI / Samsung / LG / Sony AI / NTT

Korea

Japan

Common Threads

Both Korea and Japan are leaning hard into on-device AI. NPUs are now standard in phones / cars / appliances, and cloud-LLM cost / latency / privacy issues are pushing a "do everything possible on the device" strategy.

Japan in particular has strong in-house NPU design — Renesas DRP-AI, Sony IMX500, Panasonic's vision IP, and Edgecortix's SAKURA-II have positioned themselves as global competitors to NVIDIA / Hailo / Coral.

16. Who Should Learn Edge AI — IoT / Mobile / Automotive

Finally, what tools to learn for which role.

IoT / Firmware Engineer

Mobile Engineer

SBC / Robotics Engineer

Automotive Engineer

ML Engineer / Data Scientist (transitioning to edge)

Students / Beginners

The cheapest and fastest path:

  1. Arduino Nano 33 BLE Sense ($35) + Edge Impulse (free tier) — first steps in TinyML. Keyword spotting, gesture recognition
  2. Sipeed MaixCAM or XIAO ESP32-S3 Sense ($15-$65) — camera + AI maker projects
  3. Raspberry Pi 5 + Coral USB Accelerator ($130) or Orange Pi 5 ($130) — entry to SBC
  4. Jetson Orin Nano ($249) — serious robotics / SBC

Starting with a $15 board and stepping up to a $249 Jetson over six months is the smoothest path.

17. References

Comments

No comments yet.

Sign in to leave a comment