The Real Problem: Everyone Says “On-Device AI” But Nobody Defines What That Means For Their Chip

I am going to start with something that keeps coming up in my conversations with overseas hardware founders. They tell me their product needs “on-device AI.” When I ask what exactly that means for their chip selection, power budget, BOM target, and form factor, the answer is often some version of: “you know, AI stuff — voice control, smart features, maybe some kind of assistant.”

This is not a criticism. It is a genuine reflection of how fast the landscape is moving. But from where I sit in Shenzhen — surrounded by chip vendors, module suppliers, PCBA factories, and firmware teams — this level of ambiguity is where hardware projects start going off the rails.

Here is what I want to make clear in this article: on-device AI is not a single thing. It exists in distinct tiers, each tied to specific chip compute capabilities. The tier you land on determines what your product can do, what it cannot do, what it will cost, how long the battery will last, and whether your target market will actually find the capability useful. Choosing the wrong tier — or worse, not consciously choosing one at all — is one of the most expensive mistakes I see in AI hardware development right now.

Why This Tiering Exists And Why Most Teams Ignore It Until It’s Too Late

The reason tiering matters comes down to a fundamental physics problem that no amount of model optimization can fully escape: inference costs compute, compute costs power, and power is the scarcest resource in almost every consumer electronics device except perhaps wall-plugged home appliances.

Let me break down what I mean by three distinct compute tiers, because these map directly onto real chips that you can actually buy today from suppliers here in Shenzhen:

Tier 1: Ultra-Low-Power Audio and IoT Chips — Rule-Based Lightweight Agents

We are talking about chips in the range of tens of MHz clock speed, single-digit or low-double-digit milliwatt active power draw, often running bare-metal or RTOS codebases without an OS layer. Think ESP32-C3, nRF5340’s smaller core, Ambiq Apollo4 (in its lowest-power configuration), and various specialized audio DSPs from Cadence, Synaptics, and Chinese vendors like BES and Bestechnic.

What can these chips actually do for on-device AI? Let me be precise because precision matters here. They can run rule-based or very small neural network models that fit entirely in on-chip SRAM — typically under 500KB of model weight memory. In practice, this translates to:

• Voice wake word detection. A few thousand samples of a trigger phrase, matched against a compact template or tiny CNN running continuously in a always-listening mode drawing microamps. This works. This has worked for years. Nothing new here, but it is still the foundation of every voice-enabled product.
• Simple automatic device state regulation. Temperature thresholds triggering fan speed changes. Ambient light sensors adjusting LED brightness. Motion detection toggling power modes. These are not “AI” in any modern sense, but they are marketed as intelligent behavior, and from a user perspective, the distinction is academic.
• Basic environmental awareness. Detecting whether a room is occupied. Classifying sound events — glass breaking, smoke alarm, baby crying — using pre-trained tinyML models. This is genuinely useful and genuinely achievable on this tier, provided you keep the model size and inference frequency within the power envelope.

What these chips cannot do: anything resembling natural language understanding, multi-turn dialogue, image recognition beyond trivial classifications, or any form of reasoning or task planning. If your product concept requires a user to ask a question and get a meaningful answer, Tier 1 is not enough. Full stop.

In Shenzhen, I see products built on this tier all the time — Bluetooth earbuds, basic smart speakers, wearables with simple activity detection, IoT sensors with edge processing. The BOM cost for the main processor here ranges from roughly $0.80 to $3.00 in volume, depending on the vendor and integration level. Power consumption allows multi-day or even multi-week battery life in small form factors. This is the baseline for what counts as “smart” in mass-market consumer electronics today.

Tier 2: Mid-Range Embedded Chips — Multimodal Perception With Light Inference

This is where things get interesting, and where most of the current AI hardware innovation action is happening. We are looking at chips with dedicated DSP or NPU blocks, clock speeds in the hundreds of MHz to low GHz range, power budgets of hundreds of milliwatts to a few watts, and typically running FreeRTOS, Zephyr, or lightweight Linux. Representative parts include Rockchip RK3566/RK3588 (lower power modes), MediaTek Genio series, NXP i.MX 8M mini/nano, Allwinner T5/T7, Amlogic A311D/A5, and various Chinese SoCs targeting AIoT applications like Sigmastar and Ingenic.

The capability jump from Tier 1 to Tier 2 is significant but bounded. Here is what becomes realistically achievable:

• Multimodal sensor fusion. Combining audio input with IMU data, environmental sensors, and in some cases basic camera input to build a richer context picture. A smart home hub on this tier can understand not just that someone spoke a command, but roughly where they are standing, what time of day it is, what devices are nearby, and what the ambient conditions are — all before sending anything to a cloud backend.
• Lightweight model inference for short-form interaction. Compressed language models in the 100M to 1B parameter range (heavily quantized to INT8) can handle short-sentence queries, intent classification, and simple slot-filling dialogues. Not a full conversational experience, but enough to handle “turn on the lights,” “what’s the temperature,” or “play jazz music” without cloud round-trip latency.
• Intent judgment and local device coordination decisions. This is the sweet spot for Tier 2. The device can evaluate whether a user request should be handled locally (change a setting, toggle a switch, read a local sensor) versus escalated to cloud or another device. This local decision-making capability is what transforms a remote-controlled gadget into something that feels like it has its own intelligence.

The limitations remain real. You are not running GPT-class models on a $12 SoC. Response quality will be noticeably below cloud-based alternatives. Model updates require firmware OTA, which introduces its own reliability considerations. And thermal management starts becoming a real design constraint — I have seenTier 2-based smart displays that become noticeably warm to the touch after 30 minutes of continuous voice interaction, which affects both user perception and long-term component reliability.

BOM cost for Tier 2 main processors typically falls in the $8 to $25 range at volume, depending on NPU capability, memory configuration, and peripheral integration. Battery life drops from days to hours for portable devices. But the capability gain is substantial enough to justify this trade-off for many product categories.

Tier 3: High-End Edge SoCs and Flagship Mobile Chips — General-Purpose Agents

This is the top of the current on-device AI food chain. Qualcomm Snapdragon 8-series, MediaTek Dimensity 9-series, Apple Silicon (for reference design purposes), high-end Rockchip RK3588 implementations, and specialized edge AI platforms like Nvidia Jetson Orin Nano (though this sits more at the industrial edge). These chips deliver multiple TOPS of AI compute, gigabytes of RAM, and power budgets measured in watts to tens of watts.

What becomes possible here:

• Multi-turn natural conversation. Models in the 3B to 8B parameter range (quantized) can maintain coherent conversations across several exchanges, remember context within a session, and handle follow-up questions meaningfully. This is not ChatGPT-4 quality, but for many consumer use cases, it is good enough to feel like a real assistant.
• Complex task planning and execution. “Plan a weekend trip, check the weather for each day, book restaurants for Saturday evening, and add everything to my calendar.” Breaking this down into subtasks, executing each one, handling failures and retries — this requires genuine reasoning capability that only Tier 3 can currently provide on-device.
• Cross-scene intelligent orchestration. A Tier 3 device can act as a local hub coordinating other lower-tier devices. Your smart glasses capture video and audio (Tier 2-level processing), stream compressed context to your phone or a local compute box (Tier 3), which runs the heavy model, then sends commands back to your home automation system (mix of Tier 1 and Tier 2 endpoints). This hierarchical architecture is where the industry is heading.

The cost is equally significant. Main processors at this tier start around $40-$60 and go much higher for premium options. Power consumption means either large batteries, frequent charging, or tethered operation. Thermal design requires active cooling in many cases. And the software ecosystem complexity — Android/Linux driver stacks, model runtime frameworks, OTA update infrastructure — is an order of magnitude harder than Tier 1.

But for products where the AI capability IS the value proposition — not just a feature checkbox — Tier 3 is becoming non-negotiable. Premium smart glasses, AI companion devices, portable AI compute boxes, high-end smart home hubs. These products live or die by the quality of their on-device intelligence, and that quality is directly proportional to the compute tier you choose.

Where Each Tier Shows Up In Real Products — And What Goes Wrong

Let me ground this in actual product categories I encounter regularly in Shenzhen, because the abstract tiers only matter when mapped to physical devices.

True wireless earbuds (TWS) overwhelmingly operate at Tier 1. The BES2600, Qualcomm QCC series, and similar audio SoCs deliver wake word detection, basic touch gesture recognition, and simple voice command routing. Some premium models are starting to experiment with Tier 2-capable audio chips for on-device noise classification or enhanced voice pickup, but the battery reality of a 30-50mAh earbud cell makes aggressive AI compute extremely difficult. I have reviewed TWS designs where the engineering team wanted on-device translation — a genuinely compelling feature — but the power budget simply did not allow it. The feature got cut or moved to the charging case (which has a larger battery), which created a different set of UX and connectivity challenges.

Smart glasses span all three tiers depending on ambition level. Basic AR display glasses with voice control are Tier 1 territory. Glasses adding camera-based scene description or object recognition need at least Tier 2. Glasses aiming for a true AI assistant experience — conversation, information retrieval, real-time translation — are pushing into Tier 3, which is precisely why so many smart glass projects struggle with thermal management, weight, and battery life. The gap between what founders envision and what the current chip tier can deliver within eyewear form factor constraints is probably the single largest source of failure in this category.

Smart home terminals — smart displays, speakers-with-screens, central hub devices — naturally settle at Tier 2 or Tier 3. They have access to mains power (eliminating the battery constraint), have enough internal volume for modest thermal solutions, and benefit enough from reduced cloud latency to justify higher-end silicon. The interesting tension I see here is between platform vendors who want to push everything to Tier 3 (better demo, more capable assistant) and cost-conscious brands who know that a $50 smart display cannot absorb a $50 chipset. The winners in this space are finding ways to deliver most of the perceived intelligence at Tier 2 while using selective cloud fallback for complex queries.

Portable AI compute boxes are a newer category that essentially externalizes Tier 3 capability from constrained wearable or mobile form factors. Think of a pocket-sized device with a flagship-tier SoC, running a full LLM locally, connecting to your phone or glasses via Wi-Fi or BLE. I find this architecture interesting because it acknowledges the physical reality that you cannot put a 10W thermal envelope inside a pair of glasses, while still delivering on the promise of private, low-latency, on-device AI. The commercial question is whether consumers will carry yet another device. The technical question is whether the latency and reliability of the connection between the box and the endpoint device is good enough to feel seamless. Both questions remain open.

What I’ve Learned From Watching Teams Pick The Wrong Tier

I want to share a pattern I have observed repeatedly because it illustrates why this tiering framework matters in practice.

A team comes in with a product concept that sounds like a Tier 3 application — rich conversational AI, multimodal understanding, proactive assistance. Their funding and timeline suggest a Tier 1 budget. They spend months trying to compress a Tier 3 experience onto Tier 1 hardware, burning through engineering resources, making increasingly aggressive model optimizations that degrade output quality, and eventually shipping a product that disappoints users who expected the demo-quality experience they saw in marketing materials.

The inverse also happens. A team builds a Tier 3 solution for what is fundamentally a Tier 1 use case — a smart lightbulb that does not need a 7B parameter language model to turn itself on and off. The result is inflated BOM, unnecessary power consumption, extended development time, and a price point that puts them at a competitive disadvantage against simpler products that solve the same user problem adequately.

From my position helping teams navigate Shenzhen’s supply chain, the teams that succeed tend to do one thing differently: they define their AI capability requirements first, in concrete terms — not “smart AI” but “can process X type of input, produce Y type of output, within Z milliseconds, drawing at most W milliwatts.” Then they map those requirements to a compute tier. Only then do they start selecting specific chips, evaluating suppliers, and designing the product architecture.

Teams that start with a chip selection and try to figure out what AI features they can cram onto it afterward? Those are the ones I end up having difficult conversations with during EVT when the numbers do not add up.

How Easelink Helps Teams Match Compute Tier To Product Reality

This alignment process — between AI ambition, chip capability, and manufacturing feasibility — is exactly the kind of work Easelink supports for overseas hardware teams.

• Capability-to-tier mapping. We help you translate your desired AI features into concrete compute, memory, power, and thermal requirements, then identify which tier (and which specific chip platforms within that tier) can realistically meet them. Sometimes the answer is encouraging. Sometimes the answer is “your current concept needs a different form factor or a larger battery.” Both answers are valuable if you get them early.
• Chip platform evaluation across tiers. We maintain ongoing visibility into what is available in the Shenzhen chip ecosystem across all three tiers — pricing trends, availability status, toolchain maturity, factory familiarity, and real-world performance data from projects we have supported. We know which Tier 2 chips actually deliver on their NPU promises and which ones are marketing exercises.
• BOM optimization within your target tier. Once the tier is selected, we help you optimize the complete BOM around that decision — memory sizing (a major cost driver for AI-capable devices), power management IC selection, sensor choices that match your inference pipeline, and peripheral components that support rather than constrain your AI feature set.
• Firmware and production validation coordination. AI features introduce new testing requirements beyond traditional functional test. Inference latency verification, accuracy benchmarking on production units, power profiling under AI workload, thermal characterization — we coordinate the resources needed to validate these metrics through EVT, DVT, and PVT phases.
• Supplier and factory alignment. Not every factory in Shenzhen has equal experience with AI-enabled products. Some assembly houses have never dealt with devices that require AI model flashing as part of the production flow. Others have established test stations and calibration procedures. We help you find manufacturing partners whose capabilities match your product’s compute tier and complexity level.

The Pattern That Keeps Repeating: Overseas Vision + China Execution

What I am describing here — systematic mapping between AI capability requirements and hardware compute tiers — is a microcosm of the broader dynamic I see shaping the next generation of consumer electronics.

Overseas teams bring the AI vision. They understand what users want, what interaction paradigms feel natural, what use cases create genuine value. They have model expertise, application design skills, and market insight. These are real competitive advantages.

What they need on the China side is not just manufacturing capacity — it is the systems-level knowledge of how to translate those visions into tier-appropriate hardware architectures. Which chip family matches your compute requirement? Which supplier can deliver consistent quality at your target price point? Which factory has experience with the production testing methodology your AI features demand? How do you validate that a mass-produced unit delivers the same inference quality as your engineering prototype?

The companies that will win in the on-device AI wave are the ones that stop treating the chip layer as a commodity selection and start treating it as a strategic product-defining decision. And the ones that recognize that getting this decision right requires visibility into the China-side ecosystem that most overseas teams simply do not have from a distance.

Practical Takeaways: How To Approach On-Device AI Tier Selection

• Define your AI capability in engineering terms before touching a datasheet. Input types and rates, output requirements, latency tolerance, accuracy floor, active duty cycle. These parameters determine your tier. Everything else follows.
• Accept the tier boundaries as hard constraints, not soft targets. If your use case needs multi-turn conversation, Tier 1 is not an option no matter how good your engineers are. If your battery is 40mAh, Tier 3 is physically impossible. Design within the constraints of your tier rather than fighting them.
• Consider the hierarchical architecture option seriously. Not every device needs to host its own highest-tier AI. A Tier 1 wearable streaming compressed context to a Tier 2 hub or Tier 3 phone/box may deliver better overall user experience than trying to cram everything onto a single device.
• Evaluate toolchain and ecosystem alongside raw compute specs. A Tier 2 chip with mature deployment tools and community support will ship faster and with fewer bugs than a theoretically more powerful Tier 3 chip whose SDK is held together by hope and one overworked engineer.
• Plan your AI-specific production testing from the start. Traditional functional test does not catch model degradation, inference latency drift, or power anomalies under AI workload. Build these tests into your DFT plan before EVT.
• Get real BOM pricing for your target tier before committing to the product concept. AI-capable chips carry premiums. Memory requirements for model storage add cost. Larger power management solutions add cost. Make sure the economics work at your target retail price before you invest in detailed design.
• Build in a migration path. Chip capabilities improve quickly. The Tier 2 chip that barely fits your needs today will look underpowered in 18 months. Design your system architecture so that moving up a tier (or adopting a next-generation chip in the same tier) does not require a complete redesign.

Final Thoughts

The on-device AI revolution is real. But it is not a uniform wave that lifts every product equally. It is a stratified landscape where compute tier determines capability ceiling, and capability ceiling determines product category positioning. The teams that understand this stratification — and design their products honestly within a specific tier’s constraints — will build the products that define this cycle. The ones that conflate “on-device AI” as a single undifferentiated capability will burn capital and exit.

If you are working on an AI-enabled hardware product and you are not sure which compute tier you belong to, or whether your current chip selection matches your product ambitions, that uncertainty is worth resolving before you go further. The cost of answering this question early is a fraction of the cost of discovering the answer at DVT.

I Work FOR YOU, Not Factories.

My role is to stand on your side of the table — helping you navigate the chip tier landscape, match your AI vision to manufacturable hardware reality, and execute through the Shenzhen ecosystem with eyes open.

Need Help Matching Your AI Features To The Right Compute Tier?

If you are developing an AI-enabled consumer electronics product and want an honest assessment of which chip tier fits your requirements, what that means for your BOM and timeline, and how to execute through China’s manufacturing ecosystem — this is exactly the kind of evaluation I do regularly. No obligation, no sales pitch. Just a grounded operator’s view of what your product can realistically achieve given current hardware capabilities.

Reach me at [email protected] or connect via LinkedIn. Send me your product concept and your current thinking on chip selection, and I will tell you whether the math adds up — before you spend your prototyping budget discovering it yourself.