Apple's Neural Engine and What On-Device AI Actually Means for Privacy

Apple introduced the first Neural Engine in the A11 Bionic chip in September 2017, a dedicated hardware block for neural network inference that appeared in the iPhone X alongside Face ID. The press framing at the time was almost entirely about speed: Face ID would unlock in milliseconds. The deeper significance — that Apple was staking out an architectural position in which computation happens on the device rather than in a data center — was less remarked upon.

Nine years later, that architectural position is central to Apple’s competitive identity in the AI era. The M4 chip contains a 16-core Neural Engine capable of 38 trillion operations per second. This is not interesting because 38 TOPS is impressive relative to cloud inference infrastructure (it is not, particularly). It is interesting because 38 TOPS at the edge, running permanently on a device in your pocket with no network latency and no server logging, changes what AI can be embedded in without anyone’s data leaving the device.

Understanding what this actually means for privacy requires understanding what the Neural Engine does, what the secure enclave does, and where the gap between the two lies.

What the Neural Engine Actually Is

Apple’s Neural Engine is a fixed-function matrix multiplication accelerator optimized for the inference operations that dominate neural network computation: matrix-vector products, convolutions, and element-wise nonlinearities. It is not a general-purpose processor; it cannot run arbitrary code efficiently. It is architected specifically to execute Core ML models, Apple’s neural network inference framework, with minimal power draw.

Related reading

The A17 Pro Neural Engine consumes roughly 100-200 milliwatts during active inference — a fraction of the energy that would be required to send the same data to a cloud server, receive a response, and process the result. For tasks that run continuously, like the on-device transformer model that powers Apple Intelligence’s writing assistance and Siri’s language understanding, this power profile makes continuous local inference viable in a way that cloud inference simply is not.

The inference quality achievable locally has improved dramatically with model compression research. The Apple Foundation Models, announced at WWDC 2024, use techniques including group query attention, low-bit quantization, and knowledge distillation from larger models to achieve performance comparable to GPT-3.5 on most tasks while running entirely on the Neural Engine of an iPhone 15 Pro or M-series Mac. Apple has not published the full parameter counts, but independent analysis suggests the on-device models are in the 3-7 billion parameter range.

The Secure Enclave and What It Actually Protects

The Neural Engine is separate from the Secure Enclave, and the distinction matters enormously for privacy claims. The Secure Enclave is a physically isolated coprocessor with its own memory, its own operating system (SEP OS), and cryptographic isolation from the application processor. It handles the most sensitive operations: Face ID biometric templates, device encryption keys, health data from the Secure Element.

The Neural Engine is not part of the Secure Enclave. Neural Engine computation is accessible to the application processor and, through Core ML, to third-party applications with appropriate entitlements. A camera application that uses a Core ML model for subject recognition runs that model on the Neural Engine, and the inputs to that computation — camera frames — pass through memory that the application processor can access.

This means the privacy guarantee of Neural Engine inference is not that your data is cryptographically protected from the rest of the system. It is that the data never has to leave the device to produce the inference result. The threat model being addressed is the cloud server — a third-party infrastructure that could log, store, or analyze your requests. The threat model not being addressed is the application itself, which still processes the inputs and outputs of Neural Engine inference in regular application memory.

Private Compute Cloud: The Honest Middle Ground

Apple Intelligence, launched in iOS 18 in late 2024, introduced a feature called Private Compute Cloud — a mechanism for routing more complex requests to Apple’s servers while providing verifiability that those servers cannot log or retain your data. This is not on-device inference; it is cloud inference with a specific privacy architecture.

The Private Compute Cloud uses a combination of differential privacy, hardware attestation (the server must prove it is running specific, audited software), and an onion-routing-like request structure that prevents the server from knowing which device made a specific request. Independent security researchers, including those at Johns Hopkins Information Security Institute, reviewed the cryptographic architecture and found it sound with specific caveats about what “sound” means in this context.

The honest characterization is this: for tasks the Neural Engine can handle locally (most Siri language queries, on-device writing assistance, photo categorization, real-time translation), your data does not leave the device. For tasks requiring larger models — complex multi-step reasoning, queries that benefit from the knowledge base only large models contain — your request goes to Private Compute Cloud where it is processed without being logged, but it does leave the device.

Related reading

This is meaningfully better than the alternative of sending requests to a conventional cloud server, but it is not the same as on-device inference. The marketing tends to blur this distinction, which is unfortunate because the actual architecture is interesting and defensible on its own terms.

Qualcomm’s Parallel Claim

Apple is not the only company with significant on-device AI inference hardware. Qualcomm’s Hexagon NPU in the Snapdragon 8 Gen 3, shipping since late 2023, delivers 45 TOPS — slightly more than the A17 Pro Neural Engine — and runs the Qualcomm AI Engine framework that supports third-party models including Llama 2 7B, Mistral 7B, and others.

The competitive difference is in the software layer and the market position. Apple controls both the hardware and the operating system, which allows tight integration between Core ML, the Neural Engine, and the operating system’s privacy APIs. A third-party application on iOS cannot silently exfiltrate Neural Engine inference outputs in ways that the OS does not mediate. On Android, the isolation guarantees are weaker — the fragmented ecosystem means that the specific behaviors of the Hexagon NPU pipeline depend on the device manufacturer’s software integration choices.

Qualcomm has also been more aggressive about publishing what third-party models can run on its hardware, which serves developer mindshare but does not necessarily serve user privacy. The fact that you can run Llama 2 7B on a Snapdragon 8 Gen 3 device is a capability story. Whether the application that loaded that model handles your conversation inputs responsibly is an application story that the chip has no opinion about.

The Fundamental Privacy Limit of On-Device AI

The deepest privacy guarantee that on-device AI inference provides is this: the model provider cannot log your queries. If you use Apple’s on-device writing assistant to compose a medical question to your doctor, Apple does not have a server-side log of that question. If you use an offline speech recognition model to transcribe a private conversation, the audio never touches a network.

This is real and valuable. But it addresses a specific threat model — corporate data collection — while leaving others unaddressed. If your device is compromised by malware, the malware can intercept Neural Engine inference inputs and outputs in exactly the same way it can intercept any other computation. On-device inference does not protect against device compromise. If the application requesting inference is itself malicious, it already has access to the inputs it provides.

The privacy benefit is structural for the cloud-based logging threat and irrelevant for the endpoint compromise threat. Most users conflate these two threat models when thinking about “privacy,” and Apple’s marketing (though less irresponsible than most competitors) has not always been careful to distinguish them.

Why This Still Matters

Despite the caveats, the architectural shift toward on-device inference represents a genuine improvement in the default privacy posture of AI-assisted applications. Before the Neural Engine era, almost any AI-powered feature — voice assistant queries, photo recognition, predictive text — required a network round-trip to a cloud server. The server operator had access to every query from every user, indefinitely.

Related reading

On-device inference eliminates that default data collection for tasks within the model’s capability. The transition happening right now — where more and more AI tasks become feasible locally as model compression and hardware efficiency improve — is a meaningful structural change in who has access to what data, regardless of the policy statements of any particular company.

The M4 Pro’s 38 TOPS, which would have seemed exotic compute in a data center in 2018, is now available in a laptop that millions of people carry to work every day. The trend is clear. Within five years, the Neural Engine equivalent in a mid-range smartphone will exceed the inference capacity of a 2020-vintage server GPU. At that point, the argument for sending routine AI queries to the cloud shifts from a capability argument to a business model argument — and users will increasingly be able to make the choice.

Apple’s architectural bet from 2017 was not primarily about Face ID speed. It was about who controls the compute when AI becomes ambient.