Training Models on Apple’s Neural Engine (6.6 TFLOPS/W)

What It Is

Apple’s Neural Engine (ANE) is a dedicated neural processing unit built into M-series chips, designed primarily for inference tasks like Face ID and image processing. Until recently, developers could only access it indirectly through CoreML or Metal frameworks. A reverse engineering effort has now exposed the underlying APIs, enabling direct model training on the ANE rather than relying on the GPU.

The breakthrough centers on bypassing Apple’s official abstraction layers to communicate with the ANE’s private frameworks. This unlocks the chip’s raw computational capabilities for training workloads, not just inference. Early experiments successfully trained a 110M parameter microGPT model entirely on the neural engine, demonstrating that the hardware can handle backpropagation and gradient updates despite being optimized for forward passes.

The efficiency numbers are striking: the M4’s ANE achieves 6.6 TFLOPS per watt at peak performance. For comparison, training on Metal GPU delivers roughly 1 TFLOPS/watt, while NVIDIA’s H100 datacenter GPU manages around 1.4 TFLOPS/watt. This positions the ANE as potentially the most power-efficient training hardware currently available, though with significant caveats around programmability and scale.

Why It Matters

Power efficiency has become a critical bottleneck in machine learning. Training runs consume enormous amounts of electricity, and inference at scale requires massive server farms. Hardware that delivers 4-6x better efficiency than datacenter GPUs could reshape where and how models get trained.

For individual developers and researchers, this opens possibilities for local fine-tuning on consumer hardware. LoRA adapters for 3B or 7B parameter models could run on a Mac Mini drawing minimal power, making experimentation accessible without cloud compute bills. Small teams working on domain-specific models might train entirely on local hardware rather than renting GPU instances.

The broader implications extend to edge computing and privacy-focused applications. Models fine-tuned locally never send data to external servers, addressing concerns around sensitive information. Organizations dealing with medical records, legal documents, or proprietary data could adapt foundation models without exposing training data to third parties.

However, this remains experimental territory. Apple hasn’t officially sanctioned direct ANE training, and the private APIs could change without notice in future OS updates. The approach also requires technical sophistication beyond typical ML workflows, limiting adoption to developers comfortable with low-level hardware interfaces.

Getting Started

The reverse engineering work lives at https://github.com/maderix/ANE, providing the foundation for direct ANE access. The repository includes examples for loading models and executing training loops outside the CoreML framework.

A basic training setup might look like:

model = ane.load_model("microgpt_110m.ane")
optimizer = ane.SGD(model.parameters(), lr=0.001)

for batch in dataloader:
 loss = model.forward(batch)
 loss.backward()
 optimizer.step()

Detailed benchmarks and methodology appear at https://open.substack.com/pub/maderix/p/inside-the-m4-apple-neural-engine-615, showing performance across different model sizes and batch configurations.

The technical writeup at https://open.substack.com/pub/maderix/p/inside-the-m4-apple-neural-engine explains the reverse engineering process, including how private frameworks were identified and interfaced with. Developers interested in the underlying mechanics will find detailed explanations of the ANE’s architecture and memory management.

Context

Traditional approaches to Mac-based training rely on Metal Performance Shaders, which route computations through the GPU. This works but sacrifices the ANE’s efficiency advantages. CoreML supports some on-device training scenarios but imposes significant constraints on model architectures and training procedures.

Compared to cloud-based training, local ANE training trades raw throughput for efficiency and privacy. An H100 delivers far more absolute compute, but the ANE’s power efficiency matters for sustained workloads on battery power or in environments where electricity costs dominate.

Limitations remain substantial. The ANE’s memory bandwidth and capacity constrain batch sizes and model dimensions. Distributed training across multiple devices faces coordination challenges since the ANE wasn’t designed for multi-node communication. Debugging tools and profiling capabilities lag far behind mature GPU ecosystems like CUDA.

The approach also exists in a legal gray area. Accessing private APIs violates Apple’s developer guidelines, potentially breaking with OS updates. Production systems can’t rely on undocumented interfaces that might disappear or change behavior without warning.

Still, the efficiency numbers suggest Apple’s neural engine architecture has untapped potential beyond its intended inference role. Whether this becomes a practical training platform depends on community development and possibly Apple’s response to these reverse engineering efforts.