921: NPUs vs GPUs vs CPUs for Local AI Workloads — with Dell’s Ish Shah and Shirish Gupta

Shirish Gupta and Ish Shah from Dell Technologies explore the evolving landscape of AI hardware. They discuss why Windows, enhanced by WSL 2, remains a dominant platform for developers, and delve into the distinct roles of CPUs, GPUs, and the increasingly important Neural Processing Units (NPUs). The conversation covers the trade-offs between local and cloud computing for AI workloads and introduces new hardware, like workstations with discrete NPUs, that are making on-device AI more powerful and accessible than ever.

September 9, 2025 5 min read

921: NPUs vs GPUs vs CPUs for Local AI Workloads — with Dell’s Ish Shah and Shirish Gupta — source: https://www.youtube.com/watch?v=nSBCsX_JmQ4

The Operating System Debate: Windows for AI Development

While the AI and data science communities often gravitate towards Unix-based systems, Windows remains a formidable platform for development. Statistically, Windows is the most popular OS among software developers, used by approximately 64%. Its familiarity, user-friendliness, and compatibility with essential productivity applications make it an accessible starting point. For enterprise environments, Windows offers streamlined IT management and security integrations.

A common best practice is to develop in an environment that mirrors production, and since most large-scale ML deployments run on Linux, this has traditionally been a point of friction. However, the gap is closing significantly with Windows Subsystem for Linux (WSL 2), which allows a full Linux kernel to run directly on Windows. This provides developers with the “best of both worlds”: the productivity and enterprise benefits of Windows alongside the native command-line tools and environment of Linux, eliminating the need for dual-booting.

A New Trinity of Processors: CPU, GPU, and NPU

The modern AI landscape requires a nuanced approach to processing, moving beyond a one-size-fits-all model. The choice of hardware is no longer a simple matrix; it has expanded to an “8x8” complexity of options tailored to specific needs. The key is using the right tool for the right job.

The Rise of the NPU (Neural Processing Unit)

An NPU is a specialized processor purpose-built to handle AI and ML workloads, particularly the vector math that forms the foundation of neural networks.

Efficiency is Key: NPUs are architected for maximum performance per watt. This is critical for mobile devices and laptops, where they can handle tasks like background blur or speech-to-text without draining the battery, offloading this work from the CPU.
Integrated vs. Discrete: NPUs can be integrated into the main chipset (SoC) or exist as powerful discrete cards. Dell has announced the Dell Pro Max mobile workstation, which will feature a discrete NPU, capable of running a 109-billion-parameter model locally. This is a game-changer for use cases requiring high performance in offline, secure, or latency-sensitive environments.

GPUs: The Powerhouse for Scalable Performance

GPUs remain the champions of parallel processing and are currently more versatile and scalable than NPUs for high-end AI tasks.

High-End Training and Inference: New hardware like the Nvidia Blackwell RTX Pro GPUs continues to push boundaries. For instance, the GB10 appliance will be capable of fine-tuning a 200-billion-parameter model locally.
Versatility: GPUs are not only for AI; they accelerate a wide range of tasks from CAD software to gaming, offering a dual-use benefit for many users. Their primary trade-off, especially on client devices, is higher power consumption compared to NPUs.

The CPU: The Enduring Workhorse

The CPU is still the core of the system, handling general-purpose tasks and running the operating system. If a system lacks a dedicated NPU or GPU, the CPU must also take on the new AI workloads, potentially impacting overall performance. Modern architectures like Intel’s Lunar Lake are enhancing CPU capabilities with features like on-chip memory for faster data transfer and powerful integrated GPUs that can rival some entry-level discrete cards.

The Developer Experience: Bridging Hardware and Software

To simplify the complexity of targeting these different processors, Dell has introduced Dell Pro AI Studio. This software layer abstracts away the underlying toolchains (like Intel OpenVINO) required to run models on specific silicon (Intel, AMD, Qualcomm, etc.).

Democratizing Access: It dramatically reduces development time. In one case study, a task that took a team three months to complete using standard toolchains was accomplished in just four days with Dell Pro AI Studio.
Seamless Integration: It provides an API that follows the OpenAI standard. This allows developers to switch a workload from a cloud endpoint to a local model running on a device’s accelerator with a single line of configuration change.

The Hybrid Future: Balancing Local and Cloud AI

The decision of whether to run an AI workload locally or in the cloud is use-case dependent. The future isn’t one or the other, but a hybrid model where intelligent agents route tasks to the most appropriate compute engine.

Key reasons to run workloads locally include:

Privacy and Security: Keeping sensitive or proprietary data on-device without sending it to the cloud.
Cost: Avoiding token consumption and subscription fees associated with cloud APIs.
Speed and Latency: Eliminating the round-trip time to a server for real-time inference.
Offline Capability: Ensuring applications continue to function without an internet connection.
Experimentation: Providing a “sandbox” for data scientists to quickly test hypotheses without waiting for cloud instances or queues.

Future-Proofing Your AI Hardware

As AI becomes more integrated into operating systems (e.g., Microsoft’s Co-pilot+ features running locally), having capable hardware is essential.

Three Tiers of AI PCs:
1. Essential: Entry-level NPUs (10-15 TOPS) for basic OS-level AI features.
2. Advanced: More performant NPUs (40-50 TOPS) required for features like Co-pilot+ and running smaller custom models.
3. High-Performance: Systems with high-end CPUs and discrete GPUs/NPUs for developers, data scientists, and power users.
The Windows Refresh: With Windows 10 reaching its end-of-life, an upcoming hardware refresh cycle presents a critical opportunity. Investing in the latest generation of hardware, such as devices with Blackwell GPUs, will provide significant performance gains (e.g., reducing a week-long rendering job to three days) and ensure a system is equipped for the next 4-5 years of AI advancements.