Nvidia and Microsoft Tackle a Significant Issue with Copilot+

Illustration of The Surface Laptop utilizing local AI models.
Visual representation of Computex 2024 emblem.
This narrative is part of our report on Computex, the largest computing event globally.

When Microsoft unveiled Copilot+ PCs a few weeks ago, a predominant query emerged: Why can’t I simply execute these AI applications on my GPU? At Computex 2024, Nvidia ultimately provided a response.

Nvidia and Microsoft are collaborating on an Application Programming Interface (API) that will enable developers to operate their AI-accelerated applications on RTX graphics cards. This encompasses the diverse Small Language Models (SLMs) within the Copilot runtime, which serve as the foundation for functionalities like Recall and Live Captions.

Through this toolkit, developers can authorize applications to function locally on your GPU rather than the NPU. This not only unlocks the potential for more robust AI applications, given that GPUs generally boast superior AI capabilities compared to NPUs, but also permits operation on PCs not presently falling under the Copilot+ domain.

This marks a significant advancement. Copilot+ PCs currently necessitate a Neural Processing Unit (NPU) capable of at least 40 Tera Operations Per Second (TOPS). Presently, only the Snapdragon X Elite meets this criterion. Nonetheless, GPUs exhibit substantially greater AI processing capacities, with even entry-level models achieving up to 100 TOPS, and premium selections scaling even higher.

Incorporating the GPU, the new API introduces retrieval-augmented generation (RAG) functionalities to the Copilot runtime. RAG empowers the AI model to access specific data locally, enhancing its ability to offer more effective solutions. We witnessed RAG in action with Nvidia’s Chat with RTX earlier this year.

Comparison of performance using the RTX AI toolkit.

Alongside the API, Nvidia revealed the RTX AI Toolkit at Computex. This developer suite, set to debut in June, amalgamates various tools and SDKs enabling developers to fine-tune AI models for particular applications. Nvidia asserts that leveraging the RTX AI Toolkit can render models four times faster and three times more compact compared to utilizing open-source solutions.

We are witnessing a surge in tools empowering developers to craft specialized AI applications for end users. Some of these are already manifesting in Copilot+ PCs, but it is anticipated that a plethora of AI applications will emerge by this time next year. The hardware to support these applications is in place; now, the requisite software is the missing piece.

Evan Brooks

Evan is a seasoned reporter with an insatiable curiosity for the latest gadgets and breakthroughs in science and tech.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button