Many users desire to run Large Language Models (LLMs) locally to ensure greater privacy and control. Recent advancements have made this possible without sacrificing output quality, thanks to open-weight models like OpenAI’s gpt-oss and Alibaba’s Qwen 3, which can now run directly on PCs. NVIDIA RTX PCs are engineered to accelerate these experiences, delivering fast and responsive AI performance. NVIDIA has been actively optimizing top LLM applications to leverage the Tensor Cores in RTX GPUs, unlocking maximum performance.
Getting Started with Local LLMs on RTX PCs
Here are some of the easiest ways to get started with AI locally on a PC:
- Ollama: This open-source tool provides a user-friendly interface for running and interacting with LLMs. Key features include PDF support via drag and drop, conversational chat capabilities, and multimodal understanding that incorporates text and images.
* NVIDIA has collaborated with Ollama to enhance performance and user experience, with recent developments including:
* Performance improvements on GeForce RTX GPUs for OpenAI’s gpt-oss-20B model and Google’s Gemma 3 models.
* Support for the new Gemma 3 270M and EmbeddingGemma3 models for hyper-efficient retrieval-augmented generation on the RTX AI PC.
* Improved model scheduling to maximize and accurately report memory utilization.
* Stability and multi-GPU improvements.
- AnythingLLM: This open-source application allows users to create their own AI assistants powered by any LLM. AnythingLLM can be used with Ollama to benefit from its accelerations.
- LM Studio: Powered by the llama.cpp framework, LM Studio provides a user-friendly interface for running models locally. It enables users to load different LLMs, chat with them in real-time, and serve them as local API endpoints for integration into custom projects.
* NVIDIA has optimized performance on NVIDIA RTX GPUs within llama.cpp, with the latest updates including:
* Support for the latest NVIDIA Nemotron Nano v2 9B model, which is based on the novel hybrid-mamba architecture.
* Flash Attention now turned on by default, offering up to a 20% performance improvement compared with Flash Attention being turned off.
* CUDA kernels optimizations for RMS Norm and fast-div based modulo, resulting in up to 9% performance improvements for popular models.
* Semantic versioning, making it easy for developers to adopt future releases.
Creating AI-Powered Study Buddies
Local LLMs allow you to create context-aware AI conversations, enabling greater flexibility for building conversational and generative AI-powered assistants. This opens up a number of possibilities, including personalized study assistants.
For students, managing slides, notes, labs, and past exams can be overwhelming. Local LLMs make it possible to create a personal tutor that can adapt to individual learning needs. With tools like AnythingLLM, students can load syllabi, assignments, and textbooks into RTX PCs to create an adaptive, interactive study companion.
This allows them to perform tasks like:
- Generating flashcards from lecture slides
- Asking contextual questions tied to their materials
- Creating and grading quizzes for exam prep
- Walking through tough problems step by step
Beyond academics, hobbyists and professionals can leverage AnythingLLM to prepare for certifications or other similar purposes. Running locally on RTX GPUs ensures fast, private responses without subscription costs or usage limits.
Project G-Assist: AI Assistance for Gaming PCs
Project G-Assist is an experimental AI assistant designed to help users tune, control, and optimize their gaming PCs through simple voice or text commands, eliminating the need to navigate through complex menus. A new update will be rolled out via the NVIDIA App.
Key features of the update include commands to adjust laptop settings:
- App profiles optimized for laptops: Automatically adjust games or apps for efficiency, quality, or a balance when laptops aren’t connected to chargers.
- BatteryBoost control: Activate or adjust BatteryBoost to extend battery life while keeping frame rates smooth.
- WhisperMode control: Cut fan noise by up to 50% when needed, and go back to full performance when not.
Project G-Assist is also extensible through the G-Assist Plug-In Builder, allowing users to create and customize functionality by adding new commands or connecting external tools with easy-to-create plugins. The G-Assist Plug-In Hub also allows users to discover and install plugins to expand G-Assist capabilities.
