{{opener}} With large language model (LLM) output increasingly processing sensitive information and sensitive model context in personal, corporate, and regulated settings, data must be processed in a secure environment. NVIDIA Confidential Computing (CC) offers the ability to securely perform these tasks using encrypted virtual machines (CVMs), confidential graphics processing units (GPUs), and encrypted NVIDIA NVLink. This allows AI production output to run on trusted hardware. Inference frameworks such as NVIDIA TensorRT LLM deliver outstanding AI inference by combining framework-level optimizations with NVIDIA compute acceleration. However, when these frameworks run in a CC-aware environment, safe execution changes assumptions about memory movement, timing, scheduling, and multiprocessor communication. These changes can cause performance overhead if the runtime does not adapt. Therefore, to maintain high performance, it is necessary to optimize both the inference framework and the confidential computing environment together. For AI platform engineers evaluating
x confidential inference on NVIDIA Blackwell GPUs, this post looks at CC-aware adaptations that use AI inference frameworks such as TensorRT LLM to account for secure execution while maintaining inference performance. It presents a supervised methodology that teams can apply to quantify CC overhead on their own workloads. Workload selection to identify CC overhead. The characteristics of the workload determine how noticeable the CC overhead is. High request volume can offset the fixed costs of encryption by overlapping latency with other work, making the direct effects less noticeable. To detect these effects, select a workload with long input context, long output context, and low concurrency. Long context burdens data movement during prefetch, long generation intensifies the small CC overhead per token during decoding, and low concurrency limits the ability to hide this overhead in parallel queries. NVIDIA's performance engineering team used these
characteristics for the workload evaluated here. Workload configuration: TensorRT LLM inference framework, PyTorch backend, I/O sequence length 32K/1K, concurrent queries 1, 2, 4, 8 and 16, parallelism TP=8, EP=1, PP=1, KV cache FP8. CC overhead measurement with supervised comparison of CC-on and CC-off. To isolate the performance impact of CC, run the same workload under two conditions: confidential computing disabled (CC-off) and enabled (CC-enabled), holding model, hardware, framework version, sequence lengths, parallelism, and concurrency constant so that the CC state is the only variable. At each competitive level: output performance is maintained: 100 x (token output CC-enabled ÷ token output CC-disabled). Time delay overhead per output token (TPOT): 100 x (TPOT CC-enabled ÷ TPOT CC-disabled − 1). Performance teams can use these measurements to quantify how much of the CC-off baseline is retained when CC is enabled for the target workload. The NVIDIA performance engineering team applied this comparison using
using the hardware and software configuration summarized in the table. Performance results. As shown in the figures, at concurrencies ranging from 1 to 16, CC-enabled maintained 96.1–98.2% of the token withdrawal performance of CC-off, while average TPOT remained within 1.2% to 4.3% of baseline. Identify and reduce CC overhead. The NVIDIA Blackwell Confidential Computing Architecture introduces hardware-based security paths to protect data and workloads in use. For a detailed overview of the architecture, see Hardware-Based AI Security That Doesn't Slow You Down. For TensorRT LLM users and framework developers, the following details show how these protected paths change common runtime assumptions and how TensorRT LLM adapts to reduce the resulting performance overhead. Adaptation of data movement from host to device. In the B200 CC, host-to-device transfers go through a software-encrypted buffer because the GPU cannot directly access the CVM's protected memory. This changes the behavior expected by inference frameworks: pinned memory is no longer
provides its usual benefits of asynchronous transfer, and some copies may block the calling thread. Mitigation for host-to-device transfer: TensorRT LLM uses CC-aware memory selection, choosing a memory page for affected paths instead of unconditionally using pinned memory. Mitigation for device-to-host transmission: TensorRT LLM moves re-tokens and fetch data into an asynchronous workflow, preventing protected copies of the main scheduler from blocking during decoding. Kernel autotuner time stabilization. The kernel autotuner typically uses CUDA events to compare candidate tactics. In the tested CC configuration, CUDA event timestamps produced an unstable timing signal, which could cause the selection of a slower tactic. Mitigation: TensorRT LLM uses %globaltimer GPU for tactical measurements in CC environments, keeping CUDA events outside CC. CC-aware communication selection for multi-GPUs. NVLS multicast (NVLink SHARP) is not available on B200 CC configurations. Without NVLS, NCCL_SYMMETRIC cannot provide its intended multicast benefits, but may still incur the costs of registering
memory and synchronization between ranks before using a non-multicast collective path. Mitigation: Frameworks targeting CC should detect NVLS availability and select communication algorithms that minimize latency for a given message size, topology, and load characteristics. Running with NVIDIA Confidential Computing. NVIDIA Confidential Computing extends hardware-based security through confidential virtual machines, NVIDIA Blackwell GPUs, and encrypted NVLink, protecting sensitive models, enterprise context, and sensitive cues as they are processed. Confidential computing doesn't eliminate the need for performance engineering—it makes framework awareness even more important. With CC-aware TensorRT LLM adaptations for secure data movement, auto-tuning, and multi-GPU communication, DeepSeek-R1 confidential inference retained over 96% of CC-off token inference performance while maintaining per-token latency overhead below 5% on eight NVIDIA B200 GPUs. As organizations move to private production, security configuration
and output optimization should be considered as a single deployment task. Enable confidential computing, validate the environment, and benchmark CC-enabled and CC-disabled using the exact workload you intend to serve. The material was prepared by a neural network based on the original publication. AI may make mistakes in details (numbers, names, dates, quotes) - for important information, check with the original source. Anthropic unveils Opus 5.5 with lower prices and improved performance Optimizing load balancing with NVIDIA Topograph Qualcomm announced new AI-focused Snapdragon processors for smartphones. NVIDIA introduced DLSS 5 with 3D neural rendering to create realistic graphics in games. NVIDIA has introduced a solution for multi-device execution of TensorRT on multiple GPUs.
