GPU Travelling: Efficient Confidential Collaborative Training with TEE-Enabled GPUs

Confidential collaborative machine learning (ML) enables multiple mutually distrusted data holders to jointly train an ML model while preserving the privacy of their private datasets due to regulatory or competitive reasons. However, existing works need frequent data and model exchanges during training via slower conventional links and face increasing challenges due to the exponentially growing sizes of models and datasets in modern training workloads like large language models (LLMs), resulting in prohibitively high communication costs. In this paper, we propose a novel mechanism called GPU Travelling that leverages recently emerged confidential GPUs. With our rigorous design, the GPU can securely travel to the specific data holder to load the dataset directly into the GPU’s protected memory and then return for training, eliminating the need for data transmission while ensuring confidentiality. We developed a prototype using Intel TDX and NVIDIA H100 and evaluated its performance on llm.c, demonstrated the performance and feasibility while maintaining strong security guarantees. The results shows at least 4x speed improvement when transmitting a 512 MiB chunk of dataset versus conventional transmission.