Фото: @brianbowensmith / @lofficielhommes.usa
By default, freeing memory in CUDA is expensive because it does a GPU sync. Because of this, PyTorch avoids freeing and mallocing memory through CUDA, and tries to manage it itself. When blocks are freed, the allocator just keeps them in their own cache. The allocator can then use the free blocks in the cache when something else is allocated. But if these blocks are fragmented and there isn’t a large enough cache block and all GPU memory is already allocated, PyTorch has to free all the allocator cached blocks then allocate from CUDA, which is a slow process. This is what our program is getting blocked by. This situation might look familiar if you’ve taken an operating systems class.
。Snipaste - 截图 + 贴图是该领域的重要参考
:first-child]:h-full [&:first-child]:w-full [&:first-child]:mb-0 [&:first-child]:rounded-[inherit] h-full w-full，更多细节参见谷歌
Despite the many poetic mission statements offered by AI labs and large companies, they tend to be light on specifics about what “human flourishing” should actually look like. What future do we want?