Progress on provable inference for large language models

Release

Cryptographic proofs verify that a response comes from the announced model without revealing its weights.
According to Anthropic, these proofs can help detect the substitution of a model on a compromised server.
In 4-bit, the proofs of Llama 4 Maverick take about 70 to 95 times the GPU time of inference with vLLM.
The code, the Lean proofs and the documentation are released under the Apache 2.0 license.