I wanted to call the signing function homo (short for homomorphic encryption), but went for fhe because of reasons.
The only but I'm a bit unsure of so will probably pass for version 1 (and thus break the entire premise, but at least I know how to solve it for v2) is Verifiable FHE. Since we're not doing the inference in the FHE module (since the universe isn't long enough), we need to verify that the inference engine didn't lie to us.
Basically we lean on Freivalds algorithm to verify the matrix multiplication, and we probably use merkle trees to pass the outcome of the inference into the FHE, and it can randomly sample, do ultra slow processing in the FHE to check that the weights are what it expects.
This is proper maths that I won't manage on a sunny wednesday, but if I get everything else to work I can maths it later.