Doh! Well, what about incorporating support for something like the Turbo.264HD from ElGato? It's a hardware co-processor to accelerate some h264 functions. Moving the h264 (and I suppose in the near future the h265?) stuff into hardware seems like the way to go.
Some time ago when I contacted ElGato about Linux support they seemed very open to sharing APIs and expanding the use of their devices.
There might be other similar devices as well, or FPGA/ASIC implementations that could be rolled into a device.