It’s only ever powered on for me at all – running ubuntu or buildroot – with the 12v adapter plugged in.
I don’t think another userland would make a difference. I’m using the kernel from the Globalscale buildroot image (which defaults to the performance governor), and the buildroot image doesn’t do anything to change that (so the Globalscale buildroot image would have the same behavior). The panic on trying to change governors is definitely a kernel issue, not a userland one.