That’s mostly related to the graphics acceleration. I’ve tested the Pinephone with Phosh rather early and in some later stages. It improved drastically when Phosh implemented hardware acceleration. For example pulling down the quick settings panel from the top was a night and day difference. So I’m not surprised that Android feels that responsive already.
The most issues we have now using the GNOME software stack is actually on application level. Many applications still use GTK3 which leads to worse performance overall than with GTK4. Additionally many applications are developed for a desktop PC with pretty solid single-thread performance. However on mobile you end up with multiple low powered cores. So it’s much better if intensive tasks are handled asynchronous to make the GUI responsive during load. Users will feel the difference.
Last piece is that we need to get maximum of performance out of the hardware. So for this you actually want improvements on the GPU driver, support for modern APIs (for example to utilize the new unified GPU renderer of GTK) and improvements on scheduling.
Obviously there would be less issues when the hardware was simply more powerful. There are things a Librem5 can’t do because of lacking performance. But I think for most tasks, it’s totally fine spec-wise. It just needs more optimization.