In IMAP parlance this is “subscribe” or, more accurately, “unsubscribe”. Based on internet discussion, it seems that Geary does not support that (yet).
It is vaguely possible that you could control this on the server side but only if Geary is the sole client for this mail account and only if you have the necessary access on the server side to make configuration changes and only if your IMAP server specifically caters for that in per-account configuration.
(I would like this too. The work IMAP server that I am connecting to has a whole lot of irrelevant folders. On my desktop, with Thunderbird, I can unsubscribe, and cut the noise level on the screen.)
Internet discussion seems inconclusive on this. Some very old reports of the same problem but other reports implying that it works as advertised. (Version numbering is fairly opaque here so I can’t tell how current the version of Geary that is included on the Librem 5 is.)
Did you change this via the GUI (Hamburger / Accounts / the specific account / Download mail)? Or hack it in the config file?
Any change to this parameter may not be retrospective. That is, if an old email is already downloaded, it probably won’t get removed from the local cache.
It might require a log file to get to the bottom of this and I couldn’t see any way to enable logging on the client side!
One troubleshooting approach might be to install and configure Geary on a big computer, using the same IMAP account, and see whether it exhibits the same problem.