How to disable completion icons in Tramp Mode to improve performance

What happened?

I’m currently using (vertico +icons) in my init.el, which causes poor performance when using tramp.

Is it possible to disable vertico icons (marginalia-mode) only when using tramp? So far I have this, which works for disabling but doesn’t re-enable for local files:

(after! tramp (marginalia-mode -1))

I’d also like to do this with dired (dired +icons), which I think has the same issue.