This annoyed me too, so i tried this simple thing and it worked.
I changed from Maximum to High settings, and GPU fans suddenly went down to normal.
Now i can barely hear the GPU fans in port.
Port: Designer's Table.
Tried "Fjords" Port for fun, and then the fan speed increased a bit.
RTX 3070
Ryzen 9 5900X
32.GB RAM