Little FPS in Minecraft when looking in one direction?

Hi. I've been playing with a buddy on a server for a few days. We're building something like a city. My problem is that when I look in one direction (looking at a certain building) my FPS drops noticeably. Even on a new single player world, I have a maximum of 110 FPS. I have V-Sync off and graphics mode on quickly. Strangely, it's all gone as soon as I take out my shader, but actually the PC should do it that way.

My system: Ryzen 5 3600
Geforce RTX 2070 SUPER
16 GB RAM

I have already optimized my in-game settings (visibility 10 chunks)

We play with a few mods on the server:
Enhanced Visuals Mod
Mr. Crayfishs Furniture Mod
Elevator mod
Custom NPC's Mod

In one building there are many objects of the Furniture mod but I think the PC should still be able to do it.
I would be very happy about an answer!

Could it be that the server is simply too weak?