Friends have timeouts on their own Minecraft server?

I myself cost a Minecraft server for friends from home, initially, so only 4-5 people played on it, everything went normally, meanwhile 8-9 people play on it and many time out regularly (timeout). The server runs otherwise super smoothly, only the timeouts disturb. I also increased the server's vram to 4GB, which didn't help. I have 40mbit upload and only 10 of them are occupied, even if 9 people are playing on it, so think that it is not because of it. Does anyone know what I can do best?

I think it's because of your internet line. The internet connection is so burdened by the 9 players that they lose the connection.