I'm thinking about renting a server to play with friends, but now the question arises, how much GB Ram do I need? I want to play with rather big mods (Tekkit, Hexxit). How much GB Ram do I need at most 4 players that everything runs smoothly?
With 8 GB you are on the safe side.
6-8 should be enough for most modpacks with 4 players.
Take a look at the System Requirements of the respective Mod Pack.