Use the Modpacks tab on your server to install a modpack from Modrinth in one click. It sets up the right loader and version for you.
Big modpacks need more RAM: 6–8 GB is a good start for most. If the server runs out of memory, use Upgrade on the Orders page; the new size applies after a restart.
Installing a modpack replaces the server's files, so take a backup first if you want to keep your current world.