Provider Comparison – Low Budget Cloud, Part 1

Provider Comparison – Low Budget Cloud, Part 1

Nov 1, 2022 | The Budget Cloud Project

Part 1
Part 2

A new journey

Hi, I am Alex. I am doing my own way of setting up my “business-IT” for my own “company” Firesplash Entertainment. You ask why? As a sole-proprietor who turned his hobby into a small business, availability counts (we are serving a cool overlay game running on twitch) but budget also does. I am known as someone who loves simple solutions that are actually manageable and payable for a normal (as far as IT people can be) human person. So, welcome to my Think-Outside-The-Box HA Cloud project Number two. Let’s start with a cloud provider comparison…

Err.. Wait.. The Hetzner Hybrid Cloud Project..?

Did that introduction sound quite familiar to you? Yep. It’s me. Some time ago I started creating a series on my friend’s blog where I described how we set up a hybrid cluster on Hetzner. It consisted of two hardware dedicated servers and a cloud VM used as quorum and router. We used HCloud Floating IPs and routed them over a GRE connection using OpenVSwitch. Yes, it worked. But it had quite some pitfalls I was never able to solve – for example for some reason the complete GRE-Tunnel dropped when I added a third node. The original plan was to move to cloud networks connected to vSwitches but Hetzner did not implement IPv6 for this feature (funny as they recently announced IPv6-Only servers…).
This simple setup did still cost us about 130€ per month and the nodes were idle most of the time in the end. Further Hetzner recently dropped highly availabe Cloud VMs (CEPH-Based) which are a breaking point for the concept and they raised their prices for IPv4 addresses again (including existing customers).
When I finally noticed that my Kubernetes-Cluster needed a major OS upgrade, that was the point for me to simply toss the existing solution and start elaborating a better solution.

The Idea

I always had an eye on Hetzner’s latest features on their cloud platform. They added Load Balancers, Firewalls, private networking (still only IPv4), … That basic stuff that turns a provider to a real cloud provider, you know?

For me it’s been time to check if I can migrate all my workload from my dedicated servers into the cloud. The original plan was to cancel my two dedicated servers, consolidate some VMs into fewer ones on hetzner’s cloud.

Maths – Calculations for a good Provider Comparison

Of course you always have to look at the expected cost – especially when talking about modern cloud providers, things can become wuite expensive very fast when you got non-cloud-native workloads. So I created a quick table to calculate my expected cost which shall help us on our cloud provider comparison. First of all I defined my new VMs including their sizing. I ended up with something like that:

A table showing the calculations for the provider comparison for Hetzner Cloud

As you can see in this table, I calculated including hetzner’s backup offerings. For further comparison we will take the ~86€ without those backup costs – But we will likely use another backup solution for our final systems. Also the calculation might not be 100% accurate because I think that backing up the additional volume on Collab (GitLab-Server) will cost extra.

Other Options

I also checked a few other options like moving our Kubernetes-Workloads to AWS, GKE, … And also OVH and other server providers.

One of these providers showed a very good pricing. Hello netcup! I will now show you my second calculation, including the same workloads – except the backup feature. Also the Loadbalancer has been removed and is replaced by a floating IP (which would also work on hetzner, saving us about 4€, but more details in the next part)

A table showing the original calculations for the provider comparison for netcup
Server calculation on netcub with current prices including german VAT as of feb. 2022

So at the first look the price difference is minimal. Talking about 7€ difference while Hetzner Cloud does definitely have the better user interface and UX. Also netcup’s prices are only valid for 12 month minimum contract period subscriptions while hetzner allows per hour billing.

Provider comparison results: Why netcup is still the better choice for me

If I would see this cloud provider comparison in it’s current state, honestly… I would say “go along with Hetzner“. But of course there is a difference. The VPS 2000 G9 of netcup has double the ram compared to a CPX31 on Hetzner. Same goes for most of the “machines”. Almost every single VM on this list has more power than on the hetzner table. Also local storage is much bigger which enables us to use Persistant Volumes on our Kubernetes Environment. So in the end you get a lot more bang for your buck there.

When we add our “Think-Outside-The-Box”-Manner we can violate a best practice and converge the control planes into the worker nodes (because those now got more than enough RAM). This is a security consideration but we don’t allow third parties to manage worloads on the cluster so we know what is running on it. It’s a risk I am willing to take.

You might have further noticed that I converged the Nextcloud instance which was planned as a StorageShare on hetzner into the Collab server – This is actually our current setup and is now possible because of the bigger HDD and RAM sizes on netcup.

So all put back into the table we actually end up with a saving of about 19€/month (thats 228€ per year!) while still having more ressources available for our workloads. Isn’t that awesome?

A table showing the modified calculations for the provider comparison for netcup

Completing the concept – Adding backups

Still, we need a backup strategy. We did not honor this for our cloud provider comparison as in the end that cost ist quite identical for all solutions. Right now I have not decided which way we will go here. Hetzner recently changed their storage box model to provide more storage for less. Unfortunately they still do not support NFS… We might want to use Proxmox Backup Server for our backups as we got quite good experience with it and the pbs-client also allows us to backup “foreign” vms. PBS allows us to do a complete partition-dump from inside the vm – Other way round we can quickly restore a full VM out of a live linux or only a few files locally if required. Also it does a good job on deduplication – and I mean not only per VM!

Proxmox Backup Server has minimum server requirements of 2+ cores and 2 GB RAM. We currently need about 450GB backup storage but I expect that to be a bit more as we will not backup images but filesystems from now on which is a bit more inefficient.

  1. All-Hetzner classic solution
    • StorageBox 1TB (BX11)
    • CloudVM CPX11
    • Total cost: 8,20€/month
    • Implication: Backup traffic goes directly from our prod VMs to external IPs
  2. All-netcup
    • S 1000 G7 (1,5 TB, only 1 vCore)
    • Total cost: 15,99€/month
    • Implication: We violate the minimum requirements for PBS
  3. Hybrid
    • Hetzner StorageBox 1TB (BX11)
    • netcup VPS 500 G8
    • Total cost: 8,74€/month
    • Implication: Latency between PBS and Storage can cause trouble

There is another option: Borg Backup

Borg does not allow us to backup (and restore) the full VM but is perfect for backing up individual files or folders containing required data.

At this point it is a decision: Do we want (file system) “snapshots” or data backups? Let’s check the pricing. It is as simple as one Hetzner StorageBox BX11 for 3,45€/month because BorgBackup is natively supported.
We still stick to hetzner here as their storage boxes are the cheapest backup solution and I am a fan of having last-resort backups somewhere else… Even if it’s the same city in the end 😉 – But we could also send our storage box to finland.

The big difference…

…is that PBS allows us to quickly restore the system state of our VM. All we have to do (at least in theory) is to setup a new VM (or use the old one), spin up the recovery system, install PBS-Client and restore the root partition (and probably all other partitions) to the mounted disk.

With borg we will still have to install an OS and all the software and then restore all data folders, config files etc using borg. Basically the same what we do while setting up our system + data restore.

Next steps…

At this point I have (had… This article was written months ago actually) not yet decided which backup strategy to go for. What would you do?

In the next part we will start setting up our VMs on netcup. Want to give it a try? I would appreciate if you’d use one of the referral links contained in this cloud provider comparison to support me and Firesplash Entertainment. Also for exploring netcup, I got a 5€ voucher for new customers for you: 36nc16447952840

Upcoming articles of this series will be published with some delay depending on my finding time to actually write the text…

Migrate Proxmox Backup Server

Migrate Proxmox Backup Server

Mar 6, 2022 | Proxmox Solutions

I was recently confronted with the issue that I needed to move a running Proxmox Backup server to a new host. Along the migration I also needed to update the hostname to match the new concept. However, it does not really matter if you keep the hostname or not as fortunately PBS does not really care about it’s name.

Step 1: Research

As a senior admin I know you can usually find out anything you don’t know yet using google (ar any other search enginge out there). This time however I had a hard time. Looks like virtually nobody tried to migrate a PBS yet – or maybe the fact tht it is so easy leads to noboy writing the process down.

Okay, let’s do it on our own!

The Facts

Currently PBS is running an an “old” Proxmox VE host – and I mean on the physical host. We are migrating this PBS installation to a dedicated host, a VM on Hetzner’s cloud this time. As the PBS is only occasionally taking load and the backup performance does not really matter for our project, I chose a CX21 VM with 2 vCores and 2GB of RAM. The 40 GB HDD space are enoght for the OS and PBS’s own files. Additionally a Hetzner Storage Box is available to be used as the actual datastore.

In this case the chunks (the actual backed up files) have already been stored to the Storage Box. This means in this blog post I will not migrate the actual data but if you need to, it is as simple as rsyncing the data (rsync -azh source target) from one place to the other, just take care that the mount path is the same on the new server and stop the services on both servers before migrating!

Step 2: Preparation

The Cloud VM

First of all, I booked a CX21 cloud VM from Hetzner located in the same datacenter as the storage box resides. This is very important! Using a VM in FSN1 and storage box in HEL1 has very bad performance. I mean performance like when you’re driving right behind a farm tractor on a highway.

I chose Debian 11 (and you should do so, too). Right after the setup completed, I logged in using SSH and edited /etc/hostname to include or FQDN hostname – don’t forget to add a DNS entry!

As mentioned, our chunk data is stored on a Hetzner Storage box. I copied the respective line from fstab from the old server to the new one and created the mount path using mkdir /mnt/hetzner-sb1 in my case. For reference, this is what it looks like now (username/password redacted):

//u1234567-sub1.your-storagebox.de/u1234567-sub1 /mnt/hetzner-sb1 cifs username=u1234567-sub1,password=GeneratedPaSsWoRdFromRobot,uid=34,noforceuid,gid=34,noforcegid 0 0

Then I mounted the box using mount /mnt/hetzner-sb1 and verieifed by running an ls /mnt/hetzner-sb1

Install Proxmox Backup Server software

We need to do a clean install as docuemnted here. The commands are simple and reliable:

Trust the repository key: wget https://enterprise.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg

Add the repo to apt sources: echo "deb http://download.proxmox.com/debian/pbs bullseye pbs-no-subscription">/etc/apt/sources.list.d/pbs.list

Download package lists: apt update

Install the software: apt-get install proxmox-backup

You could also install proxmox-backu-server instead of proxmox-backup. The proxmox-backup package contains all recommended extra packages, the other onwe is a minimal setup.

Disable 2FA temporarily

If you can, you should remove 2FA configuration from the root user before migrating or create a new administrative user without 2FA if WebAuthN is used. Else you might not be able to log in anymore.

If you are changing the hostname along the migration like I do you are required to deconfigure WebAuthN and reconfigure it after migration as the hostname is the "key" for WebnAuthN accounts on the token.

Stopping services on old and new server

It is important that no backups or other jobs are running while you are migrating. Run these commands on both servers:

service proxmox-backup stop

service proxmox-backup-proxy stop

Make sure that no job or task is running before stopping the services.

Step 3: Migrate config and metadata

Proxmox backup server stores its configuration in /etc/proxmox-backup and metadata (rrd stats etc) in /var/lib/proxmox-backup – We need to sync both folders to the new host (assuming the new host has IP 10.10.20.20) using rsync on the old host:

rsync -avzh /etc/proxmox-backup/* root@10.10.20.20:/etc/proxmox-backup/

rsync -avzh /var/lib/proxmox-backup/* root@10.10.20.20:/var/lib/proxmox-backup/

Step 4: Reboot the new server

To make sure all configs are applied and all services are running smoothly, we will now reboot the new server and verify our installation and migration.

After the reboot, visit https://10.10.20.20:8007 (use the IP-Address for now)

You whould see your datastore on the dashboard now – including historical usage data graph. If so, select the datastore on the left hand menu and select the content ribbon. You should see all backups there. This might look a bit like that:

Great! The worst part is over. If you preserved the hostname, you’re done and can proceed to step 7.

Step 5: Configure SSL

This step does not apply if you keep the hostname while migrating.

Now you will need to setup Let’s encrypt again (or use your preferred method). You can do this at Configuration -> Certificates. Simply click the add button and enter yor FQDN there:

You may also want to remove the migrated old certificate. Then click the “Order certificates now” button. If everything runs smoothly, you should now have a valid SSL-certificate:

Verify this by pointing your browser to https://your-hostname.your-domain.tld:8007/

If the certificate is no used immediately, restart the proxmox-backup-proxy service

Step 6: Configuring the clients

This step does not apply if you keep the hostname while migrating.

Now you have to direct your clients to use the new server. For manual backups using proxmox-backup-client you need to edit the cronjobs or backups scripts. For Proxmox VE you can simply edit /etc/pve/storage.cfg (for example using nano) and change the “server” directive of the storage block:

Proxmox VE will reload the changed file automatically after a few seconds.

Step 7: Testing

You should try to backup a client to see that everything works. In my case I ran a manual backup from my Proxmox VE server:

While the backup is running, you will also see that in the "Tasks" bade on the top right corner of your PBS server.

Done!

You should monitor your upcoming backups and verfiy them using PBS’ builtin verifications to make sure all chunks are accessible, especially if you also migrated the chunk storage.

Also don’t forget to re-enable 2FA if you disable dit before….

…and if you changed the hostname, you have to reconfigure WebAuthN under Configuration -> Other (and the other is located at the top if you click on Configuration – The did a good job in hiding that!)

I hope you were able to follow my steps to successfully migrate your PBS installation in a rush.
Tell me, did you encounter any issues?