2023-12-25, 04:40 AM
(This post was last modified: 2023-12-25, 04:42 AM by jess. Edited 1 time in total.)
Ooh I'm currently adding some new bits to my homelab, so I'll post! Its an absolute mess right now, but it's my mess ❤️
![[Image: PXL-20231217-040608354.jpg]](https://i.ibb.co/ys6hJ1V/PXL-20231217-040608354.jpg)
From the top down:
- little legacy 8 port switch that I still use for POE, and yet is still somehow still the backbone of my network -.-
- Main 24 port 1g switch. It has two 10gb ports meant for uplinks but I'm using them to direct connect my main ceph nodes right now
- patch panel
- new (to me) 48 port 10g switch! Planning to bond 2 ports per virtualization/ceph node, and reorganize my network to use this switch as the backbone! Excited to get this project underway
- Hidden away is a little nuc running proxmox + ceph which acts as a tie breaker for my size=2, min_size=2 ceph cluster
- a supermicro box I'm colocating for work, office didn't have good enough Internet, and the server room couldn't dampen it's sound enough to not bug people. So I get to host it for now
- hp dl380 with 256gb ram and 16 cores x 2 sockets; this is my main virtualization box, along with the nuc (provides redundant DNS) and a tower out of frame (the other ceph node, zwave radio, and GPU for transcoding).
- old 4u I don't use anymore, want to move the tower I'm using into this case
- New (to me) server I'll use for virtualization and highspeed ceph pool. Thing is a beast, 128 cores & 700gb ram
Typing it out makes me realize it's even more a mess than I realized haha
Running inside are my core network services (like DNS, VPN, IPXE, and Borg). But most everything else runs in a kubernetes cluster with nodes spread across the hypervisors. Things like home assistant (zwave controller) and Jellyfin (GPU) have their hardware added to the vm, and then passed into the container. I use kata microvms for services exposed to the public Internet, along with strict network policies for all containers. I make use of metallb + bgp peering for IPs, ceph for storage (and cephfs makes for great ISO storage), and manage all the config with FluxCD, Terraform, and Ansible.
So yeah! It's been a wild ride. My goals next are:
- Reconfigure everything I can to go through bonded 10g DACs
- Move to a proper PDU to manage fencing in proxmox and track power usage
- Automate most disk encryption with a separate physical device running Clevis (and using tang)
- Better backup story. Currently the important stuff gets borged to a set of mirrored 18tb disks, but I love the idea of having a box I can grab and run with.
So lots to do in the new year! Love this thread, hopefully others can share their setups to give me more ideas ^^
From the top down:
- little legacy 8 port switch that I still use for POE, and yet is still somehow still the backbone of my network -.-
- Main 24 port 1g switch. It has two 10gb ports meant for uplinks but I'm using them to direct connect my main ceph nodes right now
- patch panel
- new (to me) 48 port 10g switch! Planning to bond 2 ports per virtualization/ceph node, and reorganize my network to use this switch as the backbone! Excited to get this project underway
- Hidden away is a little nuc running proxmox + ceph which acts as a tie breaker for my size=2, min_size=2 ceph cluster
- a supermicro box I'm colocating for work, office didn't have good enough Internet, and the server room couldn't dampen it's sound enough to not bug people. So I get to host it for now
- hp dl380 with 256gb ram and 16 cores x 2 sockets; this is my main virtualization box, along with the nuc (provides redundant DNS) and a tower out of frame (the other ceph node, zwave radio, and GPU for transcoding).
- old 4u I don't use anymore, want to move the tower I'm using into this case
- New (to me) server I'll use for virtualization and highspeed ceph pool. Thing is a beast, 128 cores & 700gb ram
Typing it out makes me realize it's even more a mess than I realized haha
Running inside are my core network services (like DNS, VPN, IPXE, and Borg). But most everything else runs in a kubernetes cluster with nodes spread across the hypervisors. Things like home assistant (zwave controller) and Jellyfin (GPU) have their hardware added to the vm, and then passed into the container. I use kata microvms for services exposed to the public Internet, along with strict network policies for all containers. I make use of metallb + bgp peering for IPs, ceph for storage (and cephfs makes for great ISO storage), and manage all the config with FluxCD, Terraform, and Ansible.
So yeah! It's been a wild ride. My goals next are:
- Reconfigure everything I can to go through bonded 10g DACs
- Move to a proper PDU to manage fencing in proxmox and track power usage
- Automate most disk encryption with a separate physical device running Clevis (and using tang)
- Better backup story. Currently the important stuff gets borged to a set of mirrored 18tb disks, but I love the idea of having a box I can grab and run with.
So lots to do in the new year! Love this thread, hopefully others can share their setups to give me more ideas ^^