The Home Lab

I recently was asked by a co-worker of mine why do I continue to invest time and money into my home lab. Don’t you have better things to do after work then your home lab? Do you really spend your free time learning a new product, when you could just wait to do at work? Did you really go home to try to replicate a issue at home so you can learn how to fix it? Did you make this “How to” documentation from your home lab for work? My answer is always YES. To be completely honest I don’t feel this is abnormal for IT Professionals. My home lab allows me to support my career, provide family services, and is a educational tool for furthering and developing my professional career.

The purpose of this post is to hopefully inspire other IT professionals, Whether you’re new or seasoned.

Overview of the Home Lab

Home Lab

  • Intel NUC 8i7HVK (Hades Canyon) (x4)
    • ESXi 6.7 Update 3
    • 64GB DDR4
    • Corsair MP500 120GB NVMe
    • Samsung 970 PRO 512GB NVMe
    • Corsair 32GB USB 3.0
    • OWC Thunderbolt 3 10Gbe
  • VDI vGPU white box
    • ESXi 6.7 Update 3
    • AMD Ryzen 3900x
    • Asrock X470D4U2-2T
    • 64GB DDR4
    • Corsair MP500 120GB NVMe
    • Samsung 970 PRO 512GB NVMe
    • Tesla P4 / M60
  • Storage
    • Synology DS1517+
    • vSAN
  • Networking
    • Ubquiti UDM Pro
    • Ubquiti 16 XG
    • Ubquiti USW-Pro 24
    • Ubquiti NanoHD
  • Power
    • CyberPower UPS 1500VA 900W Rackmount
    • CyberPower 12 Outlet PDU Power/Surge Rackmount
    • CyberPower 8 Outlet Switched PDU Rackmount
    • CyberPower Environmental Sensor
    • CyberPower UPS and PDU Management Card
  • Cameras
    • Ubquiti G3 Pro (x2)
    • Ubquiti G4 Pro
    • Ubquiti G4 Bullet-Beta

DR Site

  • Intel NUC i7DNKE
    • ESXi 6.7 Update 3
    • 64GB DDR4
    • Samsung 970 PRO 512GB NVMe
    • Corsair 32GB USB 3.0
    • USB 3.0 1Gbe Adapter
  • Storage
    • DS1515+
  • Networking
    • Ubquiti USG
    • Cloud Key Gen 2
    • Ubquiti Unifi Switch 8
    • Ubquiti NanoHD
  • Power
    • CyberPower UPS 1500VA 900W Mini-Tower

Compute

My home lab consists of 4 Intel NUC 8i7HVK (Hades Canyon) and Ryzen White Box. I went with Intel NUCs for its balance between performance and power consumption. The Ryzen White Box was designed for running VDI workloads with a Telsa P4 and M60.

I currently have one Intel NUC i7DNKE at the DR site (Parents House) used for vCenter, Site Recovery Manager, F5 BIG-IP LTM VE, Unified Gateway Access, and Cloud Pod configuration. I might be expanding to another NUC here soon.

Storage

NAS

I use Synology DS1517+ with five Seagate IronWolf Pro 4TB drives in RAID-10 with Hot spare. With M.2 SSD cache configured as RAID-1. DS1517+ NAS is used to provision LUNs for iSCSI based VMFS datastores, snapshot replication, backups to DR site, and file server.

At the DR site, I use DS1515+ four Seagate Ironwolf 3TB drives in RAID-6. with 2.5 SATA SSD used for SSD Cache. The DS1515+ NAS is used to provision LUNs for iSCSI based VMFS datastores, backups, and snapshot replication.

vSAN

When William Lam released a blog about “New Thunderbolt 3 to 10GbE options for ESXi” I immediately went on amazon purchased the following:

  • Ubquiti 16 XG Switch
  • OWC Thunderbolt 3 10Gbe (x4)
  • Cache Tier – Corsair MP500 120GB NVMe (x4)
  • Capacity Tier – Samsung 970 PRO 512GB NVMe (x4)

I might end up doing another post in the near future going over setup and configuration of vSAN with the Intel NUC Hades Canyon. My current workload running on vSAN is currently VDI workloads like Desktop Pools, Application Pools, AppStacks, and Writable Volumes.

Keep in mind the SSDs I am using for home lab vSAN are not enterprise/business grade SSDs. So they don’t have same type of endurance you would find in production deployments of vSAN.

Network

My network configuration for my home lab and DR site is configured with the following:

Home Lab

  • UDM Pro
    • Firewall
    • IDS/IPS
    • DPI (Deep Packet Inspection)
    • Routing
    • Site to site VPN
    • Unifi Network Controller
    • Unifi Protect
  • 16 XG Switch
    • vSAN / vMotion Traffic
    • Aggregation
  • USW Pro 24 Switch
    • ESXi Hosts Mgmt, VM, and iSCSI Traffic
    • Synology NAS
    • PoE Cameras
  • NanoHD Access Point

DR Site

  • USG
    • Firewall
    • DPI (Deep Packet Inspection)
    • Routing
    • Site to site VPN
  • Unifi Switch 8
    • ESXi Host Mgmt, VM, and iSCSI Traffic
    • Synology NAS
  • Cloud Key Gen 2
    • Unifi Controller
  • NanoHD Access Point

Home Lab Services

Since I mainly run ESXi as my preferred hypervisor, most of my services are virtual machines. Here are a list of following services I provide in my home lab and at the DR site.

Home Lab

  • vCenter [vCSA] (Enhanced Linked Mode)
  • vSAN
  • vROPs
  • Log Insight
  • Site Recovery Manager
  • vSphere Replication Appliance
  • Domain Controller
  • DNS
  • DHCP
  • Certificate Authority
  • Windows Admin Center
  • SQL
  • Exchange
  • Microsoft Endpoint Configuration Manager (aka SCCM)
    • Task Sequence
    • Application Deployment
    • Patching
  • Horizon View (Cloud Pod)
    • Connection Server
    • Desktop Pools
    • Application Pools
    • vGPU Workloads
  • App Volumes
  • Dynamic Environment Manager
  • Unified Access Gateways
  • F5 BIG-IP LTM Virtual Edition
  • Veeam Backup Community sedition
  • PyKMIPServer
    • Act as KMS server in order to test vSphere/vSAN Encryption.

DR Site

  • vCenter [vCSA] (Enhanced Linked Mode)
  • Site Recovery Manager
  • vSphere Replication Appliance
  • Domain Controller
  • DNS
  • DHCP
  • Horizon View (Cloud Pod)
    • Connection Server
    • Desktop Pools
    • Application Pools
  • App Volumes
  • Dynamic Environment Manager
  • Unified Access Gateways
  • F5 BIG-IP LTM Virtual Edition
  • Veeam Backup Community Editon

VMUG

I just wanted to give a big shoutout to VMware User Group. Without VMUG, most of my home lab would not be possible. One of the huge benefits of being a VMUG subscriber is the 365 days license for almost every VMware product. Without VMUG, my home lab nor my professional career would be where it is today.

Future Plans

My plans for this year are to hopefully upgrade my NAS to Seagate Ironwolf 110 SSDs and 10Gbe PCI-E adapter card for the DS1517+. My Intel Hades Canyon NUCs have one more thunderbolt 3 port not being used. I might grab another OWC Thunderbolt 3 to 10Gbe for each one. Setup some LACP with vSAN and do some benchmarking. After seeing Intel announced the Intel NUC Ghost/Quartz Canyon, I am going to try to snag one and install Tesla P4 GPU in it.

As for learning new skills, I might to try to get more involved in having NSX more integrated into my home lab. vRealize Automation (vRA) is something I have always been interested in and never got around to learning it. This year might be the year I change all that. Last but not least I would like to strengthen my skills in the container arena.

Gaming Rig

Not really part of my home lab. However, I get asked about it a lot when company comes over. I find Watercooling to be very therapeutic and relaxing.

Specs

  • i9 9900K with EK WaterBlock
  • Asus z370 Maximus X Formula
  • NVIDIA RTX 2080ti with EK WaterBlock
  • Asus ROG Swift PG27UQ 144hz
  • Phantek Evolv
  • 32GB DDR4 3600Mhz
  • Samsung 970 Pro 1TB (x2)
  • ASUS Thor 850 Watt PSU
  • EK-Coolstream 360mm Radiator
  • EK-Coolstream 280mm Radiator
  • EK Vardar Fans (x5)
  • EK Reservoir/Pump Combo

9 thoughts on “The Home Lab

  1. Awesome article, and awesome setup. I have a very similar setup as you, I’ve got 4 Hades Canyon’s with 64GB running 6.7 U3 with the Atlantis driver, and I’m using the QNAP QNA-T310G1S Thunderbolt 3 to 10Gb SFP+ adapters. I am just using 10GB for VMotion right now, and then for VSAN once I get the disks. I’m having an issue I am wondering if you have seen in your lab. I’ve found the TB3 cable connections on the NUC side, are super sensitive to pretty much any movement. With my Nuc’s and the QNAP, that causes them to lose connection to my 10GB environment if I happen to be doing anything that might touch that TB3 cable. Curious if you’ve seen that at all. On mine, the connectivity lights on my switch, and the QNAP 10GB adapter, will lose link light if you wiggle that cable at all, but only on the NUC connections.

    1. Hey Andy,

      Unfortunately I have not ran across this issue using the OWC 10Gbe to Thunderbolt 3. I don’t have QNAP QNA-T310G1S on hand. If I did maybe I could do some testing with you. Do you have TB3 QNAP for every Hades Canyon you own? Does this happen on every Hades Canyon?

      Have you updated the Thunderbolt 3 firmware on the Hades Canyon?
      https://downloadcenter.intel.com/download/28525/Thunderbolt-3-Controller-Firmware-Update-Tool-for-Intel-NUC-Kit-NUC8i7HNK-NUC8i7HVK?product=126143

      Are you on the latest version the Atlantic VIB ?
      https://github.com/Aquantia/AQtion-esxi/releases

      Let me know what happens, I am actually curious!

      Thank you

      1. This was happening on each NUC unfortunately. So I am on the latest Atlantic driver, and thought I was on latest TB3 Firmware, but I was not, I was on v28 while v33 was the latest from your link. I updated all 4 to the latest, which in itself was a time consuming process, as I did not have Windows installed on any of the NUC’s. I ended up installing Windows to 1 of the NUC’s onto an external SSD, then used that SSD to boot on the other 3 to do the updates. Since they were all identical HW it was not an issue. So since the TB3 FW update, just wiggling a cable does not cause me to lose connectivity. This was on the OEM TB cables that shipped with my QNAP adapters, which felt kind of flimsy/cheap anyway. I had also ordered some new CalDigit TB cables that were a little longer to better fit my physical space, I’ve used that brand in the past, no issues, very high quality cable. With those cables I have no issues either. Thanks for pointing me to the FW update, that seems to be the trick! Also bought NVME drives for VSAN, so I have that setup now too.

  2. Awesome work. Really impressive. How did you go about getting the licenses? Are you a vExpert, did you buy the VMUG Advantage or another option?

    Thx
    Sincerely
    Max

  3. I noticed you have Ethernet gbics in your 16xg can you comment which ones your using and part number. Great work it’s an inspiration I hope to have mine up soon

  4. I seen you layout diagram, do you have a layout how you setup the IP’s and vlans, trying to get a good idea on how I did mine. If you don’t want to post it, you can message me direct at email, not asking for your actual IP schema , just similar to get an idea. I am using the Supermicro E200-8D;’s and trying to figure out the best way to do the IP for 10gb for then IP for mgmnt, and vSan setup. about week away from going full tilt, hopefully. How do you like that UDMPro? WIsh they would release the new UXG by now, not sure if I should grab one of those, or build home unit. Like sophos or pfsense. I would like to learn both, but want to focus on getting ESX environment online first.

Leave a Reply