So about a year ago a few colleagues and friends of mine suggested I start a blog for some of my “home projects”. This is my first attempt to work through what that might look like. I have frequently used Google searches and other people’s blogs to discern valuable information on whatever my current interests are. Something I’ve often thought was missing, however, was a description of the problems solved.
Oftentimes, the reason that websites like Stackoverflow are so successful is because people can state the exact behavior they are seeing, how it’s not what they are looking for, and get an answer from experts. This helps the second person compare what they see with the first and decide if it’s relevant. The same is true for the forum, both 20 years ago, and now. I want my blog focus more on the problems I’ve encountered, including the mistakes I make along the way. Will this be as entertaining to read as it was to work through?
I have been building computers since I was a teenager, where the first CPU I purchased being an AMD K-7! There are numerous older engineers, who no doubt have earlier stories than that about their first processor, but I remember being struck by how the CPU was not integrated or anything. It was a slot!
It looked like a Super Nintendo cartridge! Very odd. So did my Super Nintendo games contain processors too? I remember breaking one apart, to my parents dismay, to see what it looked like inside. I don’t have any photos from my first adventure into this. But you see here the same things I did. Nothing like what I was expecting, and what are those bulbs? (resistors, I know today), the larger cylinders? (capacitors), the chips? (SRAM, ROM, others), a battery?
I have always enjoyed the physical component of computers as much as the more virtual or even intellectual components. The entire methodology behind how multiple 4 gigabyte memory spaces are created from a linear physical memory space is fascinating. It contains essentially the entire history of x86 design along with it, warts and all. The recent memory space chip violations are essentially because of this history needing to be carried with the chips. Because of this enjoyment my home projects have almost always included a physical component. These physical components almost always start within a home lab or potentially getting built into their own servers.
My first true “home lab” was probably built in 2014. I had joined a High Frequency Trading company and needed to learn how 10G fiber optics worked. Having much more solid familiarity with processor design and high speed programming than physical networking, I wanted to learn more. So I did my research and purchased a MikroTik – CRS226-24G-2S+RM [3]. It had 2 10G SFP+ ports. I combined it with a Dell x4012 [4] a year later to expand the 10G network. This was when I put together my first 45U rack, when I added extra networking switches and a few actual servers I built at the time. Before that I tended to keep most of my servers lying around in towers (like the LIAN LI PC-Z70B [5], I’m a sucker for big black boxes as cases). I had a FedoraCore NAS I had built myself from older hard drive hot swap drive bay cradles running mdadm for software raid. These stories are on the list of potentially being told at some point as I kept documentation for myself. Back to the lab, by 2016, my lab had expanded with a few projects I had completed to challenge myself into a completely full rack!
I can tell the cable management folks among you are screaming with horror. I mean it’s not that bad, but still. I’m sorry to say I never really got a whole lot better than this. So some of you will simply have to bury your horror in the depths of your soul.
There were some additions and subtractions along the way, but since I didn’t think to document all of the changes with photos, I simply don’t have them. For instance I had a 1U crash console at one point, and then got rid of it when I realized most of my projects weren’t going to easily translate to the old PS2 keyboard and mouse.
There are several projects worth of servers, networking, planning, and more going on in this photo. Those are some of the stories I’ve been suggested to write. I will leave those for later, but feel free to ask if you are interested. In the end though, I ended up learning ESXI 5.5, 6.0, numerous versions of FedoraCore, Ubuntu, debian, freeBSD, solaris, windows, and more. These are just the operating systems I’ve touched! It was a truly worthwhile growing experience, even if the operating system isn’t as big as it used to be.
At the beginning of this year, I was doing research on the new Ryzen 3 processors from AMD. I was getting excited. Since basically my K-7 days, AMD has not been truly competitive with Intel on performance. They’ve eked out a business by being a good price per performance, especially with their multiprocessor systems at scale. Simultaneous Multi-Threading (SMT) seemed to elude them for a long time. Not to be confused with Speculative Multi-Threading (SpMT), a much more awesome idea only ever really attempted in the Sun Niagara architecture. To be fair, SMT is a difficult thing to get right, especially with security considerations. A friend of mine at Intel once told me (no idea if true) that they built the triple-fault processor reset because if the CPU screwed this up 3 times, it was better to give up the whole processor state and start over.
The Ryzen 1 hype was, perhaps, a bit more than it should have been, but it did represent AMD actually getting into the right league for tackling Intel. For too long, Intel had basically only competed with last year’s Intel processors, and the Ryzen 3 represents an actual competition with them.
I was considering building a new Ryzen based workstation since it had been about 4 years since my last upgrade, and decided maybe I should look into EPYC Rome, the server version of Ryzen. (I didn’t know the difference between Naples and Rome at the time, so I wasn’t completely up to date.) The 7232P seemed interesting. An 8-core 16-thread CPU I could build a new server out of. I toyed with the idea of the 16 Core, 32 Core, even 48 Core, but kept thinking, what exactly am I going to do with all those processors? I was being caught up in buying tech just because of excitement, with no real plan.
However, a thought occurred to me, what if I could collapse my current 5 servers down to just 1? Well 2, my workstation is almost certainly not going to be wrapped back up into one big server, but maybe someday. I also remember last time I built my ESXi virtualization server I wasn’t able to get a dedicated GPU working for the virtual machine to replace servers. This would be necessary as one of these servers is used to run all the televisions in my house. It serves as my primary way of consuming media. After a bit of research, and some wishful thinking, I determined this might not be as big a deal as it used to be. To be more accurate, I figured that after 6 years this had to have been solved.
I then discovered Linus Tech Tips (yeah yeah, I know this has been around forever, and I really enjoyed scrapyard wars, especially the one about building a water cooling system, block and all, which I actually did in my freshman year of college) and saw THIS Replaced Every PC in my House! [6] I was instantly reminded about my previous failure and thought, you know, if I add my VR PC, my Home Streaming PC, my home assistant raspberry pi, and my storage server all to one big one, I could even add an extra workstation for my partner who has been discussing starting a podcast/blog herself. Now this was a reason for upgrading to a large CPU. I’ll save money on my monthly power bill too, by collapsing these down together.
So I placed an ebay bid on a 7702P 64-core, 128-thread EPYC processor. Now the more astute of you are probably noticing this is not one of the previous considerations, but to be fair, I didn’t expect to win. This was around the time covid hit (February), and I was more playing around than actually expecting anything. I was just plain surprised I won considering I wasn’t willing to pay that much. This was probably my first mistake. Don’t bid on something you aren’t prepared to win. That being said, this was going to be fun. Let’s see how complicated I can make this.
I had intended to learn and work my way through a few deployments more of interest for work, like Kubernetes, Kafka, Redis, and a couple more smaller systems I have had deployed, but have varying degrees of hands-on experience. I’ve been an Engineering Leader by trade for nearly 9 years now, but I like to actually program, actually deploy, actually work with these tools to understand them. Especially so I’m not just always going off my colleagues’ opinions. Having a large server like this would give me an excuse to learn this and work through new problems. AWS has taken the infrastructure world by storm, but I think there is a lot of value in knowing these systems more than just through the AWS console. So, without further adieu, let’s get to upgrading my old “Home Lab”!
References
[1] https://www.youtube.com/watch?v=h4etRkXPahc
[2] https://www.youtube.com/watch?v=NLEMsw1SjDY
[3] https://mikrotik.com/product/CRS226-24G-2SplusRM
[4] https://www.dell.com/en-us/work/shop/cty/pdp/spd/networking-x-series
[5] https://www.newegg.com/p/N82E16811112327?Item=N82E16811112327