As you already know. I have created an application called _AugmentedIntelligence! Which is entirely a big data Emergent Build that is mean to calm people down about privacy and computers and technology in general.

Current Small Scale Experiment

At the time of this writing, we have a total of four servers and one NAS. The two enterprise systems were about $500 a pop. Both 1U servers are both older multi-socket Xeons with 24 threads pre 1U. SUN is the networking server with 64GB of ECC DDR3 memory and JUPITER with 72GB of ECC DDR3 memory. SUN handles Active Directory Directory Services, DHCP, DNS, IIS (this site and the forum), and Hyper-V for a single Ubuntu Dev test system (mostly so I can have an SSH and RDP connection wherever I am on the planet). Jupiter serves as a backup for DNS server and Active Directory Domain Services. And for a time served as a decent Minecraft Server. Recently, I posted about a Linux server that is dedicated LLM (Large Language Model) and Whisper Transcription services. Because Jupiter server is simply an old 1U without room for a GPU, both services are very slow; talking like 30 minutes to generate with Llama 3 instruct bad. My NVIDIA ASUS 1080TI produces the same text in a matter of milliseconds. With the app now in testing, we need a bigger boat! And with the family hinting that this would be something they would want to be a part of means we need an even larger boat!

The Plan

My workstation has 36 total threads, while generating a summary with Command-R-Plus, CPU usage is set at 50-55% and GPU usage at 45-50%, and RAM usage at 100% (64GB). Command-R-Plus is a 104 billion parameter model and can write at about the same pace as me writing this post. That would be at about the benchmark I would like to see for each person connected.

So we are looking at Xeon W9 CPUS with 72 threads a piece and 36 total users. 18 threads per user, making 8 total Xeon systems, 64 GB of RAM per user and 265GB of DDR5 ECC memory in each the systems per VPS. So Install Hyper-V Server Hypervisor as the OS and configure four VMs per Xeon. Maybe we would switch to Xeon Scalable which are Dual Socket solutions rather than workstation systems of the Xeon Ws. And of course NVIDIA PNY GPUs and four of those per system making a single GPU pre VPS.

Total would be: $67,014.83

---

Filed under: Uncategorized - @ May 1, 2024 7:38 pm