ESXi 5 on an Apple Mac Mini
Monday, May 21, 2012 at 3:33PM | Christopher Wells
I have been looking to refresh my home lab environment for some time now. Since I moved into a rental townhouse upon my move to NC, I have had power issues when my home lab is 100% powered-on. A couple of times, I have even tripped the breaker entirely. Between my desktop machine, (2) HP Proliant ML110 G5s, a Netgear ReadyNAS NVX, and (2) Netgear ReadyNAS Duos, it's just too much load for a single breaker to handle; let alone the cost of powering the whole thing.
I had my eye on the Apple Mac Mini 2011 w/Lion Server since it was released as this would make the perfect low power and low noise option. This model comes with a quad core CPU that supports up to 16GB of RAM and uses 85W (peak) of power per machine. The current servers I have are dual core, and are all limited at 8GB of RAM. The idea was to replace the (4) 8GB servers that I have with (2) Apple Mac Minis running ESXi 5.0U1. I had some minor concerns about the fact that the Mini has only a single NIC, but I don't really foresee 100% utilization, but if I do, I always have the older servers available for more capacity. Perhaps someone will come up with a Thunderbolt to Ethernet adapter to address this bottleneck.
When the new Mac Mini was released someone tried to get ESXi installed and running but had some issues getting the Gigabit on-board NIC to be recognized in ESXi. Apparently the driver for the Broadcom NIC that Apple uses didn't get included in the release of ESXi 5. As a result, I put my plans on the back burner, thinking that someone would eventually figure it out.
Well, finally it appears as though someone got it working with ESXi, by installing a custom VIB from VMware for the infamous Broadcom NIC (found here). This was posted on the following site back in January (I have been busy; what can I say). The blog post was pointed out to me on Google+ by my blogger friend & fellow Tech Field Day delegate: Shannon Snowden over at Virtualization Information.
Since he was also successful in getting it working, I thought I would take a stab at it. I did have a couple of pre-requisites around what I wanted to accomplish by doing this, however.
I wanted to have the ability to host nested ESXi servers on the machine, so that I could have an all-in-one ESXi lab cluster. In order to realistically accomplish this, I needed to have an SSD in the machine, one with high IOPS performance and high enough capacity to hold all the VMs. I currently use an OCZ Vertex 2 in my desktop machine, and decided that I would do a quick search to see the current deals at Newegg, Amazon, etc. By coincidence, Newegg had a deal (which has since expired) for a 240GB OCZ Vertex 3 with a free 32GB OCZ Onyx for $249.99 (before $20 rebate). They also had the 2x8GB Corsair SODIMMS that others have installed successfully in the Mac Mini for $99. I decided on picking up 2 of each in anticipation of a 2-node build.
The parts arrived 2 days later, so I took a lunch trip over to the Apple store and picked up the Mac Mini w/Lion Server ($939 with my NetApp discount ;) After returning, I spent 30m disassembling the Mini and swapping out the RAM and 2 HDDs. It was a bit tricky getting at the 2nd HDD, but well-documented on iFixIt. After getting the hardware in working order, I performed the install of ESXi on the 32GB drive and reserved the 240GB drive as a local VMFS datastore. In addition, I added the required Broadcom NIC driver. It only ended up taking 1-2hr total, including the hardware upgrade, to get everything working. Once it was proven that everything worked, I picked up another Mini last Fri. and performed the same operation; this time it went much faster. By about 10PM on Fri. night, I had a working, silent, 2-node ESXi cluster.
I did run into one issue that I wanted to point out... Upon powering on the 2nd machine and starting the ESXi installer, I noticed some pretty sluggish performance. The installer was taking longer to got through the motions than it had for the initial build. I thought maybe I had a memory issue or another problem. As I was moving the machine around on my desk, I noticed that there was an unreasonable amount of heat coming off the aluminum casing. I removed the cover/foot from the bottom and realized that the power connector for the fan wasn't fully-seated and thus the fan was not spinning. As this is the last component to get re-installed after replacing the HDDs, it is somewhat difficult to ensure that it is re-seated properly. Also, it's hard to hear whether the fan is spinning as the Mini is so quiet. Please check it carefully before closing the access panel, so you don't make the same mistake. Thankfully, it doesn't appear that any harm was done and the sytem is running much better and much cooler now.
Once I had everything in working physical order, I decided to start working on getting the virtual layer set up to suit my needs. I started by moving my AD, MSSQL and vCenter VMs temporarily to the local SSD storage to get an idea of the performance. It was pretty good, however, I wanted to make sure that I wasn't isolating those important VMs on local storage so I then moved them back to my iSCSI datastore on the ReadyNAS, where they now sit. This allowed me to use Update Manager to update the ESXi installs with all the latest patches, etc.
I can report that the Minis are working quite well and were successful at performing a "burn in" over the weekend. I haven't done any real load testing on them yet, but I do plan on getting the nested ESXi builds started this week. I will update you all with another blog post with the results of that testing.
Finally, I decided that this is such a cool use case of nested ESXi and pefect hardware for a home lab that I submitted an abstract last week for VMworld to talk about how to implement and get the most out of it to learn ESXi and have a very portable VMware lab solution. Please stay tuned as the public voting comes available. I would love to have you vote so that I can expose the importance and usefulness of a home lab to more and potential VMware professionals. Also stay tuned for more on the Mac Mini as I perform additional testing.
Thanks, as always, for reading!
Reader Comments (19)
I'm going to be really mad at you if you force me to go buy another Mini with this post. ;)
All seriousness, great stuff man. Can't wait to get home and try it out myself!
Hi,
Interesting idea however does only having 1 NIC really limit you? can you configure MPIO?
Im looking at twin servers with 6 NICs each for my home lab - but obviously if I could do this the power requirements would be much better - but I am worried that you are really limiting what you can setup.
Can you please post a bit more info on how this works here?
Thanks
Geoff.
@Geoff
I can definitely see your point about 1 NIC being a major limitation. Others have pointed that out as well. These weren't my only lab servers (I have 2 more older ones which have 4 NICs each). I was planning to use the Mac Mini as a mostly all-inclusive nested ESXi lab, i.e. installing virtual ESXi servers, vCenter, MSSQL and AD all within the same box on an SSD which should cut down on the network bandwidth requirements.
The surprise (albeit I knew there was a good possibility of this) is that Apple released a Thunderbolt-GigE adapter recently, that will add a 2nd 1Gb port to the system, effectively doubling the network bandwidth. Now, I would argue that with a combination of a nested ESXi configuration and 2 NICs for redundancy and load-balancing (MPIO) you have removed the main limitation of this platform and keep all of the benefits of the small form factor and miniscule power requirements. The adapters will run you about $29 each at the Apple Store.
I plan to follow-up my original post with some additional detail on how I have this set up as well as performance benchmarks at a later time, so be looking out for that!
Hi Chris,
Thanks for your reply. I am putting together a home lab right now and power is a big concern for me too as I plan to be adding a SAN (prob NetApp) and already have numerous other devices powered up. I guess you can also add NICs using the usb ports - its predicted that the new minis will be out soon so will be interesting to see what the specs are an evaluate them too.
Cheers
Geoff.
Hi, just installed ESXi 5 update 1 on my mac mini server flawlessly. Just one question: I noticed there is no audio device emulation for the virtual machines, is anyone using it? I thought being virtualized it shouldn't be a problem if ESXi does not recognize the mac mini audio hardware. Days ago I also received a reply from a post on Ars Technica forum and a user confirmed this weird behavior with update 1 of ESXi 5.0. Then I tried with 5.0 but still the same issue.
@Tommaso
I am not aware of issues with audio device emulation, but in all honesty, I don't have a need for it, so I can't speak to that. For me, the most important enhancement was the addition of the Thunderbolt GigE adapter.
I do keep my Mac Mini ESXi servers updated with VUM, so I will be sure and let you know if I see any updates related to audio. Stay tuned.
I’ve had ESXi 5.0, and now 5.1 running on my Mac Mini for about 6 months now. The only problem I’ve seen is very serious….the occasional purple screen of death, or PSOD. The system will run flawlessly for weeks or months, but if it’s pushed hard and the CPU’s ramp up, it will get a PSOD.
Here are parts of the PSOD error:
PCPU 0 locked up. Failed to ack TLB invalidate (total of 1 locked up, PCPU(s): 0).
*PCPU5:2805/hostd-worker
VMware’s KB articles don’t help me out too much. Any suggestions? Could this be a heat issue?
@oturn
I am not aware of any particular issues with PSOD. I have, on occasion, witnessed a crash a couple of time, but I run my Mini headless, which means I haven't check the console via a monitor to see the reason for the crash, I usually just reboot it. I am running everything on an SSD and can recover the system fairly easily by reinstalling if need be. I will ask around and see if anyone else I know who is running an ESXi Mac Mini has experienced your issues. In the meantime, try to monitor whether the issue might be due to heat by checking the case temperature on a regular basis, as I would guess this couple be the most likely cause of the issues.
I'm interested, but wanted to know if anyone has tried DAS via Thunderbolt. The Promise kit looks expensive, but I'd want to build separate RAID disks as I do today on my Dell servers.
I don't like NAS, especially with a single NIC, but wondered if anyone has gone this route?
Will
@Will
I would be interested as well to see if the Mini works with DAS via Thunderbolt. I suspect it will, as it's just an extension of the internal bus. If you do go this route, I'd be curious to see if you can daisy chain the Apple Thunderbolt GigE adapter off the port on the Promise enclosure. That way you'll be able to use DAS in conjunction with a 2nd GigE NIC.
Let us know your findings!
I'm struggling to get auto boot into ESXi working on my Mac Mini 4,1. The only way for me to get ESXi running seems to be to use the option key at boot time and select the efi.boot disk image that appears. What am I missing?
I'd like to give the new MacMini i7 Server/ 16GB a try to replace a HP ProLiant.
My plan is to set it up with ESXi 5 and virtualize two 2008 R2 Servers, one serving as AD and the other one as Exchange 2010. It should serve as inhouse-solution for 5-10 users.
Worked very well on the elderlish ProLiant singlesocket XEON Quadcore, so a i7-system should cause no performance problems at all. I'm still a bit sceptic... Has anyone experience with similar environments?
@Richard
I didn't have any issues with auto booting the system to ESXi. Did you install over the existing HDD that came with the Mini? Perhaps it still has the Mac OS X partition layout. I would recommend wiping it prior to install of ESXi. In my case, I started with a fresh SSD so it wasn't an issue.
Let me know if that helps...
@Mattes
I had my environment set up similar to yours. I had my AD, SQL Server and vCenter server all on the local storage of the Mac Mini. With just those 3 VMs, you shouldn't have much difficulty.
To accomodate greater amounts of I/O, I replaced the stock HDD with an SSD and it is much faster now. I don't think you'll have any limitations with the CPU. It will be limited by available memory and I/O to the hard disk.
Let me know how it works out for you...
@Christopher
Yes, I was installing over a system that had been running OS X. I managed to get things to work by rebooting using an external hard drive, running terminal and using the bless command to point to the relevant VMware partition. I'm about to deploy another mini and will try what you suggested. Thanks.
-- Rich
Hi! Great write-up.
Can you post the exact model number?
This is the setup that I want as well.
But, I want to make sure the components I get work just as you described.
Thanks.
@JL
Thanks for your comment. I believe that this will work great on the most recent Mac Mini, although I haven't tested it obviously as my Mac Mini is from the previous generation. For your reference the model number for mine is: Apple Mac mini (Intel Core i7 2.0Ghz) DDR3 - Mid 2011 (MC936LL/A). Hope that helps.
Thanks!
I'm liking my Mac Mini w/ ESXI but am curious to know if anyone knows how to monitor the device as far as temperature, fan speed, etc?
Thanks for the great writeup,
Bob
@Richard
After powering on while holding the ALT key and the boot menu appears, click once on the correct boot device icon to highlight it, then hold the CTRL key and the small up-arrow in the circle should change to a circular arrow. While continuing to hold the CTRL key, click the circle containing the arrow. The Mac should then remember your boot device selection.