Nathan's Weblog

Another Network Test

I was concerned with the previous network test that perhaps it was slower than what it should be because of the bandwidth to the outside machine. To alleviate that I bonded the two gigabit ethernet cards in that machine effectively giving it a 2 gigabit pipe. As you can see from the results it didn't make much of a difference.

As always here is the excel spreadsheet with all of the test results.

Excel Workbook

Initial Network Test

After some headache and help from Devlin I was able to do some initial testing of the networking with VMWare. Devlin helped me get IOMeter compiled in Debian. Below are the graphs. The tests were run using iometer to read and write 1 MB of data. I'm not sure how valid these tests are because I only had one non VM machine for all of the Virtual Machines to run against so the performance may have been hindered by that and not just the fact that multiple Virtual Machines were all trying to go through the same interface. Anyway below are some graphs and an excel workbook with the test results.

Excel Workbook

The importance of distributing across LUNs

As an update to what I have been working on I have been running more IOMeter tests. This time with many more Virtual Machines. To do this latest set of tests I created a new Windows 2003 Enterprise Server image that is smaller. I also learned how to change the Machine name and IP address when deploying so deployment of many machines goes a lot faster now. Any way on to the tests.

The Tests

Below you will see four graphs. The graphs labeled with Split you will notice a bump at 13 Virtual Machines. The reason for this bump is because at that point it switched to having virtual machines on two different LUNs instead of just one. Because of that bump I reran the tests and the other graphs are the result. With the second set I started with 24 virtual machines, 12 on each LUN, and alternated between LUNs for deciding which VM to remove. As a result there are no bumps and I think a more accurate representation of performance.

Files

Here are the XLS files used to make the graphs.

8-22 VMs with the bump.

24 VMs Alternating.

Graphs

VM test with IO Meter

A few days ago I was talking to Devlin and he asked why we don't just run benchmarks on the VMs to see what performance they get. After thinking about that I decided it was a good idea. So today I setup and ran some initial tests.

The test was fairly simple and is outlined below:

IO Meter Dynamo on each virtual machine
IO Meter GUI running on Tibblefork acting as the manager
Ran the default IO Meter Disk IO test
Ran 6 tests each with 1 more virtual machine

The graphs below are the results of the tests. You will notice that with only one VM running it has very good performance then drops in half almost with 2 and then just a steady decline.

(The pictures seem to be too wide for the page so you may want to right-click on them and select view image.)

The default Acess Specification for IO Meter is:

Transfer Request Size: 2 KB
Writes 33% of the time and does reads the other 67%
100% random distribution of calls instead of sequential

Here is a link to the Excel spreadsheet of the results that were used to generate the graphs.

IOMeter

I've done some further looking at IOMeter and it won't work for a monitoring tool for our overhead test. It is meant more as a benchmark tool. Meaning that it generates a work load on the system and sees how well it responds. Which makes for some interesting experiments. It may be useful to run IOMeter tests on multiple VMs at the same time to see the performance degredation. This would probably yield some interesting results that we could then use.

Overhead Test Initial Impressions and Update

The overhead test configuration is progressing. Well actually it is ready to go. Over the last couple of days I figured out how to configure our labs switch, I also enabled the web configuration module for it so configuring it is very easy now. I figured out how to work the load balancer we got from OIT, that turned out to be a little bit of a pain but not bad. What I had the most trouble with was figuring out how our test network needed to be setup so that requests made to virtual machines through the load balancer would work. With Windows 2003 server you have to have the network setup so that the load balancer is the default gateway for the server. Currently each of the virtual machines has two network cards the second one being on a separate vlan where the load balancer is the default gateway. After I disabled the first network card, which also had a default gateway set, everything started working.

My initial impressions of the test so far is that with strictly static content we can put a lot of virtual machines on one box and have them all running without much problem. The current setup is nine virtual machines each with 250MB of ram and a single CPU. One problem with VMWare is that the virtual network cards are only 10Mb/s which I think makes them become network bound before anything else. I need to try the other vmware network driver to discover if it will give each VM a bigger pipe.

Currently with all 9 machines active and serving content to about 150 clients continuously we have about 65 percent CPU usage and the network is at about 90 Mb (what you would expect with 9 VMs that each have a 10 Mb pipe).

What I need to do now is run the actual tests. Meaning have the load balancer pointing at only one machine and then increse by one machine every couple of minutes. I'm afraid though that we are going to run into an issue of being network bound though which will make the tests less meaningful, hopefully the other network driver will work and give more throughput. I'll update this when I find out.

Update

The other network driver allows the virtual machines to have 1Gb connections so it looks like I need to change all of the other VMs.

Overhead Test Progress

Dr. Windley came up with an idea to estimate the overhead created by VMWare. The setup is fairly simple. Have a load balancer in front of a group of virtual machines. The test is simple also, you run a simple download test and note machine statistics for the parent system. Start haveing the load balancer include more and more virtual machines in the test. Watch the parent machine statistics. If VMWare had no overhead the parent machine statistics should stay the same. We should be able to tell the overhead for each virtual machine by the increase in the parent machines stats.

The big assumption here is that the overhead added by the OS itself is very small. This may or may not be true. We will need to test that to see.

If you look here you can find the list of todos that we have still to work on as well as other miscellaneous thoughts regarding the test.

Feel free to comment there on under this entry.

VMFS and Test Data for the VMs

So I have learned a couple of things about VMFS. First it is a propriatary file system that is best suited for very large files, such as isos and disk images. This makes sense because its primary purpose is to host disk images that are use by virtual machines.

Second it looks like it should be possible to mount a VMFS drive under the host system. However, I have not figured out how to do that quit yet. I have discovered though that I can create a disk image that can be loaded into multiple virtual machines. What this means is that we can create a disk image of the data we want to use for benchmarks and then just have each of the virtual machines load it for use. One thing to note however is that you have to make all of the machines discard any changes they would actually make to the drive. If they don't then only one machine can power up with that drive at a time, I think it must do some sort of locking.

My current todo is

Create less beafy virtual machine
Get example websites and files (I've contacted Steve about this)
Run some initial tests and benchmarks that will help me figure out how to run tests and benchmarks

Comparing Apples to Oranges?

After some discussion between Devlin and I. I am starting to wonder if it will even be useful to load anything besides the ESX server on the other machine. The question arises on whether installing an OS on there will be useful when it comes to benchmarking.

Though it would fairly simple to install Windows 2003 Server and some services and see how well they do under load I don't think that is a very fair comparison to a similar setup under virtualization. I don't think it is for a couple of reasons. The main ones though are CPUs and Memory. Where the 6650 has 16 Gig of Ram and 4 CPUs the virtualized version has at most 2 CPUs and 3600 MB of ram.

I suppose to figure out overhead involved with virtualization we could yank the memory and CPUs but I am not sure how much Dr. Windley will want to do that. It needs some more thought I think. Feel free to comment and tell me your ideas.

Waiting for the SAN

We are at an impass currently. Our biggest limiting factor is disk space. We have aquired various ISOs but have no where to put them so that they can be loaded. We also are limited in the number of machines we can run because we don't have enough space for each machines virtual hard disk.

Currently Devlin has successfully installed Knoppix in a VM and I am trying to install Windows 2003 Server Enterprise Edition. It is taking a very long time however.

We have installed Windows 2003 Server Standard Edition on the 1750 and we can Remote Desktop into it. I have also installed the VMWare management softare on the 1750 and it can be used to manage our one 6650 that currently has the ESX server running.

Progress!! well some any way

We've made some good progress this week. We have one ESX server up and running. Devlin sucessfully launched KNoppix in a virtual machine. So we know it works. We need to go through and document what it is that we have done but I am feeling too burned out right now to do it, hopefully Monday I will feel more like writing.

Also as a general lab update, serval's firewall was totally open so I went and closed off all ports except 80, 443, 22, and some VNC ports for us to use.

The reason we are using VNC is so that we can use VMWare's console application. The way you talk to a virtual machine is through this console application that hooks up and acts as the monitor for the machine. It gives you all the functionality as if you were sitting right in front of the machine, which is pretty cool. Anyway they don't have a console for OS X and when exporting, X doesn't go between big endian and little endian very well so we run the VNC client over to serval and then use it to open the console to yampa.

Next week we have a meeting with OIT which will hopefully give us some ideas on what to benchmark with and with what OS. I already plan on benchmarking with a couple of linux distributions, but we need to find out if we need to run tests with Windows also. In all things are moving forward.

Cheaper servers?

I wonder if it would be possible to create distributed virtual machines. Meaning launch virtual machines on regular work stations that people are sitting in front of and basically syphone off the idle cpu cycles to perform the tasks normally reserved for a server, such as dhcp, mail, or dns. In a small company where those services are not heavily loaded it seems like it could be a feasable solution.

Distributed Computing and Virtualization

It seems that virtualization has much in common with distributed computing. One of the biggest problems it faces is scheduling. How to decide what to run where and when to do it.

Another problem they both seem to face is fault tolerance. How do you recover from a problem. In my experience with Dogma if a node had a problem then we just gave the job to another node and didn't do any type of recovery of already done work. The same scheme seems to be used with virtualization except the costs are much higher. Meaning it costs a lot more to shutdown the problem vm, bring up a new one and launch it than it does to just send the work to another node.