Benchmarking XenBench using HPL

I created a python script (see SVN or the python source file) that determines how many flops the system reaches without our XenBench application running to get a point of comparison. To achieve this I used the HPL (High-Performance Linpack) benchmark, which can be found here. Afterwards we determine the number of flops we can reach when our XenBench application is profiling, followed by the flops we get when the curses and the gui interfaces are used. After the benchmark, these measurements are presented in a clear overview, that states the change in performance such that conclusions about the amount of overhead can be made.

Installing the HPL Benchmarkt was pretty difficult, so a step-by-step overview is given here:

1. Download and untar HPL Linpack Benchmark from the website.
2. Make sure openmpi and the other dependencies for HPL are installed: 'sudo apt-get install gcc gcc-4.3 gfortran-4.3 make openmpi-bin libatlas-base-dev libblas-dev libopenmpi-dev'.
3. Make sure GoToBlas2 is installed, it can be found here. Untar the download and build the GoToBlas software: './quickbuild.64bit'. Copy the shared library: 'cp libgoto2.so /usr/local/lib'.
4. Download an architecture Make file for Ubuntu here, and modify its entries so the right paths are being used. Make sure to paste the file in the hpl-2.0 folder.
5. Build the HPL software: 'sudo make arch=Ubuntu'. For me, this still resulted in the following errors:

   make[2]: Entering directory `/home/kurt/Desktop/hpl-2.0/src/auxil/Ubuntu'
   Makefile:47: Make.inc: No such file or directory
   make[2]: *** No rule to make target `Make.inc'. Stop.

   and

   /usr/lib/libtbb.so not found

   The first problem was solved by removing the 'include Make.inc' lines from the different make files, a modified version of hpl can be downloaded here. The second problem on the other hand was easily solved by installing libtbb: 'sudo apt-get install libtbb-dev'.
6. Try building HPL again, it should work now! Go to 'hpl-2.0/bin/Ubuntu' and run the benchmark (modify HPL.dat to your preferences first): 'mpirun -np 8 xhpl'.

Python XenBench Plotting GUI

After a little research I decided to look into the software I found and try and make a plotting GUI proof of concept. Matplotlib, the plotting library that I used, is described as follows:

Matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. Matplotlib can be used in python scripts, the python and ipython shell (ala MATLAB or Mathematica), web application servers, and six graphical user interface toolkits.

The following steps need to be taken to install matplotlib:

1. sudo apt-get install python-dev
2. download numpy, which is the fundamental package needed for scientific computing with Python, here
3. tar -xvf numpy-1.5.1.tar.gz
4. sudo apt-get install gfortran
5. cd numpy-1.5.1
6. sudo python setup.py install
7. sudo apt-get install python-matplotlib or get it here

The following steps should be taken to install general python GUI libraries:

1. sudo apt-get install python-tk
2. download pmw here
3. tar -xvf Pmw.1.3.2.tar.gz
4. cd Pmw.1.3.2
5. cd src
6. sudo python setup.py install

I created a tabbed pane view with tabs for the CPU, Cache and I/O section of our application, I added a menubar that can be used for certain actions (such as saving a certain graph, starting and stopping the real time monitoring, ...). I added a statusbar to the application as well, and I embedded a graph in the CPU pane. I got some inspiration from snippets of code that can be found here.



Note that Glade (which can be found here) could come in handy to design the GUI. Note also that I first tried to work with GNUPlot (using gnuplot-py), but the embedding of these graphs was not fully supported.

PS: Last week I made a little progress with the Curses interface for the CPU utilization monitoring, more on that later.

Proof of Concept Curses Program

I created a proof of concept project to illustrate how we can use curses programming in Python to provide a shell program that updates regularly. This will be useful for users to monitor the cpu usage of the different VMs from within the terminal. I found inspiration here and here.



The demo project can be found on SVN as the "CursesTest" project or can be downloaded here.

Intermediate Presentation

During the last week I gave a presentation about what I've done so far (together with Sam Verboven) for my research project this year. I think this presentation went rather good, although I was quite nervous. You can download the ppt presentation here, a pdf version can be acquired as well: here.
On Wednesday I'll start doing my weekly day at the office in uni again.

Lack of Updates

Just a little post to explain the lack of updates in December and January. Well during this period at Belgian universities students need to study for and take their exams of the courses they took during the first academic semester. So, that is what I've been doing: studying and finishing other projects.
The last two weeks of the semester Sam and I discovered that the results of the CPU measurements were not correct. This had to do with the fact that in some time intervals no Xen Trace Events occurred. A couple of changes were made to the XenBaked and modified XenMon code, which made the results more reliable.
Expect more updates from me during the second semester, which starts February 14th.