Xen(Trace/Mon/Baked) => XenGatherer

I tried to create a first version of the instrumentation app for Xen VMs that already gathers the information that is made available by XenTrace. I started by having a look at the following sources:

• A tutorial which explains how to use XenTrace (and xentrace_format) and XenMon (and xenbaked) found here.
• Information about the parameters that are monitored and how the tools hierarchy (see image below) can be found here.
• A paper titled "XenMon: QoS Monitoring and Performance Profiling Tool". This whitepaper can be downloaded here.

While discovering the above mentioned tools I had a closer look at their implementations and decided that for the parameters I need that are made available by XenTrace I'll keep using XenBaked since it does not bring a lot of overhead (its output is stored in a binary file/shared memory). I adjusted the XenMon Python script such that it does not write to file constantly (less io interrupts) while gathering information through XenBaked. The source code can be found here.

If you run the XenGatherer tool, you get a CLI with the following possibilities:

• start: start gathering information by running xenbaked
• stop: stop gathering information by killing the xenbaked process
• report: get the desired data from the mmapped file (/var/run/xenq-shm) and write it to a text file
• quit: exit the application

Patching the Linux Kernel

During this week I managed to get the newer kernel version to work with oprofile: I had to make changes to a couple of source file to get it to compile. Then I tried to make a patch file by using the diff command that checks the differences that exist between nonpatched and patched kernel files. But applying this patch didn't work. So, now I googled how to create a patch file again and found some things here that I probably did wrong:

• When creating the patch, the new kernel was compiled, which explains the fact that I genereated a 200 MB patch when using the root folders of the kernels.


• "Do not ever create a patch with uneven directories" does the site state, well I guess this is what I did.

Next week I'll first try to get this patch working, afterwards I'll start writing a Python application that already gathers one of the parameters (learning the Python language in the mean time).

What's next

Tomorrow I'll try to get the test environment up and running (with a patched more recent kernel version). And during the week I'll also set this up on a machine at home, such that I can develop at home as well. If all goes well I'll start looking into creating a little python program that already gathers the information for one of the parameters.

First day & Planning

Yesterday was the first day of working on this project, the objective of the day: setting up the test environment according to a tutorial made by Sam Verboven.
An outline of this installation process:

1. Download a pv-ops enabled kernel, patch it with the xenoprof patch, compile and install it.
2. Download Xen 3.4.3-testing, compile and install it (also add a Grub entry for Xen).
3. Boot to Xen and create a new Ubuntu 9 image using the xen tools (and make sure this VM runs correctly).
4. Install (Xen)OProfile and make sure the profiling features work correctly.

An overview of the steps required to build the application:

1. Updating the test environment by installing a patched more recent kernel version
2. Make a Python (rapid prototyping, efficient for string operations) tool that gathers the desired parameters from the selected tools:
   • Xen Trace for CPU usage, blocked, waiting and allocation
   • XenoProfile for cache hits and misses
   • SSH connection to VMs (DomU Kernel) for disk block reads/writes, requests/s and average wait and service times
3. Extend the gathering tool with a parser that actually parses and formats the gathered data
4. Make sure running the tool introduces little overhead to the system
5. Make the application even more black-box by finding a way around getting the IO data through SSH connections to the VMs
6. Modify XenMon such that the data can be observed through a GUI

Performance Interference Effects in Virtual Environments

I read the paper "An Analysis of Performance Interference Effects in Virtual Environments" by Younggyun Koh, Rob Knauerhase, Paul Brett, Mic Bowman, Zhihua Wen, Calton Pu. An abstract can be found here: http://goo.gl/9pR9.
It gave a good amount of background information about the things we want to realize.

Introduction

Today I had a first meeting about my research project for this year, last year I decided to do this again with Sam Verboven as my supervisor. I look forward to working on the project and hope it will be learnful. I'll try to keep this blog up to date with the progress that is being made.

The assignment consists of building a 'blackbox' application which uses information about the applications that are running in a virtualized environment. It should make decisions about when to migrate a virtual machine for a certain application. This is the case when the VM starts to interfere with other VMs too much. When the load becomes too low on the other hand, decisions about the consolidation of VMs should be made.
The input for this application consists of the following information gathered from the different VMs/the system:
- average CPU utilization
- cache hits and misses
- virtual machine switches per second
- I/O blocks per second
- disk reading and writing time per VM
This gathering of information should have a minimal impact on the system: low resource usage of the application and should require no (or minimal) adaptation of DomU. This information about the loads could also be important for getting the required input information for my thesis project (where this data is initially considered as known).

The first steps this week will be to read the paper "An Analysis of Performance Interference Effects in Virtualized Enviroments", follow Sams tutorial to set up a test system with the required tools and patch the latest Xen kernel such that XenOProfile works fine.