VM with XenServer, Java, and Apache CloudStack
Apache CloudStack is an open-source management server for running a private cloud infrastructure. Because it’s backed by Citrix, CloudStack has enterprise-class support for scaling out VMs on XenServer hosts. The CloudStack management server controls the XenServer host instances using the Java bindings of the XenServer Management API. Since the CloudStack source code is on github, it serves as an instructive example of how to use the XenServer Java API. In this article we’ll do a code inspection on CloudStack and learn the mechanics of how to scale out a private cloud with Xen and Java.
The intended audience for this article includes Java programmers interested in Java-based clouds, and also virtualization sysadmins looking under the hood of their CloudStack and/or XenServer systems. You don’t need CloudStack or XenServer installed to follow this code inspection, but if you want to try running a program using the XenServer Java API then you will definitely need an installation of XenServer and a VM template. Setting up XenServer and the VM template is beyond the scope of this article. Also, please note: the code extracts shown here are greatly simplified from the original CloudStack code, not complete Java programs to be directly compiled. The real CloudStack code is pretty long so I’ve heavily edited the original to produce something short enough to get the point across.
So with that introduction, let’s look at some CloudStack code.
When CloudStack needs to scale out on a XenServer hypervisor, it makes a specification of what kind of VM it would like to launch and embeds the spec inside a StartCommand. So our code inspection begins with method execute(StartCommand), found in the Java source file CitrixResourceBase.java. The CloudStack code lives in Java package com.cloud, and makes use of the XenServer management library in Java package com.xensource.
Below are the essentials of CloudStack’s execute(StartCommand) method.
package com.cloud.hypervisor.xen.resource;
.
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Host;
import com.xensource.xenapi.VM;
.
import com.cloud.agent.api.StartCommand;
import com.cloud.agent.api.StartAnswer;
import com.cloud.agent.api.to.NicTO;
import com.cloud.agent.api.to.VirtualMachineTO;
import com.cloud.agent.api.to.VolumeTO;
.
public class CitrixResourceBase {
.
public StartAnswer execute(StartCommand cmd) {
VirtualMachineTO vmSpec = cmd.getVirtualMachine();
Connection conn = getConnection();
Host host = Host.getByUuid(conn, _host.uuid);
VM vm = createVmFromTemplate(conn, vmSpec, host);
for (VolumeTO disk : vmSpec.getDisks())
createVbd(conn, disk, vmName, vm, vmSpec.getBootloader());
for (NicTO nic : vmSpec.getNics())
createVif(conn, vmName, vm, nic);
startVM(conn, host, vm, vmName);
return new StartAnswer(cmd);
}
.
}When the above method is finished, your XenServer will have a brand new VM guest running and ready to process whatever tasks your cloud is designed for. Let’s dig into the steps this method follows to launch that VM.
Get a Xen Connection and Login
Line 17 above shows that the first step is to get a Connection to the XenServer. To prepare a secure, encrypted Xen connection, simply construct a new Connection object over HTTPS using the IP address of the XenServer instance:
import java.net.URL;
import com.xensource.xenapi.Connection;
...
Connection conn = new Connection(new URL("https://" + ipAddress),
NUM_SECONDS_TO_WAIT);Having a Connection in hand, next you would proceed to login, which is what makes the first XML-RPC transmission to the XenServer. With normal XenServer credentials you could login like so:
import com.xensource.xenapi.APIVersion;
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Session;
...
Session session = Session.loginWithPassword(conn, "username",
"password", APIVersion.latest().toString());Login is a tad more complicated when you have multiple XenServer instances configured in a Master/Slave resource pool. Normally you should connect only to the master, using method loginWithPassword, which produces a Session that is valid on any host in the pool. If you need to connect to a specific slave instance you would use slaveLocalLoginWithPassword, which gives you an “emergency mode” session usable only on that slave host. When presented with a XenServer pool, CloudStack’s intention is to make its permanent connection to the master. To be safe, it assumes that the IP address it’s been given to connect to could be a slave, so it does the slave local login first, then obtains the master host IP address from that login and re-authenticates with the master. (For more information on XenServer Master/Slave resource pools, see the XenServer System Recovery Guide.)
The CloudStack login code lives in XenServerConnectionPool.java. Here it is:
package com.cloud.hypervisor.xen.resource;
.
import java.net.URL;
.
import com.xensource.xenapi.APIVersion;
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Host;
import com.xensource.xenapi.Pool;
import com.xensource.xenapi.Session;
.
public class XenServerConnectionPool {
.
protected Map<String, Connection> _conns;
.
public Connection connect(String hostUuid, String poolUuid,
String ipAddress, String username, String password,
int wait) {
Host host = null;
Connection sConn = null;
Connection mConn = _conns.get(poolUuid); // Cached?
if (mConn != null) {
try {
host = Host.getByUuid(mConn, hostUuid);
}
catch (SessionInvalid e) {
Session.loginWithPassword(mConn, mConn.getUsername(),
mConn.getPassword(), APIVersion.latest().toString());
}
}
else {
Connection sConn = new Connection(
new URL("https://" + ipAddress), wait);
Session.slaveLocalLoginWithPassword(sConn, username,
password);
Pool.Record pr = Pool.getAllRecords(sConn)
.values().iterator().next(); //Just 1 pool, 1 record
String masterIp = pr.master.getAddress(sConn);
mConn = new Connection(masterIp, wait);
Session.loginWithPassword(mConn, username, password,
APIVersion.latest().toString());
_conns.put(poolUuid, mConn);
}
return mConn;
}
.
}Notice above that the XenServer host and resource pool are identified by a UUID (Universally Unique ID), which is an object naming convention used in both XenServer and CloudStack.
Create a Xen VM
Line 19 of the execute(StartCommand) method above shows the next step is to create a VM from a template which is specified in the command. The CloudStack StartCommand contains a CloudStack-specific vmSpec of type VirtualMachineTO. (“TO” means Transfer Object, which applies to data records passed in and out of the command api. Transfer Objects are not persisted to a database.) vmSpec is a specification to the cloud management server that makes a request for a VM with certain characteristics, such as number of cpus, clock speed, min/max RAM. The CloudStack createVmFromTemplate method applies the vmSpec specification and produces a XenServer Java VM object. The Xen VM class represents a guest virtual machine which can run on a XenServer instance. The VM returned by this method is not started yet.
Here are the essentials of the CloudStack createVmFromTemplate method.
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Host;
import com.xensource.xenapi.Types;
import com.xensource.xenapi.VM;
.
import com.cloud.agent.api.to.VirtualMachineTO;
.
public class CitrixResourceBase {
.
protected VM createVmFromTemplate(Connection conn,
VirtualMachineTO vmSpec, Host host) {
String guestOsTypeName = getGuestOsType(vmSpec.getOs());
VM template = VM.getByNameLabel(conn, guestOsTypeName)
.iterator().next(); //Just 1 template
VM vm = template.createClone(conn, vmSpec.getName());
.
vm.setIsATemplate(conn, false);
vm.setAffinity(conn, host); //Preferred host
vm.removeFromOtherConfig(conn, "disks");
vm.setNameLabel(conn, vmSpec.getName());
vm.setMemoryStaticMin(conn, vmSpec.getMinRam());
vm.setMemoryDynamicMin(conn, vmSpec.getMinRam());
vm.setMemoryDynamicMax(conn, vmSpec.getMinRam());
vm.setMemoryStaticMax(conn, vmSpec.getMinRam());
vm.setVCPUsMax(conn, (long)vmSpec.getCpus());
vm.setVCPUsAtStartup(conn, (long)vmSpec.getCpus());
.
Map<String, String> vcpuParams = new HashMap<String, String>();
Integer speed = vmSpec.getSpeed();
vcpuParams.put("weight", Integer.toString(
(int)(speed*0.99 / _host.speed * _maxWeight)));
vcpuParams.put("cap", Long.toString(
!vmSpec.getLimitCpuUse() ? 0
: ((long)speed * 100 * vmSpec.getCpus()) / _host.speed));
vm.setVCPUsParams(conn, vcpuParams);
.
vm.setActionsAfterCrash(conn, Types.OnCrashBehaviour.DESTROY);
vm.setActionsAfterShutdown(conn, Types.OnNormalExit.DESTROY);
vm.setPVArgs(vmSpec.getBootArgs()); //if paravirtualized guest
.
if (!guestOsTypeName.startsWith("Windows")) {
vm.setPVBootloader(conn, "pygrub");
}
}
.
// Current host, discovered with getHostInfo(Connection)
protected final XsHost _host = ... ;
.
}Notice above that every Xen VM method requires a Connection argument. Because the guest is virtually a computer, there can be many Connections to it simultaneously – so the Connection is not a property of the VM. On the other hand, a VM can only reside on one XenServer host at a time, so the VM.Record type has a field residentOn of type Host. The above CloudStack method does not set residentOn, but it does set another field of type Host, called affinity, which is a hint to CloudStack that the VM would “prefer” to be launched on a particular host. CitrixResourceBase also has a field _host of type XsHost, a CloudStack helper structure, which gets initialized with XenServer host info and uuids in a CloudStack method called getHostInfo, separately from what is shown in this code inspection.
The call to removeFromOtherConfig refers to the Citrix other-config map parameter, which is a map object that provides arbitrary key/value arguments to XenServer host commands. (See the xe command reference for some host commands that make use of the other-config parameter.) In this case the point is to strip away any initial assumptions about disks associated with the VM.
VCPUsParams weight and cap are defined in Citrix article CTX117960. A higher weight results in XenServer allocating more cpu to this VM than other guests on the host. cap is a percentage limit on how much cpu the VM can use.
Create a VBD for each disk
Line 21 of the execute(StartCommand) method invokes the CloudStack method createVbd to create a VBD (Virtual Block Device) in the XenServer guest for each disk in the vmSpec. VBD is a decorator to an underlying VDI (Virtual Disk Image). CloudStack recognizes four types of disk volumes: Root (the main drive), Swap, Datadisk (more local storage), ISO (CD-ROM). The vmSpec disk info is passed to the volume method parameter below.
Below is CloudStack method createVbd. It does not do a whole lot, relying mainly on a pre-existing VDI. If you have done a Basic Installation of CloudStack, then you’ll have the VDI already. The key invocation of interest here is VBD.create, which allocates resources on the XenServer.
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.VBD;
import com.xensource.xenapi.VDI;
import com.xensource.xenapi.VM;
.
import com.cloud.agent.api.to.VolumeTO;
import com.cloud.storage.Volume;
import com.cloud.template.VirtualMachineTemplate.BootloaderType;
.
public class CitrixResourceBase {
.
protected VBD createVbd(Connection conn, VolumeTO volume,
String vmName, VM vm, BootloaderType bootLoaderType) {
VDI vdi = VDI.getByUuid(conn, volume.getPath());
VBD.Record vbdr = new VBD.Record();
vbdr.VM = vm;
vbdr.VDI = vm;
vbdr.userdevice = Long.toString(volume.getDeviceId());
vbdr.mode = Types.VbdMode.RW;
vbdr.type = Types.VbdType.DISK;
if (volume.getType() == Volume.Type.ROOT
&& bootLoaderType == BootloaderType.PyGrub)
vbdr.bootable = true;
if (volume.getType() != Volume.Type.ROOT)
vbdr.unpluggable = true;
VBD vbd = VBD.create(conn, vbdr);
return vbd;
}
.
}Although createVbd returns a VBD, the caller execute does not save it. Presumably this is because CloudStack could look it up again later from the XenServer using static method VBD.getAllRecords(Connection).
For simplicity I edited out the logic in createVbd involving Volume.Type.ISO, which is for CD-ROM drives.
Create a VIF for each network interface
Line 23 of the execute(StartCommand) method invokes the CloudStack method createVif to create a new XenServer Java VIF (Virtual network Interface) for the NICs (Network Interface Controllers, aka network adapters) specified on the guest VM. The method appears at first to be pretty simple:
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Network;
import com.xensource.xenapi.VIF;
import com.xensource.xenapi.VM;
.
import com.cloud.agent.api.to.NicTO;
.
public class CitrixResourceBase {
.
protected VIF createVif(Connection conn, String vmName, VM vm,
NicTO nic) {
VIF.Record vifr = new VIF.Record();
vifr.VM = vm;
vifr.device = Integer.toString(nic.getDeviceId());
vifr.MAC = nic.getMac();
vifr.network = getNetwork(conn, nic);
VIF vif = VIF.create(conn, vifr);
return vif;
}
.
}But nothing is really quite simple when it comes to the network. CloudStack’s method getNetwork undergoes a nontrivial effort to piece together a Xen Network object. The overall procedure is essentially like this:
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Network;
import com.xensource.xenapi.VLAN;
.
import com.cloud.agent.api.to.NicTO;
import com.cloud.network.Networks.BroadcastDomainType;
.
public class CitrixResourceBase {
.
protected Network getNetwork(Connection conn, NicTO nic) {
BroadcastDomainType nicType = nic.getBroadcastType();
Network network = null;
Network.Record nwr = new Network.Record();
if (nic.getBroadcastType() == BroadcastDomainType.Native
|| nic.getBroadcastType() == BroadcastDomainType.LinkLocal) {
network = ...
}
else if (nicType == BroadcastDomainType.Vlan) {
network = ...
}
else if (nicType == BroadcastDomainType.Vswitch) {
network = ...
}
return network;
}
.
}The basic network scenarios above are Native/Link-Local, Vlan, and Open vSwitch.
Native networking means the guest VM will perform plain vanilla traffic to its ethernet adapter. In this case, CloudStack just takes note of the XenServer uuid for guest traffic on this VM and passes it as an argument to the Xen Network constructor. As mentioned above, CitrixResourceBase has called getHostInfo to obtain XenServer uuids and has saved them in the field called _host.
Link-Local networking, formally defined in RFC 3927, is for the networking setup where two machines are directly connected by ethernet cables with no intervening switch or router. You wouldn’t normally configure your hardware this way but in fact CloudStack does use link-local addressing to connect the XenServer host to a special system control VM called the Secondary Storage VM. When that VM starts it falls into this case, and since CloudStack already has its network adapter uuid stored in _host it makes sense to handle the Link-Local case in the same block of code as for Native networking:
protected Network getNetwork(Connection conn, NicTO nic) {
...
if (nic.getBroadcastType() == BroadcastDomainType.Native
|| nic.getBroadcastType() == BroadcastDomainType.LinkLocal) {
String uuid = null;
switch (nic.getType()) {
case Guest: uuid = _host.guestNetwork; break;
case Control: uuid = _host.linkLocalNetwork; break;
case Management: uuid = _host.privateNetwork; break;
//other cases not shown
}
network = Network.getByUuid(conn, uuid);
}
...With Vlan, CloudStack makes a network name unique by adding a tag, to allow virtual LANs to coexist on a single network. CloudStack employs an additional trick of adding a timestamp to the Vlan name in case of clustered XenServer hosts concurrently trying to create the same Vlan; they will be made to choose the Vlan created with the earliest timestamp.
protected Network getNetwork(Connection conn, NicTO nic) {
...
else if (nicType == BroadcastDomainType.Vlan) {
long tag = Long.parseLong(nic.getBroadcastUri().getHost());
nwr.nameLabel = "VLAN-" + network.getNetworkRecord(conn).uuid
+ "-" + tag;
nwr.tags = new HashSet<String>();
nwr.tags.add(generateTimeStamp());
network = Network.create(conn, nwr);
VLAN.create(conn, network.getPif(conn), tag, network);
}
...Open vSwitch, like CloudStack itself, is another open source virtualization product backed by Citrix. When the guest NIC is routed with Open vSwitch, CloudStack creates the VIF on the XenServer control domain (“dom0”) and temporarily plugs it in, which has the effect of creating a network bridge to the underlying PIF (Physical network Interface). Open vSwitch supports its own Vlan, or you can choose Tunnel-style networking to the underlying network interface.
protected Network getNetwork(Connection conn, NicTO nic) {
...
else if (nicType == BroadcastDomainType.Vswitch) {
String nwName = null;
if (nic.getBroadcastUri().getAuthority().startsWith("vlan")
nwName = "vswitch";
else {
nwName = "OVSTunnel"
+ Long.parseLong(nic.getBroadcastUri().getHost());
nwr.otherConfig = mapPair("ovs-host-setup", "");
}
nwr.nameLabel = nwName;
network = Network.create(conn, nwr);
VM dom0 = null;
for (VM vm : Host.getByUuid(conn, _host.uuid)
.getResidentVMs(conn)) {
if (vm.getIsControlDomain(conn)) {
dom0 = vm;
break;
}
}
VIF.Record vifr = new VIF.Record();
vifr.VM = dom0;
vifr.device = getLowestAvailableVIFDeviceNum(conn, dom0);
vifr.otherConfig = mapPair("nameLabel", nwName);
vifr.MAC = "FE:FF:FF:FF:FF:FF";
vifr.network = network;
VIF dom0vif = VIF.create(conn, vifr);
dom0vif.plug(conn); //XenServer creates a bridge
dom0vif.unplug(conn);
}
...Start the VM
Finally, on line 24 of execute(StartCommand), we see a call to the CloudStack method startVM. The sole job of startVM is to invoke the Xen method VM.startOnAsync. It is a potentially long-running operation, so it returns a Xen Task object that CloudStack will monitor, waiting for the VM to be done with startup. startOnAsync also takes two boolean flags, which in this case are set to start the VM in a running state (not paused), and to force startup regardless of whether the current VM configuration looks different than in its last startup.
If you ignore the try/catch handling, the process of starting a VM is nice and short:
import com.xensource.xenapi.Connection;
import com.xensource.xenapi.Host;
import com.xensource.xenapi.Task;
import com.xensource.xenapi.Types;
import com.xensource.xenapi.VM;
.
public class CitrixResourceBase {
.
void startVM(Connection conn, Host host, VM vm, String vmName) {
boolean startPaused = false;
boolean force = true;
Task task = vm.startOnAsync(conn, host, startPaused, force);
while (task.getStatus(conn) == Types.TaskStatusType.PENDING) {
Thread.sleep(1000);
}
if (task.getStatus(conn) != Types.TaskStatusType.SUCCESS) {
task.cancel(conn);
throw new Types.BadAsyncResult();
}
}
.
}And there we have it! With this code inspection we’ve walked through the Java code in which CloudStack has made a connection to a XenServer hypervisor, logged in, created the guest VM, allocated its VBDs and VIFs, and started the guest. A brand new VM is fired up and running on the private cloud.
