Common CloudStack issues and their solutions

I have been working on a proof of concept project to check a cloud solution and tested CloudStack as one of the solutions.

Although setting up CloudStack seems simple and the installation documentation is good, there were issues that I encountered and took some time to figure out as the error logging does not always tell the cause of the issue.

I have tested CloudStack with KVM hypervisor – so some of the issues may be specific to this configuration.

What to check during installation

Primary storage

When using NFS storage as the primary storage, CloudStack will try to mount the storage on the physical hosts automatically by calling Libvirt and tell it to create a new storage pool with the NFS server host and path.

Libvirt will mount the primary storage NFS at /mnt/<CloudStack’s Primary Storage GUID> on the physical host.

So, when using an NFS storage, make sure that the primary storage is NOT left mounted or configured in /etc/fstab on:

• any of the physical hosts.
• the management server (not needed).

If you use NFS, make sure to choose NFS in the protocol drop down menu (avoid the SharedMountPoint option).

Primary storage protocol

When creating a primary storage in the GUI, make sure to choose NFS from the protocol drop down menu.

The SharedMountPoint option is available for KVM hosts that use OCFS2 or other shared file systems and NOT for NFS. You will need to specify the path where the mount point is on the physical host (for example: /mnt/primary).  When this option is specified, CloudStack will not try to mount the primary storage on the physical hosts.

Note:  If ShareMountPoint is specified and it points to an existing NFS mount point, CloudStack will get confused:

It will ask Libvirt to define a storage pool for /mnt/primary (or the mount point path that was specified in the GUI) and also ask libvirt to define another storage pool again under /mnt/<CloudStack’s Primary Storage GUID>. Libvirt will fail and will put the physical host into the avoid set.

Secondary Storage

The secondary storage is an NFS share that is used to store CloudStack’s VM templates.

When configuring secondary storage in the GUI:

• In “NFS Server” field – put the IP of the NFS server as the host name (DNS will not work well in CloudStack 4).
• Put the path to the files on the NFS server.

Make sure not to mount the secondary storage on the physical hosts (or have an entry for it in /etc/fstab) – CloudStack will mount the storage automatically.

Using physical host’s local storage as primary storage

If you have plenty of storage on the physical host, you can store the VM images on the host itself instead of on a shared storage that might be slower (like NFS).

The downside of this is that it will not be possible to move VMs from one physical host to another.

To enable storing VM images on the local disks, go to “Global Settings” in CloudStack’s GUI and enable:

• system.vm.use.local.storage = true
• use.local.storage = true

When enabled, CloudStack will ask Libvirt to create a storage pool that points to “/var/lib/libvirt/images”. Make sure to create a symlink from /var/lib/libvirt/images to a disk or partition that has enough room for the VM images.

VM expunge values

When a VM is being deleted, CloudStack will keep its details in the database as well as it’s disk images on the primary storage for 24 hours.

If you want to shorten that period, go to “Global Settings” in the GUI and set the expunging delay to 60 seconds or other value:

• expunge.delay = 60
• expunge.interval = 60

Common Issues

Issue #1: When trying to remove a POD receive an error: “There are private IP addresses allocated for this POD”

Possible solutions:

Check that the IP range is not already taken by other computers on the network.

Sometimes, CloudStack will keep old VMs in the database and will not release their IPs until they are expunged (see above about shortening the expunge period).

There are two options here:

• If it is not a production environment, the fastest option is to drop the database and recreate it.
• For production environments:

Check which hosts are left over in the host table for a given POD:

select * from host where pod_id = ‘1’;

delete from host where pod_id = ‘1’;

See what is left for a given host in the op_ha_work table:

select * from op_ha_work;

delete from op_ha_work where host_id = ‘1’;

Remove the host from the vm_instance table:

delete from vm_instance where pod_id = ‘1’ and host = ‘1’;

Issue #2: In management-server.log: GetRootDir for nfs://nfs-server/secondary failred due to java.net.UnknownHostException

Solution:

Put the full name as the secondary storage host, or use IP instead.

Issue #3: Templates stuck during download

Solution:

Restart the secondary storage VMs.

Issue #4: Unable to create system VMs or other VMs. In the management-server.log: Host name: phys04, hostId: 1 is in avoid set, skipping this and trying other available hosts

Solution:

• Restart the cloud-management service.
• Check that other computers on the network do not occupy IPs in CloudStack’s IP range for VMs or system VMs.