[Rdo-list] cinder speed (slow nova?)

Hi everybody, using RDO IceHouse packages I've set up an infrastructure atop of RHEL6.6 and am seeing a very unpleasant performance for the storage. I've done some testing and here's what I get from the same storage: but different access points: cinder-volume # dd if=/dev/zero of=baloon bs=1048576 count=200 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 0.162997 s, 1.3 GB/s nova-compute # dd if=/dev/zero of=baloon bs=1048576 count=200 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 0.167905 s, 1.2 GB/s instance # dd if=/dev/zero of=baloon bs=1048576 count=200 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 10.064 s, 20.8 MB/s A bit of explanation: in above scenario I have created LV on cinder-node, then mounted it locally and ran command for "cinder-volume". Created an iSCSI target, mounted it on nova-compute, and ran command there. Then, via cinder created storage volume, booted the OS off it, and ran test from within it... Results are just miserable. going from 1.2G/s down to 20M/s seems to be a big degradation. What should I look for? I have also tried running the same command within our RHEL KVM instance and got great performance. I have checked under /var/lib/nova/instances/* and libvirt.xml seems to indicate that virtio is being employed: <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none"/> <source dev="/dev/disk/by-path/ip-192.168.46.18:3260-iscsi-iqn.2010-10.org.openstack:volume-955b25eb-bb48-43c3-a14d-222c9e8c7019-lun-1"/> <target bus="virtio" dev="vda"/> <serial>955b25eb-bb48-43c3-a14d-222c9e8c7019</serial> </disk> guest used - is rhel-guest-image-6.6-20140926.0.x86_64.qcow2 downloaded off RH site. -- Dmitry Makovey Web Systems Administrator Athabasca University (780) 675-6245 --- Confidence is what you have before you understand the problem Woody Allen When in trouble when in doubt run in circles scream and shout http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330