<div dir="ltr"><div>Hi Ruslanas / Donny,</div><div><br></div><div>We did try the below suggested steps on affected VMs but it doesn't help to boot VM properly, we still get XFS corruption errors on VM's Console. <br></div><div><br></div><div>VMs fails on random compute nodes. Let's suppose VM fails on Compute-8 then if we try to recreate a new VM using same image and flavor then subsequent attempts on VM creation for that Compute via same image will always fail unless and until we delete the problematic VM and clean image cache from compute node.</div><div><br></div><div>We always notice that on some compute nodes VM is provisioned successfully whereas it got failed on some random compute nodes. If we assume that issue with Image then we should have XFS errors on every VM but it is not the case.</div><div><br></div><div></div><div>We also try to provision around 6 VMs using CentOS and RHEL Cloud images, for those images we don't see XFS related errors on VM's Console.</div><div><br></div><div>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New";background:yellow none repeat scroll 0% 0%">KO1A3D02O131006CM03:/</span><span style="font-size:10pt;font-family:"Courier New"">var/lib/nova/instances/77e9d09f-ef12-4e65-aa40-256084597719$
LANG=C qemu-img info /var/lib/nova/instances/77e9d09f-ef12-4e65-aa40-256084597719/disk<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">image:
/var/lib/nova/instances/77e9d09f-ef12-4e65-aa40-256084597719/disk<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">file
format: qcow2<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">virtual
size: 60G (64424509440 bytes)<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">disk
size: 26M<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">cluster_size:
65536<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">backing
file: /var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">Format
specific information:<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
compat: 1.1<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
lazy refcounts: false<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
refcount bits: 16<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
corrupt: false<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New";background:yellow none repeat scroll 0% 0%">KO1A3D02O131006CM03</span><span style="font-size:10pt;font-family:"Courier New"">:/var/lib/nova/instances/_base$
LANG=C qemu-img info /var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">image:
/var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">file
format: raw<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">virtual
size: 20G (21474836480 bytes)<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">disk
size: 5.1G<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">ericsson@KO1A3D02O131006CM03:/var/lib/nova/instances/_base$<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">2.
Fsck output<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New";background:yellow none repeat scroll 0% 0%">root@KO1A3D02O131006CM03</span><span style="font-size:10pt;font-family:"Courier New"">:~# fsck
/var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c <span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck
from util-linux 2.27.1<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">e2fsck
1.42.13 (17-May-2015)<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">ext2fs_open2:
Bad magic number in super-block<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck.ext2:
Superblock invalid, trying backup blocks...<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck.ext2:
Bad magic number in super-block while trying to open
/var/lib/nova/instances/_base/86692cd1e738b8df7cf1f951967c61e92222fc4c<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">The
superblock could not be read or does not describe a valid ext2/ext3/ext4<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">filesystem.
If the device is valid and it really contains an ext2/ext3/ext4<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">filesystem
(and not swap or ufs or something else), then the superblock<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">is
corrupt, and you might try running e2fsck with an alternate superblock:<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
e2fsck -b 8193 <device><span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">or<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
e2fsck -b 32768 <device><span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New";background:yellow none repeat scroll 0% 0%">root@KO1A3D02O131006CM03</span><span style="font-size:10pt;font-family:"Courier New"">:~# fsck
/var/lib/nova/instances/77e9d09f-ef12-4e65-aa40-256084597719/disk<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck
from util-linux 2.27.1<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">e2fsck
1.42.13 (17-May-2015)<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">ext2fs_open2:
Bad magic number in super-block<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck.ext2:
Superblock invalid, trying backup blocks...<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">fsck.ext2:
Bad magic number in super-block while trying to open
/var/lib/nova/instances/77e9d09f-ef12-4e65-aa40-256084597719/disk<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">The
superblock could not be read or does not describe a valid ext2/ext3/ext4<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">filesystem.
If the device is valid and it really contains an ext2/ext3/ext4<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">filesystem
(and not swap or ufs or something else), then the superblock<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">is
corrupt, and you might try running e2fsck with an alternate superblock:<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
e2fsck -b 8193 <device><span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">or<span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New"">
e2fsck -b 32768 <device><span></span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:10pt;font-family:"Courier New""><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="color:rgb(31,73,125)">These are the only flags
available for virsh, there is no flag with </span><span style="font-family:"Courier New"">domblkshow.</span><span style="color:rgb(31,73,125)"><span></span></span></p>
<div><img src="cid:ii_k9a3pi0m2" alt="image.png" width="471" height="64"><br><br></div><div><img src="cid:ii_k9a3q1fs3" alt="image.png" width="471" height="299"><br><br></div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="color:rgb(31,73,125)"><span> </span></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> <br></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="color:rgb(31,73,125)"><span></span></span></p>
</div><div><br></div><div><br></div><div><br></div><div>
On Tue, Apr 21, 2020 at 3:35 PM Ruslanas Gžibovskis <<a href="mailto:ruslanas@lpic.lt">ruslanas@lpic.lt</a>> wrote:
</div><div>
does it fail always on same computes? <div dir="auto">once you have failed instance, have you tried:</div><div dir="auto">openstack server show uuid <br></div><div dir="auto">find instance name</div><div dir="auto">then login to compute, </div><div dir="auto">virsh domblkshow instance-######</div><div dir="auto">then you will find out, that your instance disk is in /var/lib/nova/instances/(instance_uuid)/disk...</div><div dir="auto">for example fromyour log: </div><div dir="auto"><span style="color:rgb(0,0,255);font-family:sans-serif;font-size:x-small">LANG=C qemu-img info /var/lib/nova/instances/</span><span style="color:rgb(0,0,255);font-family:sans-serif;font-size:x-small">dfa80e78-ee02-46e5-ba7a-</span><span style="color:rgb(0,0,255);font-family:sans-serif;font-size:x-small">0874fa37da56/disk</span></div><div dir="auto">literally this command</div><div dir="auto">to see which base file it uses, but it should be:</div><div dir="auto">LANG=C qemu-img info /var/lib/nova/instances/_base/image_uuid<br></div><div dir="auto"><br></div><div dir="auto">and do fscheck on those two :)) copy them first :)) and run fsck on the copy :)))</div><div dir="auto"><br></div><div dir="auto">i would check do you have space on that dir: df -h /var/lib/nova/instances/</div><div dir="auto"><br></div><div dir="auto">also if it has space.check that base image which is used by that qcow2</div><div class="gmail-yj6qo gmail-ajU"><div id="gmail-:154" class="gmail-ajR" tabindex="0"><img class="gmail-ajT" src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif"></div></div>
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 21, 2020 at 4:44 PM Donny Davis <<a href="mailto:donny@fortnebula.com">donny@fortnebula.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 21, 2020 at 4:26 AM Pradeep Antil <<a href="mailto:pradeepantil@gmail.com" target="_blank">pradeepantil@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Ruslanas / Openstack Gurus, <br></div><div><br></div><div>Please find the response inline below:</div><div><br></div><div>
<div dir="ltr"><b>is it the same image all the time?</b> -- <span style="color:rgb(255,0,0)"><span style="color:rgb(0,0,255)">Yes, we are using same image but image size is around 6GB and recently we have an oobersavtion that VMs are successfully spawned on some compute nodes but randomly failing on certain compute hosts. It is also observed that in nova compute logs, image is attempted to be resize. please refer the below snapshot, <br></span></span></div><div dir="ltr"><span style="color:rgb(255,0,0)"><span style="color:rgb(0,0,255)"></span><span style="color:rgb(0,0,255)"></span></span><span></span><span></span><br><span style="color:rgb(255,0,0)"><span style="color:rgb(0,0,255)"></span></span><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span></span><div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div>
<div><div><div><div dir="ltr"><span style="color:rgb(0,0,0)"><font size="1"><span style="font-family:"comic sans ms",sans-serif">2020-04-20 19:03:27.067 150243 DEBUG oslo_concurrency.processutils [req-1caea4a2-7cf0-4ba5-9dda-2bb90bb746d8 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Running cmd (subprocess): qemu-img resize /var/lib/nova/instances/616b1a27-8b8c-486b-b8db-57c7b91a7402/disk 64424509440 execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
2020-04-20 19:03:27.124 150243 DEBUG oslo_concurrency.processutils [req-1caea4a2-7cf0-4ba5-9dda-2bb90bb746d8 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] CMD "qemu-img resize /var/lib/nova/instances/616b1a27-8b8c-486b-b8db-57c7b91a7402/disk 64424509440" returned: 0 in 0.056s execute /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409
2020-04-20 19:03:27.160 150243 DEBUG nova.virt.disk.api [req-1caea4a2-7cf0-4ba5-9dda-2bb90bb746d8 cbabd9368dc24fea84fd2e43935fddfa 975a7d3840a141b0a20a9dc60e3da6cd - default default] Checking if we can resize image /var/lib/nova/instances/616b1a27-8b8c-486b-b8db-57c7b91a7402/disk. size=64424509440 can_resize_image /openstack/venvs/nova-17.1.12/lib/python2.7/site-packages/nova/virt/disk/api.py:216</span></font></span><br><div><br></div><div><b>try to create that instance using horizon or cli, whichever you favor more. does it boot good?</b> - <span style="color:rgb(0,0,255)"><font size="2">Yes , We did try to create an instance using the same image sometimes VMs spawned properly without any errors. If we specify VM count let's assume 6 , on some compute node it is failing to spawn VMs properly, on console we are getting XFS Metadata corruption errors. <br></font></span></div><div><br></div><div><span style="color:rgb(0,0,0)"><b>I would also, do cleanup of instances (remove all), and remove all dependent base files from here. rm -rf <span style="font-size:x-small">/var/lib/nova/instances/_base/</span> </b></span>-- <span style="color:rgb(0,0,255)"><font size="2">We used to clear the image cache from all compute nodes before initiating Stack Creation. Yes, used the same rm command to clear cache. <br></font></span></div><div><span style="color:rgb(0,0,255)"><font size="2"><br></font></span></div><div><span style="color:rgb(0,0,255)"><font size="2">I Just want let you know one more thing in my setup, my glance file system on comtroller are mounted on external NFS share with the following parameters, <br></font></span></div><div><span style="color:rgb(0,0,255)"><font size="2"><br></font></span></div><div><div><img src="cid:ii_k99n09bi0" alt="image.png" width="457" height="162"><br><div><img src="cid:ii_k99n0pqs1" alt="image.png" width="471" height="162"><br><br></div></div><span style="color:rgb(0,0,255)"></span></div><div><div>Any pointers or suggestions to resolve this issue. <br></div><div><br></div><div></div><div><br></div></div></div></div></div></div></div></div></div></div></div>
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 21, 2020 at 11:37 AM Ruslanas Gžibovskis <<a href="mailto:ruslanas@lpic.lt" target="_blank">ruslanas@lpic.lt</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">is it the same image all the time?<div><br></div><div>try to create that instance using horizon or cli, whichever you favor more. does it boot good?</div><div><br></div><div>I would also, do cleanup of instances (remove all), and remove all dependent base files from here. rm -rf <span style="color:rgb(0,0,255);font-size:x-small">/var/lib/nova/instances/_base/</span> <br><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 16 Apr 2020 at 19:08, Pradeep Antil <<a href="mailto:pradeepantil@gmail.com" target="_blank">pradeepantil@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Techies, <br></div><div><br></div><div>I have below RDO setup, <br></div><ul><li>RDO 13</li><li>Base OS for Controllers & Compute is Ubuntu</li><li>Neutron with vxlan + VLAN (for provider N/W)</li><li>Cinder backend is CHEF</li><li>HugePages and CPU Pinning for VNF's VMs<br></li></ul><div>I am trying to deploy a stack which is suppose to create 18 VMs across 11 computes nodes internal disk, but every time 3 to 4 VMs out of 18 doesn't spawned properly. At console of these VMs i am getting below errors, <br></div><div><br></div><div>Any idea and suggestion how to troubleshoot this? and resolve the issue. <br></div><div><br></div><div><span style="color:rgb(0,0,255)"><font size="1">[ 100.681552] ffff8b37f8f86020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681553] ffff8b37f8f86030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681560] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800<br>[ 100.681561] XFS (vda1): Unmount and run xfs_repair<br>[ 100.681561] XFS (vda1): First 64 bytes of corrupted metadata buffer:<br>[ 100.681562] ffff8b37f8f86000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681562] ffff8b37f8f86010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681563] ffff8b37f8f86020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681564] ffff8b37f8f86030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 100.681596] XFS (vda1): metadata I/O error: block 0x179b800 ("xfs_trans_read_buf_map") error 117 numblks 32<br>[ 100.681599] XFS (vda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.<br>[ 99.585766] [cloud-init[32m OK [0m[2530]: ] Cloud-init v. 18.2 running 'init-local' at Thu, 16 Apr 2020 10:44:21 +0000. Up 99.55 seconds.Started oVirt Guest Agent.<br><br>[ 101.086566] XFS (vda1): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x179b800<br>[ 101.092093] XFS (vda1): Unmount and run xfs_repair<br>[ 101.094660] XFS (vda1): First 64 bytes of corrupted metadata buffer:<br>[ 101.097787] ffff8b37fef07000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br>[ 101.105959] ffff8b37fef07010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</font></span><br>-- <br><div dir="ltr">Best Regards<br>Pradeep Kumar</div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div>Ruslanas Gžibovskis<br>+370 6030 7030<br></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Best Regards<br>Pradeep Kumar</div></div>
_______________________________________________<br>
dev mailing list<br>
<a href="mailto:dev@lists.rdoproject.org" target="_blank">dev@lists.rdoproject.org</a><br>
<a href="http://lists.rdoproject.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/dev</a><br>
<br>
To unsubscribe: <a href="mailto:dev-unsubscribe@lists.rdoproject.org" target="_blank">dev-unsubscribe@lists.rdoproject.org</a><br>
</blockquote></div><br clear="all"><div><br></div><div>Have you tried this with any other images? Maybe pickup a fresh image from <a href="https://cloud.centos.org/centos/7/images/" target="_blank">https://cloud.centos.org/centos/7/images/</a> and run the same test again where you launch 6 instances. </div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div>~/DonnyD</div><div>C: 805 814 6800</div><div>"No mission too difficult. No sacrifice too great. Duty First"</div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Best Regards<br>Pradeep Kumar</div></div>