<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">For the migration test, I used instances (cirros) without any volume attached, since Cinder was not available.<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 9, 2018 at 9:28 AM Cody <<a href="mailto:codeology.lab@gmail.com">codeology.lab@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Tzach,<br><br>Thank you very much for verifying and reporting the bug.<br><br><div>As I moved on to deploy with Ceph, the Cinder volume service is still unavailable. Perhaps the issue is more than just with using NFS?<span class="gmail_default" style="font-family:arial,helvetica,sans-serif"> <br></span></div><div><span class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></span></div><div><span class="gmail_default" style="font-family:arial,helvetica,sans-serif">With Ceph, all other pools for Nova (vms) and Glance (images) are working fine; only the Cinder (volumes) has problems. <br></span></div><br><div style="margin-left:40px">[heat-admin@overcloud-controller-2 ~]$ ceph status<br> cluster:<br> id: 7900258e-cb68-11e8-b7cf-002590a2d123<br> health: HEALTH_WARN<br> application not enabled on 1 pool(s)<br> <br> services:<br> mon: 3 daemons, quorum overcloud-controller-2,overcloud-controller-0,overcloud-controller-1<br> mgr: overcloud-controller-2(active), standbys: overcloud-controller-0, overcloud-controller-1<br> osd: 6 osds: 6 up, 6 in<br> <br> data:<br> pools: 4 pools, 240 pgs<br> objects: 8 objects, 12.1MiB<br> usage: 687MiB used, 10.9TiB / 10.9TiB avail<br> pgs: 240 active+clean<br></div><br>My testing environment is like following:<br><br>3 controller nodes<br>3 ceph storage nodes (non-collocated, 1 SSD for journal + 2 HDDs for OSD on each node)<br>2 compute nodes<br><br>The deployment is <span class="gmail_default" style="font-family:arial,helvetica,sans-serif">to test</span> an HA cluster (both controller HA and instance HA) with DVR. The cold and live migration would work only after I <span class="gmail_default" style="font-family:arial,helvetica,sans-serif">address</span> this issue [1].<span class="gmail_default" style="font-family:arial,helvetica,sans-serif"> Other than that, the Cinder volume is the only major issue for now.</span><br><br>[1] <a href="https://lists.rdoproject.org/pipermail/dev/2018-October/008934.html" target="_blank">https://lists.rdoproject.org/pipermail/dev/2018-October/008934.html</a><br><div><br></div><div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Thank you,</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Cody</div><br></div><br><br><br><br></div><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Oct 8, 2018 at 3:44 PM Tzach Shefi <<a href="mailto:tshefi@redhat.com" target="_blank">tshefi@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hey Cody, </div><div><br></div><div>The bad news, after our email figured I'd check Rocky's status, still not working.</div><div>I've thus opened two new bugs, I'll clone these back for Queens and Pike as well. </div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1637014" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1637014</a></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1637030" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1637030</a><br></div><div><br></div><div>Any ways per debugging, as you rightfully mentioned docker. <br>As of Queens Cinder was containerized meaning:</div><div>log location: /var/log/container/cinder/cinder-volume.log</div><div>Config file path is : /var/lib/config-data/puppet-generated/cinder/etc/cinder/</div><div>So nfs_shares should be found/reside under this path ^ path. </div><div><br></div><div>However if you login to Cinder's volume docker</div><div># docker ps | grep cinder somethign like -> openstack-cinder-volume-docker-0</div><div># docker exec -it openstack-cinder-volume-docker-0 /bin/bash </div><div>You should see cinder.conf plus the share file under /etc/cinder/</div><div>In side the docker /etc/cinder/ path is valid, outside of docker mapping goes to /var/lib/config-data.. </div><div> </div><div>I'd me more than happy to take a look at your volume log,</div><div>should you be willing to share it in public or private with me. </div><div><br></div><div>Tzach </div><div><br></div><div><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Sun, Oct 7, 2018 at 7:38 PM Cody <<a href="mailto:codeology.lab@gmail.com" target="_blank">codeology.lab@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Tzach,<br>
<br>
Thank you for getting back! I tested it again with<br>
CinderNfsMountOptions: 'rw,sync,nosharecache' in Queens, but still to<br>
no avail.<br>
<br>
I also noticed that the file /etc/cinder/nfs_shares does not exist on<br>
any controller, although in cinder.conf it has "#nfs_shares_config =<br>
/etc/cinder/nfs_shares". I am not sure if this is normal for using NFS<br>
with the containerized cinder service.<br>
<br>
Thank you,<br>
Cody<br>
<br>
<br>
On Sun, Oct 7, 2018 at 5:03 AM Tzach Shefi <<a href="mailto:tshefi@redhat.com" target="_blank">tshefi@redhat.com</a>> wrote:<br>
><br>
><br>
> Hey Cody,<br>
><br>
> I recall hitting a related problem, when both Glance and Cinder use the same NFS server, while each service uses it's own share if both shares reside on same NFS server you may hit an selinux issue.<br>
><br>
> The original bug I hit/reported, was closed EOL.<br>
> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1491597" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1491597</a><br>
> Notice 4th comment, adding nosharecache mount option helped me.<br>
><br>
> I'll re-check this on queens as well maybe need to rebug it.<br>
> Thanks<br>
><br>
> Tzach<br>
><br>
><br>
> On Sun, Oct 7, 2018 at 6:32 AM Cody <<a href="mailto:codeology.lab@gmail.com" target="_blank">codeology.lab@gmail.com</a>> wrote:<br>
>><br>
>> Hi everyone,<br>
>><br>
>> I have an issue with using TripleO (Queens) to setup an NFS backend for Cinder.<br>
>><br>
>> My storage.yaml is as follows:<br>
>><br>
>> parameter_defaults:<br>
>> CinderEnableIscsiBackend: false<br>
>> CinderEnableRbdBackend: false<br>
>> CinderEnableNfsBackend: true<br>
>> NovaEnableRbdBackend: false<br>
>> GlanceBackend: 'file'<br>
>><br>
>> CinderNfsMountOptions: 'rw,sync'<br>
>> CinderNfsServers: '192.168.24.1:/export/cinder'<br>
>><br>
>> GlanceNfsEnabled: true<br>
>> GlanceNfsShare: '192.168.24.1:/export/glance'<br>
>> GlanceNfsOptions: 'rw,sync,context=system_u:object_r:glance_var_lib_t:s0'<br>
>><br>
>> NovaNfsEnabled: true<br>
>> NovaNfsShare: '192.168.24.1:/export/nova'<br>
>> NovaNfsOptions: 'rw,sync,context=system_u:object_r:nfs_t:s0'<br>
>><br>
>> I used the undercloud node as an NFS server for testing purposes. Iptables is set accordingly. The /etc/exportfs on the NFS server is as follows:<br>
>><br>
>> /export/nova <a href="http://192.168.24.0/24(rw,no_root_squash)" rel="noreferrer" target="_blank">192.168.24.0/24(rw,no_root_squash)</a><br>
>> /export/glance <a href="http://192.168.24.0/24(rw,no_root_squash)" rel="noreferrer" target="_blank">192.168.24.0/24(rw,no_root_squash)</a><br>
>> /export/cinder <a href="http://192.168.24.0/24(rw,no_root_squash)" rel="noreferrer" target="_blank">192.168.24.0/24(rw,no_root_squash)</a><br>
>><br>
>> All three folders are set to chmod 777. Nova and Glance work as expected. Only Cinder remains problematic. I can try to upload volumes from overcloud, but nothing would show up in the cinder folder. Also Horizon gives errors like unable to retrieve volume and volume snapshots. Did I miss something here? I do plan to use Ceph later, but I wish to use NFS for now to test migration and failover. Any helps would be appreciated. Thank you!<br>
>><br>
>> Best regards,<br>
>> Cody<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> dev mailing list<br>
>> <a href="mailto:dev@lists.rdoproject.org" target="_blank">dev@lists.rdoproject.org</a><br>
>> <a href="http://lists.rdoproject.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/dev</a><br>
>><br>
>> To unsubscribe: <a href="mailto:dev-unsubscribe@lists.rdoproject.org" target="_blank">dev-unsubscribe@lists.rdoproject.org</a><br>
><br>
><br>
><br>
> --<br>
><br>
> Tzach Shefi<br>
><br>
> Senior Quality Engineer, RHCSA<br>
><br>
> Red Hat<br>
><br>
> <a href="mailto:tshefi@redaht.com" target="_blank">tshefi@redaht.com</a> M: +972-54-4701080 IM: tshefi<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-8793896553798122768m_-7164450212413392235gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>
<p style="font-weight:bold;margin:0;padding:0;font-size:14px;text-transform:uppercase;margin-bottom:0"><span>Tzach</span> <span>Shefi</span></p>
<p style="font-weight:normal;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>Senior Quality Engineer</span><span style="color:rgb(204,204,204)">, <span style="font-weight:normal;color:#aaa;margin:0">RHCSA</span></span></p>
<p style="font-weight:normal;margin:0;font-size:10px;color:#999"><a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span><br><br></span></a></p>
<p style="font-weight:normal;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px">
<a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="mailto:tshefi@redaht.com" target="_blank">tshefi@redaht.com</a> </span>
<span>M: <a href="tel:+972-54-4701080" style="color:#0088ce;font-size:11px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" target="_blank">+972-54-4701080</a> </span>
<span>IM: <span>tshefi</span></span></p>
<table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a> </td>
</tr></tbody></table>
</div><span></span></div></div></div></div>
</blockquote></div>
</blockquote></div>