[rdo-dev] Cinder NFS backend issue
Cody
codeology.lab at gmail.com
Tue Oct 9 13:31:43 UTC 2018
For the migration test, I used instances (cirros) without any volume
attached, since Cinder was not available.
On Tue, Oct 9, 2018 at 9:28 AM Cody <codeology.lab at gmail.com> wrote:
> Hi Tzach,
>
> Thank you very much for verifying and reporting the bug.
>
> As I moved on to deploy with Ceph, the Cinder volume service is still
> unavailable. Perhaps the issue is more than just with using NFS?
>
> With Ceph, all other pools for Nova (vms) and Glance (images) are working
> fine; only the Cinder (volumes) has problems.
>
> [heat-admin at overcloud-controller-2 ~]$ ceph status
> cluster:
> id: 7900258e-cb68-11e8-b7cf-002590a2d123
> health: HEALTH_WARN
> application not enabled on 1 pool(s)
>
> services:
> mon: 3 daemons, quorum
> overcloud-controller-2,overcloud-controller-0,overcloud-controller-1
> mgr: overcloud-controller-2(active), standbys: overcloud-controller-0,
> overcloud-controller-1
> osd: 6 osds: 6 up, 6 in
>
> data:
> pools: 4 pools, 240 pgs
> objects: 8 objects, 12.1MiB
> usage: 687MiB used, 10.9TiB / 10.9TiB avail
> pgs: 240 active+clean
>
> My testing environment is like following:
>
> 3 controller nodes
> 3 ceph storage nodes (non-collocated, 1 SSD for journal + 2 HDDs for OSD
> on each node)
> 2 compute nodes
>
> The deployment is to test an HA cluster (both controller HA and instance
> HA) with DVR. The cold and live migration would work only after I address
> this issue [1]. Other than that, the Cinder volume is the only major
> issue for now.
>
> [1] https://lists.rdoproject.org/pipermail/dev/2018-October/008934.html
>
>
> Thank you,
> Cody
>
>
>
>
>
>
>
> On Mon, Oct 8, 2018 at 3:44 PM Tzach Shefi <tshefi at redhat.com> wrote:
>
>> Hey Cody,
>>
>> The bad news, after our email figured I'd check Rocky's status, still not
>> working.
>> I've thus opened two new bugs, I'll clone these back for Queens and Pike
>> as well.
>> https://bugzilla.redhat.com/show_bug.cgi?id=1637014
>> https://bugzilla.redhat.com/show_bug.cgi?id=1637030
>>
>> Any ways per debugging, as you rightfully mentioned docker.
>> As of Queens Cinder was containerized meaning:
>> log location: /var/log/container/cinder/cinder-volume.log
>> Config file path is :
>> /var/lib/config-data/puppet-generated/cinder/etc/cinder/
>> So nfs_shares should be found/reside under this path ^ path.
>>
>> However if you login to Cinder's volume docker
>> # docker ps | grep cinder somethign like
>> -> openstack-cinder-volume-docker-0
>> # docker exec -it openstack-cinder-volume-docker-0 /bin/bash
>> You should see cinder.conf plus the share file under /etc/cinder/
>> In side the docker /etc/cinder/ path is valid, outside of docker mapping
>> goes to /var/lib/config-data..
>>
>> I'd me more than happy to take a look at your volume log,
>> should you be willing to share it in public or private with me.
>>
>> Tzach
>>
>>
>>
>> On Sun, Oct 7, 2018 at 7:38 PM Cody <codeology.lab at gmail.com> wrote:
>>
>>> Hi Tzach,
>>>
>>> Thank you for getting back! I tested it again with
>>> CinderNfsMountOptions: 'rw,sync,nosharecache' in Queens, but still to
>>> no avail.
>>>
>>> I also noticed that the file /etc/cinder/nfs_shares does not exist on
>>> any controller, although in cinder.conf it has "#nfs_shares_config =
>>> /etc/cinder/nfs_shares". I am not sure if this is normal for using NFS
>>> with the containerized cinder service.
>>>
>>> Thank you,
>>> Cody
>>>
>>>
>>> On Sun, Oct 7, 2018 at 5:03 AM Tzach Shefi <tshefi at redhat.com> wrote:
>>> >
>>> >
>>> > Hey Cody,
>>> >
>>> > I recall hitting a related problem, when both Glance and Cinder use
>>> the same NFS server, while each service uses it's own share if both shares
>>> reside on same NFS server you may hit an selinux issue.
>>> >
>>> > The original bug I hit/reported, was closed EOL.
>>> > https://bugzilla.redhat.com/show_bug.cgi?id=1491597
>>> > Notice 4th comment, adding nosharecache mount option helped me.
>>> >
>>> > I'll re-check this on queens as well maybe need to rebug it.
>>> > Thanks
>>> >
>>> > Tzach
>>> >
>>> >
>>> > On Sun, Oct 7, 2018 at 6:32 AM Cody <codeology.lab at gmail.com> wrote:
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> I have an issue with using TripleO (Queens) to setup an NFS backend
>>> for Cinder.
>>> >>
>>> >> My storage.yaml is as follows:
>>> >>
>>> >> parameter_defaults:
>>> >> CinderEnableIscsiBackend: false
>>> >> CinderEnableRbdBackend: false
>>> >> CinderEnableNfsBackend: true
>>> >> NovaEnableRbdBackend: false
>>> >> GlanceBackend: 'file'
>>> >>
>>> >> CinderNfsMountOptions: 'rw,sync'
>>> >> CinderNfsServers: '192.168.24.1:/export/cinder'
>>> >>
>>> >> GlanceNfsEnabled: true
>>> >> GlanceNfsShare: '192.168.24.1:/export/glance'
>>> >> GlanceNfsOptions:
>>> 'rw,sync,context=system_u:object_r:glance_var_lib_t:s0'
>>> >>
>>> >> NovaNfsEnabled: true
>>> >> NovaNfsShare: '192.168.24.1:/export/nova'
>>> >> NovaNfsOptions: 'rw,sync,context=system_u:object_r:nfs_t:s0'
>>> >>
>>> >> I used the undercloud node as an NFS server for testing purposes.
>>> Iptables is set accordingly. The /etc/exportfs on the NFS server is as
>>> follows:
>>> >>
>>> >> /export/nova 192.168.24.0/24(rw,no_root_squash)
>>> >> /export/glance 192.168.24.0/24(rw,no_root_squash)
>>> >> /export/cinder 192.168.24.0/24(rw,no_root_squash)
>>> >>
>>> >> All three folders are set to chmod 777. Nova and Glance work as
>>> expected. Only Cinder remains problematic. I can try to upload volumes from
>>> overcloud, but nothing would show up in the cinder folder. Also Horizon
>>> gives errors like unable to retrieve volume and volume snapshots. Did I
>>> miss something here? I do plan to use Ceph later, but I wish to use NFS for
>>> now to test migration and failover. Any helps would be appreciated. Thank
>>> you!
>>> >>
>>> >> Best regards,
>>> >> Cody
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> dev mailing list
>>> >> dev at lists.rdoproject.org
>>> >> http://lists.rdoproject.org/mailman/listinfo/dev
>>> >>
>>> >> To unsubscribe: dev-unsubscribe at lists.rdoproject.org
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Tzach Shefi
>>> >
>>> > Senior Quality Engineer, RHCSA
>>> >
>>> > Red Hat
>>> >
>>> > tshefi at redaht.com M: +972-54-4701080 IM: tshefi
>>>
>>
>>
>> --
>>
>> Tzach Shefi
>>
>> Senior Quality Engineer, RHCSA
>>
>> Red Hat
>>
>> <https://www.redhat.com>
>>
>> tshefi at redaht.com M: +972-54-4701080 IM: tshefi
>> <https://red.ht/sig>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20181009/14a75ff0/attachment-0001.html>
More information about the dev
mailing list