Hi, Deployed Tripleo Liberty stable on baremetal, but NetApp NFS Cinder backend is not working. It is auto-mounting no problem, and I can write to it with sudo, but the 'tripleo_netapp' backend is enabled with state 'down' as it cannot write to the mount point. cinder service-list | grep tripleo_netapp > > cinder-volume | hostgroup@tripleo_netapp | nova | enabled | down [heat-admin@overcloud-controller-0 ~]$ mount | grep cinder [ip addr]:/[mount] on /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=t cp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=[ip addr],local_lock=none,addr=[ip addr]) I can write to it - [heat-admin@overcloud-controller-0 ~]$ sudo touch /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/test [heat-admin@overcloud-controller-0 ~]$ But Cinder cannot - /var/log/cinder/volume.log:2016-04-29 09:43:49.870 56696 ERROR cinder.volume.drivers.remotefs [req-99928048-2446-4967-99ba- 0e85c2ba5712 - - - - -] Failed to created Cinder secure environment indicator file: [Errno 13] Permission denied: '/var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/.cinderSecureEn vIndicator' So this look like an issue with the user that Cinder is using to write to the export (cinder?)? I have tried setting this option in cinder.conf, but it makes no difference nas_secure_file_operations = False "Allow network-attached storage systems to operate in a secure environment where root level access is not permitted. If set to False, access is as the root user and insecure. If set to True, access is not as root. If set to auto, a check is done to determine if this is a new installation: True is used if so, otherwise False. Default is auto" Any help appreciated Thanks Charles -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

Hi, Thanks for this. 1) Yes unlikely as root can write to it. 2) Already set to permissive. 3) When we set up our previous OSP6 (Juno) environment using the same NetApp storage system, only root had permission to write to the NetApp volume and all worked fine. When our storage team set up this volume, it was also as root (same settings as the last setup). I suspect that Cinder uid usage is now enforced. I will get the storage team to make the changes and see if this helps Regards Charles On 29/04/2016 11:49, Christopher Brown wrote: > Hi Charles, > > I had similar problems with a netapp deployment. Three possibilities to > check: > > 1. Security on the export shipped by default with a missing netmask on > the export so 0.0.0.0 should be 0.0.0.0/24 or whatever you want to > restrict to. Though as you can write with sudo probably not the issue. > > 2. SELinux - I wonder if you try temporarily running setenforce 0 and > re-mounting if it has the same problem? > > 3. Cinder and Glance exports should be created with their respective > UIDs as owner. I blogged about it here: > > https://chruz.wordpress.com/2016/03/31/openstack-and-clustered-data-ont > ap/ > > Hope some of this is helpful but if not would be glad to hear of > outcome. > > Regards > > On Fri, 2016-04-29 at 11:30 +0100, Charles Short wrote: >> Hi, >> >> Deployed Tripleo Liberty stable on baremetal, but NetApp NFS Cinder >> backend is not working. >> >> It is auto-mounting no problem, and I can write to it with sudo, but >> the >> 'tripleo_netapp' backend is enabled with state 'down' as it cannot >> write >> to the mount point. >> >> cinder service-list | grep tripleo_netapp >>> cinder-volume | hostgroup@tripleo_netapp | nova | enabled | down >> [heat-admin@overcloud-controller-0 ~]$ mount | grep cinder >> [ip addr]:/[mount] on >> /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f type nfs4 >> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=t >> cp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=[ip >> addr],local_lock=none,addr=[ip addr]) >> >> I can write to it - >> >> [heat-admin@overcloud-controller-0 ~]$ sudo touch >> /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/test >> [heat-admin@overcloud-controller-0 ~]$ >> >> But Cinder cannot - >> >> /var/log/cinder/volume.log:2016-04-29 09:43:49.870 56696 ERROR >> cinder.volume.drivers.remotefs [req-99928048-2446-4967-99ba- >> 0e85c2ba5712 >> - - - - -] Failed to created Cinder secure environment indicator >> file: >> [Errno 13] Permission denied: >> '/var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/.cinderSecureEn >> vIndicator' >> >> So this look like an issue with the user that Cinder is using to >> write >> to the export (cinder?)? >> >> I have tried setting this option in cinder.conf, but it makes no >> difference >> >> nas_secure_file_operations = False >> >> "Allow network-attached storage systems to operate in a secure >> environment where root level access is not permitted. If set to >> False, >> access is as the root user and insecure. If set to True, access is >> not >> as root. If set to auto, a check is done to determine if this is a >> new >> installation: True is used if so, otherwise False. Default is auto" >> >> Any help appreciated >> >> Thanks >> >> Charles >> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > -- > Regards, > > Christopher Brown > OpenStack Engineer > OCF plc > > Tel: +44 (0)114 257 2200 > Web: www.ocf.co.uk > Blog: blog.ocf.co.uk > Twitter: @ocfplc > > Please note, any emails relating to an OCF Support request must always > be sent to support(a)ocf.co.uk for a ticket number to be generated or > existing support ticket to be updated. Should this not be done then OCF > > cannot be held responsible for requests not dealt with in a timely > manner. > > OCF plc is a company registered in England and Wales. Registered number > > 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, > > 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield S35 > 2PG. > > If you have received this message in error, please notify us > immediately and remove it from your system.

Hi Charles, On Fri, 2016-04-29 at 14:32 +0100, Charles Short wrote: > ok applying specific uid/gid 165 to the NetApp volume solved the > permission error. > Cinder now successfully writes .cinderSecureEnvIndicator to the > export. Great stuff. > But I have a new error now and the service is still down... > > /var/log/cinder/volume.log:2016-04-29 13:20:24.004 3902 ERROR > cinder.volume.manager [req-a4544310-84c6-4602-a944-7efaee5ff90f - - - > - > -] Failed to initialize driver. > ... > /var/log/cinder/volume.log:2016-04-29 13:20:24.004 3902 ERROR > cinder.volume.manager raise NaApiError('Unexpected error') > /var/log/cinder/volume.log:2016-04-29 13:20:24.004 3902 ERROR > cinder.volume.manager NaApiError: NetApp API failed. Reason - > Unexpected > error:unknown > > Have you seen this one? No but maybe the conf might be pointing to the mgmt IP rather than the data IP? Are you using 7 mode or ontap? Feel free to post the conf file (redacting security stuff obviously) if you like. Also check authentication perhaps. > Charles > > > On 29/04/2016 12:40, Charles Short wrote: >> Hi, >> >> Thanks for this. >> >> 1) Yes unlikely as root can write to it. >> >> 2) Already set to permissive. >> >> 3) When we set up our previous OSP6 (Juno) environment using the >> same >> NetApp storage system, only root had permission to write to the >> NetApp >> volume and all worked fine. When our storage team set up this >> volume, >> it was also as root (same settings as the last setup). I suspect >> that >> Cinder uid usage is now enforced. I will get the storage team to >> make >> the changes and see if this helps >> >> Regards >> >> Charles >> >> >> On 29/04/2016 11:49, Christopher Brown wrote: >>> Hi Charles, >>> >>> I had similar problems with a netapp deployment. Three >>> possibilities to >>> check: >>> >>> 1. Security on the export shipped by default with a missing >>> netmask on >>> the export so 0.0.0.0 should be 0.0.0.0/24 or whatever you want >>> to >>> restrict to. Though as you can write with sudo probably not the >>> issue. >>> >>> 2. SELinux - I wonder if you try temporarily running setenforce 0 >>> and >>> re-mounting if it has the same problem? >>> >>> 3. Cinder and Glance exports should be created with their >>> respective >>> UIDs as owner. I blogged about it here: >>> >>> https://chruz.wordpress.com/2016/03/31/openstack-and-clustered-da >>> ta-ont >>> ap/ >>> >>> Hope some of this is helpful but if not would be glad to hear of >>> outcome. >>> >>> Regards >>> >>> On Fri, 2016-04-29 at 11:30 +0100, Charles Short wrote: >>>> Hi, >>>> >>>> Deployed Tripleo Liberty stable on baremetal, but NetApp NFS >>>> Cinder >>>> backend is not working. >>>> >>>> It is auto-mounting no problem, and I can write to it with >>>> sudo, but >>>> the >>>> 'tripleo_netapp' backend is enabled with state 'down' as it >>>> cannot >>>> write >>>> to the mount point. >>>> >>>> cinder service-list | grep tripleo_netapp >>>>> cinder-volume | hostgroup@tripleo_netapp | nova | enabled >>>>> | down >>>> [heat-admin@overcloud-controller-0 ~]$ mount | grep cinder >>>> [ip addr]:/[mount] on >>>> /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f type nfs4 >>>> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,p >>>> roto=t >>>> cp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=[ip >>>> addr],local_lock=none,addr=[ip addr]) >>>> >>>> I can write to it - >>>> >>>> [heat-admin@overcloud-controller-0 ~]$ sudo touch >>>> /var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/test >>>> [heat-admin@overcloud-controller-0 ~]$ >>>> >>>> But Cinder cannot - >>>> >>>> /var/log/cinder/volume.log:2016-04-29 09:43:49.870 56696 ERROR >>>> cinder.volume.drivers.remotefs [req-99928048-2446-4967-99ba- >>>> 0e85c2ba5712 >>>> - - - - -] Failed to created Cinder secure environment >>>> indicator >>>> file: >>>> [Errno 13] Permission denied: >>>> '/var/lib/cinder/mnt/3fb6f6744c383eacbe46593911aa4b0f/.cinderSe >>>> cureEn >>>> vIndicator' >>>> >>>> So this look like an issue with the user that Cinder is using >>>> to >>>> write >>>> to the export (cinder?)? >>>> >>>> I have tried setting this option in cinder.conf, but it makes >>>> no >>>> difference >>>> >>>> nas_secure_file_operations = False >>>> >>>> "Allow network-attached storage systems to operate in a secure >>>> environment where root level access is not permitted. If set to >>>> False, >>>> access is as the root user and insecure. If set to True, access >>>> is >>>> not >>>> as root. If set to auto, a check is done to determine if this >>>> is a >>>> new >>>> installation: True is used if so, otherwise False. Default is >>>> auto" >>>> >>>> Any help appreciated >>>> >>>> Thanks >>>> >>>> Charles >>>> >>>> -- >>>> Charles Short >>>> Cloud Engineer >>>> Virtualization and Cloud Team >>>> European Bioinformatics Institute (EMBL-EBI) >>>> Tel: +44 (0)1223 494205 >>>> >>>> _______________________________________________ >>>> Rdo-list mailing list >>>> Rdo-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> -- >>> Regards, >>> >>> Christopher Brown >>> OpenStack Engineer >>> OCF plc >>> >>> Tel: +44 (0)114 257 2200 >>> Web: www.ocf.co.uk >>> Blog: blog.ocf.co.uk >>> Twitter: @ocfplc >>> >>> Please note, any emails relating to an OCF Support request must >>> always >>> be sent to support(a)ocf.co.uk for a ticket number to be generated >>> or >>> existing support ticket to be updated. Should this not be done >>> then OCF >>> >>> cannot be held responsible for requests not dealt with in a >>> timely >>> manner. >>> >>> OCF plc is a company registered in England and Wales. Registered >>> number >>> >>> 4132533, VAT number GB 780 6803 14. Registered office address: >>> OCF plc, >>> >>> 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, >>> Sheffield S35 >>> 2PG. >>> >>> If you have received this message in error, please notify us >>> immediately and remove it from your system. > -- > Charles Short > Cloud Engineer > Virtualization and Cloud Team > European Bioinformatics Institute (EMBL-EBI) > Tel: +44 (0)1223 494205 > -- Regards, Christopher Brown OpenStack Engineer OCF plc Tel: +44 (0)114 257 2200 Web: www.ocf.co.uk Blog: blog.ocf.co.uk Twitter: @ocfplc Please note, any emails relating to an OCF Support request must always be sent to support(a)ocf.co.uk for a ticket number to be generated or existing support ticket to be updated. Should this not be done then OCF cannot be held responsible for requests not dealt with in a timely manner. OCF plc is a company registered in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield S35 2PG. If you have received this message in error, please notify us immediately and remove it from your system.