[Rdo-list] Permission denied errors after installing a Storage Node in a Swift Cluster

Fri May 16 13:21:15 UTC 2014

On 05/16/2014 12:30 PM, Diogo Vieira wrote:
> On May 15, 2014, at 6:05 PM, Diogo Vieira <dfv at eurotux.com <mailto:dfv at eurotux.com>> wrote:
> 
>> On May 6, 2014, at 7:35 PM, Pádraig Brady <pbrady at redhat.com <mailto:pbrady at redhat.com>> wrote:
>>
>>> On 05/06/2014 06:47 PM, Diogo Vieira wrote:
>>>> On May 6, 2014, at 11:06 AM, Pádraig Brady <pbrady at redhat.com <mailto:pbrady at redhat.com>> wrote:
>>>>
>>>>> On 05/06/2014 09:47 AM, Diogo Vieira wrote:
>>>>>> On May 6, 2014, at 2:53 AM, Pádraig Brady <pbrady at redhat.com <mailto:pbrady at redhat.com> <mailto:pbrady at redhat.com>> wrote:
>>>>>>
>>>>>>> What version of swift are you using?
>>>>>>>
>>>>>>> swift-1.13.1.rc2 could have permissions errors,
>>>>>>> while we included a patch in the RDO icehouse swift-1.13.1 release to fix
>>>>>>> http://pad.lv/1302700 which on first glance could be related?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Pádraig.
>>>>>>
>>>>>> I'm using 1.13.1-1.fc21 (I'm using Fedora) as you can see:
>>>>>>
>>>>>>  # yum info openstack-swift
>>>>>>  Loaded plugins: priorities
>>>>>>  196 packages excluded due to repository priority protections
>>>>>>  Installed Packages
>>>>>>  Name        : openstack-swift
>>>>>>  Arch        : noarch
>>>>>>  Version     : 1.13.1
>>>>>>  Release     : 1.fc21
>>>>>>
>>>>>>
>>>>>> So the fix should already be present right?
>>>>>
>>>>> Yes, must be something else so.
>>>>>
>>>>> thanks,
>>>>> Pádraig.
>>>>
>>>> That's unfortunate then. One thing's for sure: these errors aren't supposed to happen right?
>>>>
>>>> If someone else has any idea of what could be the problem I would greatly appreciate since this is a recurring problem (even between different Openstack  and Packstack versions, since it was tested in Havana and Icehouse).
>>>>
>>>> Thank you very much,
>>>> Diogo Vieira
>>>>
>>>
>>> Ah you see this in Havana repos also, that's NB info.
>>>
>>> Pete any ideas?
>>>
>>> thanks,
>>> Pádraig.
>>
>> Yes, the issue happened with an older Havana installation and with the new Icehouse version.
>>
>> Sorry to bring this up again, but I'm really lost and have no idea what could the problem be.
>>
>> Nobody has an idea for trying to resolve the issue or issues (since I don't know if they're related)? Should I file a bug report?
>>
>> Thank you,
>>
>> Diogo Vieira <dfv at eurotux.com <mailto:dfv at eurotux.com>>
>> Programador
>> Eurotux Informática, S.A. | www.eurotux.com <http://www.eurotux.com>
>> (t) +351 253 680 300
> 
> Hello again,
> 
> I'm sorry for answering my own email but I found something I believe to be the problem. After some time trying to see what the problem was I ended up searching for what could cause the permission problems. This was one of the errors I found in the syslog:
> 
>     May 16 10:07:00 host-10-10-6-30 object-auditor: ERROR Trying to audit /srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/obj/auditor.py", line 173, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/site-packages/swift/obj/auditor.py", line 191, in object_audit#012 with df.open():#012 File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 1029, in open#012 data_file, meta_file)#012 File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 1247, in _construct_from_data_file#012 fp = open(data_file, 'rb')#012IOError: [Errno 13] Permission denied: '/srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a/1399288004.47239.data'
> 
> 
> So the problems was in '/srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a/1399288004.47239.data'. I tried the following set of commands after I made sure that the swift-object-auditor was run by the swift user (using 'ps faxu'):
> 
>     # su - swift
> 
>     -bash-4.2$ ls -lha /srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a
>     ls: cannot access /srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a/.: Permission denied
>     ls: cannot access /srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a/..: Permission denied
>     ls: cannot access /srv/node/device3/objects/134106/f2a/82f6a3461bb69f80918a1a508a8bdf2a/1399288004.47239.data: Permission denied
>     total 0
>     d????????? ? ? ? ?            ? .
>     d????????? ? ? ? ?            ? ..
>     ?????????? ? ? ? ?            ? 1399288004.47239.data
> 
>     -bash-4.2$ ls -lha /srv/node/device3/objects/134106/f2a/
>     total 0
>     drwxr--r-- 3 swift swift 45 May  5 11:06 .
>     drwxr-xr-x 3 swift swift 45 May 16 10:57 ..
>     drw-r--r-- 2 swift swift 34 May  5 11:06 82f6a3461bb69f80918a1a508a8bdf2a
> 
> 
> So you see '82f6a3461bb69f80918a1a508a8bdf2a' doesn't have execute permissions and I believe that is the problem.
> 
> My question now is, could this be caused by some misconfiguration on my part or is it a bug in swift, since it was not me that created the folder?

I'm not sure.

You might have changed after inadvertently with chmod,
or beforehand with umask.

The problematic drw-r--r-- permission above is consistent
with a umask of 0133 (unlikely) or a chmod of 0644.
Now there was a recent patch to chmod(0644): http://pad.lv/1302700
Though I don't think that's the cause as it should only be for the gz _files_

Not much help.

sorry,
Pádraig.