2:33 p.m.
I created an Icehouse cluster that uses Cinder for object storage (using the NFS driver).
I use an Isilon NL cluster and NFS mount to that. I'm getting these errors concerning
"Circular Directory Structure" and I can't create volumes because of it.
From the commands it looks like "snapshots" are supposed to be ignored but
apparently they are not.
Any suggestions would be greatly appreciated.
Thank you,
Richard
Text from /var/log/cinder/volume.log:
2014-08-12 15:24:12.035 9485 AUDIT cinder.service [-] Starting cinder-volume node (version
2014.1.1)
2014-08-12 15:24:12.036 9485 INFO cinder.volume.manager
[req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Starting volume driver NfsDriver
(1.1.0)
2014-08-12 15:24:12.329 9485 INFO cinder.volume.manager
[req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Updating volume status
2014-08-12 15:24:12.334 9485 INFO cinder.brick.remotefs.remotefs
[req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Already mounted:
/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
2014-08-12 15:24:12.428 9485 ERROR cinder.openstack.common.threadgroup [-] Unexpected
error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude
*snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
Exit code: 1
Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'
Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly
means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following
directory is part of the cycle:\n
`/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Traceback (most
recent call last):
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line
125, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup x.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line
47, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return
self.thread.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return
self._exit_event.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return
hubs.get_hub().switch()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return
self.greenlet.switch()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup result =
function(*args, **kwargs)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/service.py", line 486,
in run_service
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
service.start()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/service.py", line 103, in start
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
self.manager.init_host()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 308, in
init_host
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
self.publish_service_capabilities(ctxt)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1106, in
publish_service_capabilities
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
self._report_driver_status(context)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1095, in
_report_driver_status
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup volume_stats =
self.driver.get_volume_stats(refresh=True)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 340, in
get_volume_stats
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
self._update_volume_stats()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 359, in
_update_volume_stats
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup capacity, free,
used = self._get_capacity_info(share)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 567, in
_get_capacity_info
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
'*snapshot*', mount_point, run_as_root=True)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/utils.py", line 136, in execute
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return
processutils.execute(*cmd, **kwargs)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line
173, in execute
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup cmd='
'.join(cmd))
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
ProcessExecutionError: Unexpected error while running command.
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Command: sudo
cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot*
/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Exit code: 1
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stdout:
'26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stderr:
"/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means
that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following
directory is part of the cycle:\n
`/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Child 9485 exited
with status 0
2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Forking too fast,
sleeping
Richard Minton
Lockheed Martin - D&IS
LMICC Systems Administrator
4000 Geerdes Blvd, 13D31
King of Prussia, PA 19406
Phone: 610-354-5482