I created an Icehouse cluster that uses Cinder for object storage (using the NFS driver). I use an Isilon NL cluster and NFS mount to that. I’m getting these errors concerning “Circular Directory Structure” and I can’t create volumes because of it. From the commands it looks like “snapshots” are supposed to be ignored but apparently they are not.

Any suggestions would be greatly appreciated.

Thank you,

Richard

Text from /var/log/cinder/volume.log:

2014-08-12 15:24:12.035 9485 AUDIT cinder.service [-] Starting cinder-volume node (version 2014.1.1)

2014-08-12 15:24:12.036 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Starting volume driver NfsDriver (1.1.0)

2014-08-12 15:24:12.329 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Updating volume status

2014-08-12 15:24:12.334 9485 INFO cinder.brick.remotefs.remotefs [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Already mounted: /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03

2014-08-12 15:24:12.428 9485 ERROR cinder.openstack.common.threadgroup [-] Unexpected error while running command.

Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03

Exit code: 1

Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'

Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n  `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     x.wait()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self.thread.wait()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self._exit_event.wait()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return hubs.get_hub().switch()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self.greenlet.switch()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     result = function(*args, **kwargs)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/service.py", line 486, in run_service

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     service.start()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/service.py", line 103, in start

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self.manager.init_host()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 308, in init_host

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self.publish_service_capabilities(ctxt)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1106, in publish_service_capabilities

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self._report_driver_status(context)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1095, in _report_driver_status

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     volume_stats = self.driver.get_volume_stats(refresh=True)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 340, in get_volume_stats

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self._update_volume_stats()

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 359, in _update_volume_stats

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     capacity, free, used = self._get_capacity_info(share)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 567, in _get_capacity_info

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     '*snapshot*', mount_point, run_as_root=True)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 136, in execute

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     cmd=' '.join(cmd))

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Exit code: 1

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n  `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"

2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup

2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Child 9485 exited with status 0

2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Forking too fast, sleeping

Richard Minton

Lockheed Martin - D&IS

LMICC Systems Administrator

4000 Geerdes Blvd, 13D31

King of Prussia, PA 19406

Phone: 610-354-5482