[Rdo-list] Cinder Problem with Circular directory structure.

Minton, Rich rich.minton at lmco.com
Tue Aug 12 19:33:56 UTC 2014 

I created an Icehouse cluster that uses Cinder for object storage (using the NFS driver). I use an Isilon NL cluster and NFS mount to that. I'm getting these errors concerning "Circular Directory Structure" and I can't create volumes because of it. From the commands it looks like "snapshots" are supposed to be ignored but apparently they are not.

Any suggestions would be greatly appreciated.

Thank you,
Richard

Text from /var/log/cinder/volume.log:

2014-08-12 15:24:12.035 9485 AUDIT cinder.service [-] Starting cinder-volume node (version 2014.1.1)
2014-08-12 15:24:12.036 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Starting volume driver NfsDriver (1.1.0)
2014-08-12 15:24:12.329 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Updating volume status
2014-08-12 15:24:12.334 9485 INFO cinder.brick.remotefs.remotefs [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Already mounted: /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
2014-08-12 15:24:12.428 9485 ERROR cinder.openstack.common.threadgroup [-] Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
Exit code: 1
Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'
Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n  `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     x.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self.thread.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self._exit_event.wait()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return self.greenlet.switch()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     service.start()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/service.py", line 103, in start
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self.manager.init_host()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 308, in init_host
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self.publish_service_capabilities(ctxt)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1106, in publish_service_capabilities
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self._report_driver_status(context)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1095, in _report_driver_status
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     volume_stats = self.driver.get_volume_stats(refresh=True)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 340, in get_volume_stats
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     self._update_volume_stats()
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 359, in _update_volume_stats
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     capacity, free, used = self._get_capacity_info(share)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 567, in _get_capacity_info
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     '*snapshot*', mount_point, run_as_root=True)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 136, in execute
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup     cmd=' '.join(cmd))
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Exit code: 1
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n  `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"
2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup
2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Child 9485 exited with status 0
2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Forking too fast, sleeping

Richard Minton
Lockheed Martin - D&IS
LMICC Systems Administrator
4000 Geerdes Blvd, 13D31
King of Prussia, PA 19406
Phone: 610-354-5482

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20140812/64497174/attachment.html>

More information about the dev mailing list