<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I created an Icehouse cluster that uses Cinder for object storage (using the NFS driver). I use an Isilon NL cluster and NFS mount to that. I’m getting these errors concerning “Circular Directory Structure” and I can’t create volumes because
of it. From the commands it looks like “snapshots” are supposed to be ignored but apparently they are not.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any suggestions would be greatly appreciated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you,<o:p></o:p></p>
<p class="MsoNormal">Richard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Text from /var/log/cinder/volume.log:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.035 9485 AUDIT cinder.service [-] Starting cinder-volume node (version 2014.1.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.036 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Starting volume driver NfsDriver (1.1.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.329 9485 INFO cinder.volume.manager [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Updating volume status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.334 9485 INFO cinder.brick.remotefs.remotefs [req-36376eb6-d919-4451-9c83-a2bae72c85c5 - - - - -] Already mounted: /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 ERROR cinder.openstack.common.threadgroup [-] Unexpected error while running command.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Exit code: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory
is part of the cycle:\n `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup x.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return self.thread.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return self._exit_event.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return hubs.get_hub().switch()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return self.greenlet.switch()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup result = function(*args, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/openstack/common/service.py", line 486, in run_service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup service.start()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/service.py", line 103, in start<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup self.manager.init_host()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 308, in init_host<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup self.publish_service_capabilities(ctxt)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1106, in publish_service_capabilities<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup self._report_driver_status(context)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 1095, in _report_driver_status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup volume_stats = self.driver.get_volume_stats(refresh=True)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 340, in get_volume_stats<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup self._update_volume_stats()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 359, in _update_volume_stats<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup capacity, free, used = self._get_capacity_info(share)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/nfs.py", line 567, in _get_capacity_info<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup '*snapshot*', mount_point, run_as_root=True)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 136, in execute<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup return processutils.execute(*cmd, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup cmd=' '.join(cmd))<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Exit code: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stdout: '26\t/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03\n'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup Stderr: "/usr/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted
file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n `/var/lib/cinder/mnt/ade98462e07fef2a453d3328ce54ac03/.snapshot'\n\n"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.428 9485 TRACE cinder.openstack.common.threadgroup<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Child 9485 exited with status 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2014-08-12 15:24:12.439 2662 INFO cinder.openstack.common.service [-] Forking too fast, sleeping<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><u><span style="font-size:14.0pt;color:#990000">Richard Minton<o:p></o:p></span></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Lockheed Martin - D&IS<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">LMICC Systems Administrator<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">4000 Geerdes Blvd, 13D31<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">King of Prussia, PA 19406<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Phone: 610-354-5482<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>