<div dir="ltr">I've manually removed all gnocchi, aodh, collectd and ceilometer containers.<div>The Load average was low, but the problem was RabbitMQ, I've manually stopped all RabbitMQ containers and started them again and the cloud got back on track.</div><div><br></div><div>Yes, you are totally right, I am planning to change Gnocchi backend to file before enabling it again.</div><div><br></div><div>Thanks a lot,</div><div>Khodayar</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 27, 2020 at 5:07 PM Matthias Runge <<a href="mailto:mrunge@redhat.com">mrunge@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 27/10/2020 13:47, Khodayar Doustar wrote:<br>
> Matthias,<br>
> <br>
> I've done that like this:<br>
> <br>
> openstack overcloud deploy --templates -r ..........<br>
> -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml<br>
> <br>
> but it didn't change anything.<br>
> I can remove the containers manually, but:<br>
> <br>
> 1. I thought it should remove the container itself when you undo<br>
> something, isn't that right?<br>
<br>
I've been told, it's not.<br>
You need explicitly to describe what you want the installer to do.<br>
<br>
If you'd remove the collectd containers, you should get the cloud in a<br>
usable state quickly, since that won't overwhelm gnocchi anymore.<br>
Again, using traditional telemetry in such a large deployment is not a<br>
good idea. Use ceph as gnocchi backend, if you really need gnocchi.<br>
<br>
Matthias<br>
<br>
<br>
> 2. The main problem is that the cloud is down, CLI works but we cannot<br>
> access any machine etc. We cannot login to horizon and every 5 minutes<br>
> we have this error on all of our controllers:<br>
> <br>
> haproxy[830994]: proxy cinder has no server available!<br>
> <br>
> As I've checked this queue has no consumer in RabbitMQ:<br>
> <br>
> /]# rabbitmqctl list_queues | awk '$2 > 0'<br>
> Listing queues<br>
> cinder-scheduler_fanout_d94a6e6429db48848ca49d04ce5f4d6b11277<br>
> <br>
> Thanks,<br>
> <br>
> On Tue, Oct 27, 2020 at 10:29 AM Matthias Runge <<a href="mailto:mrunge@redhat.com" target="_blank">mrunge@redhat.com</a><br>
> <mailto:<a href="mailto:mrunge@redhat.com" target="_blank">mrunge@redhat.com</a>>> wrote:<br>
> <br>
> On 26/10/2020 22:27, Khodayar Doustar wrote:<br>
> > Hi,<br>
> ><br>
> > <br>
> ><br>
> > Maybe you remember me from the last thread:<br>
> ><br>
> > I have enable legacy telemetry services and my cloud got<br>
> overloaded and<br>
> > down.<br>
> ><br>
> > I have tried to undo all I’ve done with no luck. Load average is lower<br>
> > but gnocchi and ceilometer containers are still there, but the main<br>
> > problem is timeout on various services/endpoints.<br>
> <br>
> You need to undefine these services in tripleo-heat-templates,<br>
> comparable to [1] and also remove the containers.<br>
> <br>
> Matthias<br>
> <br>
> [1]<br>
> <a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/disable-telemetry.yaml" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/disable-telemetry.yaml</a><br>
> <<a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/disable-telemetry.yaml" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/disable-telemetry.yaml</a>><br>
> <br>
> -- <br>
> Matthias Runge <<a href="mailto:mrunge@redhat.com" target="_blank">mrunge@redhat.com</a> <mailto:<a href="mailto:mrunge@redhat.com" target="_blank">mrunge@redhat.com</a>>><br>
> <br>
> Red Hat GmbH, <a href="http://www.de.redhat.com/" rel="noreferrer" target="_blank">http://www.de.redhat.com/</a> <<a href="http://www.de.redhat.com/" rel="noreferrer" target="_blank">http://www.de.redhat.com/</a>>,<br>
> Registered seat: Grasbrunn,<br>
> Commercial register: Amtsgericht Muenchen, HRB 153243,<br>
> Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael<br>
> O'Neil<br>
> <br>
> _______________________________________________<br>
> users mailing list<br>
> <a href="mailto:users@lists.rdoproject.org" target="_blank">users@lists.rdoproject.org</a> <mailto:<a href="mailto:users@lists.rdoproject.org" target="_blank">users@lists.rdoproject.org</a>><br>
> <a href="http://lists.rdoproject.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/users</a><br>
> <<a href="http://lists.rdoproject.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/users</a>><br>
> <br>
> To unsubscribe: <a href="mailto:users-unsubscribe@lists.rdoproject.org" target="_blank">users-unsubscribe@lists.rdoproject.org</a><br>
> <mailto:<a href="mailto:users-unsubscribe@lists.rdoproject.org" target="_blank">users-unsubscribe@lists.rdoproject.org</a>><br>
> <br>
<br>
<br>
-- <br>
Matthias Runge <<a href="mailto:mrunge@redhat.com" target="_blank">mrunge@redhat.com</a>><br>
<br>
Red Hat GmbH, <a href="http://www.de.redhat.com/" rel="noreferrer" target="_blank">http://www.de.redhat.com/</a>, Registered seat: Grasbrunn,<br>
Commercial register: Amtsgericht Muenchen, HRB 153243,<br>
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil<br>
<br>
</blockquote></div>