10:46 a.m.
Hi everybody,
I am searching for a good and useful method to monitor my 40 nodes cloud.
I have tried
- Prometheus + Grafana (with
https://github.com/openstack-exporter/openstack-exporter) but it cannot
monitor nodes load and cpu usage etc.
and
- Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
and make the whole cloud completely unusable!
I've tried to use Graphite + Grafana but I failed.
Do you have any suggestions?
P.S. Where can I find good documentation for TripleO over CentOS? Redhat's
is somehow restricted to its own RHOSP. This website is down:
http://tripleo.org/
Thanks,
Khodayar
11:09 a.m.
Have you taken a look at the Prometheus Node Exporter? (
https://github.com/prometheus/node_exporter) It will expose things like
load average and disk usage for each node. These statistics can then be
scraped into Prometheus and graphed with Grafana.
Michael.
On Thu, Oct 22, 2020 at 11:46 AM Khodayar Doustar <khodayard(a)gmail.com>
wrote:
Hi everybody,
I am searching for a good and useful method to monitor my 40 nodes cloud.
I have tried
- Prometheus + Grafana (with
https://github.com/openstack-exporter/openstack-exporter) but it cannot
monitor nodes load and cpu usage etc.
and
- Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
and make the whole cloud completely unusable!
I've tried to use Graphite + Grafana but I failed.
Do you have any suggestions?
P.S. Where can I find good documentation for TripleO over CentOS? Redhat's
is somehow restricted to its own RHOSP. This website is down:
http://tripleo.org/
Thanks,
Khodayar
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
7:46 p.m.
There is no magic here, unfortunately.
Most implementations of monitoring will mean you have some agent running on
the computes providing metrics to a central system. You could have
node_exporter + Cadvisor exporting compute metrics + container metrics
towards a central Prometheus server. You can then plug Grafana into
Prometheus and have your centralized "single pane of glass" monitoring. It
will require some work to create the dashboards and arrange all the
visualization but I'm not aware of magical solutions that are free.
On Thu, Oct 22, 2020 at 12:29 PM Khodayar Doustar <khodayard(a)gmail.com>
wrote:
Yes, I’ve seen it. It’s totally simple and good, but I’d rather
something
which could be integrated/deployed with TripleO so that:
- We won’t be forced to deploy it on each node separately.
- We will be able to monitor all of cloud “as a cloud” and all of
nodes together in a single page or program.
Node exporter is good but only for individual OSs.
Khodayar
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
*From: *Michael Solberg <msolberg(a)redhat.com>
*Sent: *Donnerstag, 22. Oktober 2020 18:10
*To: *users(a)lists.rdoproject.org
*Subject: *Re: [rdo-users] TripleO Monitoring Tool/Method
Have you taken a look at the Prometheus Node Exporter? (
https://github.com/prometheus/node_exporter) It will expose things like
load average and disk usage for each node. These statistics can then be
scraped into Prometheus and graphed with Grafana.
Michael.
On Thu, Oct 22, 2020 at 11:46 AM Khodayar Doustar <khodayard(a)gmail.com>
wrote:
Hi everybody,
I am searching for a good and useful method to monitor my 40 nodes cloud.
I have tried
- Prometheus + Grafana (with
https://github.com/openstack-exporter/openstack-exporter) but it cannot
monitor nodes load and cpu usage etc.
and
- Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
and make the whole cloud completely unusable!
I've tried to use Graphite + Grafana but I failed.
Do you have any suggestions?
P.S. Where can I find good documentation for TripleO over CentOS? Redhat's
is somehow restricted to its own RHOSP. This website is down:
http://tripleo.org/
Thanks,
Khodayar
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
3:34 a.m.
Laurent,
Thanks for your answer. I will look into cAdvisor, it seems really
interesting.
On Fri, Oct 23, 2020 at 2:46 AM Laurent Dumont <laurentfdumont(a)gmail.com>
wrote:
There is no magic here, unfortunately.
Most implementations of monitoring will mean you have some agent running
on the computes providing metrics to a central system. You could have
node_exporter + Cadvisor exporting compute metrics + container metrics
towards a central Prometheus server. You can then plug Grafana into
Prometheus and have your centralized "single pane of glass" monitoring. It
will require some work to create the dashboards and arrange all the
visualization but I'm not aware of magical solutions that are free.
On Thu, Oct 22, 2020 at 12:29 PM Khodayar Doustar <khodayard(a)gmail.com>
wrote:
> Yes, I’ve seen it. It’s totally simple and good, but I’d rather something
> which could be integrated/deployed with TripleO so that:
>
> - We won’t be forced to deploy it on each node separately.
> - We will be able to monitor all of cloud “as a cloud” and all of
> nodes together in a single page or program.
>
> Node exporter is good but only for individual OSs.
>
>
>
> Khodayar
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Michael Solberg <msolberg(a)redhat.com>
> *Sent: *Donnerstag, 22. Oktober 2020 18:10
> *To: *users(a)lists.rdoproject.org
> *Subject: *Re: [rdo-users] TripleO Monitoring Tool/Method
>
>
>
> Have you taken a look at the Prometheus Node Exporter? (
> https://github.com/prometheus/node_exporter) It will expose things like
> load average and disk usage for each node. These statistics can then be
> scraped into Prometheus and graphed with Grafana.
>
>
>
> Michael.
>
>
>
>
>
> On Thu, Oct 22, 2020 at 11:46 AM Khodayar Doustar <khodayard(a)gmail.com>
> wrote:
>
> Hi everybody,
>
>
>
> I am searching for a good and useful method to monitor my 40 nodes cloud.
>
>
>
> I have tried
>
>
>
> - Prometheus + Grafana (with
> https://github.com/openstack-exporter/openstack-exporter) but it cannot
> monitor nodes load and cpu usage etc.
>
> and
>
> - Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
> and make the whole cloud completely unusable!
>
>
>
> I've tried to use Graphite + Grafana but I failed.
>
>
>
> Do you have any suggestions?
>
>
>
> P.S. Where can I find good documentation for TripleO over CentOS?
> Redhat's is somehow restricted to its own RHOSP. This website is down:
> http://tripleo.org/
>
>
>
> Thanks,
>
> Khodayar
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
>
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
>
>
>
>
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
>
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
>
3 a.m.
On 22/10/2020 17:46, Khodayar Doustar wrote:
Hi everybody,
I am searching for a good and useful method to monitor my 40 nodes cloud.
I have tried
- Prometheus + Grafana (with
https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>) but it
cannot monitor nodes load and cpu usage etc.
and
- Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
and make the whole cloud completely unusable!
I've tried to use Graphite + Grafana but I failed.
Do you have any suggestions?
Hi,
yes, I have some opinions here.
My proposal here is:
- use collectd to collect low level metrics from your baremetal machines
- use ceilometer to collect OpenStack related info, like project usage,
etc. That is nothing you'd get by using node-exporter
- hook them both together and send metrics over to something called
Service Telemetry Framework. The configuration *is* included in tripleo.
The website has documentation available
https://infrawatch.github.io/documentation
- graphite + grafana (plus collectd) is also a single node setup and
won't provide you reliability.
- collectd also provides the ability to send events, which can be acted
on. That is not included if you use node-exporter, openstack-exporter
etc. Prometheus monitoring creates events from metrics, but will be slow
to detect failed components.
Since prometheus is meant to be single server, there is no HA per se in
prometheus. That makes handling prometheus on standalone machines a bit
awkward, or you'd have a infrastructure taking care of that.
In your tests with gnocchi, collectd and grafana, I bet you used swift
as backend for gnocchi storage. That is not a good idea and may lead to
bad performance.
Matthias
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
4:09 a.m.
Matthias,
Thanks a lot for your answer.
Yes, you win the bet :) I've used swift and currently struggling to disable
collectd to make my cloud usable again! :))
I've seen this STF (Service Telemetry Framework) but it seems a little bit
too complicated. I should implement an OKD cluster to monitor my openstack,
isn't it too much work?
Have you tried it yourself?
If I understand correctly, with your first and main opinion you mean adding
this files to my overcloud deploy command:
/usr/share/openstack-tripleo-heat-templates/environments/enable-legacy-telemetry.yaml
/usr/share/openstack-tripleo-heat-templates/environments/services/collectd.yaml
and for performance tuning I've checked this page:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/...
Is that what you mean?
If so I should make my cloud usable again and just change GnocchiBackend to
a path to a file on a shared file system (i.e. NFS) because I have 4
controller nodes, because the rest is exactly what I've done up to now.
Thanks a lot,
Khodayar
On Fri, Oct 23, 2020 at 10:01 AM Matthias Runge <mrunge(a)redhat.com> wrote:
On 22/10/2020 17:46, Khodayar Doustar wrote:
> Hi everybody,
>
> I am searching for a good and useful method to monitor my 40 nodes cloud.
>
> I have tried
>
> - Prometheus + Grafana (with
> https://github.com/openstack-exporter/openstack-exporter
> <https://github.com/openstack-exporter/openstack-exporter>) but it
> cannot monitor nodes load and cpu usage etc.
> and
> - Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
> and make the whole cloud completely unusable!
>
> I've tried to use Graphite + Grafana but I failed.
>
> Do you have any suggestions?
Hi,
yes, I have some opinions here.
My proposal here is:
- use collectd to collect low level metrics from your baremetal machines
- use ceilometer to collect OpenStack related info, like project usage,
etc. That is nothing you'd get by using node-exporter
- hook them both together and send metrics over to something called
Service Telemetry Framework. The configuration *is* included in tripleo.
The website has documentation available
https://infrawatch.github.io/documentation
- graphite + grafana (plus collectd) is also a single node setup and
won't provide you reliability.
- collectd also provides the ability to send events, which can be acted
on. That is not included if you use node-exporter, openstack-exporter
etc. Prometheus monitoring creates events from metrics, but will be slow
to detect failed components.
Since prometheus is meant to be single server, there is no HA per se in
prometheus. That makes handling prometheus on standalone machines a bit
awkward, or you'd have a infrastructure taking care of that.
In your tests with gnocchi, collectd and grafana, I bet you used swift
as backend for gnocchi storage. That is not a good idea and may lead to
bad performance.
Matthias
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
8:36 a.m.
Hi,
yes of course I'm using STF, and it's not complicated.
It's always a good idea to separate your monitoring stack from the
monitored infrastructure. How would you know your stack is down, if
notifications are also sent from that stack?
With the tripleo-heat-templates you linked, you basically enable legacy
telemetry (ceilometer, aodh, gnocchi).
If you are running 40 computes, that is not a small stack anymore. I
would suggest (recommend) to use ceph as backend.
Also, depending on your use-case and your settings (for collectd) you
may want to lower the interval, the parameter is
CollectdDefaultPollingInterval, I have set it here to something like 5
secs, but in your case, I would suggest to use 600 (same as for Ceilometer).
Matthias
On 23/10/2020 11:09, Khodayar Doustar wrote:
Matthias,
Thanks a lot for your answer.
Yes, you win the bet :) I've used swift and currently struggling to
disable collectd to make my cloud usable again! :))
I've seen this STF (Service Telemetry Framework) but it seems a little
bit too complicated. I should implement an OKD cluster to monitor my
openstack, isn't it too much work?
Have you tried it yourself?
If I understand correctly, with your first and main opinion you mean
adding this files to my overcloud deploy command:
/usr/share/openstack-tripleo-heat-templates/environments/enable-legacy-telemetry.yaml
/usr/share/openstack-tripleo-heat-templates/environments/services/collectd.yaml
and for performance tuning I've checked this page:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/...
<https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/...
Is that what you mean?
If so I should make my cloud usable again and just change GnocchiBackend
to a path to a file on a shared file system (i.e. NFS) because I have 4
controller nodes, because the rest is exactly what I've done up to now.
Thanks a lot,
Khodayar
On Fri, Oct 23, 2020 at 10:01 AM Matthias Runge <mrunge(a)redhat.com
<mailto:mrunge@redhat.com>> wrote:
On 22/10/2020 17:46, Khodayar Doustar wrote:
> Hi everybody,
>
> I am searching for a good and useful method to monitor my 40 nodes
cloud.
>
> I have tried
>
> - Prometheus + Grafana (with
> https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>
> <https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>>) but it
> cannot monitor nodes load and cpu usage etc.
> and
> - Gnocchi +Collectd + Grafana but it enforces unbelievable load on
nodes
> and make the whole cloud completely unusable!
>
> I've tried to use Graphite + Grafana but I failed.
>
> Do you have any suggestions?
Hi,
yes, I have some opinions here.
My proposal here is:
- use collectd to collect low level metrics from your baremetal machines
- use ceilometer to collect OpenStack related info, like project usage,
etc. That is nothing you'd get by using node-exporter
- hook them both together and send metrics over to something called
Service Telemetry Framework. The configuration *is* included in tripleo.
The website has documentation available
https://infrawatch.github.io/documentation
<https://infrawatch.github.io/documentation>
- graphite + grafana (plus collectd) is also a single node setup and
won't provide you reliability.
- collectd also provides the ability to send events, which can be acted
on. That is not included if you use node-exporter, openstack-exporter
etc. Prometheus monitoring creates events from metrics, but will be slow
to detect failed components.
Since prometheus is meant to be single server, there is no HA per se in
prometheus. That makes handling prometheus on standalone machines a bit
awkward, or you'd have a infrastructure taking care of that.
In your tests with gnocchi, collectd and grafana, I bet you used swift
as backend for gnocchi storage. That is not a good idea and may lead to
bad performance.
Matthias
--
Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
http://lists.rdoproject.org/mailman/listinfo/users
<http://lists.rdoproject.org/mailman/listinfo/users>
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
<mailto:users-unsubscribe@lists.rdoproject.org>
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
10:47 a.m.
What do you mean by collectd and events? Is there a way to make collected
collect Rabbitmq messages/events and forward them?
On Fri, Oct 23, 2020 at 4:01 AM Matthias Runge <mrunge(a)redhat.com> wrote:
On 22/10/2020 17:46, Khodayar Doustar wrote:
> Hi everybody,
>
> I am searching for a good and useful method to monitor my 40 nodes cloud.
>
> I have tried
>
> - Prometheus + Grafana (with
> https://github.com/openstack-exporter/openstack-exporter
> <https://github.com/openstack-exporter/openstack-exporter>) but it
> cannot monitor nodes load and cpu usage etc.
> and
> - Gnocchi +Collectd + Grafana but it enforces unbelievable load on nodes
> and make the whole cloud completely unusable!
>
> I've tried to use Graphite + Grafana but I failed.
>
> Do you have any suggestions?
Hi,
yes, I have some opinions here.
My proposal here is:
- use collectd to collect low level metrics from your baremetal machines
- use ceilometer to collect OpenStack related info, like project usage,
etc. That is nothing you'd get by using node-exporter
- hook them both together and send metrics over to something called
Service Telemetry Framework. The configuration *is* included in tripleo.
The website has documentation available
https://infrawatch.github.io/documentation
- graphite + grafana (plus collectd) is also a single node setup and
won't provide you reliability.
- collectd also provides the ability to send events, which can be acted
on. That is not included if you use node-exporter, openstack-exporter
etc. Prometheus monitoring creates events from metrics, but will be slow
to detect failed components.
Since prometheus is meant to be single server, there is no HA per se in
prometheus. That makes handling prometheus on standalone machines a bit
awkward, or you'd have a infrastructure taking care of that.
In your tests with gnocchi, collectd and grafana, I bet you used swift
as backend for gnocchi storage. That is not a good idea and may lead to
bad performance.
Matthias
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
12:58 p.m.
On 23/10/2020 17:47, Laurent Dumont wrote:
What do you mean by collectd and events? Is there a way to make
collected collect Rabbitmq messages/events and forward them?
Collectd does not connect to rabbitmq (in this setup).
collectd (as collecting agents) collects metrics, but also events. An
event can be: process "nova" died, interface "eth0" went down.
These events are not handled by prometheus (as that is a metrics store).
ceilometer also creates/forwards metrics and events. In STF, this data
(metrics and events, from, both sources collectd and ceilometer) are
delivered over via amqp1 to prometheus (for metrics) and elasticsearch
(for events).
Does that answer your question?
Matthias
On Fri, Oct 23, 2020 at 4:01 AM Matthias Runge <mrunge(a)redhat.com
<mailto:mrunge@redhat.com>> wrote:
On 22/10/2020 17:46, Khodayar Doustar wrote:
> Hi everybody,
>
> I am searching for a good and useful method to monitor my 40 nodes
cloud.
>
> I have tried
>
> - Prometheus + Grafana (with
> https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>
> <https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>>) but it
> cannot monitor nodes load and cpu usage etc.
> and
> - Gnocchi +Collectd + Grafana but it enforces unbelievable load on
nodes
> and make the whole cloud completely unusable!
>
> I've tried to use Graphite + Grafana but I failed.
>
> Do you have any suggestions?
Hi,
yes, I have some opinions here.
My proposal here is:
- use collectd to collect low level metrics from your baremetal machines
- use ceilometer to collect OpenStack related info, like project usage,
etc. That is nothing you'd get by using node-exporter
- hook them both together and send metrics over to something called
Service Telemetry Framework. The configuration *is* included in tripleo.
The website has documentation available
https://infrawatch.github.io/documentation
<https://infrawatch.github.io/documentation>
- graphite + grafana (plus collectd) is also a single node setup and
won't provide you reliability.
- collectd also provides the ability to send events, which can be acted
on. That is not included if you use node-exporter, openstack-exporter
etc. Prometheus monitoring creates events from metrics, but will be slow
to detect failed components.
Since prometheus is meant to be single server, there is no HA per se in
prometheus. That makes handling prometheus on standalone machines a bit
awkward, or you'd have a infrastructure taking care of that.
In your tests with gnocchi, collectd and grafana, I bet you used swift
as backend for gnocchi storage. That is not a good idea and may lead to
bad performance.
Matthias
--
Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
http://lists.rdoproject.org/mailman/listinfo/users
<http://lists.rdoproject.org/mailman/listinfo/users>
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
<mailto:users-unsubscribe@lists.rdoproject.org>
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
1:05 p.m.
I guess I'm still not super clear on the Framework for feeding internal
Openstack events.
This is something that we are actively discussing with RH as part of my
$dayjob. We basically want to get some real-time visibility into Openstack
events (nova, neutron, cinder, glance) and are looking into ways to feed
our tooling from Rabbitmq notifications. That specific approach does not
seem to be supported and we were told that it was risky and not supported.
The notification queue does not seem to be present in OSP13 and most
documentation detail how to remove it when you are on <13.
We already have performance metrics (for computes though Collectd +
Cadvisor) that are fed to our own Prometheus.
With the STF, is there a way to limit our integration to Ceilometer and the
event forwarding system? And for ceilometer itself, what is actually
driving the feed of Openstack events to Ceilometer itself?
I'm open to running the STF as a whole, but I'd rather pick what I really
need - which is visibility into internal Openstack events (VM start, stop,
migrate, port create, delete and so forth)
Thanks!
On Fri, Oct 23, 2020 at 1:58 PM Matthias Runge <mrunge(a)redhat.com> wrote:
On 23/10/2020 17:47, Laurent Dumont wrote:
> What do you mean by collectd and events? Is there a way to make
> collected collect Rabbitmq messages/events and forward them?
>
Collectd does not connect to rabbitmq (in this setup).
collectd (as collecting agents) collects metrics, but also events. An
event can be: process "nova" died, interface "eth0" went down.
These events are not handled by prometheus (as that is a metrics store).
ceilometer also creates/forwards metrics and events. In STF, this data
(metrics and events, from, both sources collectd and ceilometer) are
delivered over via amqp1 to prometheus (for metrics) and elasticsearch
(for events).
Does that answer your question?
Matthias
> On Fri, Oct 23, 2020 at 4:01 AM Matthias Runge <mrunge(a)redhat.com
> <mailto:mrunge@redhat.com>> wrote:
>
> On 22/10/2020 17:46, Khodayar Doustar wrote:
> > Hi everybody,
> >
> > I am searching for a good and useful method to monitor my 40 nodes
> cloud.
> >
> > I have tried
> >
> > - Prometheus + Grafana (with
> > https://github.com/openstack-exporter/openstack-exporter
> <https://github.com/openstack-exporter/openstack-exporter>
> > <https://github.com/openstack-exporter/openstack-exporter
> <https://github.com/openstack-exporter/openstack-exporter>>) but it
> > cannot monitor nodes load and cpu usage etc.
> > and
> > - Gnocchi +Collectd + Grafana but it enforces unbelievable load on
> nodes
> > and make the whole cloud completely unusable!
> >
> > I've tried to use Graphite + Grafana but I failed.
> >
> > Do you have any suggestions?
>
>
> Hi,
>
> yes, I have some opinions here.
>
> My proposal here is:
>
> - use collectd to collect low level metrics from your baremetal
machines
> - use ceilometer to collect OpenStack related info, like project
usage,
> etc. That is nothing you'd get by using node-exporter
> - hook them both together and send metrics over to something called
> Service Telemetry Framework. The configuration *is* included in
tripleo.
> The website has documentation available
> https://infrawatch.github.io/documentation
> <https://infrawatch.github.io/documentation>
> - graphite + grafana (plus collectd) is also a single node setup and
> won't provide you reliability.
> - collectd also provides the ability to send events, which can be
acted
> on. That is not included if you use node-exporter, openstack-exporter
> etc. Prometheus monitoring creates events from metrics, but will be
slow
> to detect failed components.
>
> Since prometheus is meant to be single server, there is no HA per se
in
> prometheus. That makes handling prometheus on standalone machines a
bit
> awkward, or you'd have a infrastructure taking care of that.
>
> In your tests with gnocchi, collectd and grafana, I bet you used
swift
> as backend for gnocchi storage. That is not a good idea and may lead
to
> bad performance.
>
> Matthias
>
> --
> Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
>
> Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
> Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
> O'Neil
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
> http://lists.rdoproject.org/mailman/listinfo/users
> <http://lists.rdoproject.org/mailman/listinfo/users>
>
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
> <mailto:users-unsubscribe@lists.rdoproject.org>
>
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
5:18 a.m.
Hi,
that question is different.
In traditional telemetry, events are stored in panko. Depending on your
workload, you probably don't want this. That's why you'll always find
docs on how to disable it (if you have issues, you'll be looking for
solutions around panko)
ceilometer is listening to the message bus and emits events e.g to
panko, but that is also configurable to something else, e.g to stf.
In OSP 16.1 and sending data over to STF, ceilometer sends events only
over to STF, nowhere else (but that can be configured).
E.g there is a config option to publish events via amqp1[1],[2]
cadvisor and prometheus are for sending/collecting/storing metrics.
While you can also generate events from metrics (via rules, sent out via
alertmanager), this is nowhere real time.
Matthias
[1]
https://github.com/openstack/tripleo-heat-templates/blob/master/environme...
[2]
https://github.com/openstack/tripleo-heat-templates/blob/master/environme...
On 23/10/2020 20:05, Laurent Dumont wrote:
I guess I'm still not super clear on the Framework for feeding
internal
Openstack events.
This is something that we are actively discussing with RH as part of my
$dayjob. We basically want to get some real-time visibility into
Openstack events (nova, neutron, cinder, glance) and are looking into
ways to feed our tooling from Rabbitmq notifications. That specific
approach does not seem to be supported and we were told that it was
risky and not supported. The notification queue does not seem to be
present in OSP13 and most documentation detail how to remove it when you
are on <13.
We already have performance metrics (for computes though Collectd +
Cadvisor) that are fed to our own Prometheus.
With the STF, is there a way to limit our integration to Ceilometer and
the event forwarding system? And for ceilometer itself, what is actually
driving the feed of Openstack events to Ceilometer itself?
I'm open to running the STF as a whole, but I'd rather pick what I
really need - which is visibility into internal Openstack events (VM
start, stop, migrate, port create, delete and so forth)
Thanks!
On Fri, Oct 23, 2020 at 1:58 PM Matthias Runge <mrunge(a)redhat.com
<mailto:mrunge@redhat.com>> wrote:
On 23/10/2020 17:47, Laurent Dumont wrote:
> What do you mean by collectd and events? Is there a way to make
> collected collect Rabbitmq messages/events and forward them?
>
Collectd does not connect to rabbitmq (in this setup).
collectd (as collecting agents) collects metrics, but also events. An
event can be: process "nova" died, interface "eth0" went down.
These events are not handled by prometheus (as that is a metrics store).
ceilometer also creates/forwards metrics and events. In STF, this data
(metrics and events, from, both sources collectd and ceilometer) are
delivered over via amqp1 to prometheus (for metrics) and elasticsearch
(for events).
Does that answer your question?
Matthias
> On Fri, Oct 23, 2020 at 4:01 AM Matthias Runge <mrunge(a)redhat.com
<mailto:mrunge@redhat.com>
> <mailto:mrunge@redhat.com <mailto:mrunge@redhat.com>>> wrote:
>
> On 22/10/2020 17:46, Khodayar Doustar wrote:
> > Hi everybody,
> >
> > I am searching for a good and useful method to monitor my 40
nodes
> cloud.
> >
> > I have tried
> >
> > - Prometheus + Grafana (with
> > https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>
> <https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>>
> > <https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>
> <https://github.com/openstack-exporter/openstack-exporter
<https://github.com/openstack-exporter/openstack-exporter>>>) but it
> > cannot monitor nodes load and cpu usage etc.
> > and
> > - Gnocchi +Collectd + Grafana but it enforces unbelievable
load on
> nodes
> > and make the whole cloud completely unusable!
> >
> > I've tried to use Graphite + Grafana but I failed.
> >
> > Do you have any suggestions?
>
>
> Hi,
>
> yes, I have some opinions here.
>
> My proposal here is:
>
> - use collectd to collect low level metrics from your
baremetal machines
> - use ceilometer to collect OpenStack related info, like
project usage,
> etc. That is nothing you'd get by using node-exporter
> - hook them both together and send metrics over to something
called
> Service Telemetry Framework. The configuration *is* included
in tripleo.
> The website has documentation available
> https://infrawatch.github.io/documentation
<https://infrawatch.github.io/documentation>
> <https://infrawatch.github.io/documentation
<https://infrawatch.github.io/documentation>>
> - graphite + grafana (plus collectd) is also a single node
setup and
> won't provide you reliability.
> - collectd also provides the ability to send events, which can
be acted
> on. That is not included if you use node-exporter,
openstack-exporter
> etc. Prometheus monitoring creates events from metrics, but
will be slow
> to detect failed components.
>
> Since prometheus is meant to be single server, there is no HA
per se in
> prometheus. That makes handling prometheus on standalone
machines a bit
> awkward, or you'd have a infrastructure taking care of that.
>
> In your tests with gnocchi, collectd and grafana, I bet you
used swift
> as backend for gnocchi storage. That is not a good idea and
may lead to
> bad performance.
>
> Matthias
>
> --
> Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>
<mailto:mrunge@redhat.com <mailto:mrunge@redhat.com>>>
>
> Red Hat GmbH, http://www.de.redhat.com/
<http://www.de.redhat.com/> <http://www.de.redhat.com/
<http://www.de.redhat.com/>>;,
> Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
> O'Neil
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
<mailto:users@lists.rdoproject.org <mailto:users@lists.rdoproject.org>>
> http://lists.rdoproject.org/mailman/listinfo/users
<http://lists.rdoproject.org/mailman/listinfo/users>
> <http://lists.rdoproject.org/mailman/listinfo/users
<http://lists.rdoproject.org/mailman/listinfo/users>>
>
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
<mailto:users-unsubscribe@lists.rdoproject.org>
> <mailto:users-unsubscribe@lists.rdoproject.org
<mailto:users-unsubscribe@lists.rdoproject.org>>
>
--
Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neil
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
4:29 a.m.
New subject: TripleO - Renew all containers
On 26/10/2020 22:27, Khodayar Doustar wrote:
Hi,
Maybe you remember me from the last thread:
I have enable legacy telemetry services and my cloud got overloaded and
down.
I have tried to undo all I’ve done with no luck. Load average is lower
but gnocchi and ceilometer containers are still there, but the main
problem is timeout on various services/endpoints.
You need to undefine these services in tripleo-heat-templates,
comparable to [1] and also remove the containers.
Matthias
[1]
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
7:47 a.m.
New subject: TripleO - Renew all containers
Matthias,
I've done that like this:
openstack overcloud deploy --templates -r .......... -e
/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml
but it didn't change anything.
I can remove the containers manually, but:
1. I thought it should remove the container itself when you undo something,
isn't that right?
2. The main problem is that the cloud is down, CLI works but we cannot
access any machine etc. We cannot login to horizon and every 5 minutes we
have this error on all of our controllers:
haproxy[830994]: proxy cinder has no server available!
As I've checked this queue has no consumer in RabbitMQ:
/]# rabbitmqctl list_queues | awk '$2 > 0'
Listing queues
cinder-scheduler_fanout_d94a6e6429db48848ca49d04ce5f4d6b 11277
Thanks,
On Tue, Oct 27, 2020 at 10:29 AM Matthias Runge <mrunge(a)redhat.com> wrote:
On 26/10/2020 22:27, Khodayar Doustar wrote:
> Hi,
>
>
>
> Maybe you remember me from the last thread:
>
> I have enable legacy telemetry services and my cloud got overloaded and
> down.
>
> I have tried to undo all I’ve done with no luck. Load average is lower
> but gnocchi and ceilometer containers are still there, but the main
> problem is timeout on various services/endpoints.
You need to undefine these services in tripleo-heat-templates,
comparable to [1] and also remove the containers.
Matthias
[1]
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
11:07 a.m.
New subject: TripleO - Renew all containers
On 27/10/2020 13:47, Khodayar Doustar wrote:
Matthias,
I've done that like this:
openstack overcloud deploy --templates -r ..........
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml
but it didn't change anything.
I can remove the containers manually, but:
1. I thought it should remove the container itself when you undo
something, isn't that right?
I've been told, it's not.
You need explicitly to describe what you want the installer to do.
If you'd remove the collectd containers, you should get the cloud in a
usable state quickly, since that won't overwhelm gnocchi anymore.
Again, using traditional telemetry in such a large deployment is not a
good idea. Use ceph as gnocchi backend, if you really need gnocchi.
Matthias
2. The main problem is that the cloud is down, CLI works but we
cannot
access any machine etc. We cannot login to horizon and every 5 minutes
we have this error on all of our controllers:
haproxy[830994]: proxy cinder has no server available!
As I've checked this queue has no consumer in RabbitMQ:
/]# rabbitmqctl list_queues | awk '$2 > 0'
Listing queues
cinder-scheduler_fanout_d94a6e6429db48848ca49d04ce5f4d6b11277
Thanks,
On Tue, Oct 27, 2020 at 10:29 AM Matthias Runge <mrunge(a)redhat.com
<mailto:mrunge@redhat.com>> wrote:
On 26/10/2020 22:27, Khodayar Doustar wrote:
> Hi,
>
>
>
> Maybe you remember me from the last thread:
>
> I have enable legacy telemetry services and my cloud got
overloaded and
> down.
>
> I have tried to undo all I’ve done with no luck. Load average is lower
> but gnocchi and ceilometer containers are still there, but the main
> problem is timeout on various services/endpoints.
You need to undefine these services in tripleo-heat-templates,
comparable to [1] and also remove the containers.
Matthias
[1]
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
<https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
--
Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neil
_______________________________________________
users mailing list
users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
http://lists.rdoproject.org/mailman/listinfo/users
<http://lists.rdoproject.org/mailman/listinfo/users>
To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
<mailto:users-unsubscribe@lists.rdoproject.org>
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
11:25 a.m.
New subject: TripleO - Renew all containers
I've manually removed all gnocchi, aodh, collectd and ceilometer containers.
The Load average was low, but the problem was RabbitMQ, I've manually
stopped all RabbitMQ containers and started them again and the cloud got
back on track.
Yes, you are totally right, I am planning to change Gnocchi backend to file
before enabling it again.
Thanks a lot,
Khodayar
On Tue, Oct 27, 2020 at 5:07 PM Matthias Runge <mrunge(a)redhat.com> wrote:
On 27/10/2020 13:47, Khodayar Doustar wrote:
> Matthias,
>
> I've done that like this:
>
> openstack overcloud deploy --templates -r ..........
>
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml
>
> but it didn't change anything.
> I can remove the containers manually, but:
>
> 1. I thought it should remove the container itself when you undo
> something, isn't that right?
I've been told, it's not.
You need explicitly to describe what you want the installer to do.
If you'd remove the collectd containers, you should get the cloud in a
usable state quickly, since that won't overwhelm gnocchi anymore.
Again, using traditional telemetry in such a large deployment is not a
good idea. Use ceph as gnocchi backend, if you really need gnocchi.
Matthias
> 2. The main problem is that the cloud is down, CLI works but we cannot
> access any machine etc. We cannot login to horizon and every 5 minutes
> we have this error on all of our controllers:
>
> haproxy[830994]: proxy cinder has no server available!
>
> As I've checked this queue has no consumer in RabbitMQ:
>
> /]# rabbitmqctl list_queues | awk '$2 > 0'
> Listing queues
> cinder-scheduler_fanout_d94a6e6429db48848ca49d04ce5f4d6b11277
>
> Thanks,
>
> On Tue, Oct 27, 2020 at 10:29 AM Matthias Runge <mrunge(a)redhat.com
> <mailto:mrunge@redhat.com>> wrote:
>
> On 26/10/2020 22:27, Khodayar Doustar wrote:
> > Hi,
> >
> >
> >
> > Maybe you remember me from the last thread:
> >
> > I have enable legacy telemetry services and my cloud got
> overloaded and
> > down.
> >
> > I have tried to undo all I’ve done with no luck. Load average is
lower
> > but gnocchi and ceilometer containers are still there, but the main
> > problem is timeout on various services/endpoints.
>
> You need to undefine these services in tripleo-heat-templates,
> comparable to [1] and also remove the containers.
>
> Matthias
>
> [1]
>
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
> <
https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/en...
>
>
> --
> Matthias Runge <mrunge(a)redhat.com <mailto:mrunge@redhat.com>>
>
> Red Hat GmbH, http://www.de.redhat.com/ <http://www.de.redhat.com/>;,
> Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
> O'Neil
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org <mailto:users@lists.rdoproject.org>
> http://lists.rdoproject.org/mailman/listinfo/users
> <http://lists.rdoproject.org/mailman/listinfo/users>
>
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org
> <mailto:users-unsubscribe@lists.rdoproject.org>
>
--
Matthias Runge <mrunge(a)redhat.com>
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Man.Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neil
1277
days inactive
1282
days old
16 comments
4 participants
participants (4)
-
Khodayar Doustar -
Laurent Dumont -
Matthias Runge -
Michael Solberg