11:28 a.m.
(In the spirit of "Concerning Hobbits")
Ryan O'Hara and I have been investigating RabbitMQ as it pertains to RDO
recently. There has been a lot of discussion on several disparate
threads, so I wanted to try and capture it on the list for the benefit
of everyone.
Ryan has been working on getting RabbitMQ running in a multi-node HA
configuration. I won't steal his thunder, and he can speak to it better
than I can, so I'll defer to him on the details.
As for me, I've been working on el7 support and bug squashing along the
way.
The first bug[1] causes the daemon to load incredibly slow, or outright
fail by timing out. This is due to the SELinux policy disallowing
name_bind on ports lower than 32768. RabbitMQ tries to name_bind to a
port starting at 10000, and increments if it fails. So if you have
SELinux in enforcing mode, you'll get 22768 AVC denials in the log
before it finally starts.
The second bug[2] causes the daemon to intermittently fail to start due
to a race condition in the creation of the erlang cookie file. This
happens only the first time the service starts. Really this is an
Erlang bug, but there's a workaround for the RabbitMQ case.
I've submitted patches for both issues. Until those get merged in, I've
rebuilt[3] RabbitMQ for F20 which includes the fixes.
Beyond bugs, I've also built out RabbitMQ and all the build/runtime
dependencies for el7. I have a yum repo[4] on my fedorapeople page
containing all the bits. This is all the stuff that is presently
missing from EPEL7. In time, I would hope the maintainers build all
this stuff, but for now it'll work for testing. You will also need the
EPEL 7 Beta repository[5] enabled.
As a side note, I built everything using mock with a local override repo
on my workstation. I've not used copr before but it seems relevant to
this sort of thing, so if it's any benefit I'll look to rebuilt the el7
stack there for easier consumption.
Hopefully this helps get the discussion into one place, and provide a
baseline for further investigation by everyone interested in RabbitMQ.
John.
---
[1] Is really two bugzillas, but the same bug:
[1a] https://bugzilla.redhat.com/show_bug.cgi?id=998682
[1b] https://bugzilla.redhat.com/show_bug.cgi?id=1032595
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1059913
[3] http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm
[4] http://jeckersb.fedorapeople.org/rabbitmq-el7/
[5] http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/
11:44 a.m.
On Tue, Feb 04, 2014 at 12:28:31PM -0500, John Eckersberg wrote:
(In the spirit of "Concerning Hobbits")
Thanks for kicking off this thread.
Ryan O'Hara and I have been investigating RabbitMQ as it pertains
to RDO
recently. There has been a lot of discussion on several disparate
threads, so I wanted to try and capture it on the list for the benefit
of everyone.
Ryan has been working on getting RabbitMQ running in a multi-node HA
configuration. I won't steal his thunder, and he can speak to it better
than I can, so I'll defer to him on the details.
Right now I have a 3-node RabbitMQ cluster with mirrored queues. I
also put haproxy in front of this cluster and pointed all relevant
OpenStack services at the virtual IP address. This seems to work well
so far. Details instructions coming soon.
As for me, I've been working on el7 support and bug squashing
along the
way.
The first bug[1] causes the daemon to load incredibly slow, or outright
fail by timing out. This is due to the SELinux policy disallowing
name_bind on ports lower than 32768. RabbitMQ tries to name_bind to a
port starting at 10000, and increments if it fails. So if you have
SELinux in enforcing mode, you'll get 22768 AVC denials in the log
before it finally starts.
The second bug[2] causes the daemon to intermittently fail to start due
to a race condition in the creation of the erlang cookie file. This
happens only the first time the service starts. Really this is an
Erlang bug, but there's a workaround for the RabbitMQ case.
I've submitted patches for both issues. Until those get merged in, I've
rebuilt[3] RabbitMQ for F20 which includes the fixes.
Awesome.
Beyond bugs, I've also built out RabbitMQ and all the
build/runtime
dependencies for el7. I have a yum repo[4] on my fedorapeople page
containing all the bits. This is all the stuff that is presently
missing from EPEL7. In time, I would hope the maintainers build all
this stuff, but for now it'll work for testing. You will also need the
EPEL 7 Beta repository[5] enabled.
As a side note, I built everything using mock with a local override repo
on my workstation. I've not used copr before but it seems relevant to
this sort of thing, so if it's any benefit I'll look to rebuilt the el7
stack there for easier consumption.
Hopefully this helps get the discussion into one place, and provide a
baseline for further investigation by everyone interested in RabbitMQ.
John and I will be putting all of this in a wiki page on the RDO
website in the very near future. I'll send email to the list when it
is ready to be reviewed.
Ryan
John.
---
[1] Is really two bugzillas, but the same bug:
[1a] https://bugzilla.redhat.com/show_bug.cgi?id=998682
[1b] https://bugzilla.redhat.com/show_bug.cgi?id=1032595
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1059913
[3] http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm
[4] http://jeckersb.fedorapeople.org/rabbitmq-el7/
[5] http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
10:20 a.m.
A new wiki page is available here:
http://openstack.redhat.com/RabbitMQ
On Tue, Feb 04, 2014 at 11:44:20AM -0600, Ryan O'Hara wrote:
On Tue, Feb 04, 2014 at 12:28:31PM -0500, John Eckersberg wrote:
> (In the spirit of "Concerning Hobbits")
Thanks for kicking off this thread.
> Ryan O'Hara and I have been investigating RabbitMQ as it pertains to RDO
> recently. There has been a lot of discussion on several disparate
> threads, so I wanted to try and capture it on the list for the benefit
> of everyone.
>
> Ryan has been working on getting RabbitMQ running in a multi-node HA
> configuration. I won't steal his thunder, and he can speak to it better
> than I can, so I'll defer to him on the details.
Right now I have a 3-node RabbitMQ cluster with mirrored queues. I
also put haproxy in front of this cluster and pointed all relevant
OpenStack services at the virtual IP address. This seems to work well
so far. Details instructions coming soon.
> As for me, I've been working on el7 support and bug squashing along the
> way.
>
> The first bug[1] causes the daemon to load incredibly slow, or outright
> fail by timing out. This is due to the SELinux policy disallowing
> name_bind on ports lower than 32768. RabbitMQ tries to name_bind to a
> port starting at 10000, and increments if it fails. So if you have
> SELinux in enforcing mode, you'll get 22768 AVC denials in the log
> before it finally starts.
>
> The second bug[2] causes the daemon to intermittently fail to start due
> to a race condition in the creation of the erlang cookie file. This
> happens only the first time the service starts. Really this is an
> Erlang bug, but there's a workaround for the RabbitMQ case.
>
> I've submitted patches for both issues. Until those get merged in, I've
> rebuilt[3] RabbitMQ for F20 which includes the fixes.
Awesome.
> Beyond bugs, I've also built out RabbitMQ and all the build/runtime
> dependencies for el7. I have a yum repo[4] on my fedorapeople page
> containing all the bits. This is all the stuff that is presently
> missing from EPEL7. In time, I would hope the maintainers build all
> this stuff, but for now it'll work for testing. You will also need the
> EPEL 7 Beta repository[5] enabled.
>
> As a side note, I built everything using mock with a local override repo
> on my workstation. I've not used copr before but it seems relevant to
> this sort of thing, so if it's any benefit I'll look to rebuilt the el7
> stack there for easier consumption.
>
> Hopefully this helps get the discussion into one place, and provide a
> baseline for further investigation by everyone interested in RabbitMQ.
John and I will be putting all of this in a wiki page on the RDO
website in the very near future. I'll send email to the list when it
is ready to be reviewed.
Ryan
> John.
>
> ---
> [1] Is really two bugzillas, but the same bug:
> [1a] https://bugzilla.redhat.com/show_bug.cgi?id=998682
> [1b] https://bugzilla.redhat.com/show_bug.cgi?id=1032595
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1059913
> [3] http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm
> [4] http://jeckersb.fedorapeople.org/rabbitmq-el7/
> [5] http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
7 p.m.
On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
A new wiki page is available here:
http://openstack.redhat.com/RabbitMQ
Excellent stuff guys :)
So... next step would be working with the Packstack folks to get
RabbitMQ as an option for basic Packstack configs (not HA stuff, just
vanilla RabbitMQ)
And then beyond that, I know there is some work going on around HA
deployments via Foreman. Probably we should make the HA RabbitMQ
configuration that Ryan outlined an option for Foreman deployments?
Perry
7:05 p.m.
On 02/06/2014 08:00 PM, Perry Myers wrote:
On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
> A new wiki page is available here:
>
> http://openstack.redhat.com/RabbitMQ
Excellent stuff guys :)
So... next step would be working with the Packstack folks to get
RabbitMQ as an option for basic Packstack configs (not HA stuff, just
vanilla RabbitMQ)
And then beyond that, I know there is some work going on around HA
deployments via Foreman. Probably we should make the HA RabbitMQ
configuration that Ryan outlined an option for Foreman deployments?
Perry
+1, great work guys!
On the packstack work, I'd like to ask if there is a significant amount
of non-puppet work needed to make the rabbitmq configuration happen
let's have some discussion around on the list about it.
We have hit a few cases where we things deviated between packstack and
foreman simply because of a fair bit of work happening outside puppet.
If we have to do some things outside of puppet both teams need to make
sure to allocate time to do the work (or find a reasonable way to share
non-puppet code).
Thanks,
Mike
7:08 p.m.
On 02/06/2014 08:05 PM, Mike Orazi wrote:
On 02/06/2014 08:00 PM, Perry Myers wrote:
> On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
>> A new wiki page is available here:
>>
>> http://openstack.redhat.com/RabbitMQ
>
> Excellent stuff guys :)
>
> So... next step would be working with the Packstack folks to get
> RabbitMQ as an option for basic Packstack configs (not HA stuff, just
> vanilla RabbitMQ)
>
> And then beyond that, I know there is some work going on around HA
> deployments via Foreman. Probably we should make the HA RabbitMQ
> configuration that Ryan outlined an option for Foreman deployments?
>
> Perry
>
+1, great work guys!
On the packstack work, I'd like to ask if there is a significant amount
of non-puppet work needed to make the rabbitmq configuration happen
let's have some discussion around on the list about it.
We have hit a few cases where we things deviated between packstack and
foreman simply because of a fair bit of work happening outside puppet.
If we have to do some things outside of puppet both teams need to make
sure to allocate time to do the work (or find a reasonable way to share
non-puppet code).
Good point. Packstack should have very little logic in it. Nominally
we should treat it as a way to apply puppet modules to machines.
Anything we implement directly in Packstack python code would need to be
re-implemented in Foreman or other places, so we should try to minimize
that duplication
8:17 p.m.
Mike Orazi <morazi(a)redhat.com> writes:
On the packstack work, I'd like to ask if there is a significant
amount
of non-puppet work needed to make the rabbitmq configuration happen
let's have some discussion around on the list about it.
I suspect this will be pretty standard, puppet-only updates. Install
package, enable service, maybe twiddle a config or two.
John.
11:10 p.m.
On Thu, Feb 06, 2014 at 08:00:37PM -0500, Perry Myers wrote:
On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
> A new wiki page is available here:
>
> http://openstack.redhat.com/RabbitMQ
Excellent stuff guys :)
So... next step would be working with the Packstack folks to get
RabbitMQ as an option for basic Packstack configs (not HA stuff, just
vanilla RabbitMQ)
I think the basic RabbitMQ (non-HA) option in Packstack should be very
straightforward. Have packstack give the option of qpid or rabbitmq
(qpid default). If RabbitMQ is chosen, allow for the host/address and
port to be specified.
And then beyond that, I know there is some work going on around HA
deployments via Foreman. Probably we should make the HA RabbitMQ
configuration that Ryan outlined an option for Foreman deployments?
Maybe. I was forwarded a message earlier this week about mirrored
queues being a bit unstable. I'd like to see more testing with RDO and
RabbitMQ w/ mirrored queues, but I also understand that having the
means to deploy it makes it easier to test.
Ryan
11:15 p.m.
On Thu, Feb 06, 2014 at 08:05:30PM -0500, Mike Orazi wrote:
On 02/06/2014 08:00 PM, Perry Myers wrote:
> On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
>> A new wiki page is available here:
>>
>> http://openstack.redhat.com/RabbitMQ
>
> Excellent stuff guys :)
>
> So... next step would be working with the Packstack folks to get
> RabbitMQ as an option for basic Packstack configs (not HA stuff, just
> vanilla RabbitMQ)
>
> And then beyond that, I know there is some work going on around HA
> deployments via Foreman. Probably we should make the HA RabbitMQ
> configuration that Ryan outlined an option for Foreman deployments?
>
> Perry
>
+1, great work guys!
On the packstack work, I'd like to ask if there is a significant amount
of non-puppet work needed to make the rabbitmq configuration happen
let's have some discussion around on the list about it.
The packstack guys can give an official answer, but I don't think it
would require much logic. I think John's original write-up showed that
you can drop RabbitMQ in and simply replace the rpc_driver parameter
in a handful of config files (glance being a slight deviation).
We have hit a few cases where we things deviated between packstack
and
foreman simply because of a fair bit of work happening outside puppet.
If we have to do some things outside of puppet both teams need to make
sure to allocate time to do the work (or find a reasonable way to share
non-puppet code).
The other bit of outstanding work is to fix a few bugs with
rabbitmq-server. John has a good handle on this, and I think we're
just waiting to get packages built.
We can start by getting the rabbitmq puppet modules into packstack and
foreman.
Ryan
4:49 a.m.
On 07 Feb 2014, at 06:15, Ryan O'Hara <rohara(a)redhat.com> wrote:
On Thu, Feb 06, 2014 at 08:05:30PM -0500, Mike Orazi wrote:
> On 02/06/2014 08:00 PM, Perry Myers wrote:
>> On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
>>> A new wiki page is available here:
>>>
>>> http://openstack.redhat.com/RabbitMQ
>>
>> Excellent stuff guys :)
>>
>> So... next step would be working with the Packstack folks to get
>> RabbitMQ as an option for basic Packstack configs (not HA stuff, just
>> vanilla RabbitMQ)
>>
>> And then beyond that, I know there is some work going on around HA
>> deployments via Foreman. Probably we should make the HA RabbitMQ
>> configuration that Ryan outlined an option for Foreman deployments?
>
> +1, great work guys!
>
> On the packstack work, I'd like to ask if there is a significant amount
> of non-puppet work needed to make the rabbitmq configuration happen
> let's have some discussion around on the list about it.
The packstack guys can give an official answer, but I don't think it
would require much logic. I think John's original write-up showed that
you can drop RabbitMQ in and simply replace the rpc_driver parameter
in a handful of config files (glance being a slight deviation).
Unless we find something unexpected, it should only require some parameter handling code
in Python. The configuration logic would live in the Puppet module.
Best,
Alvaro
8:27 a.m.
I just want to know how long the packstack can support RabbitMQ?
On Fri, Feb 7, 2014 at 6:49 PM, Alvaro Lopez Ortega <aortega(a)redhat.com>wrote:
On 07 Feb 2014, at 06:15, Ryan O'Hara <rohara(a)redhat.com>
wrote:
On Thu, Feb 06, 2014 at 08:05:30PM -0500, Mike Orazi wrote:
On 02/06/2014 08:00 PM, Perry Myers wrote:
On 02/06/2014 11:20 AM, Ryan O'Hara wrote:
A new wiki page is available here:
http://openstack.redhat.com/RabbitMQ
Excellent stuff guys :)
So... next step would be working with the Packstack folks to get
RabbitMQ as an option for basic Packstack configs (not HA stuff, just
vanilla RabbitMQ)
And then beyond that, I know there is some work going on around HA
deployments via Foreman. Probably we should make the HA RabbitMQ
configuration that Ryan outlined an option for Foreman deployments?
+1, great work guys!
On the packstack work, I'd like to ask if there is a significant amount
of non-puppet work needed to make the rabbitmq configuration happen
let's have some discussion around on the list about it.
The packstack guys can give an official answer, but I don't think it
would require much logic. I think John's original write-up showed that
you can drop RabbitMQ in and simply replace the rpc_driver parameter
in a handful of config files (glance being a slight deviation).
Unless we find something unexpected, it should only require some parameter
handling code in Python. The configuration logic would live in the Puppet
module.
Best,
Alvaro
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
--
Shake Chen
1:18 p.m.
On 02/07/2014 03:27 PM, Shake Chen wrote:
I just want to know how long the packstack can support RabbitMQ?
Packstack supports RabbitMQ since May. Moreover, it's been the default
AMQP since then.
The CONFIG_AMQP_BACKEND configuration option, and the --amqp-backend
parameter allow to choose between qpid and rabbitmq.
Best,
Alvaro
3459
days inactive
3727
days old
11 comments
6 participants
participants (6)
-
Alvaro Lopez Ortega -
John Eckersberg -
Mike Orazi -
Perry Myers -
Ryan O'Hara -
Shake Chen