[Rdo-list] Concerning Rabbits
by John Eckersberg
(In the spirit of "Concerning Hobbits")
Ryan O'Hara and I have been investigating RabbitMQ as it pertains to RDO
recently. There has been a lot of discussion on several disparate
threads, so I wanted to try and capture it on the list for the benefit
of everyone.
Ryan has been working on getting RabbitMQ running in a multi-node HA
configuration. I won't steal his thunder, and he can speak to it better
than I can, so I'll defer to him on the details.
As for me, I've been working on el7 support and bug squashing along the
way.
The first bug[1] causes the daemon to load incredibly slow, or outright
fail by timing out. This is due to the SELinux policy disallowing
name_bind on ports lower than 32768. RabbitMQ tries to name_bind to a
port starting at 10000, and increments if it fails. So if you have
SELinux in enforcing mode, you'll get 22768 AVC denials in the log
before it finally starts.
The second bug[2] causes the daemon to intermittently fail to start due
to a race condition in the creation of the erlang cookie file. This
happens only the first time the service starts. Really this is an
Erlang bug, but there's a workaround for the RabbitMQ case.
I've submitted patches for both issues. Until those get merged in, I've
rebuilt[3] RabbitMQ for F20 which includes the fixes.
Beyond bugs, I've also built out RabbitMQ and all the build/runtime
dependencies for el7. I have a yum repo[4] on my fedorapeople page
containing all the bits. This is all the stuff that is presently
missing from EPEL7. In time, I would hope the maintainers build all
this stuff, but for now it'll work for testing. You will also need the
EPEL 7 Beta repository[5] enabled.
As a side note, I built everything using mock with a local override repo
on my workstation. I've not used copr before but it seems relevant to
this sort of thing, so if it's any benefit I'll look to rebuilt the el7
stack there for easier consumption.
Hopefully this helps get the discussion into one place, and provide a
baseline for further investigation by everyone interested in RabbitMQ.
John.
---
[1] Is really two bugzillas, but the same bug:
[1a] https://bugzilla.redhat.com/show_bug.cgi?id=998682
[1b] https://bugzilla.redhat.com/show_bug.cgi?id=1032595
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1059913
[3] http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm
[4] http://jeckersb.fedorapeople.org/rabbitmq-el7/
[5] http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/
8 years, 7 months
[Rdo-list] [OFI] Recap from last week
by Jason Rist
A bunch of us got together in Brno, CZ and cranked through some of the
core needs for the OpenStack Foreman Installer, AKA Staypuft!
Some big progress:
1.) Using 'wicked' gem, we got much of the wizard tied together,
creating underlying relevant OpenStack hostgroups, made each step tie
with one another, and created a relevant deployment model upon
completion of the 3rd step. This also includes listing deployments
(currently only 1), and the ability to delete a deployment and
associated hostgroups.
2.) mtaylor, mhulan and pchalupa made huge progress on the DynFlow
orchestration of the OpenStack deployment, to the point of successfully
launching a (mis-configured) OpenStack deployment that was able to be
opened in a browser.
3.) Big demo that we did on Friday showing our progress.
If interested, I can post the DynFlow video (ogv) somewhere.
Thanks from the Staypuft team,
Jason
--
Jason E. Rist
Senior Software Engineer
OpenStack Management UI
Red Hat, Inc.
openuc: +1.972.707.6408
mobile: +1.720.256.3933
Freenode: jrist
github/identi.ca: knowncitizen
9 years, 1 month
Re: [Rdo-list] Firewall issue/error when spawning instances on compute node
by Andrew Lau
I opened a BZ here https://bugzilla.redhat.com/show_bug.cgi?id=1082188
Did you also run into this issue too,
https://bugzilla.redhat.com/show_bug.cgi?id=1082187
On Sat, Mar 29, 2014 at 2:00 AM, St. George, Allan L. <
ALLAN.L.ST.GEORGE(a)leidos.com> wrote:
> I can confirm that I have the same issue on the hypervisor/nova-compute
> node that is hosting the instance...
>
> Chain neutron-openvswi-sd7933aba-f (1 references)
> num target prot opt source destination
> 1 RETURN all -- 10.0.0.6 0.0.0.0/0 MAC
> FA:16:3E:67:64:A4
> 2 DROP all -- 0.0.0.0/0 0.0.0.0/0
>
>
> How do we get someone to look at the problem for patching? This
> obviously wasn't identified and was carried over to the new build.
>
> - Allan
>
> ------------------------------
>
> Hi,
>
> I saw this issue too, I was just about to report it.
>
> If I understand correctly, this is because of the openvswitch iptables
> rules which are created (for security groups?)
>
> `service iptables status`
> ...
> Chain neutron-openvswi-s0ec3eb58-0 (1 references)
> num target prot opt source destination
> 1 RETURN all -- 10.0.0.12 0.0.0.0/0 MAC
> FA:16:3E:2E:B7:E3
> 2 DROP all -- 0.0.0.0/0 0.0.0.0/0
> ....
>
> In your case, the MAC address is different -- FA:16:3E:67:64:A4
>
> This issue also appears on icehouse w/ foreman, so it looks like it may
> be the puppet modules at fault here
>
> Andrew.
>
>
> On Fri, Mar 28, 2014 at 7:37 AM, St. George, Allan L. <
> ALLAN.L.ST.GEORGE(a)leidos.com> wrote:
>
>> Currently running RDO/Havana deployed via Foreman on a multi-compute
>> node stack (Controller, Neutron, and three Nova-Compute servers)
>>
>>
>>
>> When spawning an instance, it correctly spawns and reports/registers to
>> the Foreman dashboard.
>>
>>
>>
>> The problem is that the hypervisor/compute-node that is hosting the
>> instance will then begin to report:
>>
>>
>>
>> *Level*
>>
>> *Resource*
>>
>> *message*
>>
>> *err*
>>
>> Puppet
>>
>> Could not prefetch firewall provider 'iptables': Invalid address from
>> IPAddr.new: FA:16:3E:67:64:A4
>>
>> *err*
>>
>> /Firewall[001 nova compute incoming]
>>
>> Could not evaluate: Invalid address from IPAddr.new: FA:16:3E:67:64:A4
>>
>> *err*
>>
>> /Firewall[002 vxlan udp]
>>
>> Could not evaluate: Invalid address from IPAddr.new: FA:16:3E:67:64:A4
>>
>>
>>
>> When the instance is deleted, the error will disappear also.
>>
>>
>>
>> Any assistance/insight would be appreciated.
>>
>>
>>
>> Thank you.
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>>
>
9 years, 2 months
[Rdo-list] Timed out waiting for nova-conductor. w/ icehouse-3 and foreman
by Andrew Lau
Hi,
I tried to deploy RDO icehouse-3 using foreman on CentOS 6.5, previously I
had success with Havana and icehouse-2
Everything went smoothly, and quite a few of the issues I had with
icehouse-2 were gone!
The only non-puppet task I had to do was on the controller:
cat <<EOF > /etc/mysql/conf.d/innodb.cnf
[mysqld]
default-storage-engine = innodb
EOF
service mysqld restart
mysql -e 'drop database neutron; create database neutron;'
However, I've hit a blocker with nova compute not showing up into the
controller. The only thing useful I can find in the nova-compute logs,
"
Timed out waiting for nova-conductor. Is it running? Or did this service
start before nova-conductor?
"
Both nova-compute (on the compute) and nova-conductor (on the controller)
said "AMQP connected" after a service restart, so I'm not sure what's gone
wrong here.
Suggestions?
Thanks,
Andrew
9 years, 2 months
[Rdo-list] New and improved OpenStack User Survey!
by Steve Gordon
Hi all,
Just a quick note to highlight that the OpenStack User Survey is back and better than ever with a few extra sections in response to feedback from previous rounds. This data helps the community shape OpenStack so if you are an OpenStack deployer, operator, user, or application developer be sure to head over and have your say:
https://www.openstack.org/user-survey/
Thanks,
Steve
9 years, 2 months
[Rdo-list] Firewall issue/error when spawning instances on compute node
by St. George, Allan L.
Currently running RDO/Havana deployed via Foreman on a multi-compute node stack (Controller, Neutron, and three Nova-Compute servers)
When spawning an instance, it correctly spawns and reports/registers to the Foreman dashboard.
The problem is that the hypervisor/compute-node that is hosting the instance will then begin to report:
Level
Resource
message
err
Puppet
Could not prefetch firewall provider 'iptables': Invalid address from IPAddr.new: FA:16:3E:67:64:A4
err
/Firewall[001 nova compute incoming]
Could not evaluate: Invalid address from IPAddr.new: FA:16:3E:67:64:A4
err
/Firewall[002 vxlan udp]
Could not evaluate: Invalid address from IPAddr.new: FA:16:3E:67:64:A4
When the instance is deleted, the error will disappear also.
Any assistance/insight would be appreciated.
Thank you.
9 years, 2 months
[Rdo-list] Mongodb error while installing RDO icehouse-3 through packstack on F20
by chandan kumar
Hello,
I am trying to deploy RDO icehouse-3 using packstack on fedora 20.
Below is the screen error log of packstack:
10.65.193.94_osclient.pp: [ DONE ]
10.65.193.94_horizon.pp: [ DONE ]
10.65.193.94_provision.pp: [ DONE ]
Applying 10.65.193.94_mongodb.pp
10.65.193.94_mongodb.pp: [ ERROR ]
Applying Puppet manifests [ ERROR ]
ERROR : Error appeared during Puppet run: 10.65.193.94_mongodb.pp
Error: Failed to parse template mongodb/mongodb.conf.erb:
You will find full trace in log
/var/tmp/packstack/20140326-215159-urZtup/manifests/10.65.193.94_mongodb.pp.log
Please check log file
/var/tmp/packstack/20140326-215159-urZtup/openstack-setup.log for more
information
Additional information:
* Time synchronization installation was skipped. Please note that
unsynchronized time on server instances might be problem for some
OpenStack components.
* Did not create a cinder volume group, one already existed
* File /root/keystonerc_admin has been created on OpenStack client
host 10.65.193.94. To use the command line tools you need to source
the file.
* To access the OpenStack Dashboard browse to http://10.65.193.94/dashboard .
Please, find your login credentials stored in the keystonerc_admin in
your home directory.
Below is the output of
/var/tmp/packstack/20140326-215159-urZtup/manifests/10.65.193.94_mongodb.pp.log
[root@dhcp193-94 ~]# cat
/var/tmp/packstack/20140326-215159-urZtup/manifests/10.65.193.94_mongodb.pp.log
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Error: Failed to parse template mongodb/mongodb.conf.erb:
Filepath: /var/tmp/packstack/2425ebe0f2c143b3a52ed7d9756fc237/modules/mongodb/templates/mongodb.conf.erb
Line: 10
Detail: undefined method `join' for "10.65.193.94":String
at /var/tmp/packstack/2425ebe0f2c143b3a52ed7d9756fc237/modules/mongodb/manifests/server/config.pp:65
on node dhcp193-94.pnq.redhat.com
Error: Failed to parse template mongodb/mongodb.conf.erb:
Filepath: /var/tmp/packstack/2425ebe0f2c143b3a52ed7d9756fc237/modules/mongodb/templates/mongodb.conf.erb
Line: 10
Detail: undefined method `join' for "10.65.193.94":String
at /var/tmp/packstack/2425ebe0f2c143b3a52ed7d9756fc237/modules/mongodb/manifests/server/config.pp:65
on node dhcp193-94.pnq.redhat.com
To fix that, i have installed mongodb-server and started it and again
run the packstack file. After that still getting the same error.
Can someone suggest me someone workaround for this?
Thanks,
Chandan Kumar
9 years, 2 months
[Rdo-list] fail to launch instances in fedora 20
by Yogev Rabl
Hi,
Earlier I've opened a bug: Bug 1081015 - failed to launch an instance from ISO image . This is an urgent issue and it looks like a problem with the qpid that cause the (I think).
Can someone have a look in the bug and give another opinion?
thanks,
Yogev.
9 years, 2 months
