8:57 a.m.
I'm working on a slightly more in-depth article on this topic, but in
order for some people (pmyers I'm looking at you) to get started I
wanted to write up some very quick instructions. Forgive me any typos
in this email because I'd like to send it out before actually running
through everything locally: while the process is automated, an HA
deploy can still take quite a while to complete. Also, the fetch of
the undercloud image *also* takes a chunk of time; there are
instructions in the tripleo-quickstart README for caching a local copy
of the image to speed up subsequent installs.
You will need a physical host with at least 32GB of RAM. More is
better, less *may* be possible but you will probably regret it.
You will also need Ansible 2.0.x, which is what you will get if you
'pip install ansible', or install Ansible from updates-testing
(Fedora) or epel-testing (RHEL/CentOS/...).
Do *not* run Ansible HEAD from the git repository! This will lead to
sadness and disappointment.
1. Prepare your target host.
You need a user on your target host to which you can (a) log in via
ssh without a password and then (b) sudo to root without a password.
We'll refer to this user as "targetuser", below. That is, the
following should work:
ssh -tt targetuser@targethost sudo uptime
2. Clone the tripleo-quickstart repository:
git clone https://github.com/redhat-openstack/tripleo-quickstart
cd tripleo-quickstart
(Everything below is run from inside the tripleo-quickstart
directory)
2. Create an ansible inventory file.
Create an inventory file that lists your target host in the 'virthost'
and that provides ansible with the necessary connection information:
cat > inventory <<EOF
[virthost]
my.target.host ansible_user=targetuser
3. Create an ansible playbook.
cat > playbooks/ha.yml <<EOF
- hosts: virthost
roles:
- role: libvirt/teardown
- role: libvirt/setup
- hosts: localhost
roles:
- rebuild-inventory
- hosts: undercloud
roles:
- overcloud
EOF
4. Create a variables file that describes your architecture:
cat > nodes.yml <<EOF
extra_args: >-
--control-scale 3
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
--ntp-server pool.ntp.org
baremetal_vm_xml: |
<cpu mode='host-passthrough'/>
libvirt_args: --libvirt-type kvm
# Set these to values appropriate for your target system. You
# by default get three controllers, one compute node, and
# one ceph node (so this example has a memory footprint of
# 32GB, which is probably too much for a host with only
# 32GB of RAM).
control_memory: 8192
compute_memory: 4096
ceph_memory: 4096
EOF
The above configuration will enabled nested KVM on the target host.
It is possible to change the number of nodes of each type that are
created, but that's for another time.
5. Run it!
ansible-playbook playbooks/ha.yml -i inventory -e @nodes.yml
This will:
- First attempt to clean up the virtual environment from any
previous run of tripleo-quickstart
- Deploy a new virtual undercloud and virtual overcloud
- Install the undercloud
- Deploy the overcloud
- Validate the overcloud
If you don't trust your copying-and-pasting, the example files
referenced in this email are also available from:
https://gist.github.com/larsks/e02ca28982d1daacfa5d
E.g.:
git clone https://gist.github.com/e02ca28982d1daacfa5d.git
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
11:29 a.m.
On 02/19/2016 09:57 AM, Lars Kellogg-Stedman wrote:
I'm working on a slightly more in-depth article on this topic,
but in
order for some people (pmyers I'm looking at you) to get started I
/me looks around sheepishly... :)
wanted to write up some very quick instructions. Forgive me any
typos
in this email because I'd like to send it out before actually running
through everything locally: while the process is automated, an HA
deploy can still take quite a while to complete. Also, the fetch of
the undercloud image *also* takes a chunk of time; there are
instructions in the tripleo-quickstart README for caching a local copy
of the image to speed up subsequent installs.
https://github.com/redhat-openstack/tripleo-quickstart/blob/master/README...
Definitely do this.
You will need a physical host with at least 32GB of RAM. More is
better, less *may* be possible but you will probably regret it.
Wheee....
MemTotal: 65764284 kB
MemFree: 58529856 kB
You will also need Ansible 2.0.x, which is what you will get if you
'pip install ansible', or install Ansible from updates-testing
(Fedora) or epel-testing (RHEL/CentOS/...).
Ok, since I'm starting with a pretty vanilla CentOS7 Server with
libvirt, qemu, etc installed...
# yum install ansible --enablerepo epel-testing
ansible.noarch 0:2.0.0.2-1.el7
Do *not* run Ansible HEAD from the git repository! This will lead
to
sadness and disappointment.
1. Prepare your target host.
You need a user on your target host to which you can (a) log in via
ssh without a password and then (b) sudo to root without a password.
We'll refer to this user as "targetuser", below. That is, the
following should work:
ssh -tt targetuser@targethost sudo uptime
using 'admin' account, verified passwordless ssh as admin works and
passwordless sudo from admin to root works
2. Clone the tripleo-quickstart repository:
git clone https://github.com/redhat-openstack/tripleo-quickstart
cd tripleo-quickstart
(Everything below is run from inside the tripleo-quickstart
directory)
2. Create an ansible inventory file.
Create an inventory file that lists your target host in the 'virthost'
and that provides ansible with the necessary connection information:
cat > inventory <<EOF
[virthost]
my.target.host ansible_user=targetuser
duh, for those of us that are noobs... replace my.target.host with localhost
So for me its:
cat > inventory <<EOF
[virthost]
localhost ansible_user=admin
3. Create an ansible playbook.
cat > playbooks/ha.yml <<EOF
- hosts: virthost
roles:
- role: libvirt/teardown
- role: libvirt/setup
- hosts: localhost
roles:
- rebuild-inventory
- hosts: undercloud
roles:
- overcloud
EOF
4. Create a variables file that describes your architecture:
cat > nodes.yml <<EOF
extra_args: >-
--control-scale 3
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
--ntp-server pool.ntp.org
baremetal_vm_xml: |
<cpu mode='host-passthrough'/>
libvirt_args: --libvirt-type kvm
# Set these to values appropriate for your target system. You
# by default get three controllers, one compute node, and
# one ceph node (so this example has a memory footprint of
# 32GB, which is probably too much for a host with only
# 32GB of RAM).
control_memory: 8192
compute_memory: 4096
ceph_memory: 4096
url: file:///usr/share/quickstart_images/mitaka/undercloud.qcow2
(if you pre-downloaded the undercloud image as per the instructions in
the rst file link above)
EOF
The above configuration will enabled nested KVM on the target host.
It is possible to change the number of nodes of each type that are
created, but that's for another time.
5. Run it!
ansible-playbook playbooks/ha.yml -i inventory -e @nodes.yml
Ran into a small issue... since I was running as admin and needing sudo
for root escalation, this patch from larsks was necessary to apply:
http://chunk.io/f/574614d4738c460db656714931591694
Kicked off at 11am EST sharp. Currently running overcloud deploy.
Lars, let us know when this is merged :)
This will:
- First attempt to clean up the virtual environment from any
previous run of tripleo-quickstart
- Deploy a new virtual undercloud and virtual overcloud
- Install the undercloud
- Deploy the overcloud
- Validate the overcloud
If you don't trust your copying-and-pasting, the example files
referenced in this email are also available from:
https://gist.github.com/larsks/e02ca28982d1daacfa5d
E.g.:
git clone https://gist.github.com/e02ca28982d1daacfa5d.git
To login to undercloud
ssh -F ~/.quickstart/ssh.config.ansible undercloud
hewbrocca also mentions that in the future, heat.conf in the undercloud
will set by default max_resources_per_stack to -1 which should make
things go much faster
trown notes that undercloud node will be slow to deploy with only 1
vCPU. Since my box has 16 real (32 with HT), this seems like a waste of
computing power :)
Adding:
undercloud_vcpu: 4
control_vcpu: 2
to nodes.yml may make sense
At least on my machine given that I have 64gb/32 cores
Validate step finished around 70 minutes after initial run of the
ansible playbook, but it failed. larsks suggested that it might on a HA
deployment. This will need looking into.
I was able to ssh into the undercloud and from there ssh into overcloud
nodes after getting their control plane IP addresses via undercloud nova
list
I was also able to source overcloudrc and nova list, and that worked.
Probably I'll want to set up some ssh tunnels so that I can access the
overcloud horizon (just for the heck of it) without needing to be ON the
undercloud node
And for the uninitiated... there is no Undercloud Horizon. Only CLI
So far, I think this is all fairly accessible. It's really not that much
more time consuming than Packstack and I think some optimizations can be
put into place (max_resources_per_stack and undercloud vcpus) to make
things speedier.
That being said... I am running on a machine which most developers
wouldn't have, so the next steps will be to make it reasonable on a 32gb
machine and to provide a non-HA setup for those with 16gb machines.
Perry
2:09 p.m.
On 02/19/2016 12:29 PM, Perry Myers wrote:
On 02/19/2016 09:57 AM, Lars Kellogg-Stedman wrote:
> I'm working on a slightly more in-depth article on this topic, but in
> order for some people (pmyers I'm looking at you) to get started I
/me looks around sheepishly... :)
> wanted to write up some very quick instructions. Forgive me any typos
> in this email because I'd like to send it out before actually running
> through everything locally: while the process is automated, an HA
> deploy can still take quite a while to complete. Also, the fetch of
> the undercloud image *also* takes a chunk of time; there are
> instructions in the tripleo-quickstart README for caching a local copy
> of the image to speed up subsequent installs.
https://github.com/redhat-openstack/tripleo-quickstart/blob/master/README...
Definitely do this.
Also, it should work on Fedora 23 hosts too. My 32-gb minidell seems to
choke a bit on HA, but it survives (though if some manager wanted to let
me expense an ssd, well, no complaint here).
-Ryan
--
Ryan Brown / Senior Software Engineer, OpenStack / Red Hat, Inc.
6:04 p.m.
On 02/19/2016 12:29 PM, Perry Myers wrote:
On 02/19/2016 09:57 AM, Lars Kellogg-Stedman wrote:
> I'm working on a slightly more in-depth article on this topic, but in
> order for some people (pmyers I'm looking at you) to get started I
/me looks around sheepishly... :)
> wanted to write up some very quick instructions. Forgive me any typos
> in this email because I'd like to send it out before actually running
> through everything locally: while the process is automated, an HA
> deploy can still take quite a while to complete. Also, the fetch of
> the undercloud image *also* takes a chunk of time; there are
> instructions in the tripleo-quickstart README for caching a local copy
> of the image to speed up subsequent installs.
https://github.com/redhat-openstack/tripleo-quickstart/blob/master/README...
Definitely do this.
> You will need a physical host with at least 32GB of RAM. More is
> better, less *may* be possible but you will probably regret it.
Wheee....
MemTotal: 65764284 kB
MemFree: 58529856 kB
> You will also need Ansible 2.0.x, which is what you will get if you
> 'pip install ansible', or install Ansible from updates-testing
> (Fedora) or epel-testing (RHEL/CentOS/...).
Ok, since I'm starting with a pretty vanilla CentOS7 Server with
libvirt, qemu, etc installed...
# yum install ansible --enablerepo epel-testing
ansible.noarch 0:2.0.0.2-1.el7
> Do *not* run Ansible HEAD from the git repository! This will lead to
> sadness and disappointment.
>
> 1. Prepare your target host.
>
> You need a user on your target host to which you can (a) log in via
> ssh without a password and then (b) sudo to root without a password.
> We'll refer to this user as "targetuser", below. That is, the
> following should work:
>
> ssh -tt targetuser@targethost sudo uptime
using 'admin' account, verified passwordless ssh as admin works and
passwordless sudo from admin to root works
> 2. Clone the tripleo-quickstart repository:
>
> git clone https://github.com/redhat-openstack/tripleo-quickstart
> cd tripleo-quickstart
>
> (Everything below is run from inside the tripleo-quickstart
> directory)
>
> 2. Create an ansible inventory file.
>
> Create an inventory file that lists your target host in the 'virthost'
> and that provides ansible with the necessary connection information:
>
> cat > inventory <<EOF
> [virthost]
> my.target.host ansible_user=targetuser
duh, for those of us that are noobs... replace my.target.host with localhost
So for me its:
cat > inventory <<EOF
[virthost]
localhost ansible_user=admin
> 3. Create an ansible playbook.
>
> cat > playbooks/ha.yml <<EOF
> - hosts: virthost
> roles:
> - role: libvirt/teardown
> - role: libvirt/setup
>
> - hosts: localhost
> roles:
> - rebuild-inventory
>
> - hosts: undercloud
> roles:
> - overcloud
> EOF
>
> 4. Create a variables file that describes your architecture:
>
> cat > nodes.yml <<EOF
> extra_args: >-
> --control-scale 3
> -e
/usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
> --ntp-server pool.ntp.org
> baremetal_vm_xml: |
> <cpu mode='host-passthrough'/>
> libvirt_args: --libvirt-type kvm
>
> # Set these to values appropriate for your target system. You
> # by default get three controllers, one compute node, and
> # one ceph node (so this example has a memory footprint of
> # 32GB, which is probably too much for a host with only
> # 32GB of RAM).
> control_memory: 8192
> compute_memory: 4096
> ceph_memory: 4096
url: file:///usr/share/quickstart_images/mitaka/undercloud.qcow2
(if you pre-downloaded the undercloud image as per the instructions in
the rst file link above)
> EOF
>
> The above configuration will enabled nested KVM on the target host.
>
> It is possible to change the number of nodes of each type that are
> created, but that's for another time.
>
> 5. Run it!
>
> ansible-playbook playbooks/ha.yml -i inventory -e @nodes.yml
Ran into a small issue... since I was running as admin and needing sudo
for root escalation, this patch from larsks was necessary to apply:
http://chunk.io/f/574614d4738c460db656714931591694
Kicked off at 11am EST sharp. Currently running overcloud deploy.
Lars, let us know when this is merged :)
> This will:
>
> - First attempt to clean up the virtual environment from any
> previous run of tripleo-quickstart
> - Deploy a new virtual undercloud and virtual overcloud
> - Install the undercloud
> - Deploy the overcloud
> - Validate the overcloud
>
> If you don't trust your copying-and-pasting, the example files
> referenced in this email are also available from:
>
> https://gist.github.com/larsks/e02ca28982d1daacfa5d
>
> E.g.:
>
> git clone https://gist.github.com/e02ca28982d1daacfa5d.git
To login to undercloud
ssh -F ~/.quickstart/ssh.config.ansible undercloud
hewbrocca also mentions that in the future, heat.conf in the undercloud
will set by default max_resources_per_stack to -1 which should make
things go much faster
trown notes that undercloud node will be slow to deploy with only 1
vCPU. Since my box has 16 real (32 with HT), this seems like a waste of
computing power :)
Adding:
undercloud_vcpu: 4
control_vcpu: 2
to nodes.yml may make sense
At least on my machine given that I have 64gb/32 cores
Validate step finished around 70 minutes after initial run of the
ansible playbook, but it failed. larsks suggested that it might on a HA
deployment. This will need looking into.
I was able to ssh into the undercloud and from there ssh into overcloud
nodes after getting their control plane IP addresses via undercloud nova
list
I was also able to source overcloudrc and nova list, and that worked.
Probably I'll want to set up some ssh tunnels so that I can access the
overcloud horizon (just for the heck of it) without needing to be ON the
undercloud node
And for the uninitiated... there is no Undercloud Horizon. Only CLI
So far, I think this is all fairly accessible. It's really not that much
more time consuming than Packstack and I think some optimizations can be
put into place (max_resources_per_stack and undercloud vcpus) to make
things speedier.
That being said... I am running on a machine which most developers
wouldn't have, so the next steps will be to make it reasonable on a 32gb
machine and to provide a non-HA setup for those with 16gb machines.
Perry
One other thing I've noticed... If you're experimenting and going
between say... an HA setup with 3 controllers and 2 computes and then
follow that with a deploy of a 1 controller/1 compute... the other
computes and controllers hang around.
That is... on the second run, with the smaller config, it doesn't know
to go and clean up the nodes left behind from the larger config. It's
not a big deal, you just need to remember to go and virsh destroy stuff
manually. But, it's something to be aware of if you're toggling between
HA and non HA environments
Perry
6:10 p.m.
On 02/19/2016 07:04 PM, Perry Myers wrote:
On 02/19/2016 12:29 PM, Perry Myers wrote:
> On 02/19/2016 09:57 AM, Lars Kellogg-Stedman wrote:
>> I'm working on a slightly more in-depth article on this topic, but in
>> order for some people (pmyers I'm looking at you) to get started I
>
> /me looks around sheepishly... :)
>
>> wanted to write up some very quick instructions. Forgive me any typos
>> in this email because I'd like to send it out before actually running
>> through everything locally: while the process is automated, an HA
>> deploy can still take quite a while to complete. Also, the fetch of
>> the undercloud image *also* takes a chunk of time; there are
>> instructions in the tripleo-quickstart README for caching a local copy
>> of the image to speed up subsequent installs.
>
>
https://github.com/redhat-openstack/tripleo-quickstart/blob/master/README...
>
> Definitely do this.
>
>> You will need a physical host with at least 32GB of RAM. More is
>> better, less *may* be possible but you will probably regret it.
>
> Wheee....
> MemTotal: 65764284 kB
> MemFree: 58529856 kB
>
>> You will also need Ansible 2.0.x, which is what you will get if you
>> 'pip install ansible', or install Ansible from updates-testing
>> (Fedora) or epel-testing (RHEL/CentOS/...).
>
> Ok, since I'm starting with a pretty vanilla CentOS7 Server with
> libvirt, qemu, etc installed...
>
> # yum install ansible --enablerepo epel-testing
> ansible.noarch 0:2.0.0.2-1.el7
>
>> Do *not* run Ansible HEAD from the git repository! This will lead to
>> sadness and disappointment.
>>
>> 1. Prepare your target host.
>>
>> You need a user on your target host to which you can (a) log in via
>> ssh without a password and then (b) sudo to root without a password.
>> We'll refer to this user as "targetuser", below. That is, the
>> following should work:
>>
>> ssh -tt targetuser@targethost sudo uptime
>
> using 'admin' account, verified passwordless ssh as admin works and
> passwordless sudo from admin to root works
>
>> 2. Clone the tripleo-quickstart repository:
>>
>> git clone https://github.com/redhat-openstack/tripleo-quickstart
>> cd tripleo-quickstart
>>
>> (Everything below is run from inside the tripleo-quickstart
>> directory)
>>
>> 2. Create an ansible inventory file.
>>
>> Create an inventory file that lists your target host in the 'virthost'
>> and that provides ansible with the necessary connection information:
>>
>> cat > inventory <<EOF
>> [virthost]
>> my.target.host ansible_user=targetuser
>
> duh, for those of us that are noobs... replace my.target.host with localhost
>
> So for me its:
>
> cat > inventory <<EOF
> [virthost]
> localhost ansible_user=admin
>
>
>> 3. Create an ansible playbook.
>>
>> cat > playbooks/ha.yml <<EOF
>> - hosts: virthost
>> roles:
>> - role: libvirt/teardown
>> - role: libvirt/setup
>>
>> - hosts: localhost
>> roles:
>> - rebuild-inventory
>>
>> - hosts: undercloud
>> roles:
>> - overcloud
>> EOF
>>
>> 4. Create a variables file that describes your architecture:
>>
>> cat > nodes.yml <<EOF
>> extra_args: >-
>> --control-scale 3
>> -e
/usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
>> --ntp-server pool.ntp.org
>> baremetal_vm_xml: |
>> <cpu mode='host-passthrough'/>
>> libvirt_args: --libvirt-type kvm
>>
>> # Set these to values appropriate for your target system. You
>> # by default get three controllers, one compute node, and
>> # one ceph node (so this example has a memory footprint of
>> # 32GB, which is probably too much for a host with only
>> # 32GB of RAM).
>> control_memory: 8192
>> compute_memory: 4096
>> ceph_memory: 4096
>
> url: file:///usr/share/quickstart_images/mitaka/undercloud.qcow2
>
> (if you pre-downloaded the undercloud image as per the instructions in
> the rst file link above)
>
>> EOF
>>
>> The above configuration will enabled nested KVM on the target host.
>>
>> It is possible to change the number of nodes of each type that are
>> created, but that's for another time.
>>
>> 5. Run it!
>>
>> ansible-playbook playbooks/ha.yml -i inventory -e @nodes.yml
>
> Ran into a small issue... since I was running as admin and needing sudo
> for root escalation, this patch from larsks was necessary to apply:
> http://chunk.io/f/574614d4738c460db656714931591694
>
> Kicked off at 11am EST sharp. Currently running overcloud deploy.
>
> Lars, let us know when this is merged :)
>
>> This will:
>>
>> - First attempt to clean up the virtual environment from any
>> previous run of tripleo-quickstart
>> - Deploy a new virtual undercloud and virtual overcloud
>> - Install the undercloud
>> - Deploy the overcloud
>> - Validate the overcloud
>>
>> If you don't trust your copying-and-pasting, the example files
>> referenced in this email are also available from:
>>
>> https://gist.github.com/larsks/e02ca28982d1daacfa5d
>>
>> E.g.:
>>
>> git clone https://gist.github.com/e02ca28982d1daacfa5d.git
>
> To login to undercloud
> ssh -F ~/.quickstart/ssh.config.ansible undercloud
>
> hewbrocca also mentions that in the future, heat.conf in the undercloud
> will set by default max_resources_per_stack to -1 which should make
> things go much faster
>
> trown notes that undercloud node will be slow to deploy with only 1
> vCPU. Since my box has 16 real (32 with HT), this seems like a waste of
> computing power :)
>
> Adding:
> undercloud_vcpu: 4
> control_vcpu: 2
> to nodes.yml may make sense
> At least on my machine given that I have 64gb/32 cores
>
> Validate step finished around 70 minutes after initial run of the
> ansible playbook, but it failed. larsks suggested that it might on a HA
> deployment. This will need looking into.
>
> I was able to ssh into the undercloud and from there ssh into overcloud
> nodes after getting their control plane IP addresses via undercloud nova
> list
>
> I was also able to source overcloudrc and nova list, and that worked.
>
> Probably I'll want to set up some ssh tunnels so that I can access the
> overcloud horizon (just for the heck of it) without needing to be ON the
> undercloud node
>
> And for the uninitiated... there is no Undercloud Horizon. Only CLI
>
> So far, I think this is all fairly accessible. It's really not that much
> more time consuming than Packstack and I think some optimizations can be
> put into place (max_resources_per_stack and undercloud vcpus) to make
> things speedier.
>
> That being said... I am running on a machine which most developers
> wouldn't have, so the next steps will be to make it reasonable on a 32gb
> machine and to provide a non-HA setup for those with 16gb machines.
>
> Perry
One other thing I've noticed... If you're experimenting and going
between say... an HA setup with 3 controllers and 2 computes and then
follow that with a deploy of a 1 controller/1 compute... the other
computes and controllers hang around.
That is... on the second run, with the smaller config, it doesn't know
to go and clean up the nodes left behind from the larger config. It's
not a big deal, you just need to remember to go and virsh destroy stuff
manually. But, it's something to be aware of if you're toggling between
HA and non HA environments
Ah right, it will only clean up what is in nodes.yml by design. So that
if you have some utility VMs (IRC bouncer, etc) on your virthost they
dont get nuked. You can run a playbook with just the teardown role
(there is one in tree at playbooks/libvirt-teardown.yml) with the old
nodes.yml to get rid of the old setup.
Perry
6:55 a.m.
New subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
Cluster (VM based) is up and running . Keepalived is good shape on all nodes
Follow https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-con...
On each node :-
[root@hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy: active
openstack-swift-account: active
openstack-swift-container: active
openstack-swift-object: active
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN
19929/python2
tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WAIT -
tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WAIT -
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928
root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19928
swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924
root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19924
swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19929
swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
Ports are open on each node.
I am getting :-
[root@hacontroller1 ~(keystone_admin)]# swift list
Account GET failed:
http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1...
503 Service Unavailable [first 60 chars of response] <html><h1>Service
Unavailable</h1><p>The server is currently
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN
19920/python2
tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN
1569/haproxy
tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WAIT -
So , I guess it's not supposed to respond swift. I am missing something here.
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=auto 19920
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=auto 1569
controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080 , not swift-proxy-server.
/var/log/swift/swift.log is empty
First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) (client_ip:
192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
was fixed per
https://ask.openstack.org/en/question/57608/proxy-server-error-insufficie...
But , I am still getting in /var/log/messages
Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: #012Traceback (most
recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in
run_once#012 self.swift.get_account_info(self.expiring_objects_account)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 358, in
get_account_info#012 resp = self.make_request('HEAD', path, {},
acceptable_statuses)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 194, in
make_request#012 _('Unexpected response: %s') % resp.status,
resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn:
txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - [23/Feb/2016:12:51:15
+0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD
http://localhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c3"
"proxy-server 351" 0.0002 "-" 28097 -
Please, advise.
Boris.
12:02 p.m.
New subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been built per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
Recreated keystone endpoint for swift-proxy to 8090 && service httpd restart on
Controller1.
Updated proxy-swift.conf && restarted service proxy-swift on all nodes.
Opened port 8090 on all nodes. Run as admin on 192.169.142.221 (hacontroller1) :-
# swift list
in swift.log
Feb 23 20:54:50 hacontroller1 account-server: 192.169.142.222 - - [23/Feb/2016:17:54:50
+0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD
http://localhost/v1/.expiring_objects" "tx7a654a6dc9cd4aca83de0-0056cc9cea"
"proxy-server 19638" 0.0007 "-" 22509 -
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Boris
Derzhavets <bderzhavets(a)hotmail.com>
Sent: Tuesday, February 23, 2016 7:55 AM
To: Javier Pena
Cc: rdo-list
Subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per
https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
Cluster (VM based) is up and running . Keepalived is good shape on all nodes
Follow https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-con...
On each node :-
[root@hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy: active
openstack-swift-account: active
openstack-swift-container: active
openstack-swift-object: active
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN
19929/python2
tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WAIT -
tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WAIT -
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928
root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19928
swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924
root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19924
swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19929
swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
Ports are open on each node.
I am getting :-
[root@hacontroller1 ~(keystone_admin)]# swift list
Account GET failed:
http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1...
503 Service Unavailable [first 60 chars of response] <html><h1>Service
Unavailable</h1><p>The server is currently
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN
19920/python2
tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN
1569/haproxy
tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WAIT -
So , I guess it's not supposed to respond swift. I am missing something here.
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=auto 19920
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=auto 1569
controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080 , not swift-proxy-server.
/var/log/swift/swift.log is empty
First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) (client_ip:
192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
was fixed per
https://ask.openstack.org/en/question/57608/proxy-server-error-insufficie...
But , I am still getting in /var/log/messages
Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: #012Traceback (most
recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in
run_once#012 self.swift.get_account_info(self.expiring_objects_account)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 358, in
get_account_info#012 resp = self.make_request('HEAD', path, {},
acceptable_statuses)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 194, in
make_request#012 _('Unexpected response: %s') % resp.status,
resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn:
txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - [23/Feb/2016:12:51:15
+0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD
http://localhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c3"
"proxy-server 351" 0.0002 "-" 28097 -
Please, advise.
Boris.
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:18 p.m.
New subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
----- Original Message -----
Cluster (VM based) is up and running . Keepalived is good shape on all
nodes
Follow
https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-con...
On each node :-
[root@hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy: active
openstack-swift-account: active
openstack-swift-container: active
openstack-swift-object: active
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN
19929/python2
tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WAIT
-
tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WAIT
-
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928
root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19928
swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924
root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19924
swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19929
swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
Ports are open on each node.
I am getting :-
[root@hacontroller1 ~(keystone_admin)]# swift list
Account GET failed:
http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1...
503 Service Unavailable [first 60 chars of response] <html><h1>Service
Unavailable</h1><p>The server is currently
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN
19920/python2
tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN
1569/haproxy
tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WAIT
-
So , I guess it's not supposed to respond swift. I am missing something
here.
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=auto 19920
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=auto 1569
controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080 , not
swift-proxy-server.
/var/log/swift/swift.log is empty
First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
(client_ip: 192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
was fixed per
https://ask.openstack.org/en/question/57608/proxy-server-error-insufficie...
But , I am still getting in /var/log/messages
Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception:
#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in
run_once#012
self.swift.get_account_info(self.expiring_objects_account)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line
358, in get_account_info#012 resp = self.make_request('HEAD', path, {},
acceptable_statuses)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line
194, in make_request#012 _('Unexpected response: %s') % resp.status,
resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable
(txn: txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - -
[23/Feb/2016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 -
"HEAD
http://localhost/v1/.expiring_objects"
"tx088b0d9b2d814c56b3f8b-0056cc55c3"
"proxy-server 351" 0.0002 "-" 28097 -
Hi Boris,
From the errors above, this looks like a Swift configuration issue.
Where did you mount the file system for vdb? In the instructions, they are mounted under
/srv/node/vdb . If it is mounted outside /srv, I remember having SELinux issues with it.
Also, how big is the vdb disk? In my tests, I used a 8 GB virtual disk, I'm not sure
how small we could make it.
Regards,
Javier
Please, advise.
Boris.
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:39 p.m.
New subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
________________________________________
From: Javier Pena <javier.pena(a)redhat.com>
Sent: Tuesday, February 23, 2016 1:18 PM
To: rdo-list
Cc: Boris Derzhavets
Subject: Re: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been
per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
----- Original Message -----
Cluster (VM based) is up and running . Keepalived is good shape on all
nodes
Follow
https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-con...
On each node :-
[root@hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy: active
openstack-swift-account: active
openstack-swift-container: active
openstack-swift-object: active
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN
19929/python2
tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WAIT
-
tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WAIT
-
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928
root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19928
swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-object-server /etc/swift/object-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924
root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19924
swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-container-server /etc/swift/container-server.conf
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19929
swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-account-server /etc/swift/account-server.conf
Ports are open on each node.
I am getting :-
[root@hacontroller1 ~(keystone_admin)]# swift list
Account GET failed:
http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1...
503 Service Unavailable [first 60 chars of response] <html><h1>Service
Unavailable</h1><p>The server is currently
[root@hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN
19920/python2
tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN
1569/haproxy
tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WAIT
-
So , I guess it's not supposed to respond swift. I am missing something
here.
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2
/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=auto 19920
[root@hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=auto 1569
controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080 , not
swift-proxy-server.
/var/log/swift/swift.log is empty
First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
(client_ip: 192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage
192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
was fixed per
https://ask.openstack.org/en/question/57608/proxy-server-error-insufficie...
But , I am still getting in /var/log/messages
Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception:
#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in
run_once#012
self.swift.get_account_info(self.expiring_objects_account)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line
358, in get_account_info#012 resp = self.make_request('HEAD', path, {},
acceptable_statuses)#012 File
"/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line
194, in make_request#012 _('Unexpected response: %s') % resp.status,
resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable
(txn: txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - -
[23/Feb/2016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 -
"HEAD
http://localhost/v1/.expiring_objects"
"tx088b0d9b2d814c56b3f8b-0056cc55c3"
"proxy-server 351" 0.0002 "-" 28097 -
Hi Boris,
From the errors above, this looks like a Swift configuration issue.
Where did you mount the file system for vdb? In the instructions, they are mounted under
/srv/node/vdb . If it is mounted outside /srv, I remember having SELinux issues with it.
Javier,
I have updated haproxy.cfg and finally ( haproxy restart ) got back ( and understood
haproxy monitoring) :-
[root@hacontroller2 ~(keystone_admin)]# swift list
Account GET failed:
http://controller-vip.example.com:8090/v1/AUTH_acdc927b53bd43ae9a7ed657d1...
503 Service Unavailable [first 60 chars of response] <html><h1>Service
Unavailable</h1><p>The server is currently
[root@hacontroller2 ~(keystone_admin)]# netstat -antp | grep 8090
tcp 0 0 192.169.142.220:8090 0.0.0.0:* LISTEN
27323/haproxy
tcp 0 0 192.169.142.222:8090 0.0.0.0:* LISTEN
19645/python2
tcp 0 0 192.169.142.222:39922 192.169.142.220:8090 TIME_WAIT -
On all nodes
[root@hacontroller2 ~(keystone_admin)]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 41G 11G 30G 28% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 96K 1.9G 1% /dev/shm
tmpfs 1.9G 17M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vdb1 20G 33M 20G 1% /srv/node/vdb1
/dev/vda1 497M 167M 330M 34% /boot
Also, how big is the vdb disk? In my tests, I used a 8 GB virtual disk, I'm not sure
how small we could make it.
20 GB
Thank you.
Boris.
Regards,
Javier
Please, advise.
Boris.
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:21 p.m.
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Lars
Kellogg-Stedman <lars(a)redhat.com>
Sent: Friday, February 19, 2016 9:57 AM
To: RDO
Cc: Perry Myers; John Trowbridge
Subject: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
I'm working on a slightly more in-depth article on this topic, but in
order for some people (pmyers I'm looking at you) to get started I
wanted to write up some very quick instructions.
Could you be so kind to provide also setup for just
1.Controller
1.Compute
in overcloud ( as of now i7 4790 16 GB RAM)
Ansible based deployment schema. I have a chance to survive with 3VMs .
Thank you in advance.
Boris.
Forgive me any typos
in this email because I'd like to send it out before actually running
through everything locally: while the process is automated, an HA
deploy can still take quite a while to complete. Also, the fetch of
the undercloud image *also* takes a chunk of time; there are
instructions in the tripleo-quickstart README for caching a local copy
of the image to speed up subsequent installs.
You will need a physical host with at least 32GB of RAM. More is
better, less *may* be possible but you will probably regret it.
You will also need Ansible 2.0.x, which is what you will get if you
'pip install ansible', or install Ansible from updates-testing
(Fedora) or epel-testing (RHEL/CentOS/...).
Do *not* run Ansible HEAD from the git repository! This will lead to
sadness and disappointment.
1. Prepare your target host.
You need a user on your target host to which you can (a) log in via
ssh without a password and then (b) sudo to root without a password.
We'll refer to this user as "targetuser", below. That is, the
following should work:
ssh -tt targetuser@targethost sudo uptime
2. Clone the tripleo-quickstart repository:
git clone https://github.com/redhat-openstack/tripleo-quickstart
cd tripleo-quickstart
(Everything below is run from inside the tripleo-quickstart
directory)
2. Create an ansible inventory file.
Create an inventory file that lists your target host in the 'virthost'
and that provides ansible with the necessary connection information:
cat > inventory <<EOF
[virthost]
my.target.host ansible_user=targetuser
3. Create an ansible playbook.
cat > playbooks/ha.yml <<EOF
- hosts: virthost
roles:
- role: libvirt/teardown
- role: libvirt/setup
- hosts: localhost
roles:
- rebuild-inventory
- hosts: undercloud
roles:
- overcloud
EOF
4. Create a variables file that describes your architecture:
cat > nodes.yml <<EOF
extra_args: >-
--control-scale 3
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
--ntp-server pool.ntp.org
baremetal_vm_xml: |
<cpu mode='host-passthrough'/>
libvirt_args: --libvirt-type kvm
# Set these to values appropriate for your target system. You
# by default get three controllers, one compute node, and
# one ceph node (so this example has a memory footprint of
# 32GB, which is probably too much for a host with only
# 32GB of RAM).
control_memory: 8192
compute_memory: 4096
ceph_memory: 4096
EOF
The above configuration will enabled nested KVM on the target host.
It is possible to change the number of nodes of each type that are
created, but that's for another time.
5. Run it!
ansible-playbook playbooks/ha.yml -i inventory -e @nodes.yml
This will:
- First attempt to clean up the virtual environment from any
previous run of tripleo-quickstart
- Deploy a new virtual undercloud and virtual overcloud
- Install the undercloud
- Deploy the overcloud
- Validate the overcloud
If you don't trust your copying-and-pasting, the example files
referenced in this email are also available from:
https://gist.github.com/larsks/e02ca28982d1daacfa5d
E.g.:
git clone https://gist.github.com/e02ca28982d1daacfa5d.git
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
6:01 p.m.
On 02/19/2016 01:21 PM, Boris Derzhavets wrote:
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Lars
Kellogg-Stedman <lars(a)redhat.com>
Sent: Friday, February 19, 2016 9:57 AM
To: RDO
Cc: Perry Myers; John Trowbridge
Subject: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
I'm working on a slightly more in-depth article on this topic, but in
order for some people (pmyers I'm looking at you) to get started I
wanted to write up some very quick instructions.
Could you be so kind to provide also setup for just
1.Controller
1.Compute
in overcloud ( as of now i7 4790 16 GB RAM)
Ansible based deployment schema. I have a chance to survive with 3VMs .
I think Lars has a page that describes this in good detail:
http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
But the short answer to your question is to have a nodes.yml file that
looks like this:
-------------------------------------------------
extra_args: >-
--ntp-server pool.ntp.org
baremetal_vm_xml: |
<cpu mode='host-passthrough'/>
libvirt_args: --libvirt-type kvm
# Set these to values appropriate for your target system. You
# by default get three controllers, one compute node, and
# one ceph node (so this example has a memory footprint of
# 32GB, which is probably too much for a host with only
# 32GB of RAM).
control_memory: 8192
compute_memory: 4096
undercloud_vcpu: 2
url: file:///usr/share/quickstart_images/mitaka/undercloud.qcow2
overcloud_nodes:
- name: control_0
flavor: control
- name: compute_0
flavor: compute
-------------------------------------------------
6:56 p.m.
On Fri, Feb 19, 2016 at 09:57:00AM -0500, Lars Kellogg-Stedman wrote:
I'm working on a slightly more in-depth article on this topic...
Which is now available at:
http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
Cheers,
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
9:57 p.m.
is this intended to work with physical machines at any point in time? if so
what timeline?
thanks
On Sat, Feb 20, 2016 at 2:56 AM, Lars Kellogg-Stedman <lars(a)redhat.com>
wrote:
On Fri, Feb 19, 2016 at 09:57:00AM -0500, Lars Kellogg-Stedman
wrote:
> I'm working on a slightly more in-depth article on this topic...
Which is now available at:
http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
Cheers,
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @
{freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
<https://candidate.peoplecert.org/ReportsLink.aspx?argType=1&id=13D642...
*805010942448935*
<https://www.redhat.com/wapps/training/certification/verify.html?certNumbe...
*GR750055912MA*
<https://candidate.peoplecert.org/ReportsLink.aspx?argType=1&id=13D642...
*Link to me on LinkedIn <http://www.linkedin.com/in/mohammedarafa>*
6:33 a.m.
On 02/19/2016 10:57 PM, Mohammed Arafa wrote:
is this intended to work with physical machines at any point in time?
if so
what timeline?
The steps in the posted blog would not directly work for baremetal.
However, I see no reason the virtual undercloud from tripleo-quickstart
could not be used to deploy a physical overcloud.
thanks
On Sat, Feb 20, 2016 at 2:56 AM, Lars Kellogg-Stedman <lars(a)redhat.com>
wrote:
> On Fri, Feb 19, 2016 at 09:57:00AM -0500, Lars Kellogg-Stedman wrote:
>> I'm working on a slightly more in-depth article on this topic...
>
> Which is now available at:
>
>
> http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
>
> Cheers,
>
> --
> Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @
> {freenode,twitter,github}
> Cloud Engineering / OpenStack | http://blog.oddbit.com/
>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:02 p.m.
I believe i7 5690X (8 Cores ,16 threads) at least and X99 Board from Intel or ASUS to
support CPU i7 5690X
should be mentioned as well for HA testing. Just usual gamer's PC with 32/64 GB RAM
would work fine.
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Lars
Kellogg-Stedman <lars(a)redhat.com>
Sent: Friday, February 19, 2016 7:56 PM
To: RDO
Cc: Perry Myers; John Trowbridge
Subject: Re: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
On Fri, Feb 19, 2016 at 09:57:00AM -0500, Lars Kellogg-Stedman wrote:
I'm working on a slightly more in-depth article on this topic...
Which is now available at:
http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
Cheers,
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
12:52 p.m.
Sorry, typos in specs
Intel Core i7 Extreme Edition
i7-5960X
LGA 2011-3
Details here:-
http://ark.intel.com/products/82930/Intel-Core-i7-5960X-Processor-Extreme...
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Boris
Derzhavets <bderzhavets(a)hotmail.com>
Sent: Monday, March 7, 2016 1:02 PM
To: Lars Kellogg-Stedman; RDO
Cc: Perry Myers; John Trowbridge
Subject: Re: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
I believe i7 5690X (8 Cores ,16 threads) at least and X99 Board from Intel or ASUS to
support CPU i7 5690X
should be mentioned as well for HA testing. Just usual gamer's PC with 32/64 GB RAM
would work fine.
________________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Lars
Kellogg-Stedman <lars(a)redhat.com>
Sent: Friday, February 19, 2016 7:56 PM
To: RDO
Cc: Perry Myers; John Trowbridge
Subject: Re: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
On Fri, Feb 19, 2016 at 09:57:00AM -0500, Lars Kellogg-Stedman wrote:
I'm working on a slightly more in-depth article on this topic...
Which is now available at:
http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/
Cheers,
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
2:10 p.m.
On Mon, Mar 07, 2016 at 06:02:49PM +0000, Boris Derzhavets wrote:
should be mentioned as well for HA testing. Just usual gamer's PC
with 32/64 GB RAM would work fine.
I don't think it's necessary to call out specific hardware. We're not
really targeting performance here -- if that's your goal, then the
virt-only quickstart environment is probably not appropriate for you
in any case.
The real goal is just to get things up and running so that you can test
software code changes or see what the full ha stack looks like
*without* needing dedicated infrastructure.
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
3:17 p.m.
________________________________________
From: Lars Kellogg-Stedman <lars(a)redhat.com>
Sent: Monday, March 7, 2016 3:10 PM
To: Boris Derzhavets
Cc: RDO; Perry Myers; John Trowbridge
Subject: Re: [Rdo-list] Deploying an HA dev environment with tripleo-quickstart
On Mon, Mar 07, 2016 at 06:02:49PM +0000, Boris Derzhavets wrote:
should be mentioned as well for HA testing. Just usual gamer's PC
with 32/64 GB RAM would work fine.
I don't think it's necessary to call out specific hardware. We're not
really targeting performance here -- if that's your goal, then the
virt-only quickstart environment is probably not appropriate for you
in any case.
[BD] I just mentioned cheapest 8 Core CPU responsible to bring up HA Set of VMs.
of course, Haswell based 8 Core Xeon will do it better.
Memory is not the only issue for HA test, Top lined i7 4790 is not enough
to run 5-6 VMs at a time even with 32 GB RAM.
That was a point of my message.
I am not guessing. i7 4790 dies attempting to perform config from
https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
Just 3 Controllers running at a time. BIOS was tuned by myself.
I only want virt-only quickstart environment to be fairly impressing on the
box
having reasonable cost as desktop.
The real goal is just to get things up and running so that you can test
software code changes or see what the full ha stack looks like
*without* needing dedicated infrastructure.
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
3181
days inactive
3198
days old
17 comments
7 participants
participants (7)
-
Boris Derzhavets -
Javier Pena -
John Trowbridge -
Lars Kellogg-Stedman -
Mohammed Arafa -
Perry Myers -
Ryan S. Brown