[Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
Boris Derzhavets
bderzhavets at hotmail.com
Tue Feb 23 12:55:23 UTC 2016
Cluster (VM based) is up and running . Keepalived is good shape on all nodes
Follow https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-config.md
On each node :-
[root at hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy: active
openstack-swift-account: active
openstack-swift-container: active
openstack-swift-object: active
[root at hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN 19929/python2
tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WAIT -
tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WAIT -
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928
root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19928
swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924
root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19924
swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=auto 19929
swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
Ports are open on each node.
I am getting :-
[root at hacontroller1 ~(keystone_admin)]# swift list
Account GET failed: http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1309884?format=json 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
[root at hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN 19920/python2
tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN 1569/haproxy
tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WAIT -
So , I guess it's not supposed to respond swift. I am missing something here.
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=auto 19920
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=auto 1569
controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080 , not swift-proxy-server.
/var/log/swift/swift.log is empty
First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) (client_ip: 192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)
was fixed per https://ask.openstack.org/en/question/57608/proxy-server-error-insufficient-storage-10001556002sdb1/
But , I am still getting in /var/log/messages
Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in run_once#012 self.swift.get_account_info(self.expiring_objects_account)#012 File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 358, in get_account_info#012 resp = self.make_request('HEAD', path, {}, acceptable_statuses)#012 File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 194, in make_request#012 _('Unexpected response: %s') % resp.status, resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn: txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - [23/Feb/2016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD http://localhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c3" "proxy-server 351" 0.0002 "-" 28097 -
Please, advise.
Boris.
More information about the dev
mailing list