[Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been built per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md

Boris Derzhavets bderzhavets at hotmail.com
Tue Feb 23 18:02:55 UTC 2016


Recreated keystone endpoint for swift-proxy to  8090 && service httpd restart on Controller1.
Updated  proxy-swift.conf && restarted  service proxy-swift on all nodes.
Opened port 8090 on all nodes. Run as admin on 192.169.142.221 (hacontroller1) :-
# swift list
in swift.log

Feb 23 20:54:50 hacontroller1 account-server: 192.169.142.222 - - [23/Feb/2016:17:54:50 +0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD http://localhost/v1/.expiring_objects" "tx7a654a6dc9cd4aca83de0-0056cc9cea" "proxy-server 19638" 0.0007 "-" 22509 -

________________________________________
From: rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on behalf of Boris Derzhavets <bderzhavets at hotmail.com>
Sent: Tuesday, February 23, 2016 7:55 AM
To: Javier Pena
Cc: rdo-list
Subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md

Cluster (VM based)   is up and running . Keepalived is good shape on all nodes
Follow https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-config.md

On each node :-

[root at hacontroller1 ~(keystone_admin)]# openstack-status | grep swift
openstack-swift-proxy:                    active
openstack-swift-account:                active
openstack-swift-container:              active
openstack-swift-object:                    active

[root at hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202
tcp        0      0 192.169.142.221:6202    0.0.0.0:*               LISTEN      19929/python2
tcp        0      0 192.169.142.221:6202    192.169.142.222:44530   TIME_WAIT   -
tcp        0      0 192.169.142.221:6202    192.169.142.221:38513   TIME_WAIT   -

[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928

root     13985  5991  0 14:51 pts/0    00:00:00 grep --color=auto 19928
swift    19928     1  0 14:15 ?        00:00:12 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift    19981 19928  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift    19982 19928  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift    19983 19928  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924

root     14514  5991  0 14:51 pts/0    00:00:00 grep --color=auto 19924
swift    19924     1  0 14:15 ?        00:00:12 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift    19994 19924  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift    19995 19924  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf

[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929
root     14662  5991  0 14:51 pts/0    00:00:00 grep --color=auto 19929
swift    19929     1  0 14:15 ?        00:00:12 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift    19985 19929  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift    19986 19929  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf

Ports are open on each node.

I am getting :-

[root at hacontroller1 ~(keystone_admin)]# swift list
Account GET failed: http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1309884?format=json 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

[root at hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080
tcp        0      0 192.169.142.221:8080    0.0.0.0:*               LISTEN      19920/python2
tcp        0      0 192.169.142.220:8080    0.0.0.0:*               LISTEN      1569/haproxy
tcp        0      0 192.169.142.221:60969   192.169.142.220:8080    TIME_WAIT   -

So , I guess it's not supposed to respond swift.  I am missing something here.

[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920
swift    19920     1  0 14:15 ?        00:00:02 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    19996 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    19997 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    19998 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    19999 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    20000 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    20001 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    20002 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    20003 19920  0 14:15 ?        00:00:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root     29348  5991  0 14:21 pts/0    00:00:00 grep --color=auto 19920

[root at hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569
root      1569  1547  0 12:33 ?        00:00:22 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root     29973  5991  0 14:21 pts/0    00:00:00 grep --color=auto 1569

controller-vip.example.com(VIP) address is 192.169.142.220
controller-vip.example.com has haproxy listening on 8080  , not swift-proxy-server.
/var/log/swift/swift.log is empty

First error in /var/log/messages
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) (client_ip: 192.169.142.223)
Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0)

was fixed per https://ask.openstack.org/en/question/57608/proxy-server-error-insufficient-storage-10001556002sdb1/

But , I am still getting in /var/log/messages

Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in run_once#012    self.swift.get_account_info(self.expiring_objects_account)#012  File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 358, in get_account_info#012    resp = self.make_request('HEAD', path, {}, acceptable_statuses)#012  File "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line 194, in make_request#012    _('Unexpected response: %s') % resp.status, resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn: txd6ecc9e8f9eb46a284d8a-0056cc55b6)
Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - [23/Feb/2016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD http://localhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c3" "proxy-server 351" 0.0002 "-" 28097 -


Please, advise.

Boris.


_______________________________________________
Rdo-list mailing list
Rdo-list at redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

To unsubscribe: rdo-list-unsubscribe at redhat.com




More information about the dev mailing list