From bderzhavets at hotmail.com Tue Feb 23 07:55:27 2016 Content-Type: multipart/mixed; boundary="===============1342161746327161463==" MIME-Version: 1.0 From: Boris Derzhavets To: dev at lists.rdoproject.org Subject: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md Date: Tue, 23 Feb 2016 12:55:23 +0000 Message-ID: In-Reply-To: 56C75110.5070501@redhat.com --===============1342161746327161463== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Cluster (VM based) is up and running . Keepalived is good shape on all no= des Follow https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swif= t-config.md On each node :- = [root(a)hacontroller1 ~(keystone_admin)]# openstack-status | grep swift openstack-swift-proxy: active openstack-swift-account: active openstack-swift-container: active openstack-swift-object: active [root(a)hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202 tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN = 19929/python2 = tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_WA= IT - = tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_WA= IT - = = [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928 root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19928 swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/s= wift-object-server /etc/swift/object-server.conf swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-object-server /etc/swift/object-server.conf swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-object-server /etc/swift/object-server.conf swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-object-server /etc/swift/object-server.conf [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924 root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19924 swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/s= wift-container-server /etc/swift/container-server.conf swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-container-server /etc/swift/container-server.conf swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-container-server /etc/swift/container-server.conf [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929 root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19929 swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2 /usr/bin/s= wift-account-server /etc/swift/account-server.conf swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-account-server /etc/swift/account-server.conf swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-account-server /etc/swift/account-server.conf Ports are open on each node. I am getting :- [root(a)hacontroller1 ~(keystone_admin)]# swift list Account GET failed: http://controller-vip.example.com:8080/v1/AUTH_acdc927b= 53bd43ae9a7ed657d1309884?format=3Djson 503 Service Unavailable [first 60 c= hars of response]

Service Unavailable

The server is curren= tly [root(a)hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080 tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN = 19920/python2 = tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN = 1569/haproxy = tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_WA= IT - = So , I guess it's not supposed to respond swift. I am missing something he= re. [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920 swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2 /usr/bin/s= wift-proxy-server /etc/swift/proxy-server.conf root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=3Dauto 19920 [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569 root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f /etc/h= aproxy/haproxy.cfg -p /run/haproxy.pid -Ds root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=3Dauto 1569 controller-vip.example.com(VIP) address is 192.169.142.220 controller-vip.example.com has haproxy listening on 8080 , not swift-proxy= -server. /var/log/swift/swift.log is empty First error in /var/log/messages = Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.= 169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) (client_ip: = 192.169.142.223) Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage 192.= 169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) was fixed per https://ask.openstack.org/en/question/57608/proxy-server-erro= r-insufficient-storage-10001556002sdb1/ But , I am still getting in /var/log/messages Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: #012Trac= eback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/= swift/obj/expirer.py", line 169, in run_once#012 self.swift.get_account_= info(self.expiring_objects_account)#012 File "/usr/lib/python2.7/site-pack= ages/swift/common/internal_client.py", line 358, in get_account_info#012 = resp =3D self.make_request('HEAD', path, {}, acceptable_statuses)#012 Fil= e "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line = 194, in make_request#012 _('Unexpected response: %s') % resp.status, res= p)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable (txn= : txd6ecc9e8f9eb46a284d8a-0056cc55b6) Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - [23/Feb/2= 016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HEAD http://l= ocalhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c3" "proxy-= server 351" 0.0002 "-" 28097 - Please, advise. Boris. --===============1342161746327161463==--