From bderzhavets at hotmail.com Tue Feb 23 13:39:36 2016 Content-Type: multipart/mixed; boundary="===============2619553471440088200==" MIME-Version: 1.0 From: Boris Derzhavets To: dev at lists.rdoproject.org Subject: Re: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived Controller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md Date: Tue, 23 Feb 2016 18:39:30 +0000 Message-ID: In-Reply-To: 1979163399.31773659.1456251492719.JavaMail.zimbra@redhat.com --===============2619553471440088200== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ________________________________________ From: Javier Pena Sent: Tuesday, February 23, 2016 1:18 PM To: rdo-list Cc: Boris Derzhavets Subject: Re: [Rdo-list] Attempt to add Swift to 3 Node HAProxy\Keepalived C= ontroller been per https://github.com/beekhof/osp-ha-deploy/blob/master/HA-= keepalived.md ----- Original Message ----- > > Cluster (VM based) is up and running . Keepalived is good shape on all > nodes > Follow > https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/swift-con= fig.md > > On each node :- > > [root(a)hacontroller1 ~(keystone_admin)]# openstack-status | grep swift > openstack-swift-proxy: active > openstack-swift-account: active > openstack-swift-container: active > openstack-swift-object: active > > [root(a)hacontroller1 ~(keystone_admin)]# netstat -antp | grep 6202 > tcp 0 0 192.169.142.221:6202 0.0.0.0:* LISTEN > 19929/python2 > tcp 0 0 192.169.142.221:6202 192.169.142.222:44530 TIME_= WAIT > - > tcp 0 0 192.169.142.221:6202 192.169.142.221:38513 TIME_= WAIT > - > > [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19928 > > root 13985 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19928 > swift 19928 1 0 14:15 ? 00:00:12 /usr/bin/python2 > /usr/bin/swift-object-server /etc/swift/object-server.conf > swift 19981 19928 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-object-server /etc/swift/object-server.conf > swift 19982 19928 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-object-server /etc/swift/object-server.conf > swift 19983 19928 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-object-server /etc/swift/object-server.conf > [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19924 > > root 14514 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19924 > swift 19924 1 0 14:15 ? 00:00:12 /usr/bin/python2 > /usr/bin/swift-container-server /etc/swift/container-server.conf > swift 19994 19924 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-container-server /etc/swift/container-server.conf > swift 19995 19924 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-container-server /etc/swift/container-server.conf > > [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19929 > root 14662 5991 0 14:51 pts/0 00:00:00 grep --color=3Dauto 19929 > swift 19929 1 0 14:15 ? 00:00:12 /usr/bin/python2 > /usr/bin/swift-account-server /etc/swift/account-server.conf > swift 19985 19929 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-account-server /etc/swift/account-server.conf > swift 19986 19929 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-account-server /etc/swift/account-server.conf > > Ports are open on each node. > > I am getting :- > > [root(a)hacontroller1 ~(keystone_admin)]# swift list > Account GET failed: > http://controller-vip.example.com:8080/v1/AUTH_acdc927b53bd43ae9a7ed657d1= 309884?format=3Djson > 503 Service Unavailable [first 60 chars of response]

Service > Unavailable

The server is currently > > [root(a)hacontroller1 ~(keystone_admin)]# netstat -antp | grep 8080 > tcp 0 0 192.169.142.221:8080 0.0.0.0:* LISTEN > 19920/python2 > tcp 0 0 192.169.142.220:8080 0.0.0.0:* LISTEN > 1569/haproxy > tcp 0 0 192.169.142.221:60969 192.169.142.220:8080 TIME_= WAIT > - > > So , I guess it's not supposed to respond swift. I am missing something > here. > > [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 19920 > swift 19920 1 0 14:15 ? 00:00:02 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 19996 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 19997 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 19998 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 19999 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 20000 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 20001 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 20002 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > swift 20003 19920 0 14:15 ? 00:00:00 /usr/bin/python2 > /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf > root 29348 5991 0 14:21 pts/0 00:00:00 grep --color=3Dauto 19920 > > [root(a)hacontroller1 ~(keystone_admin)]# ps -ef | grep 1569 > root 1569 1547 0 12:33 ? 00:00:22 /usr/sbin/haproxy -f > /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds > root 29973 5991 0 14:21 pts/0 00:00:00 grep --color=3Dauto 1569 > > controller-vip.example.com(VIP) address is 192.169.142.220 > controller-vip.example.com has haproxy listening on 8080 , not > swift-proxy-server. > /var/log/swift/swift.log is empty > > First error in /var/log/messages > Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage > 192.169.142.222:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) > (client_ip: 192.169.142.223) > Feb 23 15:25:37 hacontroller1 proxy-server: ERROR Insufficient Storage > 192.169.142.223:6202/vdb (txn: tx511a1757780140d08cf15-0056cc4fc0) > > was fixed per > https://ask.openstack.org/en/question/57608/proxy-server-error-insufficie= nt-storage-10001556002sdb1/ > > But , I am still getting in /var/log/messages > > Feb 23 15:51:02 hacontroller1 object-expirer: Unhandled exception: > #012Traceback (most recent call last):#012 File > "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 169, in > run_once#012 > self.swift.get_account_info(self.expiring_objects_account)#012 File > "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line > 358, in get_account_info#012 resp =3D self.make_request('HEAD', path, = {}, > acceptable_statuses)#012 File > "/usr/lib/python2.7/site-packages/swift/common/internal_client.py", line > 194, in make_request#012 _('Unexpected response: %s') % resp.status, > resp)#012UnexpectedResponse: Unexpected response: 503 Service Unavailable > (txn: txd6ecc9e8f9eb46a284d8a-0056cc55b6) > Feb 23 15:51:15 hacontroller1 account-server: 192.169.142.223 - - > [23/Feb/2016:12:51:15 +0000] "HEAD /vdb/3926/.expiring_objects" 507 - "HE= AD > http://localhost/v1/.expiring_objects" "tx088b0d9b2d814c56b3f8b-0056cc55c= 3" > "proxy-server 351" 0.0002 "-" 28097 - > Hi Boris, >From the errors above, this looks like a Swift configuration issue. Where = did you mount the file system for vdb? In the instructions, they are mounte= d under /srv/node/vdb . If it is mounted outside /srv, I remember having SE= Linux issues with it. Javier, I have updated haproxy.cfg and finally ( haproxy restart ) got back ( and = understood haproxy monitoring) :- [root(a)hacontroller2 ~(keystone_admin)]# swift list Account GET failed: http://controller-vip.example.com:8090/v1/AUTH_acdc927b= 53bd43ae9a7ed657d1309884?format=3Djson 503 Service Unavailable [first 60 c= hars of response]

Service Unavailable

The server is curren= tly [root(a)hacontroller2 ~(keystone_admin)]# netstat -antp | grep 8090 tcp 0 0 192.169.142.220:8090 0.0.0.0:* LISTEN = 27323/haproxy = tcp 0 0 192.169.142.222:8090 0.0.0.0:* LISTEN = 19645/python2 = tcp 0 0 192.169.142.222:39922 192.169.142.220:8090 TIME_WA= IT - = On all nodes [root(a)hacontroller2 ~(keystone_admin)]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 41G 11G 30G 28% / devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 96K 1.9G 1% /dev/shm tmpfs 1.9G 17M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/vdb1 20G 33M 20G 1% /srv/node/vdb1 /dev/vda1 497M 167M 330M 34% /boot = Also, how big is the vdb disk? In my tests, I used a 8 GB virtual disk, I'm= not sure how small we could make it. 20 GB Thank you. Boris. Regards, Javier > > Please, advise. > > Boris. > > > _______________________________________________ > Rdo-list mailing list > Rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > --===============2619553471440088200==--