<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 14, 2017 at 4:37 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Tue, Jun 13, 2017 at 4:39 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-m_3150827675519306887gmail-">On Tue, Jun 13, 2017 at 4:18 PM, Alan Pevec <span dir="ltr"><<a href="mailto:apevec@redhat.com" target="_blank">apevec@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Numan,<br>
<br>
thanks for testing! Few questions inline:<br>
<span class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-"><br>
>>>> > I tested by deploying upstream tripleo on my local setup by making use of these rpms and it works fine for me.<br>
>>>> Did you also try upgrade tests? That was the pain-point with the previous OVS updates.<br>
>>> No. I didn't try that.<br>
</span><span class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-">> I did some testing locally and these are my findings<br>
> - After deploying a overcloud (with 3 controllers and 1 compute ) using<br>
> tripleo-quickstart, I added the OVS 2.7 repo from here [1] manually on all<br>
<br>
</span>Was [1] <a href="http://cbs.centos.org/repos/cloud7-openstack-pike-candidate/x86_64/os/" rel="noreferrer" target="_blank">http://cbs.centos.org/repos/cl<wbr>oud7-openstack-pike-candidate/<wbr>x86_64/os/</a> ?<br></blockquote><div><br></div></span><div>No. It was <a href="http://cbs.centos.org/repos/cloud7-openstack-common-candidate/x86_64/os/" target="_blank">http://cbs.centos.org/repos/cl<wbr>oud7-openstack-common-candidat<wbr>e/x86_64/os/</a></div><span class="gmail-m_3150827675519306887gmail-"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Can you confirm which openvswitch NVR you got?<br></blockquote><div><br></div></span><div>I am not sure which one. I will get back to you on this.</div><span class="gmail-m_3150827675519306887gmail-"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-"><br>
> the nodes and triggered "openstack overcloud update stack".<br>
> - For some reason update was always failing. Not sure what is going wrong<br>
> - So I manually ran "yum update" on the nodes<br>
> - OVS 2.7 was updated successfully, but the service was not restarted.<br>
<br>
</span>Numan, please report update stack failure as tripleo upstream LP bug.<br>
Adding Sofer from the upgrades team: is there upstream tripleo<br>
update/upgrade CI job which is testing with OVS 2.7 ?<br></blockquote><div><br></div></span><div>I want to test once more before I submit a bug to be really sure if I haven't done any mistake.</div></div></div></div></blockquote><div><br></div></span><div>Hi Alan, I ran the stack update again, but now with 1 controller and 1 compute node and the stack update was successful.</div><div><br></div><div>I have captured the terminal logs which you can find here - <a href="https://paste.fedoraproject.org/paste/FELBFKHNWiIwgNsd3o22~A" target="_blank">https://paste.fedoraproject.<wbr>org/paste/<wbr>FELBFKHNWiIwgNsd3o22~A</a></div><div><br></div><div>Before starting the update, I created a VM with ip 10.0.0.9. I started a ping command in the controller node from the neutron dhcp namespace. The ping was running successfully without any issues.</div><div><br></div><div>As expected, the OVS packages were updated to 2.7 and openvswitch service was not restarted. So essentially OVS 2.6 was running even after the update.</div><div><br></div><div>Then I manually restarted openvswitch service first on the compute node and then on controller node. The ping was only interrupted during the service restarts and was back again.</div><div><br></div><div>Below is the repo file I added to get the OVS 2.7</div><div><br></div><div>******************************<wbr>************************</div><div><div> cat /etc/yum.repos.d/cbs.centos.<wbr>org_repos_cloud7-openstack-<wbr>pike-candidate_x86_64_os.repo </div><div><br></div><div>[cbs.centos.org_repos_cloud7-<wbr>openstack-pike-candidate_x86_<wbr>64_os]</div><div>name=added from: <a href="http://cbs.centos.org/repos/cloud7-openstack-pike-candidate/x86_64/os" target="_blank">http://cbs.centos.org/repos/<wbr>cloud7-openstack-pike-<wbr>candidate/x86_64/os</a></div><div>baseurl=<a href="http://cbs.centos.org/repos/cloud7-openstack-pike-candidate/x86_64/os" target="_blank">http://cbs.centos.org/<wbr>repos/cloud7-openstack-pike-<wbr>candidate/x86_64/os</a></div><div>enabled=1</div><div>gpgcheck=0</div><div>includepkgs=openvswitch,<wbr>openvswitch-ovn*</div></div><div>******************************<wbr>***************************</div><div> </div><div><br></div></div></div></div></blockquote><div><br></div><div><br></div><div>Hi Alan,</div><div><br></div><div>Just an update. The test review [1] is passing for the CI job - <a href="http://logs.openstack.org/91/473191/9/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/3307cf8/" rel="nofollow" style="color:rgb(6,84,172);font-family:sans-serif">gate-tripleo-ci-centos-7-multinode-upgrades-nv</a>.</div><div>This patch updates the OVS to OVS 2.7 from the repo - <a href="http://cbs.centos.org/repos/cloud7-openstack-pike-candidate/x86_64/os" target="_blank">http://cbs.centos.org/<wbr>repos/cloud7-openstack-pike-<wbr>candidate/x86_64/os</a> and restarts it at step0 of the upgrade.</div><div><br></div><div><br></div><div>[1] - <a href="https://review.openstack.org/#/c/473191/">https://review.openstack.org/#/c/473191/</a></div><div> <a href="http://logs.openstack.org/91/473191/9/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/3307cf8/logs/subnode-2/var/log/ovs_debug.txt.gz">http://logs.openstack.org/91/473191/9/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/3307cf8/logs/subnode-2/var/log/ovs_debug.txt.gz</a></div><div> <a href="http://logs.openstack.org/91/473191/9/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/3307cf8/logs/subnode-2/var/log/openvswitch/ovsdb-server.txt.gz">http://logs.openstack.org/91/473191/9/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/3307cf8/logs/subnode-2/var/log/openvswitch/ovsdb-server.txt.gz</a></div><div><br></div><div>Thanks</div><div>Numan</div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>Thanks</div><span class="gmail-HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="gmail-h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>I don't think there is any CI job yet. I tried to be bold and submitted a patch to test it here (please note it's only for testing) - <a href="https://review.openstack.org/#/c/473191/" target="_blank">https://review.openstack.org<wbr>/#/c/473191/</a></div><div><br></div><div>In the puppet/services/openvswitch_up<wbr>grade.yaml, I tried to install the OVS 2.7 and restart it.</div><div>From the logs I could see that ovs was never updated (not sure why) and the ansible task to restart the openvswitch (still with version 2.6) makes the node loose the network completely. I will explore more on this job and see how it goes.</div><div><br></div><div>But from what I understand upgrading OVS (presently) has a down time overhead. i.e We can update the ovs package. But restarting OVS with the updated version would cause some down time or reboot. I may be wrong here. But that's what I understand. If this is the case, not sure if we can test this scenario in CI other than just updating the package without restart.</div><span class="gmail-m_3150827675519306887gmail-"><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-"><br>
> - After the update everything was working fine.<br>
<br>
</span>Working fine without ovs service restart?<br></blockquote><div><br></div></span><div>That's right.</div><span class="gmail-m_3150827675519306887gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-"><br>
> - Then I restarted all the nodes to have OVS 2.7 started up.<br>
> - After restart I was able to have my tenant networks working fine.<br>
<br>
</span>So that sounds promising but I'd like to get details on update stack<br>
failure before we proceed promotion OVS 2.7 packages to the RDO Pike<br>
repo.<br>
Also note that 2.7.0-1 RPM from v2.7.0 tag is missing > 100 backports<br>
on branch-2.7, upstream should really consider releasing 2.7.1.<br>
<br></blockquote><div><br></div></span><div>I have requested for a new tag in the OVS mailing list. Hope it would be done soon.</div><div><br></div><div>Thanks</div><span class="gmail-m_3150827675519306887gmail-HOEnZb"><font color="#888888"><div>Numan</div></font></span><span class="gmail-m_3150827675519306887gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Cheers,<br>
Alan<br>
<div class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-HOEnZb"><div class="gmail-m_3150827675519306887gmail-m_-3601032393120618961gmail-h5"><br>
> Thanks<br>
> Numan<br>
<br>
>>>> > Is it possible to cross-tag for RDO pike ?<br>
>>>> I've tagged openvswitch-2.7.0-1.el7 for pike-candidate for now so we<br>
>>>> can ran full RDO CI on it.<br>
>>>> Alfredo, David, Emilien, can you please run pipeline jobs with<br>
>>>> <a href="http://cbs.centos.org/repos/cloud7-openstack-pike-candidate/x86_64/os/" rel="noreferrer" target="_blank">http://cbs.centos.org/repos/cl<wbr>oud7-openstack-pike-candidate/<wbr>x86_64/os/</a><br>
>>>> repo added ?<br>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>