<div dir="ltr"><font color="#444444" face="Courier New, Courier, Fixed"><div style=""><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span class="" style="margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background-image:initial;background-color:transparent;background-repeat:initial"><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span class="" style="margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background-image:initial;background-color:transparent;background-repeat:initial"><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">Hi all,

I have a rdo kilo deployment, using sr-iov ports to my instances. I'm trying to configure NUMA topology and CPU pinning for some telco based workloads based on this doc: </span></font></span><span style="font-size:13.3333px;line-height:21.3333px;background-color:transparent"><font face="Courier New, Courier, Fixed"><a href="http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/">http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/</a></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="font-family:'Courier New',Courier,Fixed;font-size:13.3333px;line-height:21.3333px;background-color:transparent">I have 3 compute nodes, I'm trying to use one of them to use cpu pinning.</span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="font-family:'Courier New',Courier,Fixed;font-size:13.3333px;line-height:21.3333px;background-color:transparent">I've configured it like this:</span><br></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><b>Compute Node (total 24 cpus)</b>

<b>/etc/nova/nova.conf</b>
vcpu_pin_set=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23

Changed grub to isolate my cpus:
#grubby --update-kernel=ALL --args="isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23"</span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;font-size:10pt;vertical-align:baseline;font-stretch:normal;line-height:21.3333px;font-family:'Courier New',Courier,Fixed;overflow:auto;background-image:initial;background-repeat:initial"><span class="" style="margin:0px;padding:0px;border:0px;outline:0px;font-size:13.3333px;vertical-align:baseline;background:transparent">#grub2-install /dev/sda</span></pre><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">

<b>Controller Nodes:</b>

<b>/etc/nova/nova.conf</b>
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

<b>Created host aggregate performance </b>
#nova aggregate-create performance
#nova aggregate-set-metadata 1 pinned=true</span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">#</span></font></span><span style="font-size:13.3333px;font-family:'Courier New',Courier,Fixed;line-height:21.3333px;background-color:transparent">nova aggregate-add-host 1 compute03</span><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><br></span></font></span><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><br></span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span class="" style="margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background-image:initial;background-color:transparent;background-repeat:initial"><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><b>Created host aggregate normal</b>
#nova aggregate-create normal
#nova aggregate-set-metadata 2 pinned=false</span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">#</span></font></span><span style="font-size:13.3333px;font-family:'Courier New',Courier,Fixed;line-height:21.3333px;background-color:transparent">nova aggregate-add-host 2 compute01</span><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><br></span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">#</span></font></span><span style="font-size:13.3333px;font-family:'Courier New',Courier,Fixed;line-height:21.3333px;background-color:transparent">nova aggregate-add-host 2 compute02</span></pre></span></pre><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">
</span><b style="font-size:13.3333px;line-height:21.3333px">Created the flavor with cpu pinning</b><span style="font-size:13.3333px;line-height:21.3333px">
#nova flavor-create m1.performance 6 2048 20 4
#nova flavor-key 6 set hw:cpu_policy=dedicated
#nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true

</span><b style="font-size:13.3333px;line-height:21.3333px">The issue is:</b><span style="font-size:13.3333px;line-height:21.3333px">
With SR-IOV ports it only let's me create instances with 6 vcpus in total with the conf described above. Without SR-IOV, using OVS, I don't have that limitation. Is this a bug or something? I've seen this: <a href="https://bugs.launchpad.net/nova/+bug/1441169">https://bugs.launchpad.net/nova/+bug/1441169</a>, however I have the patch, and as I said it works for the first 6 vcpus with my configuration. 
<br></span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><b>Some relevant logs:</b></span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px"><b>/var/log/nova/nova-scheduler.log</b></span></font></span></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="font-size:13.3333px;line-height:21.3333px;font-family:'Courier New',Courier,Fixed;background-color:transparent">2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70</span><br></pre><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.957 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter PciPassthroughFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
<b>2015-11-06 11:18:17.959 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter NUMATopologyFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84</b>
</span></font></span></pre><div><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">Any help would be appreciated.</span></font></span></pre></div><pre class="" style="margin-top:0px;margin-bottom:11px;padding:0px;border:0px;outline:0px;vertical-align:baseline;font-stretch:normal;overflow:auto;background-image:initial;background-repeat:initial"><span style="background-color:transparent"><font face="Courier New, Courier, Fixed"><span style="font-size:13.3333px;line-height:21.3333px">
Thanks,
Pedro Sousa<br></span></font></span></pre></span></pre></span></pre></div></font></div>