[rdo-list] RDO TripleO Mitaka Non-HA Overcloud Failing
Christopher Brown
cbrown2 at ocf.co.uk
Wed Aug 3 19:58:18 UTC 2016
Hello,
On Wed, 2016-08-03 at 19:40 +0100, Gunjan, Milind [CTO] wrote:
> Hi All,
>
> I am currently working on Tripleo Mitaka Openstack deployment on
> baremetal servers:
> Undercloud – 1 baremetal server with 2 NIC (1 for provisioning and
> 2nd for external network connectivity)
> Controller – 1 baremetal server ( 6 NICs with each openstack VLANs on
> separate NIC)
> Compute – 1 baremetal server
>
> I followed Graeme's instructions here : https://www.redhat.com/archi
> ves/rdo-list/2016-June/msg00049.html to set up Undercloud .
> Undercloud deployment was successful and all the images required for
> overcloud deployment was properly built as per the instruction. I
> would like to mention that I used libvirt tools to modify the root
> password on overcloud-full.qcow2 and we also modified the grub file
> to include “net.ifnames=0 biosdevname=0” to restore old interface
> naming.
I don't think there is a problem doing this but the new naming
convention does allow you to specifically target nics during deployment
with reliability.
> I was able to successfully introspect 2 serves to be used for
> controller and compute nodes. Also , we added the serial device
> discovered during introspection as root device:
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/root_device='{"serial":
> "618e728372833010c79bead9066f0f9e"}'
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/root_device='{"serial":
> "618e7283728347101f2107b511603adc"}'
Sure, we use wwn values. I hacked the following which is a bit nasty
but hey, it works. There are probably easier ways to do it but...
https://github.com/cbrown2/openstack-scripts/blob/master/root_device_co
nfig.sh
> Next, we added compute and control tag to respective introspected
> node with local boot option:
>
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/capabilities='profile:control,boot_option:local'
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/capabilities='profile:compute,boot_option:local'
I would just add this parameter to the instackenv.json file - one less
thing to have to run.
>
> We used multiple NIC templates for control and compute node which has
> been attached along with network-environment.yaml file. Default
> network isolation template file has been used.
>
>
> Deployment script looks like this :
> #!/bin/bash
> DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
> template_base_dir="$DIR"
> ntpserver=<sprint.ntp.server.ip> #Sprint LAB
> openstack overcloud deploy --templates \
> -e /usr/share/openstack-tripleo-heat-templates/environments/network-
> isolation.yaml \
> -e ${template_base_dir}/environments/network-environment.yaml \
> --control-flavor control --compute-flavor compute \
> --control-scale 1 --compute-scale 1 \
> --ntp-server $ntpserver \
> --neutron-network-type vxlan --neutron-tunnel-types vxlan --debug
I'm not sure why you set ntp and templates variable but meh. I'd really
be inclined to drop network isolation and see if it still deploys.
> Heat stack deployment goes on more really long time (more than 4
> hours) and gets stuck at postdeployment configurations. Please find
> below the capture during install :
You can change the deploy timeout with:
--timeout 120
for 2 hours for example.
>
> Every 2.0s: ironic node-list && nova list && heat stack-list && heat
> resource-list -n5 overcloud | grep -vi complete
> Wed Aug 3 17:33:37 2016
>
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> | UUID | Name | Instance
> UUID | Power State | Provisioning State |
> Maintenance |
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> | 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 | None | 9e7aae15-cabc-4489-
> a1b2-778915a78df2 | power on | active | False |
> | afcfbee3-3108-48da-a6da-aba8f422642c | None | c1ab52a9-461a-4a11-
> a13e-e57ff0a3ae2a | power on | active | False |
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> | ID | Name |
> Status | Task State | Power State | Networks |
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> | 9e7aae15-cabc-4489-a1b2-778915a78df2 | overcloud-controller-0 |
> ACTIVE | - | Running | ctlplane=192.168.149.9 |
> | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a | overcloud-novacompute-0 |
> ACTIVE | - | Running | ctlplane=192.168.149.8 |
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> | id | stack_name | stack_status |
> creation_time | updated_time |
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> | 26ee0150-4cfa-4268-9107-8bfbf6712913 | overcloud | CREATE_FAILED |
> 2016-08-03T08:11:34 | None |
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> +---------------------------------------------+--------------------
> ---------------------------+-----------------------------------------
> -------------------------------
> ---------+--------------------+---------------------+--------------
> -------------------------------------------------------------------
> ------------------------------+
> | resource_name |
> physical_resource_id | resource_type
> | resource_status | updated_time |
> stack_name
> |
> +---------------------------------------------+--------------------
> ---------------------------+-----------------------------------------
> -------------------------------
> ---------+--------------------+---------------------+--------------
> -------------------------------------------------------------------
> ------------------------------+
> | ComputeNodesPostDeployment | 3797aec6-e543-4dda-
> 9cd1-c7261e827a64 | OS::TripleO::ComputePostDeployment
> | CREATE_FAILED | 2016-08-03T08:11:35 |
> overcloud
> |
> | ControllerNodesPostDeployment | 6ad9f88c-5c55-4125-
> 97f1-eb0e33329d16 | OS::TripleO::ControllerPostDeployment
> | CREATE_FAILED | 2016-08-03T08:11:35 |
> overcloud
> |
> | ComputePuppetDeployment | 8b199f85-e4f9-48ad-
> 9aee-b1cdf4900b9f | OS::Heat::StructuredDeployments
> | CREATE_FAILED | 2016-08-03T08:29:19 | overcloud-
> ComputeNodesPostDeployment-
> 6vxfu2g2qucy
> |
> | ControllerOvercloudServicesDeployment_Step4 | 15509f59-ff28-43af-
> 95dd-6247a6a32c2d | OS::Heat::StructuredDeployments
> | CREATE_FAILED | 2016-08-03T08:29:20 | overcloud-
> ControllerNodesPostDeployment-35y7uafngfwj
> |
> | 0 | 7cd0aa3d-742f-4e78-
> 99ca-b2a575913f8e | OS::Heat::StructuredDeployment
> | CREATE_IN_PROGRESS | 2016-08-03T08:30:04 | overcloud-
> ComputeNodesPostDeployment-6vxfu2g2qucy-ComputePuppetDeployment-
> cpahcct3tfw3 |
> | 0 | 5e9308f7-c3a9-4a94-
> a017-e1acb694c036 | OS::Heat::StructuredDeployment
>
>
> [stack at mitaka-uc ~]$ openstack software deployment show 5e9308f7-
> c3a9-4a94-a017-e1acb694c036
> +---------------+--------------------------------------+
> | Field | Value |
> +---------------+--------------------------------------+
> | id | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
> | server_id | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
> | config_id | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
> | creation_time | 2016-08-03T08:32:10 |
> | updated_time | |
> | status | IN_PROGRESS |
> | status_reason | Deploy data available |
> | input_values | {} |
> | action | CREATE |
> +---------------+--------------------------------------+
>
> [stack at mitaka-uc ~]$ openstack software deployment show --long
> 5e9308f7-c3a9-4a94-a017-e1acb694c036
> +---------------+--------------------------------------+
> | Field | Value |
> +---------------+--------------------------------------+
> | id | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
> | server_id | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
> | config_id | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
> | creation_time | 2016-08-03T08:32:10 |
> | updated_time | |
> | status | IN_PROGRESS |
> | status_reason | Deploy data available |
> | input_values | {} |
> | action | CREATE |
> | output_values | None |
> +---------------+--------------------------------------+
>
> [stack at mitaka-uc ~]$ openstack stack resource list 3797aec6-e543-
> 4dda-9cd1-c7261e827a64
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
> | resource_name | physical_resource_id |
> resource_type | resource_status |
> updated_time |
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
> | ComputeArtifactsConfig | a33cd04d-61ab-4429-8565-182409c2b97f |
> file:///usr/share/openstack-tripleo-heat- | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> | | |
> templates/puppet/deploy-artifacts.yaml |
> | |
> | ComputePuppetConfig | 5bb712b0-5358-46c7-a444-f9adedfedd50 |
> OS::Heat::SoftwareConfig | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> | ComputePuppetDeployment | 8b199f85-e4f9-48ad-9aee-b1cdf4900b9f |
> OS::Heat::StructuredDeployments | CREATE_FAILED |
> 2016-08-03T08:29:19 |
> | ComputeArtifactsDeploy | 1d13bf34-fc66-4bf1-a3b7-1dd815f58f5a |
> OS::Heat::StructuredDeployments | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> | ExtraConfig | |
> OS::TripleO::NodeExtraConfigPost | INIT_COMPLETE |
> 2016-08-03T08:29:19 |
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
>
> [stack at mitaka-uc ~]$ openstack stack resource list 8b199f85-e4f9-
> 48ad-9aee-b1cdf4900b9f
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> | resource_name | physical_resource_id |
> resource_type | resource_status |
> updated_time |
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> | 0 | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
> OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2016-08-
> 03T08:30:04 |
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> [stack at mitaka-uc ~]$ openstack software deployment show 7cd0aa3d-
> 742f-4e78-99ca-b2a575913f8e
> +---------------+--------------------------------------+
> | Field | Value |
> +---------------+--------------------------------------+
> | id | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
> | server_id | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a |
> | config_id | 24e5c0db-f84f-4a94-8f8e-8e38e73ccc86 |
> | creation_time | 2016-08-03T08:30:05 |
> | updated_time | |
> | status | IN_PROGRESS |
> | status_reason | Deploy data available |
> | input_values | {} |
> | action | CREATE |
> +---------------+--------------------------------------+
>
<snip>
I'd be inclined to ssh into the nodes themselves and check the logs.
Also watch:
http://www.anstack.com/blog/2016/07/22/tripleo-deep-dive-session-3.html
for further assistance with debugging from the experts. It just looks
like post deploy puppet config has gone wrong so I don't think you are
far off.
heat --help
will give the various options for drilling down to debug the failure on
the various levels.
>
> Please let me know if there is any other logs which I can provide
> that can help in troubleshooting.
>
>
> Thanks a lot in Advance for your help and support.
>
> Best Regards,
> Milind Gunjan
>
>
>
> This e-mail may contain Sprint proprietary information intended for
> the sole use of the recipient(s). Any use by others is prohibited. If
> you are not the intended recipient, please contact the sender and
> delete all copies of the message.
--
Regards,
Christopher Brown
OpenStack Engineer
OCF plc
Tel: +44 (0)114 257 2200
Web: www.ocf.co.uk
Blog: blog.ocf.co.uk
Twitter: @ocfplc
More information about the dev
mailing list