[rdo-list] RDO TripleO Mitaka Non-HA Overcloud Failing

Marius Cornea marius at remote-lab.net
Wed Aug 3 20:00:26 UTC 2016


Hi,

Could you please ssh to the nodes, gather the os-collect-config
journals (journalctl -l -u os-collect-config) and attach them here?

Thank you,
Marius

On Wed, Aug 3, 2016 at 8:40 PM, Gunjan, Milind [CTO]
<Milind.Gunjan at sprint.com> wrote:
> Hi All,
>
>
>
> I am currently working on Tripleo Mitaka Openstack deployment on baremetal
> servers:
>
> Undercloud – 1 baremetal server with 2 NIC (1 for provisioning and 2nd for
> external network connectivity)
>
> Controller – 1 baremetal server ( 6 NICs with each openstack VLANs on
> separate NIC)
>
> Compute – 1 baremetal server
>
>
>
> I  followed Graeme's instructions here :
> https://www.redhat.com/archives/rdo-list/2016-June/msg00049.html to set up
> Undercloud . Undercloud deployment was successful and all the images
> required for overcloud deployment was properly built as per the instruction.
> I would like to mention that I used libvirt tools to modify the root
> password on overcloud-full.qcow2 and we also modified the grub file to
> include “net.ifnames=0 biosdevname=0” to restore old interface naming.
>
>
>
> I was able to successfully introspect 2 serves to be used for controller and
> compute nodes. Also , we added the serial device discovered during
> introspection as root device:
>
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/root_device='{"serial": "618e728372833010c79bead9066f0f9e"}'
>
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/root_device='{"serial": "618e7283728347101f2107b511603adc"}'
>
>
>
> Next, we added compute and control tag to respective introspected node with
> local boot option:
>
>
>
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/capabilities='profile:control,boot_option:local'
>
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/capabilities='profile:compute,boot_option:local'
>
>
>
> We used multiple NIC templates for control and compute node which has been
> attached along with network-environment.yaml file. Default network isolation
> template file has been used.
>
>
>
>
>
> Deployment script looks like this :
>
> #!/bin/bash
>
> DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
>
> template_base_dir="$DIR"
>
> ntpserver=<sprint.ntp.server.ip> #Sprint LAB
>
> openstack overcloud deploy --templates \
>
> -e
> /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml
> \
>
> -e ${template_base_dir}/environments/network-environment.yaml \
>
> --control-flavor control --compute-flavor compute  \
>
> --control-scale 1 --compute-scale 1 \
>
> --ntp-server $ntpserver \
>
> --neutron-network-type vxlan --neutron-tunnel-types vxlan --debug
>
>
>
> Heat stack deployment goes on more really long time (more than 4 hours) and
> gets stuck at postdeployment configurations. Please find below the capture
> during install :
>
>
>
>
>
> Every 2.0s: ironic node-list && nova list && heat stack-list && heat
> resource-list -n5 overcloud | grep -vi complete
> Wed Aug  3 17:33:37 2016
>
>
>
> +--------------------------------------+------+--------------------------------------+-------------+--------------------+-------------+
>
> | UUID                                 | Name | Instance UUID
> | Power State | Provisioning State | Maintenance |
>
> +--------------------------------------+------+--------------------------------------+-------------+--------------------+-------------+
>
> | 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 | None |
> 9e7aae15-cabc-4489-a1b2-778915a78df2 | power on    | active             |
> False       |
>
> | afcfbee3-3108-48da-a6da-aba8f422642c | None |
> c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a | power on    | active             |
> False       |
>
> +--------------------------------------+------+--------------------------------------+-------------+--------------------+-------------+
>
> +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
>
> | ID                                   | Name                    | Status |
> Task State | Power State | Networks               |
>
> +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
>
> | 9e7aae15-cabc-4489-a1b2-778915a78df2 | overcloud-controller-0  | ACTIVE |
> -          | Running     | ctlplane=192.168.149.9 |
>
> | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a | overcloud-novacompute-0 | ACTIVE |
> -          | Running     | ctlplane=192.168.149.8 |
>
> +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
>
> +--------------------------------------+------------+---------------+---------------------+--------------+
>
> | id                                   | stack_name | stack_status  |
> creation_time       | updated_time |
>
> +--------------------------------------+------------+---------------+---------------------+--------------+
>
> | 26ee0150-4cfa-4268-9107-8bfbf6712913 | overcloud  | CREATE_FAILED |
> 2016-08-03T08:11:34 | None         |
>
> +--------------------------------------+------------+---------------+---------------------+--------------+
>
> +---------------------------------------------+-----------------------------------------------+------------------------------------------------------------------------
>
> ---------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------+
>
> | resource_name                               | physical_resource_id
> | resource_type
>
>          | resource_status    | updated_time        | stack_name
> |
>
> +---------------------------------------------+-----------------------------------------------+------------------------------------------------------------------------
>
> ---------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------+
>
> | ComputeNodesPostDeployment                  |
> 3797aec6-e543-4dda-9cd1-c7261e827a64          |
> OS::TripleO::ComputePostDeployment
>
>          | CREATE_FAILED      | 2016-08-03T08:11:35 | overcloud
> |
>
> | ControllerNodesPostDeployment               |
> 6ad9f88c-5c55-4125-97f1-eb0e33329d16          |
> OS::TripleO::ControllerPostDeployment
>
>          | CREATE_FAILED      | 2016-08-03T08:11:35 | overcloud
> |
>
> | ComputePuppetDeployment                     |
> 8b199f85-e4f9-48ad-9aee-b1cdf4900b9f          |
> OS::Heat::StructuredDeployments
>
>          | CREATE_FAILED      | 2016-08-03T08:29:19 |
> overcloud-ComputeNodesPostDeployment-6vxfu2g2qucy
> |
>
> | ControllerOvercloudServicesDeployment_Step4 |
> 15509f59-ff28-43af-95dd-6247a6a32c2d          |
> OS::Heat::StructuredDeployments
>
>          | CREATE_FAILED      | 2016-08-03T08:29:20 |
> overcloud-ControllerNodesPostDeployment-35y7uafngfwj
> |
>
> | 0                                           |
> 7cd0aa3d-742f-4e78-99ca-b2a575913f8e          |
> OS::Heat::StructuredDeployment
>
>          | CREATE_IN_PROGRESS | 2016-08-03T08:30:04 |
> overcloud-ComputeNodesPostDeployment-6vxfu2g2qucy-ComputePuppetDeployment-cpahcct3tfw3
> |
>
> | 0                                           |
> 5e9308f7-c3a9-4a94-a017-e1acb694c036          |
> OS::Heat::StructuredDeployment
>
>
>
>
>
> [stack at mitaka-uc ~]$ openstack software deployment show
> 5e9308f7-c3a9-4a94-a017-e1acb694c036
>
> +---------------+--------------------------------------+
>
> | Field         | Value                                |
>
> +---------------+--------------------------------------+
>
> | id            | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
>
> | server_id     | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
>
> | config_id     | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
>
> | creation_time | 2016-08-03T08:32:10                  |
>
> | updated_time  |                                      |
>
> | status        | IN_PROGRESS                          |
>
> | status_reason | Deploy data available                |
>
> | input_values  | {}                                   |
>
> | action        | CREATE                               |
>
> +---------------+--------------------------------------+
>
>
>
> [stack at mitaka-uc ~]$ openstack software deployment show --long
> 5e9308f7-c3a9-4a94-a017-e1acb694c036
>
> +---------------+--------------------------------------+
>
> | Field         | Value                                |
>
> +---------------+--------------------------------------+
>
> | id            | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
>
> | server_id     | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
>
> | config_id     | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
>
> | creation_time | 2016-08-03T08:32:10                  |
>
> | updated_time  |                                      |
>
> | status        | IN_PROGRESS                          |
>
> | status_reason | Deploy data available                |
>
> | input_values  | {}                                   |
>
> | action        | CREATE                               |
>
> | output_values | None                                 |
>
> +---------------+--------------------------------------+
>
>
>
> [stack at mitaka-uc ~]$ openstack stack resource list
> 3797aec6-e543-4dda-9cd1-c7261e827a64
>
> +-------------------------+--------------------------------------+-------------------------------------------------+-----------------+---------------------+
>
> | resource_name           | physical_resource_id                 |
> resource_type                                   | resource_status |
> updated_time        |
>
> +-------------------------+--------------------------------------+-------------------------------------------------+-----------------+---------------------+
>
> | ComputeArtifactsConfig  | a33cd04d-61ab-4429-8565-182409c2b97f |
> file:///usr/share/openstack-tripleo-heat-       | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
>
> |                         |                                      |
> templates/puppet/deploy-artifacts.yaml          |                 |
> |
>
> | ComputePuppetConfig     | 5bb712b0-5358-46c7-a444-f9adedfedd50 |
> OS::Heat::SoftwareConfig                        | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
>
> | ComputePuppetDeployment | 8b199f85-e4f9-48ad-9aee-b1cdf4900b9f |
> OS::Heat::StructuredDeployments                 | CREATE_FAILED   |
> 2016-08-03T08:29:19 |
>
> | ComputeArtifactsDeploy  | 1d13bf34-fc66-4bf1-a3b7-1dd815f58f5a |
> OS::Heat::StructuredDeployments                 | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
>
> | ExtraConfig             |                                      |
> OS::TripleO::NodeExtraConfigPost                | INIT_COMPLETE   |
> 2016-08-03T08:29:19 |
>
> +-------------------------+--------------------------------------+-------------------------------------------------+-----------------+---------------------+
>
>
>
> [stack at mitaka-uc ~]$ openstack stack resource list
> 8b199f85-e4f9-48ad-9aee-b1cdf4900b9f
>
> +---------------+--------------------------------------+--------------------------------+--------------------+---------------------+
>
> | resource_name | physical_resource_id                 | resource_type
> | resource_status    | updated_time        |
>
> +---------------+--------------------------------------+--------------------------------+--------------------+---------------------+
>
> | 0             | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
> OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2016-08-03T08:30:04 |
>
> +---------------+--------------------------------------+--------------------------------+--------------------+---------------------+
>
> [stack at mitaka-uc ~]$ openstack software deployment show
> 7cd0aa3d-742f-4e78-99ca-b2a575913f8e
>
> +---------------+--------------------------------------+
>
> | Field         | Value                                |
>
> +---------------+--------------------------------------+
>
> | id            | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
>
> | server_id     | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a |
>
> | config_id     | 24e5c0db-f84f-4a94-8f8e-8e38e73ccc86 |
>
> | creation_time | 2016-08-03T08:30:05                  |
>
> | updated_time  |                                      |
>
> | status        | IN_PROGRESS                          |
>
> | status_reason | Deploy data available                |
>
> | input_values  | {}                                   |
>
> | action        | CREATE                               |
>
> +---------------+--------------------------------------+
>
>
>
> Keystonerc file was not generated. Please find below openstack status
> command result on controller and compute.
>
>
>
> [heat-admin at overcloud-controller-0 ~]$ openstack-status
>
> == Nova services ==
>
> openstack-nova-api:                     active
>
> openstack-nova-compute:                 inactive  (disabled on boot)
>
> openstack-nova-network:                 inactive  (disabled on boot)
>
> openstack-nova-scheduler:               activating(disabled on boot)
>
> openstack-nova-cert:                    active
>
> openstack-nova-conductor:               active
>
> openstack-nova-console:                 inactive  (disabled on boot)
>
> openstack-nova-consoleauth:             active
>
> openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
>
> == Glance services ==
>
> openstack-glance-api:                   active
>
> openstack-glance-registry:              active
>
> == Keystone service ==
>
> openstack-keystone:                     inactive  (disabled on boot)
>
> == Horizon service ==
>
> openstack-dashboard:                    uncontactable
>
> == neutron services ==
>
> neutron-server:                         failed    (disabled on boot)
>
> neutron-dhcp-agent:                     inactive  (disabled on boot)
>
> neutron-l3-agent:                       inactive  (disabled on boot)
>
> neutron-metadata-agent:                 inactive  (disabled on boot)
>
> neutron-lbaas-agent:                    inactive  (disabled on boot)
>
> neutron-openvswitch-agent:              inactive  (disabled on boot)
>
> neutron-metering-agent:                 inactive  (disabled on boot)
>
> == Swift services ==
>
> openstack-swift-proxy:                  active
>
> openstack-swift-account:                active
>
> openstack-swift-container:              active
>
> openstack-swift-object:                 active
>
> == Cinder services ==
>
> openstack-cinder-api:                   active
>
> openstack-cinder-scheduler:             active
>
> openstack-cinder-volume:                active
>
> openstack-cinder-backup:                inactive  (disabled on boot)
>
> == Ceilometer services ==
>
> openstack-ceilometer-api:               active
>
> openstack-ceilometer-central:           active
>
> openstack-ceilometer-compute:           inactive  (disabled on boot)
>
> openstack-ceilometer-collector:         active
>
> openstack-ceilometer-notification:      active
>
> == Heat services ==
>
> openstack-heat-api:                     inactive  (disabled on boot)
>
> openstack-heat-api-cfn:                 active
>
> openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
>
> openstack-heat-engine:                  inactive  (disabled on boot)
>
> == Sahara services ==
>
> openstack-sahara-api:                   active
>
> openstack-sahara-engine:                active
>
> == Support services ==
>
> libvirtd:                               active
>
> openvswitch:                            active
>
> dbus:                                   active
>
> target:                                 active
>
> rabbitmq-server:                        active
>
> memcached:                              active
>
>
>
>
>
> [heat-admin at overcloud-novacompute-0 ~]$ openstack-status
>
> == Nova services ==
>
> openstack-nova-api:                     inactive  (disabled on boot)
>
> openstack-nova-compute:                 activating(disabled on boot)
>
> openstack-nova-network:                 inactive  (disabled on boot)
>
> openstack-nova-scheduler:               inactive  (disabled on boot)
>
> openstack-nova-cert:                    inactive  (disabled on boot)
>
> openstack-nova-conductor:               inactive  (disabled on boot)
>
> openstack-nova-console:                 inactive  (disabled on boot)
>
> openstack-nova-consoleauth:             inactive  (disabled on boot)
>
> openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
>
> == Glance services ==
>
> openstack-glance-api:                   inactive  (disabled on boot)
>
> openstack-glance-registry:              inactive  (disabled on boot)
>
> == Keystone service ==
>
> openstack-keystone:                     inactive  (disabled on boot)
>
> == Horizon service ==
>
> openstack-dashboard:                    uncontactable
>
> == neutron services ==
>
> neutron-server:                         inactive  (disabled on boot)
>
> neutron-dhcp-agent:                     inactive  (disabled on boot)
>
> neutron-l3-agent:                       inactive  (disabled on boot)
>
> neutron-metadata-agent:                 inactive  (disabled on boot)
>
> neutron-lbaas-agent:                    inactive  (disabled on boot)
>
> neutron-openvswitch-agent:              active
>
> neutron-metering-agent:                 inactive  (disabled on boot)
>
> == Swift services ==
>
> openstack-swift-proxy:                  inactive  (disabled on boot)
>
> openstack-swift-account:                inactive  (disabled on boot)
>
> openstack-swift-container:              inactive  (disabled on boot)
>
> openstack-swift-object:                 inactive  (disabled on boot)
>
> == Cinder services ==
>
> openstack-cinder-api:                   inactive  (disabled on boot)
>
> openstack-cinder-scheduler:             inactive  (disabled on boot)
>
> openstack-cinder-volume:                inactive  (disabled on boot)
>
> openstack-cinder-backup:                inactive  (disabled on boot)
>
> == Ceilometer services ==
>
> openstack-ceilometer-api:               inactive  (disabled on boot)
>
> openstack-ceilometer-central:           inactive  (disabled on boot)
>
> openstack-ceilometer-compute:           inactive  (disabled on boot)
>
> openstack-ceilometer-collector:         inactive  (disabled on boot)
>
> openstack-ceilometer-notification:      inactive  (disabled on boot)
>
> == Heat services ==
>
> openstack-heat-api:                     inactive  (disabled on boot)
>
> openstack-heat-api-cfn:                 inactive  (disabled on boot)
>
> openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
>
> openstack-heat-engine:                  inactive  (disabled on boot)
>
> == Sahara services ==
>
> openstack-sahara-all:                   inactive  (disabled on boot)
>
> == Support services ==
>
> libvirtd:                               active
>
> openvswitch:                            active
>
> dbus:                                   active
>
> rabbitmq-server:                        inactive  (disabled on boot)
>
> memcached:                              inactive  (disabled on boot)
>
>
>
>
>
>
>
> Please let me know if there is any other logs which I can provide that can
> help in troubleshooting.
>
>
>
>
>
> Thanks a lot in Advance for your help and support.
>
>
>
> Best Regards,
>
> Milind Gunjan
>
>
>
>
> ________________________________
>
> This e-mail may contain Sprint proprietary information intended for the sole
> use of the recipient(s). Any use by others is prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies of the
> message.
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com




More information about the dev mailing list