Hi Dan, Sasha,

   Thanks for your answers and hints.
   I looked up the heat/etc log files and stack/node status.
   Only thing I found by far is "timed out". I don't know what's the reason. IPMI looks good.

   Tried with HEAT_INCLUDE_PASSWORD=1 but same error message  (Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1 Authentication required)
  
   BTW. I only followed the exact instruction as shown in the guide:  (openstack overcloud deploy --templates) No more options. I thought this is good for a demo deployment. If not sufficient, which one I should follow? See some of your discussions, but not very clear. Should I follow the example from jliberma@redhat.com?

Below are my investigation:
   By runnig: $ heat resource-list overcloud
   Found that just controller and compute are failed: CREATE_FAILED
 
   Checked the reason it says: resource_status_reason | CREATE aborted

   I then logged into the running overcloud nodes (e.g. the controller):
  

[heat-admin@overcloud-controller-0 ~]$ ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::21:5eff:fecd:9df3  prefixlen 64  scopeid 0x20<link>
        ether 02:21:5e:cd:9d:f3  txqueuelen 0  (Ethernet)
        RX packets 29926  bytes 2364154 (2.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 81  bytes 25614 (25.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s29f0u2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::21:5eff:fecd:9df3  prefixlen 64  scopeid 0x20<link>
        ether 02:21:5e:cd:9d:f3  txqueuelen 1000  (Ethernet)
        RX packets 29956  bytes 1947140 (1.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 102  bytes 28620 (27.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp11s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.6.64  netmask 255.255.0.0  broadcast 10.0.255.255
        inet6 fe80::221:5eff:fec9:abd8  prefixlen 64  scopeid 0x20<link>
        ether 00:21:5e:c9:ab:d8  txqueuelen 1000  (Ethernet)
        RX packets 66256  bytes 21109918 (20.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 35938  bytes 4641202 (4.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp11s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::221:5eff:fec9:abda  prefixlen 64  scopeid 0x20<link>
        ether 00:21:5e:c9:ab:da  txqueuelen 1000  (Ethernet)
        RX packets 25429  bytes 2004574 (1.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6  bytes 532 (532.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet6 fe80::202:c902:23:baf9  prefixlen 64  scopeid 0x20<link>
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 183678  bytes 10292768 (9.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17  bytes 5380 (5.2 KiB)
        TX errors 0  dropped 7 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 138  bytes 11792 (11.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 138  bytes 11792 (11.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[heat-admin@overcloud-controller-0 ~]$ ovs-vsctl show
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied)
[heat-admin@overcloud-controller-0 ~]$ sudo ovs-vsctl show
76e6f8a7-88cf-4920-b133-b4d15a4b9092
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "enp0s29f0u2"
            Interface "enp0s29f0u2"
    ovs_version: "2.3.1"
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ ping 10.0.6.65
PING 10.0.6.65 (10.0.6.65) 56(84) bytes of data.
64 bytes from 10.0.6.65: icmp_seq=1 ttl=64 time=0.176 ms
64 bytes from 10.0.6.65: icmp_seq=2 ttl=64 time=0.195 ms
^C
--- 10.0.6.65 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.176/0.185/0.195/0.016 ms
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ ping 10.0.6.64
PING 10.0.6.64 (10.0.6.64) 56(84) bytes of data.
64 bytes from 10.0.6.64: icmp_seq=1 ttl=64 time=0.015 ms
^C
--- 10.0.6.64 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.015/0.015/0.015/0.000 ms

[heat-admin@overcloud-controller-0 ~]$ cat /etc/os-net-config/config.json
{"network_config": [{"use_dhcp": true, "type": "ovs_bridge", "name": "br-ex", "members": [{"type": "interface", "name": "nic1", "primary": true}]}]}
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ sudo os-net-config --debug -c /etc/os-net-config/config.json
[2015/10/15 07:52:08 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2015/10/15 07:52:08 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[2015/10/15 07:52:08 PM] [INFO] Ifcfg net config provider created.
[2015/10/15 07:52:08 PM] [DEBUG] network_config JSON: [{'use_dhcp': True, 'type': 'ovs_bridge', 'name': 'br-ex', 'members': [{'type': 'interface', 'name': 'nic1', 'primary': True}]}]
[2015/10/15 07:52:08 PM] [INFO] nic1 mapped to: enp0s29f0u2
[2015/10/15 07:52:08 PM] [INFO] nic2 mapped to: enp11s0f0
[2015/10/15 07:52:08 PM] [INFO] nic3 mapped to: enp11s0f1
[2015/10/15 07:52:08 PM] [INFO] nic4 mapped to: ib0
[2015/10/15 07:52:08 PM] [INFO] adding bridge: br-ex
[2015/10/15 07:52:08 PM] [DEBUG] bridge data: DEVICE=br-ex
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSBridge
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES="enp0s29f0u2"
OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3"

[2015/10/15 07:52:08 PM] [INFO] adding interface: enp0s29f0u2
[2015/10/15 07:52:08 PM] [DEBUG] interface data: DEVICE=enp0s29f0u2
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
BOOTPROTO=none

[2015/10/15 07:52:08 PM] [INFO] applying network configs...
[2015/10/15 07:52:08 PM] [DEBUG] Diff file data:
DEVICE=enp0s29f0u2
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
BOOTPROTO=none

[2015/10/15 07:52:08 PM] [DEBUG] Diff data:
DEVICE=enp0s29f0u2
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
BOOTPROTO=none

[2015/10/15 07:52:08 PM] [DEBUG] Diff file data:

[2015/10/15 07:52:08 PM] [DEBUG] Diff data:

[2015/10/15 07:52:08 PM] [DEBUG] Diff file data:
DEVICE=br-ex
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSBridge
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES="enp0s29f0u2"
OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3"

[2015/10/15 07:52:08 PM] [DEBUG] Diff data:
DEVICE=br-ex
ONBOOT=yes
HOTPLUG=no
DEVICETYPE=ovs
TYPE=OVSBridge
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES="enp0s29f0u2"
OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3"

[2015/10/15 07:52:08 PM] [DEBUG] Diff file data:

[2015/10/15 07:52:08 PM] [DEBUG] Diff data:

  


[heat-admin@overcloud-controller-0 ~]$ openstack-status
== Nova services ==
openstack-nova-api:                     inactive  (disabled on boot)
openstack-nova-cert:                    inactive  (disabled on boot)
openstack-nova-compute:                 inactive  (disabled on boot)
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               inactive  (disabled on boot)
openstack-nova-conductor:               inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   inactive  (disabled on boot)
openstack-glance-registry:              inactive  (disabled on boot)
== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
== Horizon service ==
openstack-dashboard:                    uncontactable
== neutron services ==
neutron-server:                         inactive  (disabled on boot)
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              inactive  (disabled on boot)
neutron-metering-agent:                 inactive  (disabled on boot)
== Swift services ==
openstack-swift-proxy:                  inactive  (disabled on boot)
openstack-swift-account:                inactive  (disabled on boot)
openstack-swift-container:              inactive  (disabled on boot)
openstack-swift-object:                 inactive  (disabled on boot)
== Cinder services ==
openstack-cinder-api:                   inactive  (disabled on boot)
openstack-cinder-scheduler:             inactive  (disabled on boot)
openstack-cinder-volume:                inactive  (disabled on boot)
openstack-cinder-backup:                inactive  (disabled on boot)
== Ceilometer services ==
openstack-ceilometer-api:               inactive  (disabled on boot)
openstack-ceilometer-central:           inactive  (disabled on boot)
openstack-ceilometer-compute:           inactive  (disabled on boot)
openstack-ceilometer-collector:         inactive  (disabled on boot)
openstack-ceilometer-alarm-notifier:    inactive  (disabled on boot)
openstack-ceilometer-alarm-evaluator:   inactive  (disabled on boot)
openstack-ceilometer-notification:      inactive  (disabled on boot)
== Heat services ==
openstack-heat-api:                     inactive  (disabled on boot)
openstack-heat-api-cfn:                 inactive  (disabled on boot)
openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
openstack-heat-engine:                  inactive  (disabled on boot)
== Support services ==
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
rabbitmq-server:                        inactive  (disabled on boot)
memcached:                              inactive  (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced





Thanks,

Erming
  

On 10/14/15, 5:23 PM, Dan Sneddon wrote:
On 10/14/2015 03:03 PM, Erming Pei wrote:
Hi,

  I am deploying the overcloud in baremetal way and after a couple of
hours, it showed:

$ openstack overcloud deploy --templates
Deploying templates in the directory
/usr/share/openstack-tripleo-heat-templates
^[[A^[[BERROR: openstack ERROR: Authentication failed. Please try again
with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required


But I checked the nodes are now running:

[stack@gcloudcon-3 ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------+

| ID                                   | Name                    |
Status | Task State | Power State | Networks           |
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------+

| 1ba04ac0-fe2b-4318-aa31-2e5f4d8422a6 | overcloud-controller-0  |
ACTIVE | -          | Running     | ctlplane=10.0.6.60 |
| c152ba59-3aed-4fb0-81fa-e3fed7e35cf6 | overcloud-novacompute-0 |
ACTIVE | -          | Running     | ctlplane=10.0.6.61 |
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------+


1. Should I re-deploy the nodes or there is a way to do update/makeup
for the authentication issue?

2.
I don't know how to access to the nodes.
There is not an overcloudrc file produced.

$ ls overcloud*
overcloud-env.json  overcloud-full.initrd  overcloud-full.qcow2
overcloud-full.vmlinuz

overcloud-full.d:
dib-manifests

Is it via ssh key or password? Should I set the authentication method
somewhere?



Thanks,

Erming


_______________________________________________
Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

To unsubscribe: rdo-list-unsubscribe@redhat.com
This error generally means that something in the deployment got stuck,
and the deployment hung until the token expired after 4 hours. When
that happens, there is no overcloudrc generated (because there is not a
working overcloud). You won't be able to recover with a stack update,
you'll need to perform a stack-delete and redeploy once you know what
went wrong.

Generally a deployment shouldn't take anywhere near that long, a bare
metal deployment with 6 hosts takes me less than an hour, and less than
2 including a Ceph deployment. In fact, I usually set a timeout using
the --timeout option, because if it hasn't finished after, say 90
minutes (depending on how complicated the deployment is), then I want
it to bomb out so I can diagnose what went wrong and redeploy.

Often when a deployment times out it is because there were connectivity
issues between the nodes. Since you can log in to the hosts, you might
want to do some basic network troubleshooting, such as:

$ ip address   # check to see that all the interfaces are there, and
that the IP addresses have been assigned

$ sudo ovs-vsctl show     # make sure that the bridges have the proper
interfaces, vlans, and that all the expected bridges show up

$ ping <other overcloud nodes>  # you can try this on all VLANs to make
sure that any VLAN trunks are working properly

$ sudo ovs-appctl bond/show  # if running bonding, check to see the
bond status

$ sudo os-net-config --debug -c /etc/os-net-config/config.json   # run
the network configuration script again to make sure that it is able to
configure the interfaces without error. WARNING, MAY BE DISRUPTIVE as
this will reset the network interfaces, run on console if possible.

However, I want to first double-check that you had a valid command
line. You only show "openstack deploy overcloud --templates" in your
original email. You did have a full command-line, right? Refer to the
official installation guide for the right parameters.



-- 
---------------------------------------------
 Erming Pei, Ph.D 
 Senior System Analyst; Grid/Cloud Specialist

 Research Computing Group
 Information Services & Technology
 University of Alberta, Canada  

 Tel: +1 7804929914        Fax: +1 7804921729
---------------------------------------------