Re: [Rdo-list] [rdo-manager] Authentication required during overcloud deployment

Hi Dan, Sasha, Thanks for your answers and hints. I looked up the heat/etc log files and stack/node status. Only thing I found by far is "timed out". I don't know what's the reason. IPMI looks good. Tried with HEAT_INCLUDE_PASSWORD=1 but same error message (Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1 Authentication required) BTW. I only followed the exact instruction as shown in the guide: (openstack overcloud deploy --templates) No more options. I thought this is good for a demo deployment. If not sufficient, which one I should follow? See some of your discussions, but not very clear. Should I follow the example from jliberma(a)redhat.com? Below are my investigation: By runnig: $ heat resource-list overcloud Found that just controller and compute are failed: CREATE_FAILED Checked the reason it says: resource_status_reason | CREATE aborted I then logged into the running overcloud nodes (e.g. the controller): [heat-admin@overcloud-controller-0 ~]$ ifconfig br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::21:5eff:fecd:9df3 prefixlen 64 scopeid 0x20<link> ether 02:21:5e:cd:9d:f3 txqueuelen 0 (Ethernet) RX packets 29926 bytes 2364154 (2.2 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 81 bytes 25614 (25.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s29f0u2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::21:5eff:fecd:9df3 prefixlen 64 scopeid 0x20<link> ether 02:21:5e:cd:9d:f3 txqueuelen 1000 (Ethernet) RX packets 29956 bytes 1947140 (1.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 102 bytes 28620 (27.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp11s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.6.64 netmask 255.255.0.0 broadcast 10.0.255.255 inet6 fe80::221:5eff:fec9:abd8 prefixlen 64 scopeid 0x20<link> ether 00:21:5e:c9:ab:d8 txqueuelen 1000 (Ethernet) RX packets 66256 bytes 21109918 (20.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 35938 bytes 4641202 (4.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp11s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::221:5eff:fec9:abda prefixlen 64 scopeid 0x20<link> ether 00:21:5e:c9:ab:da txqueuelen 1000 (Ethernet) RX packets 25429 bytes 2004574 (1.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6 bytes 532 (532.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 inet6 fe80::202:c902:23:baf9 prefixlen 64 scopeid 0x20<link> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8). infiniband 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand) RX packets 183678 bytes 10292768 (9.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 17 bytes 5380 (5.2 KiB) TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 138 bytes 11792 (11.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 138 bytes 11792 (11.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [heat-admin@overcloud-controller-0 ~]$ ovs-vsctl show ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied) [heat-admin@overcloud-controller-0 ~]$ sudo ovs-vsctl show 76e6f8a7-88cf-4920-b133-b4d15a4b9092 Bridge br-ex Port br-ex Interface br-ex type: internal Port "enp0s29f0u2" Interface "enp0s29f0u2" ovs_version: "2.3.1" [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ ping 10.0.6.65 PING 10.0.6.65 (10.0.6.65) 56(84) bytes of data. 64 bytes from 10.0.6.65: icmp_seq=1 ttl=64 time=0.176 ms 64 bytes from 10.0.6.65: icmp_seq=2 ttl=64 time=0.195 ms ^C --- 10.0.6.65 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.176/0.185/0.195/0.016 ms [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ ping 10.0.6.64 PING 10.0.6.64 (10.0.6.64) 56(84) bytes of data. 64 bytes from 10.0.6.64: icmp_seq=1 ttl=64 time=0.015 ms ^C --- 10.0.6.64 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.015/0.015/0.015/0.000 ms [heat-admin@overcloud-controller-0 ~]$ cat /etc/os-net-config/config.json {"network_config": [{"use_dhcp": true, "type": "ovs_bridge", "name": "br-ex", "members": [{"type": "interface", "name": "nic1", "primary": true}]}]} [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ [heat-admin@overcloud-controller-0 ~]$ sudo os-net-config --debug -c /etc/os-net-config/config.json [2015/10/15 07:52:08 PM] [INFO] Using config file at: /etc/os-net-config/config.json [2015/10/15 07:52:08 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml [2015/10/15 07:52:08 PM] [INFO] Ifcfg net config provider created. [2015/10/15 07:52:08 PM] [DEBUG] network_config JSON: [{'use_dhcp': True, 'type': 'ovs_bridge', 'name': 'br-ex', 'members': [{'type': 'interface', 'name': 'nic1', 'primary': True}]}] [2015/10/15 07:52:08 PM] [INFO] nic1 mapped to: enp0s29f0u2 [2015/10/15 07:52:08 PM] [INFO] nic2 mapped to: enp11s0f0 [2015/10/15 07:52:08 PM] [INFO] nic3 mapped to: enp11s0f1 [2015/10/15 07:52:08 PM] [INFO] nic4 mapped to: ib0 [2015/10/15 07:52:08 PM] [INFO] adding bridge: br-ex [2015/10/15 07:52:08 PM] [DEBUG] bridge data: DEVICE=br-ex ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSBridge OVSBOOTPROTO=dhcp OVSDHCPINTERFACES="enp0s29f0u2" OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3" [2015/10/15 07:52:08 PM] [INFO] adding interface: enp0s29f0u2 [2015/10/15 07:52:08 PM] [DEBUG] interface data: DEVICE=enp0s29f0u2 ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-ex BOOTPROTO=none [2015/10/15 07:52:08 PM] [INFO] applying network configs... [2015/10/15 07:52:08 PM] [DEBUG] Diff file data: DEVICE=enp0s29f0u2 ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-ex BOOTPROTO=none [2015/10/15 07:52:08 PM] [DEBUG] Diff data: DEVICE=enp0s29f0u2 ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-ex BOOTPROTO=none [2015/10/15 07:52:08 PM] [DEBUG] Diff file data: [2015/10/15 07:52:08 PM] [DEBUG] Diff data: [2015/10/15 07:52:08 PM] [DEBUG] Diff file data: DEVICE=br-ex ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSBridge OVSBOOTPROTO=dhcp OVSDHCPINTERFACES="enp0s29f0u2" OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3" [2015/10/15 07:52:08 PM] [DEBUG] Diff data: DEVICE=br-ex ONBOOT=yes HOTPLUG=no DEVICETYPE=ovs TYPE=OVSBridge OVSBOOTPROTO=dhcp OVSDHCPINTERFACES="enp0s29f0u2" OVS_EXTRA="set bridge br-ex other-config:hwaddr=02:21:5e:cd:9d:f3" [2015/10/15 07:52:08 PM] [DEBUG] Diff file data: [2015/10/15 07:52:08 PM] [DEBUG] Diff data: [heat-admin@overcloud-controller-0 ~]$ openstack-status == Nova services == openstack-nova-api: inactive (disabled on boot) openstack-nova-cert: inactive (disabled on boot) openstack-nova-compute: inactive (disabled on boot) openstack-nova-network: inactive (disabled on boot) openstack-nova-scheduler: inactive (disabled on boot) openstack-nova-conductor: inactive (disabled on boot) == Glance services == openstack-glance-api: inactive (disabled on boot) openstack-glance-registry: inactive (disabled on boot) == Keystone service == openstack-keystone: inactive (disabled on boot) == Horizon service == openstack-dashboard: uncontactable == neutron services == neutron-server: inactive (disabled on boot) neutron-dhcp-agent: inactive (disabled on boot) neutron-l3-agent: inactive (disabled on boot) neutron-metadata-agent: inactive (disabled on boot) neutron-lbaas-agent: inactive (disabled on boot) neutron-openvswitch-agent: inactive (disabled on boot) neutron-metering-agent: inactive (disabled on boot) == Swift services == openstack-swift-proxy: inactive (disabled on boot) openstack-swift-account: inactive (disabled on boot) openstack-swift-container: inactive (disabled on boot) openstack-swift-object: inactive (disabled on boot) == Cinder services == openstack-cinder-api: inactive (disabled on boot) openstack-cinder-scheduler: inactive (disabled on boot) openstack-cinder-volume: inactive (disabled on boot) openstack-cinder-backup: inactive (disabled on boot) == Ceilometer services == openstack-ceilometer-api: inactive (disabled on boot) openstack-ceilometer-central: inactive (disabled on boot) openstack-ceilometer-compute: inactive (disabled on boot) openstack-ceilometer-collector: inactive (disabled on boot) openstack-ceilometer-alarm-notifier: inactive (disabled on boot) openstack-ceilometer-alarm-evaluator: inactive (disabled on boot) openstack-ceilometer-notification: inactive (disabled on boot) == Heat services == openstack-heat-api: inactive (disabled on boot) openstack-heat-api-cfn: inactive (disabled on boot) openstack-heat-api-cloudwatch: inactive (disabled on boot) openstack-heat-engine: inactive (disabled on boot) == Support services == libvirtd: active openvswitch: active dbus: active rabbitmq-server: inactive (disabled on boot) memcached: inactive (disabled on boot) == Keystone users == Warning keystonerc not sourced Thanks, Erming On 10/14/15, 5:23 PM, Dan Sneddon wrote: > On 10/14/2015 03:03 PM, Erming Pei wrote: >> Hi, >> >> I am deploying the overcloud in baremetal way and after a couple of >> hours, it showed: >> >> $ openstack overcloud deploy --templates >> Deploying templates in the directory >> /usr/share/openstack-tripleo-heat-templates >> ^[[A^[[BERROR: openstack ERROR: Authentication failed. Please try again >> with option --include-password or export HEAT_INCLUDE_PASSWORD=1 >> Authentication required >> >> >> But I checked the nodes are now running: >> >> [stack@gcloudcon-3 ~]$ nova list >> +--------------------------------------+-------------------------+--------+------------+-------------+--------------------+ >> >> | ID | Name | >> Status | Task State | Power State | Networks | >> +--------------------------------------+-------------------------+--------+------------+-------------+--------------------+ >> >> | 1ba04ac0-fe2b-4318-aa31-2e5f4d8422a6 | overcloud-controller-0 | >> ACTIVE | - | Running | ctlplane=10.0.6.60 | >> | c152ba59-3aed-4fb0-81fa-e3fed7e35cf6 | overcloud-novacompute-0 | >> ACTIVE | - | Running | ctlplane=10.0.6.61 | >> +--------------------------------------+-------------------------+--------+------------+-------------+--------------------+ >> >> >> 1. Should I re-deploy the nodes or there is a way to do update/makeup >> for the authentication issue? >> >> 2. >> I don't know how to access to the nodes. >> There is not an overcloudrc file produced. >> >> $ ls overcloud* >> overcloud-env.json overcloud-full.initrd overcloud-full.qcow2 >> overcloud-full.vmlinuz >> >> overcloud-full.d: >> dib-manifests >> >> Is it via ssh key or password? Should I set the authentication method >> somewhere? >> >> >> >> Thanks, >> >> Erming >> >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > This error generally means that something in the deployment got stuck, > and the deployment hung until the token expired after 4 hours. When > that happens, there is no overcloudrc generated (because there is not a > working overcloud). You won't be able to recover with a stack update, > you'll need to perform a stack-delete and redeploy once you know what > went wrong. > > Generally a deployment shouldn't take anywhere near that long, a bare > metal deployment with 6 hosts takes me less than an hour, and less than > 2 including a Ceph deployment. In fact, I usually set a timeout using > the --timeout option, because if it hasn't finished after, say 90 > minutes (depending on how complicated the deployment is), then I want > it to bomb out so I can diagnose what went wrong and redeploy. > > Often when a deployment times out it is because there were connectivity > issues between the nodes. Since you can log in to the hosts, you might > want to do some basic network troubleshooting, such as: > > $ ip address # check to see that all the interfaces are there, and > that the IP addresses have been assigned > > $ sudo ovs-vsctl show # make sure that the bridges have the proper > interfaces, vlans, and that all the expected bridges show up > > $ ping <other overcloud nodes> # you can try this on all VLANs to make > sure that any VLAN trunks are working properly > > $ sudo ovs-appctl bond/show # if running bonding, check to see the > bond status > > $ sudo os-net-config --debug -c /etc/os-net-config/config.json # run > the network configuration script again to make sure that it is able to > configure the interfaces without error. WARNING, MAY BE DISRUPTIVE as > this will reset the network interfaces, run on console if possible. > > However, I want to first double-check that you had a valid command > line. You only show "openstack deploy overcloud --templates" in your > original email. You did have a full command-line, right? Refer to the > official installation guide for the right parameters. > -- --------------------------------------------- Erming Pei, Ph.D Senior System Analyst; Grid/Cloud Specialist Research Computing Group Information Services & Technology University of Alberta, Canada Tel: +1 7804929914 Fax: +1 7804921729 ---------------------------------------------