Hi Boris, I think this is related to a regression in the python-yaql package. The same issue we hit upstream here: https://bugs.launchpad.net/tripleo/+bug/1750032 You can work around the issue by downgrading the python-yaql package to the previous version. Or apply this fix to t-h-t: https://review.openstack.org/#/c/545498/ Hope this helps, Dan

On 02/16/2018 02:21 PM, Boris Derzhavets wrote: I can't even get that far. My configuration runs up until TASK [Add virthost] task path: /home/stack/.quickstart/playbooks/quickstart.yml:8 and then starts to tear down the non-root user (which is running the deploy), at which time the whole process dies. If I try to run it as root it outright deletes the "stack" user completely and then exits without having completed the deploy. I have tried both the https://www.anstack.com/blog/2018/01/05/tripleo-quickstart-cheatsheet.html and http://dbaxps.blogspot.com/2018/02/attempt-of-evaluation-tripleo.html steps.

In meantime script overcloud_prep_containers.sh crashes with log log. This behavior stays the same for about several days already. ----------------------------------------------- overcloud_prep_containers.log ----------------------------------------------- 2018-02-23 07:21:27 | Trying paths: ['/home/stack/.docker/config.json', '/home/stack/.dockercfg'] 2018-02-23 07:21:27 | No config file found 2018-02-23 07:21:27 | http://localhost:None "GET /version HTTP/1.1" 200 299 2018-02-23 07:21:27 | Running skopeo inspect docker://docker.io/tripleomaster/centos-binary-gnocchi-api:current-tripleo-rdo 2018-02-23 07:21:28 | Extracting 2018-02-23 07:21:28 | Extracting 2018-02-23 07:21:28 | None: None 2018-02-23 07:21:28 | Traceback (most recent call last): 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand 2018-02-23 07:21:28 | result = cmd.run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run 2018-02-23 07:21:28 | super(Command, self).run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run 2018-02-23 07:21:28 | return super(Command, self).run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run 2018-02-23 07:21:28 | return_code = self.take_action(parsed_args) or 0 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 66, in take_action 2018-02-23 07:21:28 | uploader.upload() 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 94, in upload 2018-02-23 07:21:28 | uploader.run_tasks() 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 354, in run_tasks 2018-02-23 07:21:28 | for result in p.map(docker_upload, self.upload_tasks): 2018-02-23 07:21:28 | File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map 2018-02-23 07:21:28 | return self.map_async(func, iterable, chunksize).get() 2018-02-23 07:21:28 | File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get 2018-02-23 07:21:28 | raise self._value 2018-02-23 07:21:28 | ReadTimeoutError: None: None 2018-02-23 07:21:28 | clean_up UploadImage: None: None 2018-02-23 07:21:28 | Traceback (most recent call last): 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run 2018-02-23 07:21:28 | ret_val = super(OpenStackShell, self).run(argv) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run 2018-02-23 07:21:28 | result = self.run_subcommand(remainder) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand 2018-02-23 07:21:28 | ret_value = super(OpenStackShell, self).run_subcommand(argv) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand 2018-02-23 07:21:28 | result = cmd.run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run 2018-02-23 07:21:28 | super(Command, self).run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run 2018-02-23 07:21:28 | return super(Command, self).run(parsed_args) 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run 2018-02-23 07:21:28 | return_code = self.take_action(parsed_args) or 0 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 66, in take_action 2018-02-23 07:21:28 | uploader.upload() 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 94, in upload 2018-02-23 07:21:28 | uploader.run_tasks() 2018-02-23 07:21:28 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 354, in run_tasks 2018-02-23 07:21:28 | for result in p.map(docker_upload, self.upload_tasks): 2018-02-23 07:21:28 | File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map 2018-02-23 07:21:28 | return self.map_async(func, iterable, chunksize).get() 2018-02-23 07:21:28 | File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get 2018-02-23 07:21:28 | raise self._value 2018-02-23 07:21:28 | ReadTimeoutError: None: None 2018-02-23 07:21:28 | 2018-02-23 07:21:28 | END return value: 1 Thanks. Boris ________________________________ From: dev <dev-bounces(a)lists.rdoproject.org&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; Sent: Friday, February 16, 2018 10:21 PM To: dev(a)lists.rdoproject.org Subject: [rdo-dev] Attempt to reproduce Carlos Camacho simplest deployment (controller+compute) now fails as well. Follow https://www.anstack.com/blog/2018/01/05/tripleo-quickstart-cheatsheet.html <https://www.anstack.com/blog/2018/01/05/tripleo-quickstart-cheatsheet.htm... just via remote connection to VIRTHOST (32 GB) from F27 wks . Overcloud deployment fails with error :- ---------------------------- overcloud_deploy.log ---------------------------- 2018-02-16 19:02:51 | 2018-02-16 19:02:39Z [overcloud.Compute.0]: CREATE_COMPLETE Stack CREATE completed successfully 2018-02-16 19:02:51 | 2018-02-16 19:02:40Z [overcloud.Compute.0]: CREATE_COMPLETE state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:40Z [overcloud.Compute]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-02-16 19:02:51 | 2018-02-16 19:02:40Z [overcloud.Controller.0.ControllerExtraConfigPre]: CREATE_COMPLETE state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:40Z [overcloud.Controller.0.NodeExtraConfig]: CREATE_IN_PROGRESS state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:41Z [overcloud.Compute]: CREATE_COMPLETE state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:42Z [overcloud.ComputeServers]: CREATE_IN_PROGRESS state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:42Z [overcloud.ComputeServers]: CREATE_COMPLETE state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:42Z [overcloud]: CREATE_FAILED list index out of range 2018-02-16 19:02:51 | 2018-02-16 19:02:42Z [overcloud.Controller.0.NodeExtraConfig]: CREATE_COMPLETE state changed 2018-02-16 19:02:51 | 2018-02-16 19:02:51 | Stack overcloud CREATE_FAILED 2018-02-16 19:02:51 | 2018-02-16 19:02:51 | + status_code=1 2018-02-16 19:02:51 | + openstack stack list 2018-02-16 19:02:51 | + grep -q overcloud 2018-02-16 19:02:55 | + openstack stack list 2018-02-16 19:02:55 | + grep -Eq '(CREATE|UPDATE)_COMPLETE' 2018-02-16 19:02:57 | + openstack stack failures list overcloud --long 2018-02-16 19:03:01 | ++ openstack stack resource list --nested-depth 5 overcloud 2018-02-16 19:03:01 | ++ grep FAILED 2018-02-16 19:03:01 | ++ grep 'StructuredDeployment ' 2018-02-16 19:03:01 | ++ cut -d '|' -f3 2018-02-16 19:03:16 | + exit 1 Thanks Boris.

At the moment I am able to commit deployment and get back to error :- 2018-02-24 12:13:53Z [overcloud]: CREATE_FAILED list index out of range How long takes for commit "Do not use the 3rd argument of yaql groupBy" I2887011f6baf4867d422579b116b5e143acf5679<https://review.openstack.org... to get into current-passed-ci trunk ? In meantime [root@undercloud yum.repos.d]# cat delorean.repo [delorean] name=delorean baseurl=https://trunk.rdoproject.org/centos7-master/17/a5/17a57d99b10f587... gpgcheck=0 enabled=1 priority=20 which points to package tripleo-heat-templates been built on 01/22/18 Same package in https://trunk.rdoproject.org/centos7-master-head/report.html has date 2018-02-23 23:28:24, which obviously has critical patch. Thanks Boris. ________________________________ From: Michele Baldessari <michele(a)acksyn.org&gt; Sent: Saturday, February 24, 2018 11:16 AM To: Boris Derzhavets Cc: dev(a)lists.rdoproject.org Subject: Re: [rdo-dev] Attempt to reproduce Carlos Camacho simplest deployment (controller+compute) . It fails when running overcloud_prep_containers.sh On Fri, Feb 23, 2018 at 07:52:49AM +0000, Boris Derzhavets wrote: FWIW I have observed this one too (although not 100% of time, it is frequent enough): https://bugs.launchpad.net/tripleo/+bug/1749663 Bug #1749663 “openstack overcloud container image upload fails f...” : Bugs : tripleo<https://bugs.launchpad.net/tripleo/+bug/1749663> bugs.launchpad.net Via https://bugs.launchpad.net/tripleo/+bug/1746305 and https://review.openstack.org/#/c/539383/ we made the image upload a bit more resilient. I seem to be still hitting it rather often: Extracting Extracting Extracting Pull complete imagename: docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo-rdo Trying paths: ['/home/stack/.docker/config.json', '/home/stack/.dockercfg'] No config file found None: None Traceback (most recent call last): File "/usr/lib/python2.7/site... Rerunning the command a few times gets past it. -- Michele Baldessari <michele(a)acksyn.org&gt; C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D

On 03/07/2018 09:13 AM, Boris Derzhavets wrote: Getting failures on my end as well.  I got past the issues with the initial undercloud deploy with the alternate baseos_rhel_libvirt.yml from Dan Trainor, but the overcloud deploys fail to complete. Saved all the logs from stack@undercloud, pasted a couple to pastebin (the one in .instack is too big, I'd have to parse out part of t if you needed) hostname.sh.log instack install_packages.sh.log overcloud_create_ssl_cert.log overcloud_prep_flavors.log overcloud_prep_images.log overcloud_prep_network.log repo_setup.log undercloud_install.log /home/stack/.instack/install-undercloud.log (too large for pastebin) /home/stack/install-undercloud.log http://pastebin.test.redhat.com/563321 output from "openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates" http://pastebin.test.redhat.com/563323 This is on RHEL 7.4, all packages updated and the git repo for tripleo-quickstart is updated.