January 2015 - dev - Rdo List Archives

by Steve Linabery

I have been struggling with the amount of information to convey and what level of detail to include. Since I can't seem to get it perfect to my own satisfaction, here is the imperfect (and long, sorry) version to begin discussion. This is an overview of where things stand (rdopkg CI 'v0.1'). Terminology: 'Release' refers to an OpenStack release (e.g. havana,icehouse,juno) 'Dist' refers to a distro supported by RDO (e.g. fedora-20, epel-6, epel-7) 'phase1' is the initial smoketest for an update submitted via `rdopkg update` 'phase2' is a full-provision test for accumulated updates that have passed phase1 'snapshot' means an OpenStack snapshot of a running instance, i.e. a disk image created from a running OS instance. The very broad strokes: ----------------------- rdopkg ci is triggered when a packager uses `rdopkg update`. When a review lands in the rdo-update gerrit project, a 'phase1' smoketest is initiated via jenkins for each Release/Dist combination present in the update (e.g. if the update contains builds for icehouse/fedora-20 and icehouse/epel-6, each set of RPMs from each build will be smoketested on an instance running the associated Release/Dist). If *all* supported builds from the update pass phase1, then the update is merged into rdo-update. Updates that pass phase1 accumulate in the updates/ directory in the rdo-update project. Periodically, a packager may run 'phase2'. This takes everything in updates/ and uses those RPMs + RDO production repo to provision a set of base images with packstack aio. Again, a simple tempest test is run against the packstack aio instances. If all pass, then phase2 passes, and the `rdopkg update` yaml files are moved from updates/ to ready/. At that point, someone with the keys to the stage repos will push the builds in ready/ to the stage repo. If CI against stage repo passes, stage is rsynced to production. Complexity, Part 1: ------------------- Rdopkg CI v0.1 was designed around the use of OpenStack VM disk snapshots. On a periodic basis, we provision two nodes for each supported combination in [Releases] X [Dists] (e.g. "icehouse, fedora-20" "juno, epel-7" etc). One node is a packstack aio instance built against RDO production repos, and the other is a node running tempest. After a simple tempest test passes for all the packstack aio nodes, we would snapshot the set of instances. Then when we want to do a 'phase1' test for e.g. "icehouse, fedora-20", we can spin up the instances previously snapshotted and save the time of re-running packstack aio. Using snapshots saves approximately 30 min of wait time per test run by skipping provisioning. Using snapshots imposes a few substantial costs/complexities though. First and most significant, snapshots need to be reinstantiated using the same IP addresses that were present when packstack and tempest were run during the provisioning. This means we have to have concurrency control around running only one phase1 run at a time; otherwise an instance might fail to provision because its 'static' IP address is already in use by another run. The second cost is that in practice, a) our OpenStack infrastructure has been unreliable, b) not all Release/Dist combinations reliably provision. So it becomes hard to create a full set of snapshots reliably. Additionally, some updates (e.g. when an update comes in for openstack-puppet-modules) prevent the use of a previously-provisioned packstack instance. Continuing with the o-p-m example: that package is used for provisioning. So simply updating the RPM for that package after running packstack aio doesn't tell us anything about the package sanity (other than perhaps if a new, unsatisfied RPM dependency was introduced). Another source of complexity comes from the nature of the rdopkg update 'unit'. Each yaml file created by `rdopkg update` can contain multiple builds for different Release,Dist combinations. So there must be a way to 'collate' the results of each smoketest for each Release,Dist and pass phase1 only if all updates pass. Furthermore, some combinations of Release,Dist are known (at times, for various ad hoc reasons) to fail testing, and those combinations sometimes need to be 'disabled'. For example, if we know that icehouse/f20 is 'red' on a given day, we might want an update containing icehouse/fedora-20,icehouse/epel-6 to test only the icehouse/epel-6 combination and pass if that passes. Finally, pursuant to the previous point, there need to be 'control' structure jobs for provision/snapshot, phase1, and phase2 runs that pass (and perform some action upon passing) only when all their 'child' jobs have passed. The way we have managed this complexity to date is through the use of the jenkins BuildFlow plugin. Here's some ASCII art (courtesy of 'tree') to show how the jobs are structured now (these are descriptive job names, not the actual jenkins job names). BuildFlow jobs are indicated by (bf). . `-- rdopkg_master_flow (bf) |-- provision_and_snapshot (bf) | |-- provision_and_snapshot_icehouse_epel6 | |-- provision_and_snapshot_icehouse_f20 | |-- provision_and_snapshot_juno_epel7 | `-- provision_and_snapshot_juno_f21 |-- phase1_flow (bf) | |-- phase1_test_icehouse_f20 | `-- phase1_test_juno_f21 `-- phase2_flow (bf) |-- phase2_test_icehouse_epel6 |-- phase2_test_icehouse_f20 |-- phase2_test_juno_epel7 `-- phase2_test_juno_f21 When a change comes in from `rdopkg update`, the rdopkg_master_flow job is triggered. It's the only job that gets triggered from gerrit, so it kicks off phase1_flow. phase1_flow runs 'child' jobs (normal jenkins jobs, not buildflow) for each Release,Dist combination present in the update. provision_and_snapshot is run by manually setting a build parameter (BUILD_SNAPS) in the rdopkg_master_flow job, and triggering the build of rdopkg_master_flow. phase2 is invoked similar to the provision_and_snapshot build, by checking 'RUN_PHASE2' in the rdopkg_master_flow build parameters before executing a build thereof. Concurrency control is a side effect of requiring the user or gerrit to execute rdopkg_master_flow for every action. There can be only one rdopkg_master_flow build executing at any given time. Complexity, Part 2: ------------------- In addition to the nasty complexity of using nested BuildFlow type jobs, each 'worker' job (i.e. the non-buildflow type jobs) has some built in complexity that is reflected in the amount of logic in each job's bash script definition. Some of this has been alluded to in previous points. For instance, each job in the phase1 flow needs to determine, for each update, if the update contains a package that requires full packstack aio provisioning from a base image (e.g. openstack-puppet-modules). This 'must provision' list needs to be stored somewhere that all jobs can read it, and it needs to be dynamic enough to add to it as requirements dictate. But additionally, for package sets not requiring provisioning a base image, phase1 job needs to query the backing OpenStack instance to see if there exists a 'known good' snapshot, get the images' UUIDs from OpenStack, and spin up the instances using the snapshot images. This baked-in complexity in the 'worker' jenkins jobs has made it difficult to maintain the job definitions, and more importantly difficult to run using jjb or in other more 'orthodox' CI-type ways. The rdopkg CI stuff is a bastard child of a fork. It lives in its own mutant gene pool. A Way Forward...? ---------------- Wes Hayutin had a good idea that might help reduce some of the complexity here as we contemplate a) making rdopkg CI public, b) moving toward rdopkg CI 0.2. His idea was a) stop using snapshots since the per-test-run savings doesn't seem to justify the burden they create, b) do away with BuildFlow by including the 'this update contains builds for (Release1,Dist2),...,(ReleaseN,DistM)' information in the gerrit change topic. I think that's a great idea, but I have a superstitious gut feeling that we may lose some 'transaction'y-ness from the current setup. For example, what happens if phase1 and phase2 overlap their execution? It's not that I have evidence that this will be a problem; it's more that we had these issues worked out fairly well with rdopkg CI 0.1, and I think the change warrants some scrutiny/thought (which clearly I have not done!). We'd still need to work out a way to execute phase2, though. There would be no `rdopkg update` event to trigger phase2 runs. I'm not sure how we'd do that without a BuildFlow. BuildFlow jobs also allow parallelization of the child jobs, and I'm not sure how we could replicate that without using that type of job. Whew. I hope this was helpful. I'm saving a copy of this text to http://slinabery.fedorapeople.org/rdopkg-overview.txt Cheers, Steve Linabery (freenode: eggmaster) Senior Software Engineer, Red Hat, Inc.

9 years, 2 months

5
7
0 / 0

[Rdo-list] Problem with Cinder (Juno) and Ceph

by Walter Valenti

Hi, we're trying Juno with Ceph (Giant) integration on Centos7. The problem is that Cinder-Volume ignores, the directives for use rbd protocol, but every uses LVM. This is our rbd configuration in cinder.conf: [DEFAULT] volume_driver=cinder.volume.drivers.rbd.RBDDriver [rbd] rbd_pool=volumes rbd_user=cinder rbd_ceph_conf=etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot=false rbd_secret_uuid=ac94d7df-4271-4080-b1fd-8cf577bf7471 rbd_max_clone_depth=5 rbd_store_chunk_size=4 rados_connect_timeout=-1 We also tried with all configuration in "rdb" section, and with all configuration in DEFAULT section, but the problem is the same. Any ideas?? Thanks Walter

9 years, 2 months

2
2
0 / 0

[Rdo-list] Hangout: What's coming for Ceilometer in Kilo. February 9th

by Rich Bowen

Mark your calendars: On February 9th, at 14:00 UTC, Eoghan Glynn will present a hangout covering what's coming for Ceilometer in the Kilo release of OpenStack. https://plus.google.com/events/cht3k5nr5u73pv3d08i7vq5m570 Kilo milestone 2 is due on February 5, so this is traditionally around the time that mid-cycle meetings happen to evaluate the status of projects, and set expectations for what will actually make it into the release. So I hope that we'll be able to schedule more of these hangouts in time between Milestone 2 and the release on April 30th. (See https://wiki.openstack.org/wiki/Kilo_Release_Schedule for the release schedule.) -- Rich Bowen - rbowen(a)redhat.com OpenStack Community Liaison http://openstack.redhat.com/

9 years, 2 months

1
1
0 / 0

[Rdo-list] staypuft on centos 6

by Mohammed Arafa

hi i am trying to install staypuft on centos 6 and it is failing. it also tells me to look in the logs for "ERROR" messages. which i cant find. would appreciate some help. the steps i took to reach the failure yum -y update yum -y install centos-release-SCL.x86_64 ftp://ftp.muug.mb.ca/mirror/fedora/epel/6/i386/epel-release-6-8.noarch.rpm http://yum.theforeman.org/releases/latest/el6/x86_64/foreman-release.rpm yum -y update hostname -f #verify hostname is in fqdn format in /etc/sysconfig/network and /etc/hosts if there is no dns yum -y install foreman-installer-staypuft staypuft-installer --foreman-plugin-discovery-install-images=true the error output is 1 Starting networking setup Networking setup has finished Preparing installation Done Something went wrong! Check the log for ERROR-level output * Foreman is running at https://staypuft.marafa.vm Initial credentials are admin / ru4jvHvkewz4Lfdz * Foreman Proxy is running at https://staypuft.marafa.vm:8443 * Puppetmaster is running at port 8140 The full log is at /var/log/foreman-installer/foreman-installer.log Not running provisioning configuration since installation encountered errors, exit code was 1 Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/foreman-installer.log [root@staypuft ~]# cat /var/log/foreman-installer/foreman-installer.log |grep -i error [root@staypuft ~]# netstat -tupane |grep -E "8443|8140" if it is useful to know this is of course a VM also .. nothing was installed so nothing is running on ports 8443 nor 8140. which means the output is 100% misleading your help and input is much appreciated. thank you -- <https://candidate.peoplecert.org/ReportsLink.aspx?argType=1&id=13D642E995...> *805010942448935* <https://www.redhat.com/wapps/training/certification/verify.html?certNumbe...> *GR750055912MA* <https://candidate.peoplecert.org/ReportsLink.aspx?argType=1&id=13D642E995...> *Link to me on LinkedIn <http://www.linkedin.com/in/mohammedarafa>*

9 years, 2 months

1
1
0 / 0

[Rdo-list] RDO meetup at FOSDEM?

by Rich Bowen

In Paris, we had a great meeting of RDO enthusiasts, to talk about community involvement and related issues. However, we were in a very loud place, and most people found it very difficult to hear what was going on. Will there be enough people at FOSDEM to try to do this again there? It should be a little easier to get a room than it was in Paris, so we shouldn't have to shout quite so loud to be heard. Anyone interested enough to set aside time for this. (I know how crazy busy FOSDEM can be.) -- Rich Bowen - rbowen(a)redhat.com OpenStack Community Liaison http://openstack.redhat.com/

9 years, 2 months

4
6
0 / 0

[Rdo-list] [meeting] RDO Packaging meeting minutes (2015-01-28)

by Alan Pevec

================================ #rdo: RDO packaging (2015-01-28) ================================ Meeting started by number80 at 15:01:05 UTC. The full logs are available at http://meetbot.fedoraproject.org/rdo/2015-01-28/rdo.2015-01-28-15.01.log.... . Meeting summary --------------- * roll call (number80, 15:01:15) * FPCA legal stuff (number80, 15:05:01) * IDEA: moving primary RDO dist-git completely to github/openstack-packages/ then sync to Fedora (number80, 15:07:48) * LINK: http://ltsi.linuxfoundation.org/developers/signed-process (number80, 15:09:17) * ACTION: apevec request input on s-o-b policy on rdo m-l (number80, 15:12:14) * trunk.rdoproject.org (number80, 15:14:42) * ACTION: jruzicka to poke csim about trunk.rdoproject.org vhost on rdo site (apevec, 15:20:18) * installer for trunk aka Packstack against Delorean repo (apevec, 15:21:18) * ACTION: gchamoul will fix the beast (apevec, 15:22:44) * Packstack master to work with Delorean repo by EOW (apevec, 15:27:11) * rdopkg CI docs (apevec, 15:27:26) * expecting CI doc today (apevec, 15:29:43) * moving to CBS builds for RDO Juno EL7 updates (apevec, 15:32:28) * ACTION: jruzicka to add rdoupdate cbs source (apevec, 15:36:57) * ACTION: jruzicka to acquire CBS account (jruzicka, 15:37:41) * Juno/EL6 packages (apevec, 15:41:39) Meeting ended at 15:48:26 UTC. Action Items ------------ * apevec request input on s-o-b policy on rdo m-l * jruzicka to poke csim about trunk.rdoproject.org vhost on rdo site * gchamoul will fix the beast * jruzicka to add rdoupdate cbs source * jruzicka to acquire CBS account Action Items, by person ----------------------- * apevec * apevec request input on s-o-b policy on rdo m-l * gchamoul * gchamoul will fix the beast * jruzicka * jruzicka to poke csim about trunk.rdoproject.org vhost on rdo site * jruzicka to add rdoupdate cbs source * jruzicka to acquire CBS account * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * apevec (97) * number80 (35) * gchamoul (17) * jruzicka (11) * zodbot (5) * kashyap (5) * eggmaster (4) * trown (1) * ryansb (1) * chandankumar (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot

9 years, 3 months

1
0
0 / 0

[Rdo-list] Current Status on RDO HA deployment w/ or w/o Foreman (documentation)

by Arash Kaffamanesh

Hello together, Any new news about RDO HA deployment guide with or without Foreman? Thanks! Arah

9 years, 3 months

1
0
0 / 0

[Rdo-list] RDO/OpenStack meetups coming up (January 26, 2015)

by Rich Bowen

The following are the meetups I'm aware of in the coming week where RDO enthusiasts are likely to be present. If you know of others, please let me know, and/or add them to http://openstack.redhat.com/Events If there's a meetup in your area, please consider attending. It's the best way to find out what interesting things are going on in the larger community, and a great way to make contacts that will help you solve your own problems in the future. And don't forget to blog about it, tweet about it, G+ about it. --Rich * Tuesday, January 27 in Mountain View, CA, US: Online Meetup: Auto-scaling MySQL Database Services on OpenStack with ScaleDB - http://www.meetup.com/Cloud-Online-Meetup/events/219983542/ * Tuesday, January 27 in Sydney, AU: Do you want to help organise a Melbourne OpenStack Meetup in 2015? - http://www.meetup.com/Australian-OpenStack-User-Group/events/219282263/ * Tuesday, January 27 in Budapest, HU: OpenStack 2015 Jan - http://www.meetup.com/OpenStack-Hungary-Meetup-Group/events/220088735/ * Tuesday, January 27 in Tel Aviv-Yafo, IL: Auto-scaling MySQL Database Services on OpenStack with ScaleDB, Ronen Ariely - http://www.meetup.com/OpenStack-Israel/events/219391328/ * Wednesday, January 28 in Mountain View, CA, US: Online Meetup: MidoNet gives OpenStack Neutron a Boost - http://www.meetup.com/Cloud-Online-Meetup/events/220006786/ * Wednesday, January 28 in Mountain View, CA, US: Online Meetup: Unlock Your Cloud Potential w/ Mirantis OpenStack & Cumulus Linux - http://www.meetup.com/Cumulus-Linux-Open-Networking-User-Group-Bay-Area/e... * Wednesday, January 28 in Vancouver, BC, CA: OpenStack plays well with others - http://www.meetup.com/Vancouver-OpenStack-Meetup/events/219622935/ * Wednesday, January 28 in Las Vegas, NV, US: Platform as a Service and Identity Management - http://www.meetup.com/Las-Vegas-Red-Hat-User-Group/events/218703467/ * Thursday, January 29 in Mountain View, CA, US: Online Meetup: The Latest on OpenStack Trove in the Upcoming Kilo Release - http://www.meetup.com/Cloud-Online-Meetup/events/220007088/ * Thursday, January 29 in Sydney, AU: Do you want to help organise a Sydney OpenStack Meetup in 2015? - http://www.meetup.com/Australian-OpenStack-User-Group/events/219814136/ * Thursday, January 29 in Whittier, CA, US: Introduction to Red Hat and OpenShift (cohost with OC MongoDB UG) - http://www.meetup.com/Greater-Los-Angeles-Area-Red-Hat-User-Group-RHUG/ev... * Friday, January 30 in Pittsburgh, PA, US: OpenStack Projects: Ideas and Implementations - http://www.meetup.com/openstack-pittsburgh/events/220036464/ * Saturday, January 31 in Xian, CN: OpenStack Xian Meetup 2015 启航 - http://www.meetup.com/Xian-OpenStack-Meetup/events/220086059/ * Saturday, January 31 in Delhi, IN: Intros and Kickoff - http://www.meetup.com/SDN-OpenDayLight-Delhi-User-Group/events/219875181/ * Monday, February 02 in Seattle, WA, US: OpenStack Seattle Meetup: Tech Duo Featuring PLUMgrid and Tesora - http://www.meetup.com/OpenStack-Seattle/events/198405792/ -- Rich Bowen - rbowen(a)redhat.com OpenStack Community Liaison http://openstack.redhat.com/

9 years, 3 months

1
0
0 / 0

[Rdo-list] [Fedocal] Reminder meeting : RDO packaging meeting

by hguemar＠fedoraproject.org

Dear all, You are kindly invited to the meeting: RDO packaging meeting on 2015-01-28 from 15:00:00 to 16:00:00 UTC The meeting will be about: RDO packaging irc meeting ([agenda](https://etherpad.openstack.org/p/RDO-Packaging)) Every week on #rdo on freenode Source: https://apps.fedoraproject.org/calendar//meeting/2024/

9 years, 3 months

1
0
0 / 0

[Rdo-list] Juno EPEL packages

by David Hill

Hi guys, I'm volunteering to maintain Centos/RHEL/OEL 6 packages for Juno if nobody's doing it yet. How do I proceed to get involved in this project? We're already building packages internally with a set of custom patches and I feel like moving to RHEL 7.x will take lots of effort to lots of people. There's already a python-2.7 package in EPEL (I think) if Juno/Kilo/Lima/Mike requires python-2.7 there's always ways of getting around those requirements. BTW, I would like to know if there're lots of people using RHEL 6.X or are we only a small subset of the community? Thank you very much, Dave

9 years, 3 months

8
14
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

dev January 2015