[rdo-list] Multiple tools for deploying and testing TripleO

Ronelle Landy rlandy at redhat.com
Wed Aug 3 13:57:02 UTC 2016 

> From: "Wesley Hayutin" <whayutin at redhat.com>
> To: "Frank Zdarsky" <fzdarsky at redhat.com>
> Cc: "Raoul Scarazzini" <rscarazz at redhat.com>, rdo-list at redhat.com
> Sent: Wednesday, August 3, 2016 9:00:46 AM
> Subject: Re: [rdo-list] Multiple tools for deploying and testing TripleO
> 
> 
> 
> On Wed, Aug 3, 2016 at 5:23 AM, Frank Zdarsky < fzdarsky at redhat.com > wrote:
> 
> 
> 
> On Wed, Aug 3, 2016 at 9:28 AM, Tal Kammer < tkammer at redhat.com > wrote:
> 
> 
> 
> Thanks Arie for starting this discussion! (and sorry for joining in late)
> 
> Some detailed explanations for those not familiar with InfraRed and would
> like to get the highlights: (Feel free to skip to my inline comments if you
> are familiar / too long of a post :) ).
> 
> The InfraRed project is an Ansible based project comprised from three
> distinct tools: (currently under the InfraRed[1] project, and being split
> into their own standalone project soon).
> 
> 1. ir-provisioner - responsible for building up a custom environment - you
> can affect the memory, CPU, HDD size, number of HDD each node has + number
> of networks (with/without DHCP) so for example one can deploy the following
> topology: (only an example to show the versatile options)
> 1 undercloud on RHEL 7 with 16GB of ram + 60GB of HDD with 3 network
> interfaces.
> 3 controllers on Centos 7 with 8GB of ram + 40GB of HDD
> 2 compute on Centos 7 with 6GB of ram + 60GB HDD
> 3 ceph nodes on RHEL 7 with 4GB of ram + 2 HDD one with 20GB + one with 40GB
> 
> Example usage: (setting up the above VMs with four different HW specs)
> ir-provisioner virsh --host-address=<host.address>
> --host-key=<host.private.key>
> --topology-nodes=undercloud:1,controller:3,compute:2,ceph:3
> 
> *Note: while it is written "controller/compute/ceph" this is just setting up
> VMs, the names act more as a reference to the user of what is the role of
> each node.
> The installation of Openstack is done with a dedicated tool called
> `ir-installer` (next)
> 
> 2. ir-installer - responsible for installing the product - supports
> "quickstart" mode (setting up a working environment in ~30 minutes) or E2E
> mode which does a full installation in ~1h.
> The installation process is completely customized. You can supply your own
> heat templates / overcloud invocation / undercloud.conf file to use / etc.
> You can also just run a specific task (using ansible --tags), so if you have
> a deployment ready and just need to run say, the introspection phase, you
> can fully choose what to run and what to skip even.
> 
> 3. ir-tester - responsible for installing / configuring / running the tests -
> this project is meant to hold all testing tools we use so a user will be
> able to run any testing utility he would like without the need to "dive in".
> we supply a simple interface requesting simple to choose the testing tool
> and the set of tests one wishes to run and we'll do the work for him :)
> 
> More info about InfraRed can be found here[1] (though I must admit that we
> still need some "love" around our docs)
> 
> [1] - http://infrared.readthedocs.io/en/latest/
> 
> 
> I agree cleanly separating and modularizing infrastructure provisioining (for
> some use cases like NFV ideally with mixed VM and baremetal environments),
> OS installation and testing is a good approach. In that context, what
> happened to Khaleesi [0]? Haven't seen that mentioned on the original list.
> Nor Apex [1], the installer from OPNFV that does not only address OpenStack
> itself, but also integration with complementary projects like ODL?
> 
> I consider provisioning baremetal for either libvirt hosts or baremetal
> installs is a separate project and conversation from the CI tools used to
> deploy tripleo.
> To be a little more clear, provisioning imho should be completely uncoupled
> from the CI code that deploys tripleo. Any team should be able to use
> provisioner tools that best fit their needs.
> If there is interest in collaborating on provisioners maybe we can do that in
> another thread. Why in another thread? Again my thought is they should be
> uncoupled completely from our CI source code.
> 
> The rdo infra team is using two provisioners, both completely uncoupled from
> tripleo-quickstart.
> 
> 1. Ansible extra heat provisioner: for ovb, written by Mathieu Bultel
> https://github.com/ansible/ansible-modules-extras/commit/c6a45234e094922786de78a83db3028804cc9141

Link to the Ansible role for Tripleo-Quickstart that uses the heat provisioner:
https://github.com/redhat-openstack/ansible-role-tripleo-provision-heat (was forked originally from https://github.com/redhat-openstack/ansible-ovb)

I am also looking into implementing Goneri's solution to run OVB on public clouds.

> 
> 2. cico, the official client for ci.centos written by David Simard
> http://python-cicoclient.readthedocs.io/en/latest/cli_usage.html
> 
> Our baremetal deployments do not require any provisioning outside of what is
> done w/ ironic and tripleo.

Our current CI baremetal jobs do not require hardware provisioning as we are using a virt undercloud (as we were advised this is pretty standard practice for many users).
This role preps the baremetal host machine to run a VM to serve as the undercloud:

https://github.com/redhat-openstack/ansible-role-tripleo-baremetal-prep-virthost

Other related roles to complete the baremetal workflow (includes some validations):
https://github.com/redhat-openstack/ansible-role-tripleo-validate-ipmi
https://github.com/redhat-openstack/ansible-role-tripleo-baremetal-overcloud

> 
> Upstream tripleo ci provisioning is done via node pool and ovb.

I have these steps and am replicating this workflow on a standard host cloud for comparison and testing.
> 
> FYI.. khaleesi has been deprecated and is no longer used by RDO infra.
> 
> Thanks
> 
> 
> 
> 
> [0] https://github.com/redhat-openstack/khaleesi
> [1] https://wiki.opnfv.org/display/apex/Apex
> 
> 
> 
> 
> On Tue, Aug 2, 2016 at 9:52 PM, Wesley Hayutin < whayutin at redhat.com > wrote:
> 
> 
> 
> 
> 
> On Tue, Aug 2, 2016 at 1:51 PM, Arie Bregman < abregman at redhat.com > wrote:
> 
> 
> 
> On Tue, Aug 2, 2016 at 3:53 PM, Wesley Hayutin < whayutin at redhat.com > wrote:
> > 
> > 
> > On Tue, Aug 2, 2016 at 4:58 AM, Arie Bregman < abregman at redhat.com > wrote:
> >> 
> >> It became a discussion around the official installer and how to
> >> improve it. While it's an important discussion, no doubt, I actually
> >> want to focus on our automation and CI tools.
> >> 
> >> Since I see there is an agreement that collaboration does make sense
> >> here, let's move to the hard questions :)
> >> 
> >> Wes, Tal - there is huge difference right now between infrared and
> >> tripleo-quickstart in their structure. One is all-in-one project and
> >> the other one is multiple micro projects managed by one project. Do
> >> you think there is a way to consolidate or move to a different model
> >> which will make sense for both RDO and RHOSP? something that both
> >> groups can work on.
> > 
> > 
> > I am happy to be part of the discussion, and I am also very willing to help
> > and try to drive suggestions to the tripleo-quickstart community.
> > I need to make a point clear though, just to make sure we're on the same
> > page.. I do not own oooq, I am not a core on oooq.
> > I can help facilitate a discussion but oooq is an upstream tripleo tool
> > that
> > replaces instack-virt-setup [1].
> > It also happens to be a great tool for easily deploying TripleO end to end
> > [3]
> > 
> > What I *can* do is show everyone how to manipulate tripleo-quickstart and
> > customize it with composable ansible roles, templates, settings etc..
> > This would allow any upstream or downstream project to override the native
> > oooq roles and *any* step that does not work for another group w/ 3rd party
> > roles [2].
> > These 3rd party roles can be free and opensource or internal only, it works
> > either way.
> > This was discussed in depth as part of the production chain meetings, the
> > message may have been lost unfortunately.
> > 
> > I hope this resets your expectations of what I can and can not do as part
> > of
> > these discussions.
> > Let me know where and when and I'm happy to be part of the discussion.
> 
> Thanks for clarifying :)
> 
> Next reasonable step would probably be to propose some sort of
> blueprint for tripleo-quickstart to include some of InfraRed features
> and by that have one tool driven by upstream development that can be
> either cloned downstream or used as it is with an internal data
> project.
> 
> Sure.. a blueprint would help everyone understand the feature and the
> motivation.
> You could also just plug in the feature you are looking for to oooq and see
> if it meets
> your requirements. See below.
> 
> While I think a blueprint is a good starting point, I'm afraid that our
> approach for provisioning machines is completely different so I'm not sure
> how to propose such a blueprint as it will probably require quite the design
> change from today's approach.
> 
> 
> 
> 
> 
> 
> OR
> 
> have InfraRed pushed into tripleo/openstack namespace and expose it to
> the RDO community (without internal data of course). Personally, I
> really like the pluggable[1] structure (which allows it to actually
> consume tripleo-quickstart) so I'm not sure if it can be really merged
> with tripleo-quickstart as proposed in the first option.
> 
> I must admit that I like this option better as it introduces a tool to
> upstream and let the community drive it further / get more feedback on how
> to improve.
> A second benefit might be that by introducing a new concept / design we can
> take what is best from two worlds and improve.
> I would love to see an open discussion on the tool upstream and how we can
> improve the overall process.
> 
> 
> 
> 
> 
> 
> 
> 
> The way oooq is built one can plugin or override any part at run time
> with custom playbooks, roles, and config. There isn't anything that needs to
> be
> checked in directly to oooq to use it.
> 
> It's designed such that third parties can make their own decisions to use
> something
> native to quickstart, something from our role library, or something
> completely independent.
> This allows teams, individuals or whom ever to do what they need to with out
> having to fork or re-roll the entire framework.
> 
> The important step is to note that these 3rd party roles or (oooq-extras)
> incubate, mature and then graduate to github/openstack.
> The upstream openstack community should lead, evaluate, and via blueprints
> vote on the canonical CI tool set.
> 
> We can record a demonstration if required, but there is nothing stopping
> anyone right now from
> doing this today. I'm just browsing the role library for an example, I had no
> idea [1] existed.
> Looks like Raoul had a requirement and just made it work.
> 
> Justin, from the browbeat project has graciously created some documentation
> regarding 3rd party roles.
> It has yet to merge, but it should help illustrate how these roles are used.
> [2]
> 
> Thanks Arie for leading the discussion.
> 
> [1]
> https://github.com/redhat-openstack/ansible-role-tripleo-overcloud-validate-ha
> [2] https://review.openstack.org/#/c/346733/
> 
> 
> 
> 
> 
> 
> 
> I like the second option, although it still forces us to have two
> tools, but after a period of time, I believe it will be clear what the
> community prefers, which will allow us to remove one of the projects
> eventually.
> 
> So, unless there are other ideas, I think the next move should be made by
> Tal.
> 
> Tal, I'm willing to help with whatever is needed.
> 
> Thanks Arie for starting this discussion again, I believe we have still much
> work ahead of us but this is definitely a step in the right direction.
> 
> 
> 
> 
> 
> 
> [1] http://infrared.readthedocs.io/en/latest
> 
> > 
> > Thanks
> > 
> > [1] https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart
> > [2]
> > https://github.com/redhat-openstack/?utf8=%E2%9C%93&query=ansible-role-tripleo
> > [3[ https://www.rdoproject.org/tripleo/
> > 
> > 
> >> 
> >> 
> >> Raoul - I totally agree with you, especially with "difficult for
> >> anyone to start contributing and collaborate". This is exactly why
> >> this discussion started. If we can agree on one set of tools, it will
> >> make everyone's life easier - current groups, new contributors, folks
> >> that just want to deploy TripleO quickly. But I'm afraid some
> >> sacrifices need to be made by both groups.
> >> 
> >> David - I thought WeiRDO is used only for packstack, so I apologize I
> >> didn't include it. It does sound like an anther testing project, is
> >> there a place to merge it with another existing testing project? like
> >> Octario for example or one of TripleO testing projects. Or does it
> >> make sense to keep it a standalone project?
> >> 
> >> 
> >> 
> >> 
> >> On Tue, Aug 2, 2016 at 11:12 AM, Christopher Brown < cbrown2 at ocf.co.uk >
> >> wrote:
> >> > Hello RDOistas (I think that is the expression?),
> >> > 
> >> > Another year, another OpenStack deployment tool. :)
> >> > 
> >> > On Mon, 2016-08-01 at 18:59 +0100, Ignacio Bravo wrote:
> >> >> If we are talking about tools, I would also want to add something
> >> >> with regards to user interface of these tools. This is based on my
> >> >> own experience:
> >> >> 
> >> >> I started trying to deploy Openstack with Staypuft and The Foreman.
> >> >> The UI of The Foreman was intuitive enough for the discovery and
> >> >> provisioning of the servers. The OpenStack portion, not so much.
> >> > 
> >> > This is exactly mine also. I think this works really well in very large
> >> > enterprise environments where you need to split out services over more
> >> > than three controllers. You do need good in-house puppet skills though
> >> > so better for enterprise with a good sysadmin team.
> >> > 
> >> >> Forward a couple of releases and we had a TripleO GUI (Tuskar, I
> >> >> believe) that allowed you to graphically build your Openstack cloud.
> >> >> That was a reasonable good GUI for Openstack.
> >> > 
> >> > Well, I found it barely usable. It was only ever good as a graphical
> >> > representiation of what the build was doing. Interacting with it was
> >> > not great.
> >> > 
> >> >> Following that, TripleO become a script based installer, that
> >> >> required experience in Heat templates. I know I didn’t have it and
> >> >> had to ask in the mailing list about how to present this or change
> >> >> that. I got a couple of installs working with this setup.
> >> > 
> >> > Works well now that I understand all the foibles and have invested time
> >> > into understanding heat templates and puppet modules. Its good in that
> >> > it forces you to learn about orchestration which is such an important
> >> > end-goal of cloud environments.
> >> > 
> >> >> In the last session in Austin, my goal was to obtain information on
> >> >> how others were installing Openstack. I was pointed to Fuel as an
> >> >> alternative. I tried it up, and it just worked. It had the
> >> >> discovering capability from The Foreman, and the configuration
> >> >> options from TripleO. I understand that is based in Ansible and
> >> >> because of that, it is not fully CentOS ready for all the nodes (at
> >> >> least not in version 9 that I tried). In any case, as a deployer and
> >> >> installer, it is the most well rounded tool that I found.
> >> > 
> >> > This is interesting to know. I've heard of Fuel of course but there are
> >> > some politics involved - it still has the team:single-vendor tag but
> >> > from what I see Mirantis are very keen for it to become the default
> >> > OpenStack installer. I don't think being Ansible-based should be a
> >> > problem - we are deploying OpenShift on OpenStack which uses Openshift-
> >> > ansible - this recently moved to Ansible 2.1 without too much
> >> > disruption.
> >> > 
> >> >> I’d love to see RDO moving into that direction, and having an easy to
> >> >> use, end user ready deployer tool.
> >> > 
> >> > If its as good as you say its definitely worth evaluating. From our
> >> > point of view, we want to be able to add services to the pacemaker
> >> > cluster with some ease - for example Magnum and Sahara - and it looks
> >> > like there are steps being taken with regards to composable roles and
> >> > simplification of the pacemaker cluster to just core services.
> >> > 
> >> > But if someone can explain that better I would appreciate it.
> >> > 
> >> > Regards
> >> > 
> >> >> IB
> >> >> 
> >> >> 
> >> >> __
> >> >> Ignacio Bravo
> >> >> LTG Federal, Inc
> >> >> www.ltgfederal.com
> >> >> 
> >> >> 
> >> >> > On Aug 1, 2016, at 1:07 PM, David Moreau Simard < dms at redhat.com >
> >> >> > wrote:
> >> >> > 
> >> >> > The vast majority of RDO's CI relies on using upstream
> >> >> > installation/deployment projects in order to test installation of
> >> >> > RDO
> >> >> > packages in different ways and configurations.
> >> >> > 
> >> >> > Unless I'm mistaken, TripleO Quickstart was originally created as a
> >> >> > mean to "easily" install TripleO in different topologies without
> >> >> > requiring a massive amount of hardware.
> >> >> > This project allows us to test TripleO in virtual deployments on
> >> >> > just
> >> >> > one server instead of, say, 6.
> >> >> > 
> >> >> > There's also WeIRDO [1] which was left out of your list.
> >> >> > WeIRDO is super simple and simply aims to run upstream gate jobs
> >> >> > (such
> >> >> > as puppet-openstack-integration [2][3] and packstack [4][5])
> >> >> > outside
> >> >> > of the gate.
> >> >> > It'll install dependencies that are expected to be there (i.e,
> >> >> > usually
> >> >> > set up by the openstack-infra gate preparation jobs), set up the
> >> >> > trunk
> >> >> > repositories we're interested in testing and the rest is handled by
> >> >> > the upstream project testing framework.
> >> >> > 
> >> >> > The WeIRDO project is /very/ low maintenance and brings an
> >> >> > exceptional
> >> >> > amount of coverage and value.
> >> >> > This coverage is important because RDO provides OpenStack packages
> >> >> > or
> >> >> > projects that are not necessarily used by TripleO and the reality
> >> >> > is
> >> >> > that not everyone deploying OpenStack on CentOS with RDO will be
> >> >> > using
> >> >> > TripleO.
> >> >> > 
> >> >> > Anyway, sorry for sidetracking but back to the topic, thanks for
> >> >> > opening the discussion.
> >> >> > 
> >> >> > What honestly perplexes me is the situation of CI in RDO and OSP,
> >> >> > especially around TripleO/Director, is the amount of work that is
> >> >> > spent downstream.
> >> >> > And by downstream, here, I mean anything that isn't in TripleO
> >> >> > proper.
> >> >> > 
> >> >> > I keep dreaming about how awesome upstream TripleO CI would be if
> >> >> > all
> >> >> > that effort was spent directly there instead -- and then that all
> >> >> > work
> >> >> > could bear fruit and trickle down downstream for free.
> >> >> > Exactly like how we keep improving the testing coverage in
> >> >> > puppet-openstack-integration, it's automatically pulled in RDO CI
> >> >> > through WeIRDO for free.
> >> >> > We make the upstream better and we benefit from it simultaneously:
> >> >> > everyone wins.
> >> >> > 
> >> >> > [1]: https://github.com/rdo-infra/weirdo
> >> >> > [2]: https://github.com/rdo-infra/ansible-role-weirdo-puppet-openst
> >> >> > ack
> >> >> > [3]: https://github.com/openstack/puppet-openstack-integration#desc
> >> >> > ription
> >> >> > [4]: https://github.com/rdo-infra/ansible-role-weirdo-packstack
> >> >> > [5]: https://github.com/openstack/packstack#packstack-integration-t
> >> >> > ests
> >> >> > 
> >> >> > David Moreau Simard
> >> >> > Senior Software Engineer | Openstack RDO
> >> >> > 
> >> >> > dmsimard = [irc, github, twitter]
> >> >> > 
> >> >> > David Moreau Simard
> >> >> > Senior Software Engineer | Openstack RDO
> >> >> > 
> >> >> > dmsimard = [irc, github, twitter]
> >> >> > 
> >> >> > 
> >> >> > On Mon, Aug 1, 2016 at 11:21 AM, Arie Bregman < abregman at redhat.com >
> >> >> > wrote:
> >> >> > > Hi,
> >> >> > > 
> >> >> > > I would like to start a discussion on the overlap between tools
> >> >> > > we
> >> >> > > have for deploying and testing TripleO (RDO & RHOSP) in CI.
> >> >> > > 
> >> >> > > Several months ago, we worked on one common framework for
> >> >> > > deploying
> >> >> > > and testing OpenStack (RDO & RHOSP) in CI. I think you can say it
> >> >> > > didn't work out well, which eventually led each group to focus on
> >> >> > > developing other existing/new tools.
> >> >> > > 
> >> >> > > What we have right now for deploying and testing
> >> >> > > --------------------------------------------------------
> >> >> > > === Component CI, Gating ===
> >> >> > > I'll start with the projects we created, I think that's only fair
> >> >> > > :)
> >> >> > > 
> >> >> > > * Ansible-OVB[1] - Provisioning Tripleo heat stack, using the OVB
> >> >> > > project.
> >> >> > > 
> >> >> > > * Ansible-RHOSP[2] - Product installation (RHOSP). Branch per
> >> >> > > release.
> >> >> > > 
> >> >> > > * Octario[3] - Testing using RPMs (pep8, unit, functional,
> >> >> > > tempest,
> >> >> > > csit) + Patching RPMs with submitted code.
> >> >> > > 
> >> >> > > === Automation, QE ===
> >> >> > > * InfraRed[4] - provision install and test. Pluggable and
> >> >> > > modular,
> >> >> > > allows you to create your own provisioner, installer and tester.
> >> >> > > 
> >> >> > > As far as I know, the groups is working now on different
> >> >> > > structure of
> >> >> > > one main project and three sub projects (provision, install and
> >> >> > > test).
> >> >> > > 
> >> >> > > === RDO ===
> >> >> > > I didn't use RDO tools, so I apologize if I got something wrong:
> >> >> > > 
> >> >> > > * About ~25 micro independent Ansible roles[5]. You can either
> >> >> > > choose
> >> >> > > to use one of them or several together. They are used for
> >> >> > > provisioning, installing and testing Tripleo.
> >> >> > > 
> >> >> > > * Tripleo-quickstart[6] - uses the micro roles for deploying
> >> >> > > tripleo
> >> >> > > and test it.
> >> >> > > 
> >> >> > > As I said, I didn't use the tools, so feel free to add more
> >> >> > > information you think is relevant.
> >> >> > > 
> >> >> > > === More? ===
> >> >> > > I hope not. Let us know if are familiar with more tools.
> >> >> > > 
> >> >> > > Conclusion
> >> >> > > --------------
> >> >> > > So as you can see, there are several projects that eventually
> >> >> > > overlap
> >> >> > > in many areas. Each group is basically using the same tasks
> >> >> > > (provision
> >> >> > > resources, build/import overcloud images, run tempest, collect
> >> >> > > logs,
> >> >> > > etc.)
> >> >> > > 
> >> >> > > Personally, I think it's a waste of resources. For each task
> >> >> > > there is
> >> >> > > at least two people from different groups who work on exactly the
> >> >> > > same
> >> >> > > task. The most recent example I can give is OVB. As far as I
> >> >> > > know,
> >> >> > > both groups are working on implementing it in their set of tools
> >> >> > > right
> >> >> > > now.
> >> >> > > 
> >> >> > > On the other hand, you can always claim: "we already tried to
> >> >> > > work on
> >> >> > > the same framework, we failed to do it successfully" - right, but
> >> >> > > maybe with better ground rules we can manage it. We would
> >> >> > > defiantly
> >> >> > > benefit a lot from doing that.
> >> >> > > 
> >> >> > > What's next?
> >> >> > > ----------------
> >> >> > > So first of all, I would like to hear from you if you think that
> >> >> > > we
> >> >> > > can collaborate once again or is it actually better to keep it as
> >> >> > > it
> >> >> > > is now.
> >> >> > > 
> >> >> > > If you agree that collaboration here makes sense, maybe you have
> >> >> > > ideas
> >> >> > > on how we can do it better this time.
> >> >> > > 
> >> >> > > I think that setting up a meeting to discuss the right
> >> >> > > architecture
> >> >> > > for the project(s) and decide on good review/gating process,
> >> >> > > would be
> >> >> > > a good start.
> >> >> > > 
> >> >> > > Please let me know what do you think and keep in mind that this
> >> >> > > is not
> >> >> > > about which tool is better!. As you can see I didn't mention the
> >> >> > > time
> >> >> > > it takes for each tool to deploy and test, and also not the full
> >> >> > > feature list it supports.
> >> >> > > If possible, we should keep it about collaborating and not
> >> >> > > choosing
> >> >> > > the best tool. Our solution could be the combination of two or
> >> >> > > more
> >> >> > > tools eventually (tripleo-red, infra-quickstart? :D )
> >> >> > > 
> >> >> > > "You may say I'm a dreamer, but I'm not the only one. I hope some
> >> >> > > day
> >> >> > > you'll join us and the infra will be as one" :)
> >> >> > > 
> >> >> > > [1] https://github.com/redhat-openstack/ansible-ovb
> >> >> > > [2] https://github.com/redhat-openstack/ansible-rhosp
> >> >> > > [3] https://github.com/redhat-openstack/octario
> >> >> > > [4] https://github.com/rhosqeauto/InfraRed
> >> >> > > [5] https://github.com/redhat-openstack?utf8=%E2%9C%93&query=ansi
> >> >> > > ble-role
> >> >> > > [6] https://github.com/openstack/tripleo-quickstart
> >> >> > > 
> >> >> > > _______________________________________________
> >> >> > > rdo-list mailing list
> >> >> > > rdo-list at redhat.com
> >> >> > > https://www.redhat.com/mailman/listinfo/rdo-list
> >> >> > > 
> >> >> > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >> >> > _______________________________________________
> >> >> > rdo-list mailing list
> >> >> > rdo-list at redhat.com
> >> >> > https://www.redhat.com/mailman/listinfo/rdo-list
> >> >> > 
> >> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >> >> 
> >> > --
> >> > Regards,
> >> > 
> >> > Christopher Brown
> >> > OpenStack Engineer
> >> > OCF plc
> >> > 
> >> > Tel: +44 (0)114 257 2200
> >> > Web: www.ocf.co.uk
> >> > Blog: blog.ocf.co.uk
> >> > Twitter: @ocfplc
> >> > 
> >> > Please note, any emails relating to an OCF Support request must always
> >> > be sent to support at ocf.co.uk for a ticket number to be generated or
> >> > existing support ticket to be updated. Should this not be done then OCF
> >> > cannot be held responsible for requests not dealt with in a timely
> >> > manner.
> >> > 
> >> > OCF plc is a company registered in England and Wales. Registered number
> >> > 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc,
> >> > 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield S35
> >> > 2PG.
> >> > 
> >> > This message is private and confidential. If you have received this
> >> > message in error, please notify us immediately and remove it from your
> >> > system.
> >> > 
> >> > _______________________________________________
> >> > rdo-list mailing list
> >> > rdo-list at redhat.com
> >> > https://www.redhat.com/mailman/listinfo/rdo-list
> >> > 
> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >> 
> >> 
> >> 
> >> --
> >> Arie Bregman
> >> Red Hat Israel
> >> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview
> > 
> > 
> 
> 
> 
> --
> Arie Bregman
> Red Hat Israel
> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview
> 
> 
> 
> 
> --
> Tal Kammer
> Associate manager, automation and infrastracture, Openstack platform.
> Red Hat Israel
> Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659
> 
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com
> 
> 
> 
> --
> {Kind regards | Mit besten Grüßen},
> 
> Frank
> 
> ________________________________
> Frank Zdarsky | NFV Partner Engineering | Office of Technology | Red Hat
> e: fzdarsky at redhat.com | irc: fzdarsky @freenode | m: +49 175 82 11 64 4 | t:
> +49 711 96 43 70 02
> 
> 
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com

More information about the dev mailing list