Re: [rdo-list] Multiple tools for deploying and testing TripleO

On Wed, Aug 3, 2016 at 9:28 AM, Tal Kammer <tkammer(a)redhat.com&gt; wrote: > Thanks Arie for starting this discussion! (and sorry for joining in late) > > Some detailed explanations for those not familiar with InfraRed and would > like to get the highlights: (Feel free to skip to my inline comments if > you are familiar / too long of a post :) ). > > The InfraRed project is an Ansible based project comprised from three > distinct tools: (currently under the InfraRed[1] project, and being split > into their own standalone project soon). > > 1. ir-provisioner - responsible for building up a custom environment - > you can affect the memory, CPU, HDD size, number of HDD each node has + > number of networks (with/without DHCP) so for example one can deploy the > following topology: (only an example to show the versatile options) > 1 undercloud on RHEL 7 with 16GB of ram + 60GB of HDD with 3 network > interfaces. > 3 controllers on Centos 7 with 8GB of ram + 40GB of HDD > 2 compute on Centos 7 with 6GB of ram + 60GB HDD > 3 ceph nodes on RHEL 7 with 4GB of ram + 2 HDD one with 20GB + one with > 40GB > > Example usage: (setting up the above VMs with four different HW specs) > ir-provisioner virsh --host-address=<host.address> > --host-key=<host.private.key> > --topology-nodes=undercloud:1,controller:3,compute:2,ceph:3 > > *Note: while it is written "controller/compute/ceph" this is just setting > up VMs, the names act more as a reference to the user of what is the role > of each node. > The installation of Openstack is done with a dedicated tool called > `ir-installer` (next) > > 2. ir-installer - responsible for installing the product - supports > "quickstart" mode (setting up a working environment in ~30 minutes) or E2E > mode which does a full installation in ~1h. > The installation process is completely customized. You can supply your > own heat templates / overcloud invocation / undercloud.conf file to use / > etc. You can also just run a specific task (using ansible --tags), so if > you have a deployment ready and just need to run say, the introspection > phase, you can fully choose what to run and what to skip even. > > 3. ir-tester - responsible for installing / configuring / running the > tests - this project is meant to hold all testing tools we use so a user > will be able to run any testing utility he would like without the need to > "dive in". we supply a simple interface requesting simple to choose the > testing tool and the set of tests one wishes to run and we'll do the work > for him :) > > More info about InfraRed can be found here[1] (though I must admit that > we still need some "love" around our docs) > > [1] - http://infrared.readthedocs.io/en/latest/ > > I agree cleanly separating and modularizing infrastructure provisioining (for some use cases like NFV ideally with mixed VM and baremetal environments), OS installation and testing is a good approach. In that context, what happened to Khaleesi [0]? Haven't seen that mentioned on the original list. Nor Apex [1], the installer from OPNFV that does not only address OpenStack itself, but also integration with complementary projects like ODL?

[0] https://github.com/redhat-openstack/khaleesi [1] https://wiki.opnfv.org/display/apex/Apex > > On Tue, Aug 2, 2016 at 9:52 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; > wrote: > >> >> >> On Tue, Aug 2, 2016 at 1:51 PM, Arie Bregman <abregman(a)redhat.com&gt; >> wrote: >> >>> On Tue, Aug 2, 2016 at 3:53 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; >>> wrote: >>> > >>> > >>> > On Tue, Aug 2, 2016 at 4:58 AM, Arie Bregman <abregman(a)redhat.com&gt; >>> wrote: >>> >> >>> >> It became a discussion around the official installer and how to >>> >> improve it. While it's an important discussion, no doubt, I actually >>> >> want to focus on our automation and CI tools. >>> >> >>> >> Since I see there is an agreement that collaboration does make sense >>> >> here, let's move to the hard questions :) >>> >> >>> >> Wes, Tal - there is huge difference right now between infrared and >>> >> tripleo-quickstart in their structure. One is all-in-one project and >>> >> the other one is multiple micro projects managed by one project. Do >>> >> you think there is a way to consolidate or move to a different model >>> >> which will make sense for both RDO and RHOSP? something that both >>> >> groups can work on. >>> > >>> > >>> > I am happy to be part of the discussion, and I am also very willing >>> to help >>> > and try to drive suggestions to the tripleo-quickstart community. >>> > I need to make a point clear though, just to make sure we're on the >>> same >>> > page.. I do not own oooq, I am not a core on oooq. >>> > I can help facilitate a discussion but oooq is an upstream tripleo >>> tool that >>> > replaces instack-virt-setup [1]. >>> > It also happens to be a great tool for easily deploying TripleO end >>> to end >>> > [3] >>> > >>> > What I *can* do is show everyone how to manipulate tripleo-quickstart >>> and >>> > customize it with composable ansible roles, templates, settings etc.. >>> > This would allow any upstream or downstream project to override the >>> native >>> > oooq roles and *any* step that does not work for another group w/ 3rd >>> party >>> > roles [2]. >>> > These 3rd party roles can be free and opensource or internal only, it >>> works >>> > either way. >>> > This was discussed in depth as part of the production chain >>> meetings, the >>> > message may have been lost unfortunately. >>> > >>> > I hope this resets your expectations of what I can and can not do as >>> part of >>> > these discussions. >>> > Let me know where and when and I'm happy to be part of the discussion. >>> >>> Thanks for clarifying :) >>> >>> Next reasonable step would probably be to propose some sort of >>> blueprint for tripleo-quickstart to include some of InfraRed features >>> and by that have one tool driven by upstream development that can be >>> either cloned downstream or used as it is with an internal data >>> project. >>> >> >> Sure.. a blueprint would help everyone understand the feature and the >> motivation. >> You could also just plug in the feature you are looking for to oooq and >> see if it meets >> your requirements. See below. >> > > While I think a blueprint is a good starting point, I'm afraid that our > approach for provisioning machines is completely different so I'm not sure > how to propose such a blueprint as it will probably require quite the > design change from today's approach. > > >> >> >>> >>> OR >>> >>> have InfraRed pushed into tripleo/openstack namespace and expose it to >>> the RDO community (without internal data of course). Personally, I >>> really like the pluggable[1] structure (which allows it to actually >>> consume tripleo-quickstart) so I'm not sure if it can be really merged >>> with tripleo-quickstart as proposed in the first option. >>> >> > I must admit that I like this option better as it introduces a tool to > upstream and let the community drive it further / get more feedback on how > to improve. > A second benefit might be that by introducing a new concept / design we > can take what is best from two worlds and improve. > I would love to see an open discussion on the tool upstream and how we > can improve the overall process. > > >> The way oooq is built one can plugin or override any part at run time >> with custom playbooks, roles, and config. There isn't anything that >> needs to be >> checked in directly to oooq to use it. >> >> It's designed such that third parties can make their own decisions to >> use something >> native to quickstart, something from our role library, or something >> completely independent. >> This allows teams, individuals or whom ever to do what they need to with >> out having to fork or re-roll the entire framework. >> >> The important step is to note that these 3rd party roles or >> (oooq-extras) incubate, mature and then graduate to github/openstack. >> The upstream openstack community should lead, evaluate, and via >> blueprints vote on the canonical CI tool set. >> >> We can record a demonstration if required, but there is nothing stopping >> anyone right now from >> doing this today. I'm just browsing the role library for an example, I >> had no idea [1] existed. >> Looks like Raoul had a requirement and just made it work. >> >> Justin, from the browbeat project has graciously created some >> documentation regarding 3rd party roles. >> It has yet to merge, but it should help illustrate how these roles are >> used. [2] >> >> Thanks Arie for leading the discussion. >> >> [1] >> https://github.com/redhat-openstack/ansible-role-tripleo-overcloud-valida... >> [2] https://review.openstack.org/#/c/346733/ >> >> >> >> >> >> >>> >>> I like the second option, although it still forces us to have two >>> tools, but after a period of time, I believe it will be clear what the >>> community prefers, which will allow us to remove one of the projects >>> eventually. >>> >>> So, unless there are other ideas, I think the next move should be made >>> by Tal. >>> >>> Tal, I'm willing to help with whatever is needed. >>> >> > Thanks Arie for starting this discussion again, I believe we have still > much work ahead of us but this is definitely a step in the right direction. > > >> >>> [1] http://infrared.readthedocs.io/en/latest >>> >>> > >>> > Thanks >>> > >>> > [1] https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart >>> > [2] >>> > >>> https://github.com/redhat-openstack/?utf8=%E2%9C%93&query=ansible-rol... >>> > [3[ https://www.rdoproject.org/tripleo/ >>> > >>> > >>> >> >>> >> >>> >> Raoul - I totally agree with you, especially with "difficult for >>> >> anyone to start contributing and collaborate". This is exactly why >>> >> this discussion started. If we can agree on one set of tools, it will >>> >> make everyone's life easier - current groups, new contributors, folks >>> >> that just want to deploy TripleO quickly. But I'm afraid some >>> >> sacrifices need to be made by both groups. >>> >> >>> >> David - I thought WeiRDO is used only for packstack, so I apologize I >>> >> didn't include it. It does sound like an anther testing project, is >>> >> there a place to merge it with another existing testing project? like >>> >> Octario for example or one of TripleO testing projects. Or does it >>> >> make sense to keep it a standalone project? >>> >> >>> >> >>> >> >>> >> >>> >> On Tue, Aug 2, 2016 at 11:12 AM, Christopher Brown < >>> cbrown2(a)ocf.co.uk&gt; >>> >> wrote: >>> >> > Hello RDOistas (I think that is the expression?), >>> >> > >>> >> > Another year, another OpenStack deployment tool. :) >>> >> > >>> >> > On Mon, 2016-08-01 at 18:59 +0100, Ignacio Bravo wrote: >>> >> >> If we are talking about tools, I would also want to add something >>> >> >> with regards to user interface of these tools. This is based on my >>> >> >> own experience: >>> >> >> >>> >> >> I started trying to deploy Openstack with Staypuft and The >>> Foreman. >>> >> >> The UI of The Foreman was intuitive enough for the discovery and >>> >> >> provisioning of the servers. The OpenStack portion, not so much. >>> >> > >>> >> > This is exactly mine also. I think this works really well in very >>> large >>> >> > enterprise environments where you need to split out services over >>> more >>> >> > than three controllers. You do need good in-house puppet skills >>> though >>> >> > so better for enterprise with a good sysadmin team. >>> >> > >>> >> >> Forward a couple of releases and we had a TripleO GUI (Tuskar, I >>> >> >> believe) that allowed you to graphically build your Openstack >>> cloud. >>> >> >> That was a reasonable good GUI for Openstack. >>> >> > >>> >> > Well, I found it barely usable. It was only ever good as a >>> graphical >>> >> > representiation of what the build was doing. Interacting with it >>> was >>> >> > not great. >>> >> > >>> >> >> Following that, TripleO become a script based installer, that >>> >> >> required experience in Heat templates. I know I didn’t have it and >>> >> >> had to ask in the mailing list about how to present this or change >>> >> >> that. I got a couple of installs working with this setup. >>> >> > >>> >> > Works well now that I understand all the foibles and have invested >>> time >>> >> > into understanding heat templates and puppet modules. Its good in >>> that >>> >> > it forces you to learn about orchestration which is such an >>> important >>> >> > end-goal of cloud environments. >>> >> > >>> >> >> In the last session in Austin, my goal was to obtain information >>> on >>> >> >> how others were installing Openstack. I was pointed to Fuel as an >>> >> >> alternative. I tried it up, and it just worked. It had the >>> >> >> discovering capability from The Foreman, and the configuration >>> >> >> options from TripleO. I understand that is based in Ansible and >>> >> >> because of that, it is not fully CentOS ready for all the nodes >>> (at >>> >> >> least not in version 9 that I tried). In any case, as a deployer >>> and >>> >> >> installer, it is the most well rounded tool that I found. >>> >> > >>> >> > This is interesting to know. I've heard of Fuel of course but >>> there are >>> >> > some politics involved - it still has the team:single-vendor tag >>> but >>> >> > from what I see Mirantis are very keen for it to become the default >>> >> > OpenStack installer. I don't think being Ansible-based should be a >>> >> > problem - we are deploying OpenShift on OpenStack which uses >>> Openshift- >>> >> > ansible - this recently moved to Ansible 2.1 without too much >>> >> > disruption. >>> >> > >>> >> >> I’d love to see RDO moving into that direction, and having an >>> easy to >>> >> >> use, end user ready deployer tool. >>> >> > >>> >> > If its as good as you say its definitely worth evaluating. From our >>> >> > point of view, we want to be able to add services to the pacemaker >>> >> > cluster with some ease - for example Magnum and Sahara - and it >>> looks >>> >> > like there are steps being taken with regards to composable roles >>> and >>> >> > simplification of the pacemaker cluster to just core services. >>> >> > >>> >> > But if someone can explain that better I would appreciate it. >>> >> > >>> >> > Regards >>> >> > >>> >> >> IB >>> >> >> >>> >> >> >>> >> >> __ >>> >> >> Ignacio Bravo >>> >> >> LTG Federal, Inc >>> >> >> www.ltgfederal.com >>> >> >> >>> >> >> >>> >> >> > On Aug 1, 2016, at 1:07 PM, David Moreau Simard <dms(a)redhat.com >>> > >>> >> >> > wrote: >>> >> >> > >>> >> >> > The vast majority of RDO's CI relies on using upstream >>> >> >> > installation/deployment projects in order to test installation >>> of >>> >> >> > RDO >>> >> >> > packages in different ways and configurations. >>> >> >> > >>> >> >> > Unless I'm mistaken, TripleO Quickstart was originally created >>> as a >>> >> >> > mean to "easily" install TripleO in different topologies without >>> >> >> > requiring a massive amount of hardware. >>> >> >> > This project allows us to test TripleO in virtual deployments on >>> >> >> > just >>> >> >> > one server instead of, say, 6. >>> >> >> > >>> >> >> > There's also WeIRDO [1] which was left out of your list. >>> >> >> > WeIRDO is super simple and simply aims to run upstream gate jobs >>> >> >> > (such >>> >> >> > as puppet-openstack-integration [2][3] and packstack [4][5]) >>> >> >> > outside >>> >> >> > of the gate. >>> >> >> > It'll install dependencies that are expected to be there (i.e, >>> >> >> > usually >>> >> >> > set up by the openstack-infra gate preparation jobs), set up the >>> >> >> > trunk >>> >> >> > repositories we're interested in testing and the rest is >>> handled by >>> >> >> > the upstream project testing framework. >>> >> >> > >>> >> >> > The WeIRDO project is /very/ low maintenance and brings an >>> >> >> > exceptional >>> >> >> > amount of coverage and value. >>> >> >> > This coverage is important because RDO provides OpenStack >>> packages >>> >> >> > or >>> >> >> > projects that are not necessarily used by TripleO and the >>> reality >>> >> >> > is >>> >> >> > that not everyone deploying OpenStack on CentOS with RDO will be >>> >> >> > using >>> >> >> > TripleO. >>> >> >> > >>> >> >> > Anyway, sorry for sidetracking but back to the topic, thanks for >>> >> >> > opening the discussion. >>> >> >> > >>> >> >> > What honestly perplexes me is the situation of CI in RDO and >>> OSP, >>> >> >> > especially around TripleO/Director, is the amount of work that >>> is >>> >> >> > spent downstream. >>> >> >> > And by downstream, here, I mean anything that isn't in TripleO >>> >> >> > proper. >>> >> >> > >>> >> >> > I keep dreaming about how awesome upstream TripleO CI would be >>> if >>> >> >> > all >>> >> >> > that effort was spent directly there instead -- and then that >>> all >>> >> >> > work >>> >> >> > could bear fruit and trickle down downstream for free. >>> >> >> > Exactly like how we keep improving the testing coverage in >>> >> >> > puppet-openstack-integration, it's automatically pulled in RDO >>> CI >>> >> >> > through WeIRDO for free. >>> >> >> > We make the upstream better and we benefit from it >>> simultaneously: >>> >> >> > everyone wins. >>> >> >> > >>> >> >> > [1]: https://github.com/rdo-infra/weirdo >>> >> >> > [2]: >>> https://github.com/rdo-infra/ansible-role-weirdo-puppet-openst >>> >> >> > ack >>> >> >> > [3]: >>> https://github.com/openstack/puppet-openstack-integration#desc >>> >> >> > ription >>> >> >> > [4]: https://github.com/rdo-infra/ansible-role-weirdo-packstack >>> >> >> > [5]: >>> https://github.com/openstack/packstack#packstack-integration-t >>> >> >> > ests >>> >> >> > >>> >> >> > David Moreau Simard >>> >> >> > Senior Software Engineer | Openstack RDO >>> >> >> > >>> >> >> > dmsimard = [irc, github, twitter] >>> >> >> > >>> >> >> > David Moreau Simard >>> >> >> > Senior Software Engineer | Openstack RDO >>> >> >> > >>> >> >> > dmsimard = [irc, github, twitter] >>> >> >> > >>> >> >> > >>> >> >> > On Mon, Aug 1, 2016 at 11:21 AM, Arie Bregman < >>> abregman(a)redhat.com&gt; >>> >> >> > wrote: >>> >> >> > > Hi, >>> >> >> > > >>> >> >> > > I would like to start a discussion on the overlap between >>> tools >>> >> >> > > we >>> >> >> > > have for deploying and testing TripleO (RDO & RHOSP) in CI. >>> >> >> > > >>> >> >> > > Several months ago, we worked on one common framework for >>> >> >> > > deploying >>> >> >> > > and testing OpenStack (RDO & RHOSP) in CI. I think you can >>> say it >>> >> >> > > didn't work out well, which eventually led each group to >>> focus on >>> >> >> > > developing other existing/new tools. >>> >> >> > > >>> >> >> > > What we have right now for deploying and testing >>> >> >> > > -------------------------------------------------------- >>> >> >> > > === Component CI, Gating === >>> >> >> > > I'll start with the projects we created, I think that's only >>> fair >>> >> >> > > :) >>> >> >> > > >>> >> >> > > * Ansible-OVB[1] - Provisioning Tripleo heat stack, using the >>> OVB >>> >> >> > > project. >>> >> >> > > >>> >> >> > > * Ansible-RHOSP[2] - Product installation (RHOSP). Branch per >>> >> >> > > release. >>> >> >> > > >>> >> >> > > * Octario[3] - Testing using RPMs (pep8, unit, functional, >>> >> >> > > tempest, >>> >> >> > > csit) + Patching RPMs with submitted code. >>> >> >> > > >>> >> >> > > === Automation, QE === >>> >> >> > > * InfraRed[4] - provision install and test. Pluggable and >>> >> >> > > modular, >>> >> >> > > allows you to create your own provisioner, installer and >>> tester. >>> >> >> > > >>> >> >> > > As far as I know, the groups is working now on different >>> >> >> > > structure of >>> >> >> > > one main project and three sub projects (provision, install >>> and >>> >> >> > > test). >>> >> >> > > >>> >> >> > > === RDO === >>> >> >> > > I didn't use RDO tools, so I apologize if I got something >>> wrong: >>> >> >> > > >>> >> >> > > * About ~25 micro independent Ansible roles[5]. You can either >>> >> >> > > choose >>> >> >> > > to use one of them or several together. They are used for >>> >> >> > > provisioning, installing and testing Tripleo. >>> >> >> > > >>> >> >> > > * Tripleo-quickstart[6] - uses the micro roles for deploying >>> >> >> > > tripleo >>> >> >> > > and test it. >>> >> >> > > >>> >> >> > > As I said, I didn't use the tools, so feel free to add more >>> >> >> > > information you think is relevant. >>> >> >> > > >>> >> >> > > === More? === >>> >> >> > > I hope not. Let us know if are familiar with more tools. >>> >> >> > > >>> >> >> > > Conclusion >>> >> >> > > -------------- >>> >> >> > > So as you can see, there are several projects that eventually >>> >> >> > > overlap >>> >> >> > > in many areas. Each group is basically using the same tasks >>> >> >> > > (provision >>> >> >> > > resources, build/import overcloud images, run tempest, collect >>> >> >> > > logs, >>> >> >> > > etc.) >>> >> >> > > >>> >> >> > > Personally, I think it's a waste of resources. For each task >>> >> >> > > there is >>> >> >> > > at least two people from different groups who work on exactly >>> the >>> >> >> > > same >>> >> >> > > task. The most recent example I can give is OVB. As far as I >>> >> >> > > know, >>> >> >> > > both groups are working on implementing it in their set of >>> tools >>> >> >> > > right >>> >> >> > > now. >>> >> >> > > >>> >> >> > > On the other hand, you can always claim: "we already tried to >>> >> >> > > work on >>> >> >> > > the same framework, we failed to do it successfully" - right, >>> but >>> >> >> > > maybe with better ground rules we can manage it. We would >>> >> >> > > defiantly >>> >> >> > > benefit a lot from doing that. >>> >> >> > > >>> >> >> > > What's next? >>> >> >> > > ---------------- >>> >> >> > > So first of all, I would like to hear from you if you think >>> that >>> >> >> > > we >>> >> >> > > can collaborate once again or is it actually better to keep >>> it as >>> >> >> > > it >>> >> >> > > is now. >>> >> >> > > >>> >> >> > > If you agree that collaboration here makes sense, maybe you >>> have >>> >> >> > > ideas >>> >> >> > > on how we can do it better this time. >>> >> >> > > >>> >> >> > > I think that setting up a meeting to discuss the right >>> >> >> > > architecture >>> >> >> > > for the project(s) and decide on good review/gating process, >>> >> >> > > would be >>> >> >> > > a good start. >>> >> >> > > >>> >> >> > > Please let me know what do you think and keep in mind that >>> this >>> >> >> > > is not >>> >> >> > > about which tool is better!. As you can see I didn't mention >>> the >>> >> >> > > time >>> >> >> > > it takes for each tool to deploy and test, and also not the >>> full >>> >> >> > > feature list it supports. >>> >> >> > > If possible, we should keep it about collaborating and not >>> >> >> > > choosing >>> >> >> > > the best tool. Our solution could be the combination of two or >>> >> >> > > more >>> >> >> > > tools eventually (tripleo-red, infra-quickstart? :D ) >>> >> >> > > >>> >> >> > > "You may say I'm a dreamer, but I'm not the only one. I hope >>> some >>> >> >> > > day >>> >> >> > > you'll join us and the infra will be as one" :) >>> >> >> > > >>> >> >> > > [1] https://github.com/redhat-openstack/ansible-ovb >>> >> >> > > [2] https://github.com/redhat-openstack/ansible-rhosp >>> >> >> > > [3] https://github.com/redhat-openstack/octario >>> >> >> > > [4] https://github.com/rhosqeauto/InfraRed >>> >> >> > > [5] >>> https://github.com/redhat-openstack?utf8=%E2%9C%93&query=ansi >>> >> >> > > ble-role >>> >> >> > > [6] https://github.com/openstack/tripleo-quickstart >>> >> >> > > >>> >> >> > > _______________________________________________ >>> >> >> > > rdo-list mailing list >>> >> >> > > rdo-list(a)redhat.com >>> >> >> > > https://www.redhat.com/mailman/listinfo/rdo-list >>> >> >> > > >>> >> >> > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >> > _______________________________________________ >>> >> >> > rdo-list mailing list >>> >> >> > rdo-list(a)redhat.com >>> >> >> > https://www.redhat.com/mailman/listinfo/rdo-list >>> >> >> > >>> >> >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >> >>> >> > -- >>> >> > Regards, >>> >> > >>> >> > Christopher Brown >>> >> > OpenStack Engineer >>> >> > OCF plc >>> >> > >>> >> > Tel: +44 (0)114 257 2200 >>> >> > Web: www.ocf.co.uk >>> >> > Blog: blog.ocf.co.uk >>> >> > Twitter: @ocfplc >>> >> > >>> >> > Please note, any emails relating to an OCF Support request must >>> always >>> >> > be sent to support(a)ocf.co.uk for a ticket number to be generated >>> or >>> >> > existing support ticket to be updated. Should this not be done >>> then OCF >>> >> > cannot be held responsible for requests not dealt with in a timely >>> >> > manner. >>> >> > >>> >> > OCF plc is a company registered in England and Wales. Registered >>> number >>> >> > 4132533, VAT number GB 780 6803 14. Registered office address: OCF >>> plc, >>> >> > 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield >>> S35 >>> >> > 2PG. >>> >> > >>> >> > This message is private and confidential. If you have received this >>> >> > message in error, please notify us immediately and remove it from >>> your >>> >> > system. >>> >> > >>> >> > _______________________________________________ >>> >> > rdo-list mailing list >>> >> > rdo-list(a)redhat.com >>> >> > https://www.redhat.com/mailman/listinfo/rdo-list >>> >> > >>> >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >>> >> >>> >> >>> >> -- >>> >> Arie Bregman >>> >> Red Hat Israel >>> >> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview >>> > >>> > >>> >>> >>> >>> -- >>> Arie Bregman >>> Red Hat Israel >>> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview >>> >> >> > > > -- > Tal Kammer > Associate manager, automation and infrastracture, Openstack platform. > Red Hat Israel > Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659 > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > -- {Kind regards | Mit besten Grüßen}, Frank ________________________________ Frank Zdarsky | NFV Partner Engineering | Office of Technology | Red Hat e: fzdarsky(a)redhat.com | irc: fzdarsky @freenode | m: +49 175 82 11 64 4 | t: +49 711 96 43 70 02