[rdo-list] Multiple tools for deploying and testing TripleO

Wed Aug 3 07:28:54 UTC 2016

Thanks Arie for starting this discussion! (and sorry for joining in late)

Some detailed explanations for those not familiar with InfraRed and would
like to get the highlights:  (Feel free to skip to my inline comments if
you are familiar / too long of a post :) ).

The InfraRed project is an Ansible based project comprised from three
distinct tools: (currently under the InfraRed[1] project, and being split
into their own standalone project soon).

1. ir-provisioner - responsible for building up a custom environment - you
can affect the memory, CPU,  HDD size, number of HDD each node has + number
of networks (with/without DHCP) so for example one can deploy the following
topology: (only an example to show the versatile options)
1 undercloud on RHEL 7 with 16GB of ram + 60GB of HDD with 3 network
interfaces.
3 controllers on Centos 7 with 8GB of ram  + 40GB of HDD
2 compute on Centos 7 with 6GB of ram + 60GB HDD
3 ceph nodes on RHEL 7 with 4GB of ram + 2 HDD one with 20GB + one with 40GB

Example usage: (setting up the above VMs with four different HW specs)
ir-provisioner virsh --host-address=<host.address>
--host-key=<host.private.key>
--topology-nodes=undercloud:1,controller:3,compute:2,ceph:3

*Note: while it is written "controller/compute/ceph" this is just setting
up VMs, the names act more as a reference to the user of what is the role
of each node.
The installation of Openstack is done with a dedicated tool called
`ir-installer` (next)

2. ir-installer - responsible for installing the product - supports
"quickstart" mode (setting up a working environment in ~30 minutes) or E2E
mode which does a full installation in ~1h.
The installation process is completely customized. You can supply your own
heat templates / overcloud invocation / undercloud.conf file to use / etc.
You can also just run a specific task (using ansible --tags), so if you
have a deployment ready and just need to run say, the introspection phase,
you can fully choose what to run and what to skip even.

3. ir-tester - responsible for installing / configuring / running the tests
- this project is meant to hold all testing tools we use so a user will be
able to run any testing utility he would like without the need to "dive
in". we supply a simple interface requesting simple to choose the testing
tool and the set of tests one wishes to run and we'll do the work for him :)

More info about InfraRed can be found here[1] (though I must admit that we
still need some "love" around our docs)

[1] - http://infrared.readthedocs.io/en/latest/

On Tue, Aug 2, 2016 at 9:52 PM, Wesley Hayutin <whayutin at redhat.com> wrote:

>
>
> On Tue, Aug 2, 2016 at 1:51 PM, Arie Bregman <abregman at redhat.com> wrote:
>
>> On Tue, Aug 2, 2016 at 3:53 PM, Wesley Hayutin <whayutin at redhat.com>
>> wrote:
>> >
>> >
>> > On Tue, Aug 2, 2016 at 4:58 AM, Arie Bregman <abregman at redhat.com>
>> wrote:
>> >>
>> >> It became a discussion around the official installer and how to
>> >> improve it. While it's an important discussion, no doubt, I actually
>> >> want to focus on our automation and CI tools.
>> >>
>> >> Since I see there is an agreement that collaboration does make sense
>> >> here, let's move to the hard questions :)
>> >>
>> >> Wes, Tal - there is huge difference right now between infrared and
>> >> tripleo-quickstart in their structure. One is all-in-one project and
>> >> the other one is multiple micro projects managed by one project. Do
>> >> you think there is a way to consolidate or move to a different model
>> >> which will make sense for both RDO and RHOSP? something that both
>> >> groups can work on.
>> >
>> >
>> > I am happy to be part of the discussion, and I am also very willing to
>> help
>> > and try to drive suggestions to the tripleo-quickstart community.
>> > I need to make a point clear though, just to make sure we're on the same
>> > page..  I do not own oooq, I am not a core on oooq.
>> > I can help facilitate a discussion but oooq is an upstream tripleo tool
>> that
>> > replaces instack-virt-setup [1].
>> > It also happens to be a great tool for easily deploying TripleO end to
>> end
>> > [3]
>> >
>> > What I *can* do is show everyone how to manipulate tripleo-quickstart
>> and
>> > customize it with composable ansible roles, templates, settings etc..
>> > This would allow any upstream or downstream project to override the
>> native
>> > oooq roles and *any* step that does not work for another group w/ 3rd
>> party
>> > roles [2].
>> > These 3rd party roles can be free and opensource or internal only, it
>> works
>> > either way.
>> > This was discussed in depth as part of the production chain meetings,
>> the
>> > message may have been lost unfortunately.
>> >
>> > I hope this resets your expectations of what I can and can not do as
>> part of
>> > these discussions.
>> > Let me know where and when and I'm happy to be part of the discussion.
>>
>> Thanks for clarifying :)
>>
>> Next reasonable step would probably be to propose some sort of
>> blueprint for tripleo-quickstart to include some of InfraRed features
>> and by that have one tool driven by upstream development that can be
>> either cloned downstream or used as it is with an internal data
>> project.
>>
>
> Sure.. a blueprint would help everyone understand the feature and the
> motivation.
> You could also just plug in the feature you are looking for to oooq and
> see if it meets
> your requirements.  See below.
>

While I think a blueprint is a good starting point, I'm afraid that our
approach for provisioning machines is completely different so I'm not sure
how to propose such a blueprint as it will probably require quite the
design change from today's approach.

>
>
>>
>> OR
>>
>> have InfraRed pushed into tripleo/openstack namespace and expose it to
>> the RDO community (without internal data of course). Personally, I
>> really like the pluggable[1] structure (which allows it to actually
>> consume tripleo-quickstart) so I'm not sure if it can be really merged
>> with tripleo-quickstart as proposed in the first option.
>>
>
I must admit that I like this option better as it introduces a tool to
upstream and let the community drive it further / get more feedback on how
to improve.
A second benefit might be that by introducing a new concept / design we can
take what is best from two worlds and improve.
I would love to see an open discussion on the tool upstream and how we can
improve the overall process.

> The way oooq is built one can plugin or override any part at run time
> with custom playbooks, roles, and config.  There isn't anything that needs
> to be
> checked in directly to oooq to use it.
>
> It's designed such that third parties can make their own decisions to use
> something
> native to quickstart, something from our role library, or something
> completely independent.
> This allows teams, individuals or whom ever to do what they need to with
> out having to fork or re-roll the entire framework.
>
> The important step is to note that these 3rd party roles or (oooq-extras)
> incubate, mature and then graduate to github/openstack.
> The upstream openstack community should lead, evaluate, and via blueprints
> vote on the canonical CI tool set.
>
> We can record a demonstration if required, but there is nothing stopping
> anyone right now from
> doing this today.  I'm just browsing the role library for an example, I
> had no idea [1] existed.
> Looks like Raoul had a requirement and just made it work.
>
> Justin, from the browbeat project has graciously created some
> documentation regarding 3rd party roles.
> It has yet to merge, but it should help illustrate how these roles are
> used. [2]
>
> Thanks Arie for leading the discussion.
>
> [1]
> https://github.com/redhat-openstack/ansible-role-tripleo-overcloud-validate-ha
> [2] https://review.openstack.org/#/c/346733/
>
>
>
>
>
>
>>
>> I like the second option, although it still forces us to have two
>> tools, but after a period of time, I believe it will be clear what the
>> community prefers, which will allow us to remove one of the projects
>> eventually.
>>
>> So, unless there are other ideas, I think the next move should be made by
>> Tal.
>>
>> Tal, I'm willing to help with whatever is needed.
>>
>
Thanks Arie for starting this discussion again, I believe we have still
much work ahead of us but this is definitely a step in the right direction.

>
>> [1] http://infrared.readthedocs.io/en/latest
>>
>> >
>> > Thanks
>> >
>> > [1] https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart
>> > [2]
>> >
>> https://github.com/redhat-openstack/?utf8=%E2%9C%93&query=ansible-role-tripleo
>> > [3[ https://www.rdoproject.org/tripleo/
>> >
>> >
>> >>
>> >>
>> >> Raoul - I totally agree with you, especially with "difficult for
>> >> anyone to start contributing and collaborate". This is exactly why
>> >> this discussion started. If we can agree on one set of tools, it will
>> >> make everyone's life easier - current groups, new contributors, folks
>> >> that just want to deploy TripleO quickly. But I'm afraid some
>> >> sacrifices need to be made by both groups.
>> >>
>> >> David - I thought WeiRDO is used only for packstack, so I apologize I
>> >> didn't include it. It does sound like an anther testing project, is
>> >> there a place to merge it with another existing testing project? like
>> >> Octario for example or one of TripleO testing projects. Or does it
>> >> make sense to keep it a standalone project?
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Aug 2, 2016 at 11:12 AM, Christopher Brown <cbrown2 at ocf.co.uk>
>> >> wrote:
>> >> > Hello RDOistas (I think that is the expression?),
>> >> >
>> >> > Another year, another OpenStack deployment tool. :)
>> >> >
>> >> > On Mon, 2016-08-01 at 18:59 +0100, Ignacio Bravo wrote:
>> >> >> If we are talking about tools, I would also want to add something
>> >> >> with regards to user interface of these tools. This is based on my
>> >> >> own experience:
>> >> >>
>> >> >> I started trying to deploy Openstack with Staypuft and The Foreman.
>> >> >> The UI of The Foreman was intuitive enough for the discovery and
>> >> >> provisioning of the servers. The OpenStack portion, not so much.
>> >> >
>> >> > This is exactly mine also. I think this works really well in very
>> large
>> >> > enterprise environments where you need to split out services over
>> more
>> >> > than three controllers. You do need good in-house puppet skills
>> though
>> >> > so better for enterprise with a good sysadmin team.
>> >> >
>> >> >> Forward a couple of releases and we had a TripleO GUI (Tuskar, I
>> >> >> believe) that allowed you to graphically build your Openstack cloud.
>> >> >> That was a reasonable good GUI for Openstack.
>> >> >
>> >> > Well, I found it barely usable. It was only ever good as a graphical
>> >> > representiation of what the build was doing. Interacting with it was
>> >> > not great.
>> >> >
>> >> >> Following that, TripleO become a script based installer, that
>> >> >> required experience in Heat templates. I know I didn’t have it and
>> >> >> had to ask in the mailing list about how to present this or change
>> >> >> that. I got a couple of installs working with this setup.
>> >> >
>> >> > Works well now that I understand all the foibles and have invested
>> time
>> >> > into understanding heat templates and puppet modules. Its good in
>> that
>> >> > it forces you to learn about orchestration which is such an important
>> >> > end-goal of cloud environments.
>> >> >
>> >> >> In the last session in Austin, my goal was to obtain information on
>> >> >> how others were installing Openstack. I was pointed to Fuel as an
>> >> >> alternative. I tried it up, and it just worked. It had the
>> >> >> discovering capability from The Foreman, and the configuration
>> >> >> options from TripleO. I understand that is based in Ansible and
>> >> >> because of that, it is not fully CentOS ready for all the nodes (at
>> >> >> least not in version 9 that I tried). In any case, as a deployer and
>> >> >> installer, it is the most well rounded tool that I found.
>> >> >
>> >> > This is interesting to know. I've heard of Fuel of course but there
>> are
>> >> > some politics involved - it still has the team:single-vendor tag but
>> >> > from what I see Mirantis are very keen for it to become the default
>> >> > OpenStack installer. I don't think being Ansible-based should be a
>> >> > problem - we are deploying OpenShift on OpenStack which uses
>> Openshift-
>> >> > ansible - this recently moved to Ansible 2.1 without too much
>> >> > disruption.
>> >> >
>> >> >> I’d love to see RDO moving into that direction, and having an easy
>> to
>> >> >> use, end user ready deployer tool.
>> >> >
>> >> > If its as good as you say its definitely worth evaluating. From our
>> >> > point of view, we want to be able to add services to the pacemaker
>> >> > cluster with some ease - for example Magnum and Sahara - and it looks
>> >> > like there are steps being taken with regards to composable roles and
>> >> > simplification of the pacemaker cluster to just core services.
>> >> >
>> >> > But if someone can explain that better I would appreciate it.
>> >> >
>> >> > Regards
>> >> >
>> >> >> IB
>> >> >>
>> >> >>
>> >> >> __
>> >> >> Ignacio Bravo
>> >> >> LTG Federal, Inc
>> >> >> www.ltgfederal.com
>> >> >>
>> >> >>
>> >> >> > On Aug 1, 2016, at 1:07 PM, David Moreau Simard <dms at redhat.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> > The vast majority of RDO's CI relies on using upstream
>> >> >> > installation/deployment projects in order to test installation of
>> >> >> > RDO
>> >> >> > packages in different ways and configurations.
>> >> >> >
>> >> >> > Unless I'm mistaken, TripleO Quickstart was originally created as
>> a
>> >> >> > mean to "easily" install TripleO in different topologies without
>> >> >> > requiring a massive amount of hardware.
>> >> >> > This project allows us to test TripleO in virtual deployments on
>> >> >> > just
>> >> >> > one server instead of, say, 6.
>> >> >> >
>> >> >> > There's also WeIRDO [1] which was left out of your list.
>> >> >> > WeIRDO is super simple and simply aims to run upstream gate jobs
>> >> >> > (such
>> >> >> > as puppet-openstack-integration [2][3] and packstack [4][5])
>> >> >> > outside
>> >> >> > of the gate.
>> >> >> > It'll install dependencies that are expected to be there (i.e,
>> >> >> > usually
>> >> >> > set up by the openstack-infra gate preparation jobs), set up the
>> >> >> > trunk
>> >> >> > repositories we're interested in testing and the rest is handled
>> by
>> >> >> > the upstream project testing framework.
>> >> >> >
>> >> >> > The WeIRDO project is /very/ low maintenance and brings an
>> >> >> > exceptional
>> >> >> > amount of coverage and value.
>> >> >> > This coverage is important because RDO provides OpenStack packages
>> >> >> > or
>> >> >> > projects that are not necessarily used by TripleO and the reality
>> >> >> > is
>> >> >> > that not everyone deploying OpenStack on CentOS with RDO will be
>> >> >> > using
>> >> >> > TripleO.
>> >> >> >
>> >> >> > Anyway, sorry for sidetracking but back to the topic, thanks for
>> >> >> > opening the discussion.
>> >> >> >
>> >> >> > What honestly perplexes me is the situation of CI in RDO and OSP,
>> >> >> > especially around TripleO/Director, is the amount of work that is
>> >> >> > spent downstream.
>> >> >> > And by downstream, here, I mean anything that isn't in TripleO
>> >> >> > proper.
>> >> >> >
>> >> >> > I keep dreaming about how awesome upstream TripleO CI would be if
>> >> >> > all
>> >> >> > that effort was spent directly there instead -- and then that all
>> >> >> > work
>> >> >> > could bear fruit and trickle down downstream for free.
>> >> >> > Exactly like how we keep improving the testing coverage in
>> >> >> > puppet-openstack-integration, it's automatically pulled in RDO CI
>> >> >> > through WeIRDO for free.
>> >> >> > We make the upstream better and we benefit from it simultaneously:
>> >> >> > everyone wins.
>> >> >> >
>> >> >> > [1]: https://github.com/rdo-infra/weirdo
>> >> >> > [2]:
>> https://github.com/rdo-infra/ansible-role-weirdo-puppet-openst
>> >> >> > ack
>> >> >> > [3]:
>> https://github.com/openstack/puppet-openstack-integration#desc
>> >> >> > ription
>> >> >> > [4]: https://github.com/rdo-infra/ansible-role-weirdo-packstack
>> >> >> > [5]:
>> https://github.com/openstack/packstack#packstack-integration-t
>> >> >> > ests
>> >> >> >
>> >> >> > David Moreau Simard
>> >> >> > Senior Software Engineer | Openstack RDO
>> >> >> >
>> >> >> > dmsimard = [irc, github, twitter]
>> >> >> >
>> >> >> > David Moreau Simard
>> >> >> > Senior Software Engineer | Openstack RDO
>> >> >> >
>> >> >> > dmsimard = [irc, github, twitter]
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Aug 1, 2016 at 11:21 AM, Arie Bregman <
>> abregman at redhat.com>
>> >> >> > wrote:
>> >> >> > > Hi,
>> >> >> > >
>> >> >> > > I would like to start a discussion on the overlap between tools
>> >> >> > > we
>> >> >> > > have for deploying and testing TripleO (RDO & RHOSP) in CI.
>> >> >> > >
>> >> >> > > Several months ago, we worked on one common framework for
>> >> >> > > deploying
>> >> >> > > and testing OpenStack (RDO & RHOSP) in CI. I think you can say
>> it
>> >> >> > > didn't work out well, which eventually led each group to focus
>> on
>> >> >> > > developing other existing/new tools.
>> >> >> > >
>> >> >> > > What we have right now for deploying and testing
>> >> >> > > --------------------------------------------------------
>> >> >> > > === Component CI, Gating ===
>> >> >> > > I'll start with the projects we created, I think that's only
>> fair
>> >> >> > > :)
>> >> >> > >
>> >> >> > > * Ansible-OVB[1] - Provisioning Tripleo heat stack, using the
>> OVB
>> >> >> > > project.
>> >> >> > >
>> >> >> > > * Ansible-RHOSP[2] - Product installation (RHOSP). Branch per
>> >> >> > > release.
>> >> >> > >
>> >> >> > > * Octario[3] - Testing using RPMs (pep8, unit, functional,
>> >> >> > > tempest,
>> >> >> > > csit) + Patching RPMs with submitted code.
>> >> >> > >
>> >> >> > > === Automation, QE ===
>> >> >> > > * InfraRed[4] - provision install and test. Pluggable and
>> >> >> > > modular,
>> >> >> > > allows you to create your own provisioner, installer and tester.
>> >> >> > >
>> >> >> > > As far as I know, the groups is working now on different
>> >> >> > > structure of
>> >> >> > > one main project and three sub projects (provision, install and
>> >> >> > > test).
>> >> >> > >
>> >> >> > > === RDO ===
>> >> >> > > I didn't use RDO tools, so I apologize if I got something wrong:
>> >> >> > >
>> >> >> > > * About ~25 micro independent Ansible roles[5]. You can either
>> >> >> > > choose
>> >> >> > > to use one of them or several together. They are used for
>> >> >> > > provisioning, installing and testing Tripleo.
>> >> >> > >
>> >> >> > > * Tripleo-quickstart[6] - uses the micro roles for deploying
>> >> >> > > tripleo
>> >> >> > > and test it.
>> >> >> > >
>> >> >> > > As I said, I didn't use the tools, so feel free to add more
>> >> >> > > information you think is relevant.
>> >> >> > >
>> >> >> > > === More? ===
>> >> >> > > I hope not. Let us know if are familiar with more tools.
>> >> >> > >
>> >> >> > > Conclusion
>> >> >> > > --------------
>> >> >> > > So as you can see, there are several projects that eventually
>> >> >> > > overlap
>> >> >> > > in many areas. Each group is basically using the same tasks
>> >> >> > > (provision
>> >> >> > > resources, build/import overcloud images, run tempest, collect
>> >> >> > > logs,
>> >> >> > > etc.)
>> >> >> > >
>> >> >> > > Personally, I think it's a waste of resources. For each task
>> >> >> > > there is
>> >> >> > > at least two people from different groups who work on exactly
>> the
>> >> >> > > same
>> >> >> > > task. The most recent example I can give is OVB. As far as I
>> >> >> > > know,
>> >> >> > > both groups are working on implementing it in their set of tools
>> >> >> > > right
>> >> >> > > now.
>> >> >> > >
>> >> >> > > On the other hand, you can always claim: "we already tried to
>> >> >> > > work on
>> >> >> > > the same framework, we failed to do it successfully" - right,
>> but
>> >> >> > > maybe with better ground rules we can manage it. We would
>> >> >> > > defiantly
>> >> >> > > benefit a lot from doing that.
>> >> >> > >
>> >> >> > > What's next?
>> >> >> > > ----------------
>> >> >> > > So first of all, I would like to hear from you if you think that
>> >> >> > > we
>> >> >> > > can collaborate once again or is it actually better to keep it
>> as
>> >> >> > > it
>> >> >> > > is now.
>> >> >> > >
>> >> >> > > If you agree that collaboration here makes sense, maybe you have
>> >> >> > > ideas
>> >> >> > > on how we can do it better this time.
>> >> >> > >
>> >> >> > > I think that setting up a meeting to discuss the right
>> >> >> > > architecture
>> >> >> > > for the project(s) and decide on good review/gating process,
>> >> >> > > would be
>> >> >> > > a good start.
>> >> >> > >
>> >> >> > > Please let me know what do you think and keep in mind that this
>> >> >> > > is not
>> >> >> > > about which tool is better!. As you can see I didn't mention the
>> >> >> > > time
>> >> >> > > it takes for each tool to deploy and test, and also not the full
>> >> >> > > feature list it supports.
>> >> >> > > If possible, we should keep it about collaborating and not
>> >> >> > > choosing
>> >> >> > > the best tool. Our solution could be the combination of two or
>> >> >> > > more
>> >> >> > > tools eventually (tripleo-red, infra-quickstart? :D )
>> >> >> > >
>> >> >> > > "You may say I'm a dreamer, but I'm not the only one. I hope
>> some
>> >> >> > > day
>> >> >> > > you'll join us and the infra will be as one" :)
>> >> >> > >
>> >> >> > > [1] https://github.com/redhat-openstack/ansible-ovb
>> >> >> > > [2] https://github.com/redhat-openstack/ansible-rhosp
>> >> >> > > [3] https://github.com/redhat-openstack/octario
>> >> >> > > [4] https://github.com/rhosqeauto/InfraRed
>> >> >> > > [5]
>> https://github.com/redhat-openstack?utf8=%E2%9C%93&query=ansi
>> >> >> > > ble-role
>> >> >> > > [6] https://github.com/openstack/tripleo-quickstart
>> >> >> > >
>> >> >> > > _______________________________________________
>> >> >> > > rdo-list mailing list
>> >> >> > > rdo-list at redhat.com
>> >> >> > > https://www.redhat.com/mailman/listinfo/rdo-list
>> >> >> > >
>> >> >> > > To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >> >> > _______________________________________________
>> >> >> > rdo-list mailing list
>> >> >> > rdo-list at redhat.com
>> >> >> > https://www.redhat.com/mailman/listinfo/rdo-list
>> >> >> >
>> >> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >> >>
>> >> > --
>> >> > Regards,
>> >> >
>> >> > Christopher Brown
>> >> > OpenStack Engineer
>> >> > OCF plc
>> >> >
>> >> > Tel: +44 (0)114 257 2200
>> >> > Web: www.ocf.co.uk
>> >> > Blog: blog.ocf.co.uk
>> >> > Twitter: @ocfplc
>> >> >
>> >> > Please note, any emails relating to an OCF Support request must
>> always
>> >> > be sent to support at ocf.co.uk for a ticket number to be generated or
>> >> > existing support ticket to be updated. Should this not be done then
>> OCF
>> >> > cannot be held responsible for requests not dealt with in a timely
>> >> > manner.
>> >> >
>> >> > OCF plc is a company registered in England and Wales. Registered
>> number
>> >> > 4132533, VAT number GB 780 6803 14. Registered office address: OCF
>> plc,
>> >> > 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield
>> S35
>> >> > 2PG.
>> >> >
>> >> > This message is private and confidential. If you have received this
>> >> > message in error, please notify us immediately and remove it from
>> your
>> >> > system.
>> >> >
>> >> > _______________________________________________
>> >> > rdo-list mailing list
>> >> > rdo-list at redhat.com
>> >> > https://www.redhat.com/mailman/listinfo/rdo-list
>> >> >
>> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >>
>> >>
>> >>
>> >> --
>> >> Arie Bregman
>> >> Red Hat Israel
>> >> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview
>> >
>> >
>>
>>
>>
>> --
>> Arie Bregman
>> Red Hat Israel
>> Component CI: https://mojo.redhat.com/groups/rhos-core-ci/overview
>>
>
>

-- 
Tal Kammer
Associate manager, automation and infrastracture, Openstack platform.
Red Hat Israel
Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20160803/338a47e2/attachment.html>