9:15 a.m.
Hi,
TripleO CI team had a design session yesterday on the tasks for the
current sprint, which is addressing infra.
There are some open questions left, because we had important roles in
the team unavailable, and because we had to create and reprioritize some
of the cards yesterday, due to design decisions.
I think the questions involve the rdo infra people, especially the
second. But in general, I'd like to understand what level of
integration we want between the two groups regarding all the things
related to infrastructure servers.
1) We spoke about bastion host. We forgot to create a card for it during
planning, and yesterday we failed to agree on the scope of the task.
Bastion host is a precise security element, with certain rules which have
to be respected for it to be useful. The bastion host is the single
access point to all the other servers on the infrastructure, it's the
only one possessing a public ip, and that means that *EVERYTHING* needs
to get through it: logs exposure, web pages access (sova included), ssh
access. Implementing it will take more time, also because we have to
understand for example if sova will work behind a reverse proxy
without modifications.
The card tracking the bastion is here
https://trello.com/c/xuRVmCDr/622-setup-teardown-tooling-bastion-host
So the question is: is the bastion host in scope, or we have something
different in mind when we talked about it ?
We also more or less agreed that the bastion host will be present only
on the infra tenant, not on the nodepool tenant, in which it would
serve to bastion only the te-broker. So the te-broker will be
completely exposed, and it will have a public ip
2) We understood that we need a common set of roles that will set up
common aspect of every server in infra: credentials, networks,
continuous deployment setup. The role will store a file with keys for
example, and this file will be used to setup all the other servers (or
only the bastion host)
This task is tracked here, but we had to reprioritize the card, as this
is the common ground to all the other "server setup" cards.
https://trello.com/c/BxUTetSu/612-tooling-to-setup-networks-keys-etc-in-o...
While beginning working on this card, yesterday I asked a question to
dmsimard, and it showed me the common roles that are already in place to
setup RDO infra servers. These common roles cover 70-80% of this card
already.
The question is: what level of integration we want to achieve between
our two groups ? Can we modify these roles to be a bit more generic so
they can be used in setting up our infrastructure too ? Can we for
example modify the log server to accept logs from remote journald in our
servers, so we can already use the log server to expose our promotion
logs ?
I would certainly hope so. We already agree that rdo infra team needs to
have access to our server in case of need, it makes only sense that we
use something they are already familiar with.
My implementation will be probably very very very similar to what I see
in rdo-infra repository anyway.
Thanks for any feedback
11:11 a.m.
On Tue, Mar 6, 2018 at 10:15 AM, Gabriele Cerami <gcerami(a)redhat.com> wrote:
Hi,
TripleO CI team had a design session yesterday on the tasks for the
current sprint, which is addressing infra.
There are some open questions left, because we had important roles in
the team unavailable, and because we had to create and reprioritize some
of the cards yesterday, due to design decisions.
I think the questions involve the rdo infra people, especially the
second. But in general, I'd like to understand what level of
integration we want between the two groups regarding all the things
related to infrastructure servers.
1) We spoke about bastion host. We forgot to create a card for it during
planning, and yesterday we failed to agree on the scope of the task.
Bastion host is a precise security element, with certain rules which have
to be respected for it to be useful. The bastion host is the single
access point to all the other servers on the infrastructure, it's the
only one possessing a public ip, and that means that *EVERYTHING* needs
It would make more sense to me to have any of the dashboard or web servers
have a public ip but everything else locked down, even ssh if needed.
I don't think we should try to run webservices via a reverse proxy on that
bastion.
The bastion *should* be used for instances that we *do* need to log into.
to get through it: logs exposure, web pages access (sova included),
ssh
access. Implementing it will take more time, also because we have to
understand for example if sova will work behind a reverse proxy
without modifications.
The card tracking the bastion is here
https://trello.com/c/xuRVmCDr/622-setup-teardown-tooling-bastion-host
So the question is: is the bastion host in scope, or we have something
different in mind when we talked about it ?
We also more or less agreed that the bastion host will be present only
on the infra tenant, not on the nodepool tenant, in which it would
serve to bastion only the te-broker. So the te-broker will be
completely exposed, and it will have a public ip
2) We understood that we need a common set of roles that will set up
common aspect of every server in infra: credentials, networks,
continuous deployment setup. The role will store a file with keys for
example, and this file will be used to setup all the other servers (or
only the bastion host)
This task is tracked here, but we had to reprioritize the card, as this
is the common ground to all the other "server setup" cards.
https://trello.com/c/BxUTetSu/612-tooling-to-setup-networks-
keys-etc-in-openstack-nodepool-and-tripleo-infra-tenant
While beginning working on this card, yesterday I asked a question to
dmsimard, and it showed me the common roles that are already in place to
setup RDO infra servers. These common roles cover 70-80% of this card
already.
The question is: what level of integration we want to achieve between
our two groups ? Can we modify these roles to be a bit more generic so
they can be used in setting up our infrastructure too ? Can we for
example modify the log server to accept logs from remote journald in our
servers, so we can already use the log server to expose our promotion
logs ?
I would certainly hope so. We already agree that rdo infra team needs to
have access to our server in case of need, it makes only sense that we
use something they are already familiar with.
My implementation will be probably very very very similar to what I see
in rdo-infra repository anyway.
Thanks for any feedback
_______________________________________________
dev mailing list
dev(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev
To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org
5:57 a.m.
On 06 Mar, Gabriele Cerami wrote:
https://trello.com/c/xuRVmCDr/622-setup-teardown-tooling-bastion-host
So the question is: is the bastion host in scope, or we have something
different in mind when we talked about it ?
We also more or less agreed that the bastion host will be present only
on the infra tenant, not on the nodepool tenant, in which it would
serve to bastion only the te-broker. So the te-broker will be
completely exposed, and it will have a public ip
I had to see this through in details. Read around more about the bastion
host and implementations. Revisited what I did in the past jobs.
I proceeded with some analysis on the requirements. We currently have 7
servers in our infra
- promoter
- te-broker
- 4 sova instances
- dns
The 4 sova instances are not a problem, they just gather informations,
they are not active on anything and they don't have power to change,
only reporting. Having them compromised is not a big problem. It can
stay outside of the bastion.
We really should have only one so we can simplify and use less floating
ips, but this is an issue for a different sprint.
The dns server is just a forwarding server, again, no data in there and
no changes from it. It also needs to be accessible from the instances
with a floating IP. It has to stay outside of the bastion.
At this point promoter and te-broker share a similar condition: They are
both the only critical servers in their tenant. We prefer not to use a
bastion on the nodepool tenant to not have a bastion handle a single
server. But at this point this is valid for the promoter server too.
So here's my comprehensive scenario and proposal after the analysis. We
can use it as pessimistic approach, discuss on it and see what
trade-offs we need for time and resources constraints, and decide what
to do in this spint, what leave for later, and what we prefer not to
implement:
- A bastion makes sense only with a properly configured firewall.
- We will configure the security groups in this way
** sova instances, ALLOW output to the logs server and repo mirrors.
DENY rest. ALLOW input from all to http port on public ip, from bastion
internal ip to ssh. DENY the rest
** promoter server. ALLOW output to repo mirrors, images server,
containers registry, dlrn api. DENY rest. ALLOW input from bastion to
ssh in internal network. Push promoter logs to log server for exposure.
NO Floating IP.
** dns, ALLOW OUTPUT repo mirrors. ALLOW input tcp/udp 53, SSH only from
bastion host internal ip
** te-broker. ALLOW output to internal tenant and repo mirrors. DENY
rest. Allow input ssh from the public ip for the bastion host in infra
tenant. Port knocking may be used, so only the bastion can open the ssh
and log in.
** bastion host. Allow out to repo mirrors and all the servers it
handles. ALLOW INPUT ssh from all. Maybe with port knocking too.
** instances in nodepool tenant: ALLOW output to repo and registry
mirrors, DENY rest. ALLOW input ssh only from the bastion public ip
I know output whitelisting may be too difficult, so I consider it
a soft requirement.
- The bastion host will act primarily as jump host, so no service
masking, as we don't expose dangerous services. NO PUBLIC KEYS WILL BE
ALLOWED ON THE BASTION HOST. Only proxy ssh or key forwarding will be
allowed.
It will handle the infra servers in the nodepool tenant too, and should
be used for when the test instances in the tenant are put on hold for
live debugging.
Please provide feedback.
Thanks.
12:54 p.m.
Hi,
Thanks for starting this thread and apologies for not replying sooner.
TL;DR:
By means of "integration" between the two teams, I see this mostly as:
- Let's talk to each other more (specs, discussing what you need, etc)
- Let's do this in the open (this thread is a great start)
- Let's collaborate and maintain a common set of roles and playbooks
to drive the entire infrastructure rather than siloed components
Comments in-line:
I think the questions involve the rdo infra people, especially the
second.
Thanks for involving us, I'm convinced we can find something that'll
be great for everyone.
But in general, I'd like to understand what level of
integration we want between the two groups regarding all the things
related to infrastructure servers.
The way I see things is that the individuals managing and maintaining
the infrastructure are there to support you.
In an ideal world, we have the human and monetary resources to meet
your needs and requirements -- let's say we're not quite there yet but
let's say that increased collaboration between the two teams can only
be beneficial.
We're not there to block anyone or anything, we're there to support you
and provide tooling and infrastructure you can rely on. I like to
pretend we have been steadily improving over the past ~2 years given
the circumstances.
1) We spoke about bastion host. We forgot to create a card for it
during
planning, and yesterday we failed to agree on the scope of the task.
Bastion host is a precise security element, with certain rules which have
to be respected for it to be useful. The bastion host is the single
access point to all the other servers on the infrastructure, it's the
only one possessing a public ip, and that means that *EVERYTHING* needs
to get through it: logs exposure, web pages access (sova included), ssh
access.
Your definition of a bastion host doesn't match what we already use today
and what I felt we tried to convey when we mentioned a "bastion host" during
that last meeting.
What may have been described as a bastion host isn't a bastion host or a
jump box in the literal sense -- at least in the way we are currently
using them.
The upstream infrastructure team has a server called
"puppetmaster.openstack.org" which, ironically, no longer runs the puppetmaster
service but runs Ansible playbooks on a cron instead.
These Ansible playbooks end up doing local puppet apply on the different
servers of the infrastructure but let's not focus on that for now.
The general idea is that there is this central "bastion" host that,
yes, has access
to everything (by necessity) to deploy things on them through Ansible.
However, system administrators ("infra-root") have named users with sudo
privileges directly on each node -- for example I can login as
dmsimard(a)zuul.openstack.org.
Software Factory has a similar mechanism through the "managesf" node which
also leverages Ansible to deploy and configure the different nodes of the
deployment.
This is what we may have described as a "bastion" host but not in the literal
sense -- I hope this clarifies things.
I don't personally think that a literal jump box or bastion host is necessary
for our needs.
Let's try to keep things as simple as possible to make our lives and maintenance
easier.
While beginning working on this card, yesterday I asked a question
to
dmsimard, and it showed me the common roles that are already in place to
setup RDO infra servers. These common roles cover 70-80% of this card
already.
Yes, as per mentioned in different places (most recently in this review[1]), we
already have standardized roles to bootstrap standardized things across all of
our servers such as users, ssh keys and monitoring.
These roles should ideally be ran on every single server.
The question is: what level of integration we want to achieve
between
our two groups ? Can we modify these roles to be a bit more generic so
they can be used in setting up our infrastructure too ?
We are in the process of merging all our playbooks and roles back into
rdo-infra-playbooks [2] which is already integration tested under code
review [3].
The next step would be to formalize the notion of a central host that
continuously
deploy each server with the latest code.
The benefit to this is that anyone can submit a change, it will be tested,
reviewed and deployed automatically.
No one /needs/ access to the servers to install them, they'd be automatically
installed and configured through this mechanism.
This can also help provide a certain amount of guarantee that servers have not
been manually tempered with which can lead to a lot of confusion.
Can we for example modify the log server to accept logs from remote
journald in our servers, so we can already use the log server to expose our
promotion logs ?
That said, I think our existing roles are generic enough -- especially the base
and monitoring roles. If you have improvements for these, feel free to submit
them !
For this particular case, the logserver role that exists today [4] is
actually not
the one that was used to deploy our current logserver implementation which
is more in line with a WIP patch here [5].
I think we need to work on describing what we need better, agree on a solution
and then go from there.
In this case, our need is to expose the promotion logs -- this can be provided
in a number of ways ranging from our existing logstash setup to simply
doing a "passthrough" of the logs over a local httpd server.
In the upstream infrastructure, these kind of proposals are discussed as
specs [6]. It allows everyone a chance to give their opinion which not only
results in a better spec in the end but it also allows to avoid starting with
bad assumptions.
For example, you might draft a spec to deploy a new component in the
infrastructure but it turns out we already have something exactly like that but
you didn't know about it.
At the end of the day, let's not forget that RDO is an open source
community that anyone can participate in just like the upstream OpenStack
one. You can show up tomorrow in #openstack-infra and start sending and
reviewing patches if you'd like.
The upstream infra-root and config-core teams are more or less volounteers
that have accepted to take on this role regardless of their organization or
affiliation. They do not really have any privileged or closed channels.
Everything is in the open, everyone can contribute.
What I'm trying to say is that if someone from outside of Red Hat comes and
volunteers to help with the management of the RDO infrastructure, I'm definitely
not going to turn that person down.
So the scope here is not so much "integration between two teams" but instead
looking at the opportunities to have everyone collaborating to improve the
infrastructure in general.
Hope that makes sense.
[1]:
https://review.rdoproject.org/r/#/c/12777/1/ci-scripts/infra-setup/roles/...
[2]: https://github.com/rdo-infra/rdo-infra-playbooks
[3]: https://review.rdoproject.org/r/#/q/project:rdo-infra/rdo-infra-playbooks
[4]: https://github.com/rdo-infra/ansible-role-logserver
[5]: https://review.rdoproject.org/r/#/c/10814/
[6]: https://github.com/openstack-infra/infra-specs
David Moreau Simard
Senior Software Engineer | OpenStack RDO
dmsimard = [irc, github, twitter]
6:57 a.m.
----- Original Message -----
Hi,
Thanks for starting this thread and apologies for not replying sooner.
TL;DR:
By means of "integration" between the two teams, I see this mostly as:
- Let's talk to each other more (specs, discussing what you need, etc)
- Let's do this in the open (this thread is a great start)
- Let's collaborate and maintain a common set of roles and playbooks
to drive the entire infrastructure rather than siloed components
Hi,
I understand there is a lot of confusion at the moment as we are closing our
sprint, and pushing our reviews.
I started this thread with the best intentions, but I started it too late.
We did not formally discuss about this integration in details during our planning
meeting, and especially after the last reply, I realized that to integrate how
we would like to, we would have needed to start planning way before.
We needed to have our full requirements ready, then point out similarities in
our needs and differences, and plan the changes in the existing to reach the
level of functionality we needed.
After David's reply I spent some time trying to reuse part of the existing
infrastructure playbooks, but it was making too many assumptions that
turned out to be wrong for us, or would just taken too much work to complete.
I tried to see how we could take advantage of the existing logserver, but again
adapting our custom scripts needed too much time.
All this while we were also trying to better define *OUR* requirements for the
sprint especially from a security point of view.
At this point, we can assume that the integration, as I proposed, and envisioned,
is not happening, certainly not in this sprint. And for various reasons:
1) It wasn't formally accepted from both parts. I appreciated the openings from
David, but we didn't reach any consensus. I didn't hear from anyone except David.
There was informal acceptance in our team, but we didn't have time to set a roadmap.
It's not fair to take this thread as roadmap, and ask why we are not following
this path: our teams never really agreed to do what's written here. To be honest
I really thought this wasn't the direction we wanted to move to, since there
wasn't
a definitive OK. I thought this was still just a proposal.
2) As I stated, this would have required a lot more efforts at least in the
initial phase. My proposal came at the end of the week reserved to the design
part of the sprint, because that's when I realized we could have done a lot
together, but it was opening too many fronts, required a different direction, and
it was too late to steer the whole sprint.
3) There is a lot more to iron out on some topics, like the bastion/jumphost, as
they are mutually exclusive visions, and our requirements here conflict.
I would really have liked this to happen, but we really need a lot more discussion
in this thread than actually happened until now, before we can consider this the
effective path.
7:26 a.m.
On Tue, Mar 20, 2018 at 12:57 PM, Gabriele Cerami <gcerami(a)redhat.com> wrote:
I understand there is a lot of confusion at the moment as we are
closing our
sprint, and pushing our reviews.
Thanks for the reply, let's keep this discussion going on the list.
I started this thread with the best intentions, but I started it too
late.
We did not formally discuss about this integration in details during our planning
meeting, and especially after the last reply, I realized that to integrate how
we would like to, we would have needed to start planning way before.
We needed to have our full requirements ready, then point out similarities in
our needs and differences, and plan the changes in the existing to reach the
level of functionality we needed.
That why the suggestion is to use the process which works well for
openstack-infra,
where every change starts with the spec and is discussed on the list.
I know there is some design done in the TripleO CI trello board, but
IMHO Trello is best used for tracking work and linking to resources
outside, not for writing the full proposal text in the card since it's
missing versioning, voting etc. which you get in Gerrit reviews.
After David's reply I spent some time trying to reuse part of the
existing
infrastructure playbooks, but it was making too many assumptions that
turned out to be wrong for us, or would just taken too much work to complete.
Do you have notes about your investigation?
It would be bad to have this effort lost, we definitely want to have
proper generic infra.
RDO Infra backlog is tracked in BZ
https://bugzilla.redhat.com/buglist.cgi?component=Infrastructure&prod...
please file bugs there for specific issues you have noticed.
I tried to see how we could take advantage of the existing logserver,
but again
adapting our custom scripts needed too much time.
All this while we were also trying to better define *OUR* requirements for the
sprint especially from a security point of view.
At this point, we can assume that the integration, as I proposed, and envisioned,
is not happening, certainly not in this sprint. And for various reasons:
1) It wasn't formally accepted from both parts. I appreciated the openings from
David, but we didn't reach any consensus. I didn't hear from anyone except
David.
There was informal acceptance in our team, but we didn't have time to set a roadmap.
It's not fair to take this thread as roadmap, and ask why we are not following
this path: our teams never really agreed to do what's written here. To be honest
I really thought this wasn't the direction we wanted to move to, since there
wasn't
a definitive OK. I thought this was still just a proposal.
2) As I stated, this would have required a lot more efforts at least in the
initial phase. My proposal came at the end of the week reserved to the design
part of the sprint, because that's when I realized we could have done a lot
together, but it was opening too many fronts, required a different direction, and
it was too late to steer the whole sprint.
3) There is a lot more to iron out on some topics, like the bastion/jumphost, as
they are mutually exclusive visions, and our requirements here conflict.
I would really have liked this to happen, but we really need a lot more discussion
in this thread than actually happened until now, before we can consider this the
effective path.
I understand the pressure to deliver in the sprint, I'm just concerned
that we are adding more tech debt which will never be revisited.
Could we plan some future sprints dedicated to tech debt removal?
Cheers,
Alan
2228
days inactive
2242
days old
5 comments
4 participants
participants (4)
-
Alan Pevec -
David Moreau Simard -
Gabriele Cerami -
Wesley Hayutin