Re: [Rdo-list] [RDO-Manager] Rewriting instack scripts into python?
by Hugh O. Brock
(moving this thread upstream where it belongs. It should probably in
fact be on openstack-dev, I'll leave that step to someone else however.)
For context: This is a debate over whether to have a common library for
combining instack deployment commands, and hoe the unified CLI plays
with that library. I have edited a bit for length, apologies if I have
distorted anyone's meaning.
On Thu, Apr 23, 2015 at 08:06:47AM +0200, Jaromir Coufal wrote:
>
>
> On 23/04/15 04:07, James Slagle wrote:
> >On Wed, Apr 22, 2015 at 01:20:25PM -0500, Jacob Liberman wrote:
> >>
> >>
> >>On 4/22/15 11:46 AM, Ben Nemec wrote:
> >>>>I am very concerned about this single call action which is doing all the
> >>>>>magic in the background but gives user zero flexibility. It will not
> >>>>>help neither educate users about the project.
> >>>Our job is not to educate the users about all the implementation details
> >>>of the deployment process. Our job is to write a deployment tool that
> >>>simplifies that process to the point where an ordinary human can
> >>>actually complete it. In theory you could implement the entire
> >>>deployment process in documentation without any code whatsoever, and in
> >>>fact upstream devtest tries to do exactly that:
> >>>http://docs.openstack.org/developer/tripleo-incubator/devtest_overcloud.html
> >>>
> >>>And let me tell you - as someone who has tried to follow those docs -
> >>>it's a horrible user experience. Fortunately we have a tool in
> >>>instack-undercloud that rolls up those 100+ steps from devtest into
> >>>maybe a dozen or so steps that combine the logically related bits into
> >>>single commands. Moving back toward the devtest style is heading in the
> >>>wrong direction IMNSHO.
> >>>
> >>>Does instack-undercloud/rdo-manager need to be more flexible?
> >>>Absolutely. Does that mean we should throw out our existing code and
> >>>convert it to documentation? I don't believe so.
> >>>
> >>
> >>Definitely get input from our field folks on this topic.
> >>
[snip]
> >>
> >>A GUI and unified deployment scripts are nice to have but are not
> >>replacements for complete CLIs + docs.
> >
> >I'm just replying to the thread, I'm not picking on your specific point :-).
> >Your feedback is really good and something that we need to keep in mind.
> >
> >However, this discussion isn't actually about POC vs production vs flexibility.
> >That's pretty much a strawman to the discussion given that the 'openstack
> >flavor' command is already there, and will always be there. No one is saying
> >you shouldn't use it, or we shouldn't document why and how we use flavors. Or
> >for that matter that any amount of customization via flavors wouldn't
> >eventually be possible.
>
> But this is our primary focus - production ready flow. Which we should test
> as soon as we can and it is still not out there. So this discussion actually
> is about it. And also about production ready people's user experience.
>
>
> >There's also going to be a big difference between our end user documentation
> >that just gets you any repeatable process (what we have now), and advanced
> >workflows we might document for the field or consultants, or hide behind a "not
> >officially supported without RH consultants" banner.
>
> End user documentation is not what we have now. What we have now is very
> narrow restricted flow for people to get started with 1 controller and 1
> compute -- which is just POC. With zero knowledge of what is happening in
> the background.
>
>
> >Moreso, the point is that:
> >
> >The shell script we currently have, the proposed Python code, and the proposed
> >documentation change all are 100% equivalent in terms of functionality and
> >flexiblity. The proposed documentation change doesn't even offer anything other
> >than replacing 1 command with 6, with no explanation of why or how you might
> >customize them (that's why I say it's worse).
>
> First of all -- reason why it *has* to run 6 commands is how was written the
> instack deployment script which requires 3 flavors with very specific name
> for each role, despite the fact that one of the roles is not deployed.
>
> If user follows regular way (when we get out of the deployment scripts), he
> would have to create *one* single flavor (2 commands) and in these commands
> is specifically listed what features are being registered with the flavor
> (ram, disk, vcpu). So it is not hidden from user.
>
> This is very important. If you even want to improve this flow, we should
> suggest flavors to user and improve unified CLI.
>
> >Here's really my main point I guess:
> >
> >If it takes 12 (eventually) CLI commands to create flavors, do we expect people
> >to *type* those into their shell? I hope not.
> >
> >Let's suppose we document it that way...in my experience the most likely thing
> >someone would do (especially if they're repeating the process) would be to
> >copy/paste those 12 commands out of the documentation, and write their own
> >shell script/deployment tool to execute them, or just copy/paste them
> >straight into their shell, while perhaps customizing them along the way.
> >
> >That would certainlly be a totally valid way to do it.
>
> If user has homogeneous environment, he can have just one simple flavor. If
> he has heterogeneous or he wants to get more specific, he will take
> additional actions (create more flavors or deal with edeploy matching).
>
> You are overstating - at the moment the problem with those 6 commands is
> really how the instack scripts are written. So that I could get us out of
> scripts and replace flavors script I had to create three flavors instead of
> one.
>
> >So...why don't we just give them that shell script to start off with?
> >Better yet, let's write something we'd actually like to support long term (in
> >Python) and is just as flexible, perhaps taking json (or more likely YAML tbh)
> >as input, with a nice Python program to log stuff and offer --help's along the
> >way. Something that's actually supportable so we could ask: What input did you
> >provide to this tool and what was the output? VS. How did you customize these
> >12 commands, now go please run these other commands so we can figure out what
> >happened.
>
> Yes, let's write something what we support longer term. Didn't we agree it
> is unified CLI? Didn't we agree we should support and improve it? Feeding
> another yaml file to create a single flavor? I disagree that this is better
> user experience.
>
>
> >I'm already seeing developers and external people saying that it's too much
> >as-is and that they're going to write custom tooling to help out. Why wouldn't
> >we just offer that already (especially when 90% of the code is already written
> >upstream), and still be able to keep all the flexibility, as long we actually
> >document how it works, and what commands you'd run to customize it?
>
> Are these developers real world users in production environments? Ask field
> guys. The feedback you are getting is from people who are running these
> commands 20 times a day. Then it is valid point that it is too many steps
> for them. But this is *completely* different use case than production
> environments.
>
> And we will not get out of the use case that people will write automation
> scripts. But instack will not help them with that because they will write
> very specific automation scripts on top of their production environments and
> they will use our CLI as the basis.
>
> We should focus on production environments first and then simplify for POCs.
> Not vice-versa. And not improving the scripts.
>
> -- Jarda
I'm sorry to say this Jarda but I think I am with James and Ben
here -- although this discussion is way too short on specifics, which I
think is part of the problem.
What I think Ben is proposing is a Python library that will encapsulate
the business logic that we need to do a deployment. If that's the case,
then I think that yes, that is exactly what we need. If I understand it
correctly the first version of this library will have some things
like flavor matching hard-coded, but we will add the ability to
configure those things as the library matures. I think this makes a ton
of sense.
I don't believe that having this library in any way precludes use of the
unified CLI, although it may change the way the CLI works -- certainly,
we would want the CLI to be able to take advantage of the business logic
in the library.
Finally, I completely support your desire that the individual operations
the manager is taking during deployment be well documented. However, I
don't think that need for documentation also means that the recommended
path for users should be to follow each individual step.
I think the only genuine disagreement here is over the usability of the
manager CLI -- the idea that "instack is never going to go away and the
unified CLI will never become the primary interface" is a red herring
and I think we should stop chasing it. Given that, if we're having a
debate about usability, let's please get some actual examples on this
thread and get some folks like Jacob to see for themselves what we're
talking about.
Thanks,
--Hugh
--
== Hugh Brock, hbrock(a)redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey
10 years
[Rdo-list] no valid host was found
by pauline phaure
hey everyone, I hope this time somebody would help me. I still getting the
same error when I try to spawn a VM which is no valid host was found,
knowing that I have already spawned two successful VMs (one small and one
xlarge). I'm sure it's not a problem of ressources I have 16 cores on each
of my servers, 12 Gb of RAM and 1.7 TeraB of disk space available. But
still my openstack is acting like if he lacks of ressources because when I
delete when of the existing VM, nova responds correctly and I can then
Launch my instance. PLZ found attahced My logs.
[root@localhost ~]# lscpu
Architecture : x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Boutisme : Little Endian
Processeur(s) : 16
Liste de processeur(s) en ligne : 0-15
Thread(s) par cœur : 2
Cœur(s) par socket : 4
Socket(s) : 2
Nœud(s) NUMA : 2
Identifiant constructeur : GenuineIntel
Famille de processeur : 6
Modèle : 26
Nom de modèle : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
Révision : 5
Vitesse du processeur en MHz : 2533.000
BogoMIPS : 5065.94
Virtualisation : VT-x
Cache L1d : 32K
Cache L1i : 32K
Cache L2 : 256K
Cache L3 : 8192K
Nœud NUMA 0 de processeur(s) : 0,2,4,6,8,10,12,14
Nœud NUMA 1 de processeur(s) : 1,3,5,7,9,11,13,15
[root@localhost ~]# lscpu
Architecture : x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Boutisme : Little Endian
Processeur(s) : 16
Liste de processeur(s) en ligne : 0-15
Thread(s) par cœur : 2
Cœur(s) par socket : 4
Socket(s) : 2
Nœud(s) NUMA : 2
Identifiant constructeur : GenuineIntel
Famille de processeur : 6
Modèle : 26
Nom de modèle : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
Révision : 5
Vitesse du processeur en MHz : 1600.000
BogoMIPS : 5065.93
Virtualisation : VT-x
Cache L1d : 32K
Cache L1i : 32K
Cache L2 : 256K
Cache L3 : 8192K
Nœud NUMA 0 de processeur(s) : 0,2,4,6,8,10,12,14
Nœud NUMA 1 de processeur(s) : 1,3,5,7,9,11,13,15
[root@localhost ~]# df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/centos-root 1,8T 2,8G 1,8T 1% /
devtmpfs 5,8G 0 5,8G 0% /dev
tmpfs 5,8G 0 5,8G 0% /dev/shm
tmpfs 5,8G 8,9M 5,8G 1% /run
tmpfs 5,8G 0 5,8G 0% /sys/fs/cgroup
/dev/sda1 497M 126M 372M 26% /boot
/dev/mapper/centos-home 50G 33M 50G 1% /home
[root@localhost ~]# df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/centos-root 1,5T 3,3G 1,5T 1% /
devtmpfs 5,8G 0 5,8G 0% /dev
tmpfs 5,8G 4,0K 5,8G 1% /dev/shm
tmpfs 5,8G 8,9M 5,8G 1% /run
tmpfs 5,8G 0 5,8G 0% /sys/fs/cgroup
/dev/loop0 1,9G 6,1M 1,7G 1% /srv/node/swiftloopback
/dev/sda1 497M 171M 327M 35% /boot
/dev/mapper/centos-home 50G 33M 50G 1% /home
tmpfs 5,8G 8,9M 5,8G 1% /run/netns
MemTotal: 12126860 kB
MemFree: 11487076 kB
MemAvailable: 11457400 kB
Buffers: 764 kB
Cached: 154492 kB
SwapCached: 0 kB
Active: 203440 kB
Inactive: 111920 kB
Active(anon): 169672 kB
Inactive(anon): 8260 kB
Active(file): 33768 kB
Inactive(file): 103660 kB
Unevictable: 84820 kB
Mlocked: 84820 kB
SwapTotal: 6160380 kB
SwapFree: 6160380 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 244968 kB
Mapped: 33992 kB
Shmem: 8940 kB
Slab: 84552 kB
SReclaimable: 28472 kB
SUnreclaim: 56080 kB
KernelStack: 5984 kB
PageTables: 6404 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12223808 kB
Committed_AS: 658040 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 109428 kB
VmallocChunk: 34355304448 kB
HardwareCorrupted: 0 kB
AnonHugePages: 77824 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 110716 kB
DirectMap2M: 12462080 kB
cat /proc/meminfo
MemTotal: 12126860 kB
MemFree: 7088560 kB
MemAvailable: 7148508 kB
Buffers: 1156 kB
Cached: 280384 kB
SwapCached: 0 kB
Active: 4320672 kB
Inactive: 202656 kB
Active(anon): 4251044 kB
Inactive(anon): 8704 kB
Active(file): 69628 kB
Inactive(file): 193952 kB
Unevictable: 84884 kB
Mlocked: 84884 kB
SwapTotal: 6160380 kB
SwapFree: 6160380 kB
Dirty: 40 kB
Writeback: 0 kB
AnonPages: 4326216 kB
Mapped: 49868 kB
Shmem: 9072 kB
Slab: 135476 kB
SReclaimable: 43228 kB
SUnreclaim: 92248 kB
KernelStack: 12976 kB
PageTables: 118868 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12223808 kB
Committed_AS: 13439552 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 110196 kB
VmallocChunk: 34355295000 kB
HardwareCorrupted: 0 kB
AnonHugePages: 317440 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 114812 kB
DirectMap2M: 12457984 kB
[image: Images intégrées 1]
10 years
[Rdo-list] rdo-manager failures: instack-install-undercloud failing for non-obvious reasons
by Lars Kellogg-Stedman
Running "instack-install-undercloud" is failing for me:
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure
phase. [Command '['dib-run-parts',
'/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
status 6]
Unfortunately, the failure doesn't provide much in the way of useful
information. If I scroll up several pages, I find:
Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff'
Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired!
Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired!
Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired!
Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired!
Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired!
But again, that doesn't really tell me what is failing either (a
command is still failing? Which command?).
It looks like rabbitmq is having some problems:
[stack@localhost ~]$ sudo rabbitmqctl status
Status of node rabbit@localhost ...
Error: unable to connect to node rabbit@localhost: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit@localhost]
rabbit@localhost:
* connected to epmd (port 4369) on localhost
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: rabbitmqctl20640@stack
- home dir: /var/lib/rabbitmq
- cookie hash: 4DA3U2yua3rw7wYLr+PbiQ==
If I manually stop and then start rabbitmq:
sudo systemctl stop rabbitmq-server
sudo systemctl start rabbitmq-server
It seems to work:
# rabbitmqctl status
Status of node rabbit@stack ...
[{pid,20946},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
...
After manually starting rabbit and re-running
instack-install-undercloud, the process is able to successfully create
the rabbitmq_user resources and completes successfully.
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
10 years
[Rdo-list] rdoproject.org plans
by Rich Bowen
This has taken some doing, but I wanted to follow up on some earlier
conversations about the rdoproject.org wiki and possible better tools.
Shortly after the OpenStack Summit, we'll be migrating the website over
to us Middleman, rather than the current Vanilla + Mediawiki toolset.
Existing wiki documents, and the blog content in Vanilla, will be
migrated over to markdown files that will then be editable via Github.
(Exact details to be determined.)
We're not the guinea pigs in this process - the ovirt.org site is
undergoing a similar migration, so hopefully by the time they're done
we'll know all the things that can go wrong and avoid then.
This solves two immediate problems.
One is that the Google-based auth on the website is broken, and at this
time there's no working plugin that fixes the problem that we're having
there.
The other is simply bringing the site content management more into line
with the tools that we're using for everything else, and putting it
revision control along the way.
I will have more details, and dates, in a few weeks, but I wanted to let
you know that I haven't dropped this - it's just taken a while to work
out the process of importing and converting all of the content into the
new format.
Thanks again for your patience.
--
Rich Bowen - rbowen(a)redhat.com
OpenStack Community Liaison
http://rdoproject.org/
10 years
[Rdo-list] rdo-manager failures: failed to deploy Nova instances
by Lars Kellogg-Stedman
The stack deployment implemented by "instack-deploy-overcloud
--tuskar" failed because Heat was unable to create Nova instances:
$ nova list
+----...-+------...-+--------+------------+-------------+---------------------+
| ID ... | Name ... | Status | Task State | Power State | Networks |
+----...-+------...-+--------+------------+-------------+---------------------+
| 56f... | ov-k2... | ERROR | - | NOSTATE | ctlplane=192.0.2.11 |
| f1f... | ov-oc... | ERROR | - | NOSTATE | ctlplane=192.0.2.10 |
+----...-+------...-+--------+------------+-------------+---------------------+
Running "nova show" reveals the following error:
"message": "No valid host was found. There are not enough hosts available."
Running "instack-ironic-deployment --show-profile" seems to indicate that the
appropriate nodes were discovered and were assigned the correct roles:
Querying assigned profiles ...
f32c7e5c-d06c-4811-b0ba-76da571f91da
"profile:control,boot_option:local"
5a04f842-7d6d-4d41-b26e-e71f121259d6
"profile:compute,boot_option:local"
DONE.
--
Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack | http://blog.oddbit.com/
10 years
[Rdo-list] Fwd: failed to allocate network
by pauline phaure
---------- Forwarded message ----------
From: pauline phaure <phaurep(a)gmail.com>
Date: 2015-04-22 9:04 GMT+02:00
Subject: Fwd: [Rdo-list] failed to allocate network
To: Rhys Oxenham <roxenham(a)redhat.com>, rdo-list <rdo-list(a)redhat.com>
---------- Forwarded message ----------
From: pauline phaure <phaurep(a)gmail.com>
Date: 2015-04-22 9:02 GMT+02:00
Subject: Re: [Rdo-list] failed to allocate network
To: Rhys Oxenham <roxenham(a)redhat.com>
Cc: rdo-list <rdo-list(a)redhat.com>
As I can't connect to my controller node ( I lost my connection to it) I'm
uploading here the files I found on my compute node.
2015-04-21 18:40 GMT+02:00 Rhys Oxenham <roxenham(a)redhat.com>:
> Hi Pauline,
>
> > On 21 Apr 2015, at 15:44, pauline phaure <phaurep(a)gmail.com> wrote:
> >
> > 2015-04-21 14:49:05.268 7743 TRACE nova.compute.manager [instance:
> 6d4a32d5-ac96-4143-8f09-fc16f149db52] VirtualInterfaceCreateException:
> Virtual Interface creation failed
>
> Can you please upload sanitised configuration files for Nova and Neutron
> (with relevant plugin configuration files)?
>
> Many thanks
> Rhys
10 years
[Rdo-list] failed to allocate network
by pauline phaure
hey every one.
Yesterday I had I discovered that I was running out of space disk as I
misspartioned my disks so I decided to start from scratch. Now I have two
baremetal servers both with 1.8TB allocated to root. But , I still have the
message *no valid host was found* when I try to spawn a big instance. When
I try to do it with a small instance I have the error *Failed to allocate
the network(s),* not rescheduling. any ideas that might help me??
plz find attached my logs.
[root@localhost ~]# df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/centos-root 1,8T 2,3G 1,8T 1% /
devtmpfs 5,8G 0 5,8G 0% /dev
tmpfs 5,8G 0 5,8G 0% /dev/shm
tmpfs 5,8G 17M 5,8G 1% /run
tmpfs 5,8G 0 5,8G 0% /sys/fs/cgroup
/dev/sda1 7,0G 133M 6,9G 2% /boot
/dev/mapper/centos-home 52G 33M 52G 1% /home
tmpfs 5,8G 17M 5,8G 1% /run/netns
[root@localhost ~]# free -m
total used free shared buff/cache
available
Mem: 11842 9943 208 32 1690
1386
Swap: 10239 1 10238
10 years
