The longer term launch of Pink Hat OpenStack Platform director will carry some adjustments to how the overcloud nodes are configured throughout the deployment and the way it makes it sooner with customized Ansible technique plugins.
Notice: if you happen to haven’t examine “config-download” but, we advise you check out this earlier put up (“Larger management of Pink Hat OpenStack Platform deployment with Ansible integration”) earlier than studying this one.
This put up goes to take a deep dive on the adjustments we made concerning how Ansible technique plugins can influence the way in which overcloud nodes are deployed at a big scale, and current a brand new characteristic which permits a certain quantity of nodes to fail throughout a deployment or day 2 operation.
How the Overcloud Deployment Executes
When a buyer is deploying their cloud, the Pink Hat OpenStack Platform director will assemble a set of Ansible playbooks to execute the required actions and software program configurations to handle their cloud. The Ansible playbook execution is ruled by the duty methods outlined as a part of the playbooks.
The default Ansible execution technique is the “linear” technique. This technique begins with the primary process outlined within the playbook and proceeds to execute it throughout all of the goal programs previous to transferring on to the subsequent process. As soon as all duties have been accomplished, the cloud is absolutely deployed.
Ansible Technique Plugins Defined
By default Ansible runs every process on all hosts focused by a process in a play earlier than beginning the subsequent process on any host. This linear technique describes the way in which that the general process execution is carried out. The duty execution is finished per host batch as outlined by the serial parameter (default is “all”). It is going to execute every process on the identical time for the collection of hosts till this host batch is finished. Then it goes to the subsequent process. Methods additionally deal with failure logic as a part of the execution.
Ansible provides different methods, which might be discovered on GitHub. Listed below are a brief description of how they work:
-
debug: Following the execution as linear does, however managed by an interactive debug session.
-
free: the duty execution won’t await different hosts to complete the present process and queue extra duties for different hosts which are finished with that process. Subsequently, new duties for hosts that are finished with the earlier process received’t be blocked and in addition sluggish nodes received’t decelerate the execution of the remaining duties in a play for the opposite nodes.
Technique Impacts at Scale
The linear technique works effectively in case you are describing a particular order of duties throughout a whole atmosphere. The technique begins to influence the general execution efficiency when these duties are lengthy operating or there are extra nodes than accessible Ansible playbook forks. Because the linear technique has to attend for all the duties to finish on all the hosts previous to transferring on, lots of the programs will sit idle both ready for a process or having already accomplished their process whereas the opposite nodes are executing. If there are not any inter-node dependencies between the duties, this may result in wasted time ready for a process as an alternative of engaged on the subsequent bit of labor.
As a result of the linear technique will not be best for some playbook executions, Ansible offers a ‘free’ technique that may transfer on to the subsequent process for a number with out ready for the whole set of goal hosts to finish a process. An finish consumer can specify a particular set of duties that may be run freely to completion by altering the playbook technique to make use of the ‘free’ technique. This may velocity up the general execution of a playbook by higher parallelizing the execution throughout their nodes.
Deploying OpenStack
The deployment and configuration of OpenStack requires some coordination at particular occasions throughout the set up, replace, and improve processes. As a result of there are occasions when we now have execution dependencies throughout the nodes, we can not merely change to the free technique to run on a regular basis.
For sure occasions throughout the deployment, we would like the nodes to replace in a managed trend previous to transferring on to the subsequent part of the deployment. Traditionally this was managed by the idea of our deployment steps within the Overcloud deployment. These steps have lengthy been documented within the tripleo-heat-templates as follows:
-
Load Balancer and Cluster providers
-
Core Shared service setup (Database, Messaging, Time)
-
Core OpenStack service setup (Authentication, and many others.)
-
Different OpenStack service setup (Storage, Compute, Networking, and many others.)
-
Service activation and ultimate configurations
We will parallelize the execution of every of those steps, nonetheless we have to pause the general execution of the deployment on the finish of every of the steps till all hosts have completed. If we didn’t pause on the finish of the steps, we might attempt to entry a service that has but to be enabled on a special host.
In earlier variations of Pink Hat OpenStack Platform (13 and under), Warmth was used to deal with this a part of the general deployment orchestration. Warmth would set off the Steps to be executed throughout the cloud and wait till the Step was accomplished previous to transferring on to the subsequent Step. Beginning with OpenStack Platform 14, we transformed the general deployment course of to be dealt with by Ansible. The preliminary conversion to Ansible misplaced a few of this step parallelization due to the beforehand talked about technique execution. This was high-quality for smaller scale deployments nonetheless as we transfer on to bigger scales (500+ nodes), this parallelization as a part of the deployment is a should have.
Creating New Methods in TripleO
As a part of our investigation to enhance the general execution at scale, we began trying into how we will enhance the Ansible playbook execution. We developed an analogous technique to the upstream Anisble “free” technique that extra intently aligned with our expectation across the steps being parallel however failures ought to cease the general execution.
The general deployment steps are outlined as particular person “performs” within the general execution. This enables the technique to halt on the finish of a play if there’s a failure, whereas nonetheless permitting the opposite nodes to finish their duties. Moreover whereas investigating methods, we additionally recognized different areas that may very well be improved with using customized methods (e.g. tolerating failures higher).
We developed a “tripleo_free” technique that permits Ansible to execute on the goal hosts freely for a given playbook. These duties shall be executed assuming no dependencies on the opposite hosts which are operating the identical duties. Failures on any given host won’t cease future duties from executing on different hosts, however it is going to trigger the general playbook execution to cease on the finish of the play. This technique is used for the execution of the deployment steps in order that duties run on all hosts (primarily based on the variety of staff) freely, with out locking at every process.
We additionally developed a “tripleo_linear” technique that operates very equally to the Ansible “linear” technique, nonetheless it’s used to deal with a configurable failure proportion primarily based on a TripleO node position. That is additional detailed within the later part round this matter.
Outcomes (earlier than / after)
Throughout the Ussuri cycle, we deployed a number of clouds to point out the influence of the variety of nodes on the general execution time. By parallelizing the steps, we had been in a position to cut back the general execution time and cut back the influence of including further nodes to the cloud.
|
Complete Node Depend |
Structure |
Linear (default) |
TripleO Free (new!) |
|
4 |
three controllers, 1 compute |
0:54:55.900 |
0:35:37.660 |
|
13 |
three controllers, 10 computes |
1:17:13.682 |
0:40:22.737 |
|
20 |
three controllers, 17 computes |
1:30:17.337 |
0:44:34.960 |
|
30 |
three controllers, 27 computes |
1:55:08.668 |
0:49:47.799 |
From the desk, you may see that we had been in a position to cut back the general deployment time of a fundamental 30 node cloud to much less time than it used to take to deploy a Four node cloud. Moreover the time it takes to deploy for every further 10 nodes which are deployed.
From the chart, you may see in blue the time it takes with linear and in crimson with tripleo free. We will observe that beside the very fact it’s sooner with tripleo free, when including extra compute nodes the development isn’t rising as quick as with the default linear technique.
Future Work: Failure Tolerance
By implementing our personal methods, we’re in a position to implement particular logic to deal with deployment failures primarily based on node roles within the cloud. Many shoppers have expressed the need to have the ability to deploy a cloud and permit for some failures on Compute associated nodes. With the technique efforts we’re in a position to perceive when a failure happens on a node that’s not a important part of the cloud (e.g. a Compute can fail however a Controller can not).
We applied further logic inside the methods that appears up the TripleO node position for the system that the duty was executing on when it failed and checks if the variety of failed nodes exceeds a particular threshold. This enables a consumer to outline a suitable quantity of failures for a node sort earlier than fully failing the general deployment. That is most helpful for Compute associated nodes as generally the {hardware} or programs might fail for some unrelated cause however the finish consumer wish to configure what is out there.
For example, let’s say that we wish to deploy a 13 node cloud. This cloud is configured with three Controllers and 10 Computes. Because the proprietor, that you must get the cloud up as quickly as attainable however solely actually need 5 Compute nodes instantly.
You can configure ComputeMaxFailPercentage: 40 as a part of the deployment. Whereas executing the deployment associated duties towards the cloud, three of the nodes had an incorrect community configuration in order that they failed whereas attempting to execute the community setup and have become unreachable.
Previous to this technique work, the general cloud deployment would fail on the preliminary community setup for the whole cloud. You would need to troubleshoot the failed community on these nodes earlier than persevering with once more. With the outlined failure proportion the general deployment ought to proceed. On the finish of the deployment the cloud would find yourself consisting of three Controller and seven Compute nodes. The cloud can be operational and the deployer is free to troubleshoot the failed nodes earlier than operating a deployment replace so as to add the sources into the cloud.
Roadmap
Whereas specializing in the deployment steps, we realized that different operations might take good thing about this work, such because the updates and upgrades operations. That is at the moment below overview and testing, however we expect that the identical methods may very well be re-used to execute these playbooks. Our hope is to make these operations sooner at scale, and doubtlessly cut back the upkeep window for our prospects.
We’re investigating backporting all these options all the way down to OpenStack Platform 16; which isn’t nearly code backports but additionally testing (together with efficiency benchmarks) and documentation. We hope to launch these good enhancements in a future OpenStack Platform 16 minor launch!
