Faster Red Hat OpenStack Platform deployments with Ansible strategy plugins

The longer term launch of Pink Hat OpenStack Platform director will carry some adjustments to how the overcloud nodes are configured throughout the deployment and the way it makes it sooner with customized Ansible technique plugins.

Notice: if you happen to haven’t examine “config-download” but, we advise you check out this earlier put up (“Larger management of Pink Hat OpenStack Platform deployment with Ansible integration”) earlier than studying this one.

This put up goes to take a deep dive on the adjustments we made concerning how Ansible technique plugins can influence the way in which overcloud nodes are deployed at a big scale, and current a brand new characteristic which permits a certain quantity of nodes to fail throughout a deployment or day 2 operation.

How the Overcloud Deployment Executes

When a buyer is deploying their cloud, the Pink Hat OpenStack Platform director will assemble a set of Ansible playbooks to execute the required actions and software program configurations to handle their cloud. The Ansible playbook execution is ruled by the duty methods outlined as a part of the playbooks.

The default Ansible execution technique is the “linear” technique. This technique begins with the primary process outlined within the playbook and proceeds to execute it throughout all of the goal programs previous to transferring on to the subsequent process. As soon as all duties have been accomplished, the cloud is absolutely deployed.

Ansible Technique Plugins Defined

By default Ansible runs every process on all hosts focused by a process in a play earlier than beginning the subsequent process on any host. This linear technique describes the way in which that the general process execution is carried out. The duty execution is finished per host batch as outlined by the serial parameter (default is “all”). It is going to execute every process on the identical time for the collection of hosts till this host batch is finished. Then it goes to the subsequent process. Methods additionally deal with failure logic as a part of the execution.

Ansible provides different methods, which might be discovered on GitHub. Listed below are a brief description of how they work:

• debug: Following the execution as linear does, however managed by an interactive debug session.

• free: the duty execution won’t await different hosts to complete the present process and queue extra duties for different hosts which are finished with that process. Subsequently, new duties for hosts that are finished with the earlier process received’t be blocked and in addition sluggish nodes received’t decelerate the execution of the remaining duties in a play for the opposite nodes.

Technique Impacts at Scale

The linear technique works effectively in case you are describing a particular order of duties throughout a whole atmosphere. The technique begins to influence the general execution efficiency when these duties are lengthy operating or there are extra nodes than accessible Ansible playbook forks. Because the linear technique has to attend for all the duties to finish on all the hosts previous to transferring on, lots of the programs will sit idle both ready for a process or having already accomplished their process whereas the opposite nodes are executing. If there are not any inter-node dependencies between the duties, this may result in wasted time ready for a process as an alternative of engaged on the subsequent bit of labor.

As a result of the linear technique will not be best for some playbook executions, Ansible offers a ‘free’ technique that may transfer on to the subsequent process for a number with out ready for the whole set of goal hosts to finish a process. An finish consumer can specify a particular set of duties that may be run freely to completion by altering the playbook technique to make use of the ‘free’ technique. This may velocity up the general execution of a playbook by higher parallelizing the execution throughout their nodes.

Deploying OpenStack

The deployment and configuration of OpenStack requires some coordination at particular occasions throughout the set up, replace, and improve processes. As a result of there are occasions when we now have execution dependencies throughout the nodes, we can not merely change to the free technique to run on a regular basis.

For sure occasions throughout the deployment, we would like the nodes to replace in a managed trend previous to transferring on to the subsequent part of the deployment. Traditionally this was managed by the idea of our deployment steps within the Overcloud deployment. These steps have lengthy been documented within the tripleo-heat-templates as follows:

1. Load Balancer and Cluster providers

2. Core Shared service setup (Database, Messaging, Time)

3. Core OpenStack service setup (Authentication, and many others.)

4. Different OpenStack service setup (Storage, Compute, Networking, and many others.)

5. Service activation and ultimate configurations

We will parallelize the execution of every of those steps, nonetheless we have to pause the general execution of the deployment on the finish of every of the steps till all hosts have completed. If we didn’t pause on the finish of the steps, we might attempt to entry a service that has but to be enabled on a special host.

In earlier variations of Pink Hat OpenStack Platform (13 and under), Warmth was used to deal with this a part of the general deployment orchestration. Warmth would set off the Steps to be executed throughout the cloud and wait till the Step was accomplished previous to transferring on to the subsequent Step. Beginning with OpenStack Platform 14, we transformed the general deployment course of to be dealt with by Ansible. The preliminary conversion to Ansible misplaced a few of this step parallelization due to the beforehand talked about technique execution. This was high-quality for smaller scale deployments nonetheless as we transfer on to bigger scales (500+ nodes), this parallelization as a part of the deployment is a should have.

Creating New Methods in TripleO

As a part of our investigation to enhance the general execution at scale, we began trying into how we will enhance the Ansible playbook execution. We developed an analogous technique to the upstream Anisble “free” technique that extra intently aligned with our expectation across the steps being parallel however failures ought to cease the general execution.

The general deployment steps are outlined as particular person “performs” within the general execution. This enables the technique to halt on the finish of a play if there’s a failure, whereas nonetheless permitting the opposite nodes to finish their duties. Moreover whereas investigating methods, we additionally recognized different areas that may very well be improved with using customized methods (e.g. tolerating failures higher).

We developed a “tripleo_free” technique that permits Ansible to execute on the goal hosts freely for a given playbook. These duties shall be executed assuming no dependencies on the opposite hosts which are operating the identical duties. Failures on any given host won’t cease future duties from executing on different hosts, however it is going to trigger the general playbook execution to cease on the finish of the play. This technique is used for the execution of the deployment steps in order that duties run on all hosts (primarily based on the variety of staff) freely, with out locking at every process.

We additionally developed a “tripleo_linear” technique that operates very equally to the Ansible “linear” technique, nonetheless it’s used to deal with a configurable failure proportion primarily based on a TripleO node position. That is additional detailed within the later part round this matter.

Outcomes (earlier than / after)

Throughout the Ussuri cycle, we deployed a number of clouds to point out the influence of the variety of nodes on the general execution time. By parallelizing the steps, we had been in a position to cut back the general execution time and cut back the influence of including further nodes to the cloud.