Contrail Cloud 16 Update Guide Published
2023-09-25

Juniper Networks, Inc.
1133 Innovation Way
Sunnyvale, California 94089
USA
408-745-2000
www.juniper.net
Juniper Networks, the Juniper Networks logo, Juniper, and Junos are registered trademarks of Juniper Networks, Inc.
in the United States and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners.
Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.
Contrail Cloud 16 Update Guide
Copyright © 2023 Juniper Networks, Inc. All rights reserved.
The information in this document is current as of the date on the title page.
YEAR 2000 NOTICE
Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related limitations through the year 2038. However, the NTP application is known to have some difficulty in the year 2036.
END USER LICENSE AGREEMENT
The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networks software. Use of such software is subject to the terms and conditions of the End User License
Agreement (“EULA”) posted at https://support.juniper.net/support/eula/. By downloading, installing or using such software, you agree to the terms and conditions of that EULA.

About This Guide

Use this guide to update to the latest version of Juniper Networks Contrail Cloud 16. For a summary of Contrail Cloud 16 components and versions, see, Contrail Cloud 16 Release Notes.

Before You Update

Overview
This update guide covers a detailed update procedure for Contrail Cloud 16. Using this guide, a Contrail Cloud user will be able to:

• “Verify an existing Contrail Cloud setup” on page 2
• “Raise an update request ” on page 4
• “Understand the update plan” on page 4
• “Deploy the update configuration files” on page 11

Prerequisites

IN THIS SECTION

• Verify the Configuration Files | 2
• Check the Cloud Health | 3
• Back up the Undercloud and Overcloud | 3
• Pause and Shutdown Business Services | 3

The prerequisites to update Contrail Cloud 16 are as follows:

Verify the Configuration Files
Verify the Contrail Cloud deployment configuration files to ensure all configuration settings are accurate and adhere to a proper schema.

• Verify all the YAML files in the /var/lib/contrail_cloud/config directory to ensure all values match the expected results.
•  Run a schema check against your configuration files: /var/lib/contrail_cloud/scripts/node-configuration.py schema

Check the Cloud Health
Verify that Contrail Cloud is healthy and all services are active. Problems with the cloud health can cause errors during the update process. Use the resources below to help verify your cloud health:

• Contrail Cloud Deployment Guide.
• Use the “Verify Quorum and Node Health” section in the Node Reboot and Health Check topic.
• See also, Red Hat OpenStack Using the Validation Framework.

Back up the Undercloud and Overcloud
Use the following resources for backup instructions:

• Backing Up and Restoring the Undercloud and Control Plane Nodes
• Backing Up the Overcloud Control Plane Services
• How to Backup and Restore Contrail Databases in JSON Format in Openstack Environments

Pause and Shutdown Business Services
The integrity of API calls is not guaranteed during the update. It is recommended to pause or shutdown the external business services to ensure a smooth update. This helps to prevent data loss or a workload error. See the documentation for your specific service instructions.
It is recommended to do the following:

• Quiesce all external API requests. For example, Horizon.
• Perform a graceful shutdown on any vulnerable workloads.

Raise an Update Request
Send an update request to Juniper to adjust your unique key to the latest version of Contrail Cloud 16.
Juniper will adjust your unique key for the required components and version of Contrail Cloud.
The steps to raise an update request are as follows:

1. Send an e-mail message to contrail_cloud_subscriptions@juniper.net requesting for Contrail Cloud 16 update.
2. Provide the following information in an email:
   • Share your Contrail Cloud activation key in the email request. State the version you want to update your Contrail Cloud 16 to.
   • Specify the time and date for your update. Juniper will make necessary preparations during the update window you specify.
   You will receive an acknowledgment from Juniper that your key has been adjusted for the update.

Understand the Update Plan
IN THIS SECTION

• Overview | 5
• Configure Your Update Plan | 5

There is an update_plan section in the Contrail Cloud configuration file that is used for overcloud update (which has three stages). It is not used for updating any of the other components. It is very important for the user to understand the idea of update plan and how to define batches of nodes for update. Use this section to understand the update plan, how it works, and how to configure the update plan in your site.yml configuration file.

Overview
The update plan controls the overcloud update sequencing. It allows you to define which node batch to update with each run of the contrail-cloud-update- overcloud-step2.sh script.
The overcloud update contrail-cloud-update-overcloud-step2.sh script will:

• Compare the lock files with your defined batch list to find the next batch that has not already been completed.
• Update all nodes per batch as you defined in the update plan nodes list.
• Update packages and containers for each node you defined in the current batch.
•  Update one batch per script run, as defined in the update plan.
• Automatically create a lock file when the batch is processed. This is how the script knows to move on to the next batch.
• Rerun until there are no longer any batches left to update.

Configure Your Update Plan
Use the following sample to configure the update plan in the config/site.yml file. The sample would help you to:
• Understand the configuration hierarchy.
• Determine the configuration options.
• Identify proper indentations and empty space.
• Understand how to target specific nodes to be updated as workloads are migrated.

Copyright 2023 Juniper Networks, Inc. All rights reserved.

Licensed under the Juniper Networks Script Software License (the “License”).

You may not use this script file except in compliance with the License,

which is located at

http://www.juniper.net/support/legal/scriptlicense/

Unless required by applicable law or otherwise agreed to in writing by the

parties,

software distributed under the License is distributed on an “AS IS” BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

The update_plan provides the blueprint for working through the deployment

update.

The plan is defined as a series of “batches”. Each batch is processed by the

contrail-cloud-update-overcloud-step2.sh, one batch for each run. Once a

batch is

successfully completed a lock file is created to indicate that state. The

next run

of the contrail-cloud-update-overcloud-step2.sh script will find the next

unfinished

batch to process.

update_plan:

reboot_computes:

parallel: all computes from batch will be rebooted at one time

sequence: all computes from batch will be rebooted one by one

disabled: computes after update will not be rebooted, you need to reboot

manually

Each batch defines:

name: a unique batch name (used for lock file naming)

update_type: either “sequence” (one node at a time) or

“parallel” (all nodes together)

nodes_list: a list of nodes to apply the update to. The special

keyword “ControlPlane” represents the Openstack controllers.

For each of these roles, the update

will be performed on a single instance at a time sequentially

until all roles instances are updated.

reboot_computes: parallel
batches:

The first batch should always be the control plane.

– name: controlplane
update_type: sequence nodes_list:
– ControlPlane

This batch is used to update all the DPDK and kernel computes in rack0 in

parallel.
– name: computenodes_rack0 update_type: parallel nodes_list:
– overcloudx51-compdpdk0hw2-0
– overcloudx51-compkernel0hw0-0
– overcloudx51-compkernel0hw1-0

This batch is used to update all the DPDK and kernel computes in rack1 in

parallel.
– name: computenodes_rack1 update_type: parallel nodes_list:
– overcloudx51-compdpdk1hw3-0
– overcloudx51-compkernel1hw0-0
– overcloudx51-compkernel1hw1-0

This batch is used to update all the SRIOV computes in sequence (just an

example).
– name: sriovnodes
update_type: sequence nodes_list:
– overcloudx51-compsriov0hw4-0
– overcloudx51-compsriov1hw5-0

Batch to update ceph storage nodes. To ensure the integrity of the ceph

cluster,

each role instance will be updated sequentially.

– name: cephnodes
update_type: sequence nodes_list:
– overcloudx51-cephstorage0hw6-0
– overcloudx51-cephstorage0hw6-1
– overcloudx51-cephstorage1hw7-0
Edit the above update plan to match your deployment for each targeted batch.
Name Your Unique Batch
During update, you can control the batch order in your update_plan. You choose a unique batch name to identify each unique batch, and for lock file naming. Choose a unique name that you define for each batch.
The contrail-cloud-update-overcloud-step2.sh script identifies the first unique named batch, which has not already been updated. The script creates a lockfile after each successful batch update to identify it as being completed. To start, you might configure it to look like this:

update_plan:
batches:

unique batch name

– name: controller_nodes

Define the Update Type for the Named Node Role
The update_type determines the update order of the nodes in the named batch run. You have the option of running the update in sequence. The sequence update will update one node at a time, per batch run.
You can also choose to run the update in parallel. The parallel update will update all nodes together, per batch run.
Use the following sample to update the nodes in the named batch in sequence:
update_plan:
batches:

update_type: either “sequence” all positions from batch list will be

processed one by one

“parallel” batch will update all positions from list at the same time

update_type: sequence
Define the Nodes in the Named Batch
Define the nodes in the nodes_list that you would like to update in the named batch run. These nodes are the specific node names from your environment. Group by role when you define your batches. As an example, group Ceph servers together, and computes together.
Run the following commands to collect the server names from the undercloud:
(undercloud) [stack@undercloud ~]$ openstack server list –name ceph -c Name -f value | sort
(undercloud) [stack@undercloud ~]$ openstack server list –name comp -c Name -f value | sort
Following is the sample output from the CLI input above to collect Ceph and compute server names:
overcloudw9o-cephstorage0hw6-0
overcloudw9o-cephstorage1hw7-0
overcloudw9o-cephstorage2hw6-0
overcloudw9o-compdpdk2hw2-0
overcloudw9o-compkernel0hw0-0
overcloudw9o-compsriov1hw3-0
The ControlPlane is a special keyword understood by the update plan. The ControlPlane represents the OpenStack controllers. The ControlPlane updates in sequence by default. One OpenStack controller instance is updated at a time.
Always define the ControlPlane as the first entry in the first batch in your defined batch list.
update_plan:
batches:
– name: controller_nodes update_type: sequence nodes_list:
– ControlPlane
Finish Defining Your Remaining Batches
Define the rest of your batches. Ensure that you target all nodes for update. The update_plan sample shows an example of how you might define compute batches by rack, or compute type. Remember that the ControlPlane should always be the first batch to update. The next batches should include the compute nodes. The last batches should consist of the storage nodes.

Update Contrail Cloud 16

SUMMARY
Use the procedure shared below to update to the latest version of Contrail Cloud 16. This delivers updated Contrail containers, Red Hat RHEL/RHOSP/Ceph content, and kernel version. Refer to the Contrail Cloud 16 Release Notes for the associated component versions for your release and Contrail Cloud 16 Deployment Guide for the sample configuration files.
NOTE: There will be a disruption in service during the update. Whole services are taken down during the update process. Plan for a period of disruption during the update process. The update preserves existing overcloud configurations, for example: images, projects, networks, volumes, virtual machines, and so on.

The update procedure includes:

• “Update the Jump Host” on page 11
• “Update the Undercloud” on page 12
•  “Update Red Hat Virtualization Manager” on page 13
•  “Update Contrail Containers” on page 13
• “Update RHEL on the Kubernetes Nodes” on page 14
•  “Update Kubernetes Clusters” on page 14
• “Update the Overcloud” on page 15
•  “Update Contrail Insights” on page 16

Update the Jump Host
To update the jump host, run the following scripts and commands in the CLI of the jump host as a Contrail user (su – contrail from root).

1. Run the jump host update script to update the RHEL running on the jump host. This is a DNF update.
   A reboot is required after the DNF update.
   /var/lib/contrail_cloud/scripts/contrail-cloud-update-jumphost.sh -d

2.  Verify the jump host is on the correct RHEL kernel version for Contrail Cloud release.
   uname -a

3. Verify the jump host is on the correct RHEL version for Contrail Cloud release.
   cat /etc/redhat-release

Update the Undercloud
Contrail Cloud 16.3 uses secured registry and has vault-data.yml file with all the passwords related to your deployment environment. You must verify that the vault-data.yml file has your container image registry credentials. See, samples/vault-data.yml file, which can be read with the following command:
ansible-vault view samples/vault-data.yml
After going through the vault-data.yml file, perform the following procedure.

1. Run the following script from the jump host as the Contrail user to update the packages and containers on the undercloud virtual machine on the jump host.
   /var/lib/contrail_cloud/scripts/contrail-cloud-update-undercloud.sh -d
   The script will:
   • Update the undercloud virtual machine registration.
   • Update the operating system on the undercloud virtual machine.
   • Update the Red Hat OpenStack registry with the latest containers.
   • Update the Red Hat OpenStack Director.
   • Reboot the undercloud virtual machine once the update is complete.

Update Red Hat Virtualization Manager
Red Hat Virtualization (RHV) Manager is used for the OpenStack control host hypervisors in Contrail Cloud 16. To update the RHV Manager components, run the following command:
contrail-cloud-update-rhvm.sh -d
The script will:

• Update the RHV Manager virtual machine.
• Update Red Hat Virtualization Manager.
• Update the control host operating system.
•  Reboot each control host one by one and starts vms on control hosts after they come up.
  Log in to RHV Manager to verify it successfully updated. Access the GUI by entering https:///ovirt-engine/ in your browser. The default user is admin. The default password is c0ntrail123.

Update Contrail Containers
To update the Contrail containers, run the following command: contrail-cloud- update-contrail.sh -d
The script will:

• Check Contrail status before the update starts.
•  Update Contrail hosted on top Kubernetes cluster to the latest version.
•  Re-register the jump host on the satellite.
• Populate new Contrail containers to the registry.
•  Register Kubernetes cluster nodes on the satellite.
• Replace the Tungsten Fabric Operator (tf-operator) with an updated version using a zero impact update (ZIU) update process. Contrail status is validated after the tf-operator has been updated.
• Replace vRouter containers, one at a time. Contrail status is validated after all containers have been processed.

Update RHEL on the Kubernetes Nodes
To update the RHEL on the Kubernetes nodes, run the following command:
contrail-cloud-update-k8s-rhel.sh -d
The script will:

• Update the Kubernetes cluster nodes to the latest RHEL version. Each Kubernetes node reboots and processed one at a time.

• Skip Kubernetes packages. Packages related to Kubernetes are excluded during this script run.
  Kubernetes packages are updated during the Kubernetes cluster update script.

• Re-register the Kubernetes cluster nodes.

Update Kubernetes Clusters
To update the Kubernetes cluster, run the following command: contrail-cloud- update-k8s-cluster.sh -d
The script will:

• Update the Kubernetes cluster to the latest supported version for your Contrail Cloud release.
• Update through the Kubespray versions until the latest supported version in Contrail Cloud is reached.
• For each Kubespray and Kubernetes version:
• Register on the satellite for proper access to the Kubespray package, Kubernetes artifacts, and Kubernetes containers.
•  Edit the registry on the jump host with the proper Kubernetes version.
• Register the Kubernetes cluster and install the proper packages for the specific Kubespray and Kubernetes version.
• Finalize the Kubernetes cluster update.
•  Update through the Kubernetes versions until the latest supported version in Contrail Cloud is reached.

Update the Overcloud
Make sure you understand the “update plan” on page 4 and have added the update plan details in your site.yml configuration file.
To update overcloud, run the following commands:

1. contrail-cloud-update-overcloud-step1.sh -d
   The script will:
   • Generate an ansible configurations needed for the overcloud update process.
   • Prepare the services for update.
   • Run the prepare-update.sh script from the undercloud virtual machine.

2. contrail-cloud-update-overcloud-step2.sh -d
   Choose a unique name that you define for each batch. You must manually run the update script for each unique named batch. Rerun the update script for each named batch until all batches have successfully updated. The script creates a lockfile for the named batch after it is processed in the script run. The script will:
   • Update all nodes as defined in the update_plan[‘batches’][][‘nodes_list’] and is configured in the config/site.yml file.
   • Update node packages and containers according to the batch update_type.
   • Update one batch per script run, as configured in the update_plan.
   • Automatically create a lock file when the batch has been processed. This is how the script knows to move on to the next batch the next time the script is run.

3.  contrail-cloud-update-overcloud-step3.sh -d
   The script will:
   • Ensure that the stack resource structure aligns with the new packages and configurations.
   • Update Ceph cluster configuration, containers, and services.
   • Synchronize the OpenStack database.
   • Switch all services from update to fully operational state.
   • Remove all lock files.

Update Contrail Insights
This step is only required if you have Contrail Insights (formerly AppFormix) deployed in your environment.

1. Update Contrail Insights:
   contrail-cloud-update-appformix.sh -d
   The script will update the Contrail Insights monitoring and analytics software.

References

• Script Software License Agreement - Juniper Networks
• Backing up and restoring the undercloud and control plane nodes Red Hat OpenStack Platform 16.2 | Red Hat Customer Portal
• Backing up the Overcloud control plane services — TripleO 3.0.0 documentation
• Contrail Cloud Documentation | Juniper Networks
• How to Backup and Restore Contrail Databases in JSON Format in Openstack Environments Using the Openstack 16.1 Director Deployment | Contrail Networking 21 | Juniper Networks
• Contrail Cloud Deployment Guide | Contrail Cloud 16 | Juniper Networks
• Appendix A: Sample Configuration Files | Contrail Cloud 16 | Juniper Networks

Read User Manual Online (PDF format)

Read User Manual Online (PDF format)  >>

Download This Manual (PDF format)

Download this manual  >>