APIC System Management Configuration Guide

Cisco APIC System Management Configuration Guide, Release 6.0(x)
First Published: 2022-06-17
Americas Headquarters
Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000
800 553-NETS (6387) Fax: 408 527-0883

© 2022 Cisco Systems, Inc. All rights reserved.

Trademarks
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS REFERENCED IN THIS DOCUMENTATION ARE SUBJECT TO CHANGE WITHOUT NOTICE. EXCEPT AS MAY OTHERWISE BE AGREED BY CISCO IN WRITING, ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS DOCUMENTATION ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
The Cisco End User License Agreement and any supplemental license terms govern your use of any Cisco software, including this product documentation, and are located at: http://www.cisco.com/go/softwareterms.Cisco product warranty information is available at http://www.cisco.com/go/warranty. US Federal Communications Commission Notices are found here http://www.cisco.com/c/en/us/products/us-fcc-notice.html.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any products and features described herein as in development or available at a future date remain in varying stages of development and will be offered on a when-and if-available basis. Any such product or feature roadmaps are subject to change at the sole discretion of Cisco and Cisco will have no liability for delay in the delivery or failure to deliver any products or feature roadmap items that may be set forth in this document.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)
Cisco APIC System Management Configuration Guide, Release 6.0(x) iii

Trademarks

Trademarks

Cisco APIC System Management Configuration Guide, Release 6.0(x) iv

CONTENTS

PREFACE CHAPTER 1 CHAPTER 2
CHAPTER 3

Trademarks iii
New and Changed Information 1 New and Changed Information 1
Alias, Annotations, and Tags 3 Alias, Annotations, and Tags 3 Alias 3 Creating a Name Alias or Global Alias 4 Annotations 5 Creating an Annotation 5 Policy Tags 6 Creating a Policy Tag 6
Precision Time Protocol 7 About PTP 7 PTP Clock Types 8 PTP Topology 10 Master and Client Ports 10 Passive Ports 11 Announce Messages 12 PTP Topology With Various PTP Node Types 14 PTP Topology With Only End-to-End Boundary Clocks 14 PTP Topology With a Boundary Clock and End-to-End Transparent Clocks 14 PTP BMCA 15 PTP BMCA Parameters 15

Cisco APIC System Management Configuration Guide, Release 6.0(x) v

Contents

PTP BMCA Examples 16 PTP BMCA Failover 18 PTP Alternate BMCA (G.8275.1) 20 PTP Alternate BMCA Parameters 20 PTP Alternate BMCA Examples 21 PTP Clock Synchronization 23 PTP and meanPathDelay 24 meanPathDelay Measurement 25 PTP Multicast, Unicast, and Mixed Mode 28 PTP Transport Protocol 30 PTP Signaling and Management Messages 31 PTP Management Messages 32 PTP Profiles 34 Cisco ACI and PTP 35 Cisco ACI Software and Hardware Requirements 37 Supported Software for PTP 37 Supported Hardware for PTP 38 PTP Connectivity 39 Supported PTP Node Connectivity 39 Supported PTP Interface Connectivity 40 Grandmaster Deployments 41 PTP Limitations 46 Configuring PTP 48 PTP Configuration Basic Flow 48 Configuring the PTP Policy Globally and For the Fabric Interfaces Using the GUI 48 Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy
Group Using the GUI 49 Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the GUI 50 Enabling PTP on EPG Static Ports Using the GUI 50 Enabling PTP on L3Out Interfaces Using the GUI 51 Configuring the PTP Policy Globally and For the Fabric Interfaces Using the REST API 51 Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy
Group Using the REST API 52 Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the REST API 53

Cisco APIC System Management Configuration Guide, Release 6.0(x) vi

Contents

CHAPTER 4
CHAPTER 5 CHAPTER 6 CHAPTER 7

Enabling PTP on EPG Static Ports Using the REST API 53 Enabling PTP on L3Out Interfaces Using the REST API 54 PTP Unicast, Multicast, and Mixed Mode on Cisco ACI 55 PTP Unicast Mode Limitations on Cisco ACI 55 PTP PC and vPC Implementation on Cisco ACI 56 PTP Packet Filtering and Tunneling 57 PTP Packet Filtering 57 Cisco ACI As a PTP Boundary Clock or PTP-Unaware Tunnel 58 PTP and NTP 60 PTP Verification 61
Synchronous Ethernet (SyncE) 67 About Synchronous Ethernet (SyncE) 67 Guidelines and Limitations for SyncE 68 Configuring Synchronous Ethernet 69 Creating a Synchronous Ethernet Node Policy 69 Creating a Synchronous Ethernet Interface Policy 70 QL Mapping with ACI Configuration Options 72
HTTP/HTTPS Proxy Policy 77 About the HTTP/HTTPS Proxy Policy 77 Cisco APIC Features That Use the HTTP/HTTPS Proxy 77 Configuring the HTTP/HTTPS Proxy Policy Using the GUI 78
Process Statistics 79 Viewing the Statistics for Processes Using the GUI 79 Configuring the Statistics Policy for All Processes for the First Time Using the GUI 81 Configuring the Statistics Policy for All Processes After Configuring the Policy the First Time Using the GUI 82
Basic Operations 85 Troubleshooting APIC Crash Scenarios 85 Cluster Troubleshooting Scenarios 85 Cluster Faults 88

Cisco APIC System Management Configuration Guide, Release 6.0(x) vii

Contents

Troubleshooting Fabric Node and Process Crash 90 APIC Process Crash Verification and Restart 91 Troubleshooting an APIC Process Crash 93 Cisco APIC Troubleshooting Operations 95 Shutting Down the APIC System 95 Shutting Down the APIC Controller Using the GUI 95 Using the APIC Reload Option Using the GUI 96 Controlling the LED Locator Using the GUI 96 Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI 97 Manually Removing Disabled Interfaces and Decommissioned Switches from the GUI 97 Decommissioning and Recommissioning Switches 97 Decommissioning and Recommissioning Switches 97 Performing a Rebuild of the Fabric 98 Rebuilding the Fabric 98 Recovering a Disconnected Leaf 100 Recovering a Disconnected Leaf Using the REST API 100 Troubleshooting a Loopback Failure 100 Identifying a Failed Line Card 100 Removing Unwanted ui Objects 102 Removing Unwanted ui Objects Using the REST API 103 Cisco APIC SSD Replacement 103 Replacing the Solid-State Drive in Cisco APIC 103 Viewing CRC Error Counters 105 Viewing CRC and Stomped CRC Error Counters 105 Viewing CRC Errors Using the GUI 105 Viewing CRC Errors Using the CLI 105

Cisco APIC System Management Configuration Guide, Release 6.0(x) viii

1 C H A P T E R

New and Changed Information

· New and Changed Information, on page 1

New and Changed Information

The following table provides an overview of the significant changes to the organization and features in this guide up to the specified release. The table does not provide an exhaustive list of all changes made to the guide or of the new features up to that release.
Table 1: New Features and Changed Behavior in Cisco APIC Release 6.0(1)

Feature or Change

Description

Where Documented

Support for PTP G.8275.1 on remote leaf switch peer links and on vPCs

You can use the PTP Telecom Supported PTP Interface profile (G.8275.1) on virtual port Connectivity, on page 40 channels (vPCs) and on remote leaf switch peer links.

Support for SyncE on vPCs and on You can use SyncE on vPCs and Guidelines and Limitations for

remote leaf switch peer links

on remote leaf switch peer links. SyncE, on page 68

Cisco APIC System Management Configuration Guide, Release 6.0(x) 1

New and Changed Information

New and Changed Information

Cisco APIC System Management Configuration Guide, Release 6.0(x) 2

2 C H A P T E R

Alias, Annotations, and Tags

· Alias, Annotations, and Tags, on page 3
Alias, Annotations, and Tags
To simplify the identifying, addressing, and grouping of objects, ACI provides several methods for the user to add label metadata to objects. These methods are summarized in the list below:
· Name Alias: A cosmetic substitute for a GUI entity. · Global Alias: A label, unique within the fabric, that can serve as a substitute for an object’s Distinguished
Name (DN). · Tag Instance / Annotation: A simple note or description. · Policy Tag: A label for grouping of objects, which need not be of the same class.

Alias

In the ACI object model, every object has a unique Distinguished Name (DN), which is an often lengthy identifier that includes the names of its parent object hierarchy and itself. For example, consider a tenant named Tenant2468 that contains an application profile named ap13, which contains an application endpoint group named aepg35. The DN of that application endpoint group, generated by APIC, is uni/tn-Tenant2468/ap-ap13/epg-aepg35. After each of these objects is created, ACI typically does not allow their names to be changed, as that would cause a change in the DNs of all descendant objects of the renamed object. To overcome this inconvenience, ACI provides two alias functions — Name Alias for the GUI and Global Alias for the API.
Name Alias
The Name Alias feature (or simply “Alias” where the setting appears in the GUI) changes the displayed name of objects in the APIC GUI. While the underlying object name cannot be changed, the administrator can override the displayed name by entering the desired name in the Alias field of the object properties menu. In the GUI, the alias name then appears along with the actual object name in parentheses, as name_alias (object_name). Many object types, such as tenants, application profiles, bridge domains, and EPGs, support the alias property. In the object model, the name alias property is objectClass.nameAlias. The property for a tenant object, for example, is fvTenant.nameAlias.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 3

Creating a Name Alias or Global Alias

Alias, Annotations, and Tags

Using the preceding example of a tenant, suppose the administrator prefers to see the tenant name “AcmeManufacturing” instead of “Tenant2468.” By entering the preferred name in the Alias field of the Tenant2468 tenant properties, the GUI would now display AcmeManufacturing (Tenant2468). The name alias property is purely cosmetic for the APIC GUI. The alias need not be unique in any scope, and the same value can be used as name alias for other objects.
Global Alias The Global Alias feature simplifies querying a specific object in the API. When querying an object, you must specify a unique object identifier, which is typically the object’s DN. As an alternative, this feature allows you to assign to an object a label that is unique within the fabric. Using the preceding example, without a global alias, you would query the application endpoint by its DN using this API request:
GET: https://APIC_IP/api/mo/uni/tn-Tenant2468/ap-ap13/epg-aepg35.json
By configuring a simpler yet unique name in the Global Alias field of the object properties menu, you can use the global alias along with a different API command to query the object:
GET: https://APIC_IP/api/alias/global_alias.json
Using the preceding example, by entering “AcmeEPG35” in the Global Alias field of the application endpoint group’s configuration properties, the query URL would now be:
GET: https://APIC_IP/api/alias/AcmeEPG35.json
In the APIC object model, the global alias is a child object (tagAliasInst) attached to the object that is being aliased. In the preceding example, the global alias object would be a child object of the application endpoint group object. For additional information, see the “Tags and Aliases” chapter of the APIC REST API Configuration Guide.
Creating a Name Alias or Global Alias
This example procedure shows you how to create a name alias and a global alias for an application profile of a tenant. Many other objects support these alias features using the same procedure after navigating to the object.

Step 1 Step 2 Step 3
Step 4
Step 5

On the menu bar, choose Tenants and select the applicable Tenant. In the Navigation pane, expand tenant_name > Application Profiles > application_profile_name. In the Work pane, click the Policy tab. The Properties page of the application profile appears.
In the Alias field, enter a name alias. The alias need not be unique in any scope.
In the Global Alias field, enter an alias for the distinguished name (DN) of the application profile. The global alias must be unique within the fabric.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 4

Alias, Annotations, and Tags

Annotations

Step 6 Click Submit.

If you configured a name alias, the application profile is now identified in the Navigation pane as alias (name). For example, if the Name is ap1234 and you configured an Alias as SanJose, the application profile appears as SanJose (ap1234).
If you configured a global alias, you can now substitute that value for the distinguished name (DN) of the application profile in API commands that support the global alias.
Annotations
You can add arbitrary key:value pairs of metadata to an object as annotations (tagAnnotation). Annotations are provided for the user’s custom purposes, such as descriptions, markers for personal scripting or API calls, or flags for monitoring tools or orchestration applications such as Cisco Multi-Site Orchestrator (MSO). Because APIC ignores these annotations and merely stores them with other object data, there are no format or content restrictions imposed by APIC.
Evolution of Annotations
APIC support for user-defined annotation information has changed over time in the following steps:
· Prior to Cisco APIC Release 4.2(4), APIC supported tag instances (tagInst), which stored a simple string. In APIC GUI menus, these were labeled as “Tags.”
· In Cisco APIC Release 4.2(4), because many modern systems use a key and value pair as a label, changes were made to move to key:value annotations (tagAnnotation) as the main label option for API. The shortcut API to query objects via tag instances (/api/tag/your_tag.json) was deprecated. The APIC GUI continued to use the simple string tag instances (tagInst), labeled as “Tags.”.
· In Cisco APIC Release 5.1(1), tag instances (tagInst) were deprecated in the GUI. GUI menus still used the term “Tags,” but actually configured annotations (tagAnnotation). Also beginning with this release, a list of all annotations can be viewed from Fabric > Fabric policies > Tags.
· In Cisco APIC release 5.2(1), GUI menu labels were changed from “Tags” to “Annotations.” This change was made to avoid confusion with Policy Tags.
Creating an Annotation
This example procedure shows you how to create an annotation for a tenant. Many other objects support the annotation feature using the same procedure after navigating to the object.

Step 1 Step 2 Step 3
Step 4 Step 5 Step 6

On the menu bar, choose Tenants and select the applicable Tenant. In the Navigation pane, click the tenant_name. In the Work pane, click the Policy tab. The properties menu of the tenant appears.
Next to Annotations, click the + symbol to add a new annotation. In the annotation key box, choose an existing key or type a new key. In the annotation value box, type a value.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 5

Policy Tags

Alias, Annotations, and Tags

Step 7

Allowed characters for key and value are a-z, A-Z, 0-9, period, colon, dash, or underscore. Click the symbol to save the annotation. You can add more annotations by repeating these steps.

Policy Tags
Policy tags (tagTag), or simply tags, are user-definable key and value pairs for use by ACI features. You can configure multiple tags on a single object, and you can apply the same tag on multiple objects. Because many object classes support policy tags, you can use policy tags to group disparate objects. For example, a policy tag can be used to group endpoints, subnets, and VMs together as one Endpoint Security Group (ESG) using ESG tag selectors in Cisco APIC Release 5.2(1). ACI features using policy tags include:
· Endpoint Security Group (ESG)
Creating a Policy Tag
This example procedure shows you how to create a policy tag for a static endpoint. Several other objects support policy tags using the same procedure after navigating to the object.

Step 1 Step 2 Step 3
Step 4 Step 5 Step 6
Step 7

On the menu bar, choose Tenants and select the applicable Tenant. In the Navigation pane, expand tenant_name > Application Profiles > application_profile_name > Application EPGs > application_epg_name > Static Endpoint. In the Work pane, double-click the static endpoint to be tagged. The Static Endpoint properties dialog box appears.
Next to Policy Tags, click the + symbol to add a new policy tag. In the tag key box, choose an existing key or type a new key. In the tag value box, type a tag value. Allowed characters for key and value are a-z, A-Z, 0-9, period, colon, dash, or underscore.
Click the symbol to save the tag.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 6

3 C H A P T E R
Precision Time Protocol
· About PTP, on page 7 · Cisco ACI and PTP, on page 35
About PTP
The Precision Time Protocol (PTP) is a time synchronization protocol defined in IEEE 1588 for nodes distributed across a network. With PTP, you can synchronize distributed clocks with an accuracy of less than 1 microsecond using Ethernet networks. PTP’s accuracy comes from the hardware support for PTP in the Cisco Application Centric Infrastructure (ACI) fabric spine and leaf switches. The hardware support allows the protocol to compensate accurately for message delays and variation across the network.
Note This document uses the term “client” for what the IEEE1588-2008 standard refers to as the “slave.” The exception is instances in which the word “slave” is embedded in the Cisco Application Policy Infrastructure Controller (APIC) CLI commands or GUI.
PTP is a distributed protocol that specifies how real-time PTP clocks in the system synchronize with each other. These clocks are organized into a master- client synchronization hierarchy with the grandmaster clock, which is the clock at the top of the hierarchy, determining the reference time for the entire system. Synchronization is achieved by exchanging PTP timing messages, with the members using the timing information to adjust their clocks to the time of their master in the hierarchy. PTP operates within a logical scope called a PTP domain. The PTP process consists of two phases: establishing the master-client hierarchy and synchronizing the clocks. Within a PTP domain, each port of an ordinary or boundary clock uses the following process to determine its state: 1. Establish the master-client hierarchy using the Best Master Clock Algorithm (BMCA):
· Examine the contents of all received Announce messages (issued by ports in the master state). · Compare the data sets of the foreign master (in the Announce message) and the local clock for priority,
clock class, accuracy, and so on. · Determine its own state as either master or client.
2. Synchronize the clocks:
Cisco APIC System Management Configuration Guide, Release 6.0(x) 7

PTP Clock Types

Precision Time Protocol

· Use messages, such as Sync and Delay_Req, to synchronize the clock between the master and clients.
PTP Clock Types
The following illustration shows the hierarchy of the PTP clock types:

PTP has the following clock types:

Type

Description

Grandmaster Clock (GM, GMC)

The source of time for the entire PTP topology. The grandmaster clock is selected by the Best Master Clock Algorithm (BMCA).

Cisco APIC System Management Configuration Guide, Release 6.0(x) 8

Precision Time Protocol

PTP Clock Types

Type Boundary Clock (BC) Transparent Clock (TC)
Ordinary Clock (OC)

Description
A device with multiple PTP ports. A PTP boundary clock participates in the BMCA and each port has a status, such as master or client. A boundary clock synchronizes with its parent/master so that the client clocks behind itself synchronize to the PTP boundary clock itself. To ensure that, a boundary clock terminates PTP messages and replies by itself instead of forwarding the messages. This eliminates the delay caused by the node forwarding PTP messages from one port to another.
A device with multiple PTP ports. A PTP transparent clock does not participate in the BMCA. This clock type only transparently forwards PTP messages between the master clock and client clocks so that they can synchronize directly with one another. A transparent clock appends the residence time to the PTP messages passing by so that the clients can take the forwarding delay within the transparent clock device into account.
In the case of a peer-to-peer delay mechanism, a PTP transparent clock terminates PTP Pdelay_xxx messages instead of forwarding the messages.
Note Switches in the ACI mode cannot be a transparent clock.
A device that may serve a source of time as a grandmaster clock or that may synchronize to another clock (such as a master) with the role as a client (a PTP client).

Cisco APIC System Management Configuration Guide, Release 6.0(x) 9

PTP Topology
PTP Topology
Master and Client Ports

Precision Time Protocol

The master and client ports work as follows: · Each PTP node directly or indirectly synchronizes its clock to the grandmaster clock that has the best source of time, such as GPS (Clock 1 in the figure). · One grandmaster is selected for the entire PTP topology (domain) based on the Best Master Clock Algorithm (BMCA). The BMCA is calculated on each PTP node individually, but the algorithm makes sure that all nodes in the same domain select the same clock as the grandmaster. · In each path between PTP nodes, based on the BMCA, there will be one master port and at least one client port. There will be multiple client ports if the path is point-to-multipoints, but each PTP node can have only one client port. Each PTP node uses its client port to synchronize to the master port on the other end. By repeating this, all PTP nodes eventually synchronize to the grandmaster directly or indirectly. · From Switch 1’s point of view, Clock 1 is the master and the grandmaster.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 10

Precision Time Protocol

Passive Ports

· From Switch 2’s point of view, Switch 1 is the master and Clock 1 is the grandmaster.
· Each PTP node should have only one client port, behind which exists the grandmaster. The grandmaster can be multiple hops away.
· The exception is a PTP transparent clock, which does not participate in BMCA. If Switch 3 was a PTP transparent clock, the clock would not have a port status, such as master and client. Clock 3, Clock 4, and Switch 1 would establish a master and client relationship directly.

Passive Ports

The BMCA can select another PTP port that is in the passive state on top of the master and client. A passive port does not generate any PTP messages, with a few exceptions such as PTP Management messages as a response to Management messages from other nodes.
Example 1

If a PTP node has multiple ports towards the grandmaster, only one of them will be the client port. The other ports toward the grandmaster will be passive ports.
Example 2

Cisco APIC System Management Configuration Guide, Release 6.0(x) 11

Announce Messages

Precision Time Protocol

If a PTP node detects two master only clocks (grandmaster candidates), the port toward the candidate selected as the grandmaster becomes a client port and the other becomes a passive port. If the other clock can be a client, it forms a master and client relation instead of passive.
Example 3

If a master-only clock (grandmaster candidate) detects another master-only clock that is better than itself, the clock puts itself in a passive state. This happens when two grandmaster candidates are on the same communication path without a PTP boundary clock in between.
Announce Messages
The Announce message is used to calculate the Best Master Clock Algorithm (BMCA) and establish the PTP topology (master-client hierarchy).

Cisco APIC System Management Configuration Guide, Release 6.0(x) 12

Precision Time Protocol

Announce Messages

The message works as follows: · PTP master ports send PTP Announce messages to IP address 224.0.1.129 in the case of PTP over IPv4 UDP. · Each node uses information in the PTP Announce messages to automatically establish the synchronization hierarchy (master/client relations or passive) based on the BMCA. · Some of the information that PTP Announce messages contain is as follows: · Grandmaster priority 1 · Grandmaster clock quality (class, accuracy, variance) · Grandmaster priority 2 · Grandmaster identity · Step removed · PTP Announce messages are sent with an interval based on 2logAnnounceInterval seconds.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 13

PTP Topology With Various PTP Node Types

Precision Time Protocol

PTP Topology With Various PTP Node Types
PTP Topology With Only End-to-End Boundary Clocks
In this topology, the boundary clock nodes terminate all multicast PTP messages, except for Management messages.

This ensures that each node processes the Sync messages from the closest parent master clock, which helps the nodes to achieve high accuracy.
PTP Topology With a Boundary Clock and End-to-End Transparent Clocks
In this topology, the boundary clock nodes terminate all multicast PTP messages, except for Management messages.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 14

Precision Time Protocol

PTP BMCA

End-to-end (E2E) transparent clock nodes do not terminate PTP messages, but simply add a residence time (the time the packet took to go through the node) in the PTP message correction field as the packets pass by so that clients can use them to achieve better accuracy. But, this has lower scalability as the number of PTP messages that need to be handled by one boundary clock node increases.

PTP BMCA

PTP BMCA Parameters

Each clock has the following parameters defined in IEEE 1588-2008 that are used in the Best Master Clock Algorithm (BMCA):

Order

Parameter

Possible Values

Description

1

Priority 1

0 to 255

A user configurable number. The value is normally 128 or lower for grandmaster-candidate clocks (master-capable devices) and 255 for client-only devices.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 15

PTP BMCA Examples

Precision Time Protocol

Order 2
3 4 5 6 7

Parameter Clock Quality – Class
Clock Quality Accuracy Clock Quality Variance Priority 2
Clock Identity
Steps Removed

Possible Values

Description

0 to 255

Represents the status of the clock devices. For example, 6 is for devices with a primary reference time source, such as GPS. 7 is for devices that used to have a primary reference time source. 127 or lower are for master-only clocks (grandmaster candidates). 255 is for client-only devices.

0 to 255

The accuracy of the clock. For example, 33 (0x21) means < 100 ns, while 35 (0x23) means < 1 us.

0 to 65535

The precision of the timestamps encapsulated in the PTP messages.

0 to 255

Another user-configurable number. This parameter is typically used when the setup has two grandmaster candidates with identical clock quality and one is a standby.

This is an 8-byte value This parameter serves as the final tie that is typically formed breaker, and is typically a MAC address. using a MAC address

Not configurable

This parameter represents the number of hops from the announced clock and is the last tie breaker when receiving the clock of the same grandmaster from two different ports. If the steps removed is the same for the candidates, the port ID and number is used as a tiebreaker.
You cannot configure the value of this parameter.

These parameters of the grandmaster clock are carried by the PTP Announce messages. Each PTP node compares these values in the order as listed in the table from all Announce messages that the node receives and also the node’s own values. For all parameters, the lower number wins. Each PTP node will then create Announce messages using the parameters of the best clock among the ones the node is aware of, and the node will send the messages from its own master ports to the next client devices.

Note For more information about each parameter, see clause 7.6 in IEEE 1588-2008.
PTP BMCA Examples
In the following example, Clock 1 and Clock 4 are the grandmaster candidates for this PTP domain:

Cisco APIC System Management Configuration Guide, Release 6.0(x) 16

Precision Time Protocol

PTP BMCA Examples

Clock 1 has the following parameter values: Parameter Priority 1 Clock Quality – Class Clock Quality – Accuracy Clock Quality – Variance Priority 2 Clock Identity Step Removed Clock 4 has the following parameter values:

Value 127 6 0x21 (< 100ns) 15652 128 0000.1111.1111 *

Cisco APIC System Management Configuration Guide, Release 6.0(x) 17

PTP BMCA Failover

Precision Time Protocol

Parameter Priority 1 Clock Quality – Class Clock Quality – Accuracy Clock Quality – Variance Priority 2 Clock Identity Step Removed

Value 127 6 0x21 (< 100ns) 15652 129 0000.1111.2222 *

Both clocks send PTP Announce messages, then each PTP node compares the values in the messages. In this example, because the first four parameters have the same value, Priority 2 decides the active grandmaster, which is Clock 1.
After all switches (1, 2, and 3) have recognized that Clock 1 is the best master clock (that is, Clock 1 is the grandmaster), those switches send PTP Announce messages with the parameters of Clock 1 from their master ports. On Switch 3, the port connected to Clock 4 (a grandmaster candidate) becomes a passive port because the port is receiving PTP Announce messages from a master-only clock (class 6) with parameters that are not better than the current grandmaster that is received by another port.
The Step Removed parameter indicates the number of hops (PTP boundary clock nodes) from the grandmaster. When a PTP boundary clock node sends PTP Announce messages, it increments the Step Removed value by 1 in the message. In this example, Switch 2 receives the PTP Announce message from Switch 1 with the parameters of Clock 1 and a Step Removed value of 1. Clock 2 receives the PTP Announce message with a Step Removed value of 2. This value is used only when all the other parameters in the PTP Announce messages are the same, which happens when the messages are from the same grandmaster candidate clock.

PTP BMCA Failover
If the current active grandmaster (Clock 1) becomes unavailable, each PTP port recalculates the Best Master Clock Algorithm (BMCA).

Cisco APIC System Management Configuration Guide, Release 6.0(x) 18

Precision Time Protocol

PTP BMCA Failover

The availability is checked using the Announce messages. Each PTP port declares the timeout of the Announce messages after the Announce messages were consecutively missing for Announce Receipt Timeout times. In other words, for Announce Receipt Timeout x 2logAnnounceInterval seconds. This timeout period should be uniform throughout a PTP domain as mentioned in Clause 7.7.3 in IEEE 1588-2008. When the timeout is detected, each switch starts recalculating the BMCA on all PTP ports by sending Announce messages with the new best master clock data. The recalculation can result in a switch initially determining that the switch itself is the best master clock, because most of the switches are aware of only the previous grandmaster.
When the client port connected toward the grandmaster goes down, the node (or the ports) does not need to wait for the announce timeout and can immediately start re-calculating the BMCA by sending Announce messages with the new best master clock data.
The convergence can take several seconds or more depending on the size of the topology, because each PTP port recalculates the BMCA from the beginning individually to find the new best clock. Prior to the failure of the active grandmaster, only Switch 3 knows about Clock 4, which should take over the active grandmaster role.
Also, when the port status changes to master from non-master, the port changes to the PRE_MASTER status first. The port takes Qualification Timeout seconds for the port to become the actual master, which is typically equal to:
Cisco APIC System Management Configuration Guide, Release 6.0(x) 19

PTP Alternate BMCA (G.8275.1)

Precision Time Protocol

(Step Removed + 1) x the announce interval
This means that if the other grandmaster candidate is connected to the same switch as (or close to) the active grandmaster, the port status changes will be minimum and the convergence time will be shorter. See Clause 9.2 in IEEE 1588-2008 for details.

PTP Alternate BMCA (G.8275.1)
The PTP Telecom profile (G.8275.1) uses the alternate Best Master Clock Algorithm (BMCA) defined in G.8275.1, which has a different algorithm than the regular BMCA defined in IEEE 1588-2008. One of the biggest differences is that if there are two grandmaster candidates with the same quality, the alternate BMCA from G.8275.1 allows each PTP node to pick the closest grandmaster instead of forcing all PTP nodes to pick the same clock as the grandmaster by comparing Steps Removed before Clock Identity. Another difference is the new parameter Local Priority, which provides users with manual control over which port to be preferred as the client port. This makes it easier to select the same port as the source for both the PTP Telecom profile and SyncE on each node, which is often preferred for the hybrid mode operation.

PTP Alternate BMCA Parameters

Each clock has the following parameters defined in G.8275.1 that are used in the alternate Best Master Clock Algorithm (BMCA) for the PTP Telecom profile (G.8275.1):

Order

Parameter

Possible Values

Description

1

Clock Quality – Class 0 to 255

Represents the status of the clock devices. For example, 6 is for devices with a primary reference time source, such as GPS. 7 is for devices that used to have a primary reference time source. 127 and lower are for master-only clocks (grandmaster candidates). 255 is for client-only devices.

2

Clock Quality –

0 to 255

Accuracy

The accuracy of the clock. For example, 33 (0x21) means < 100 ns, while 35 (0x23) means < 1 us.

3

Clock Quality –

0 to 65535

Variance

The precision of the timestamps encapsulated in the PTP messages.

4

Priority 2

0 to 255

A user-configurable number. This parameter is typically used when the setup has two grandmaster candidates with identical clock quality and one is a standby.

5

Local Priority

1 to 255

The clock of the node itself uses the clock local priority configured on the node. A clock received from another node is given the local priority configured for incoming port.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 20

Precision Time Protocol

PTP Alternate BMCA Examples

Order 6

Parameter Steps Removed

7

Clock Identity

8

Steps Removed

Possible Values

Description

Not configurable

This parameter represents the number of hops from the announced clock. The comparison of this allows each Telecom boundary clock to synchronize with a different and closer grandmaster when there are multiple active grandmaster candidates. If the steps removed is the same for the candidates, the port ID and number is used as a tiebreaker.
This comparison is performed only when the Clock Quality – Class value is 127 or less, which indicates that the clock is a grandmaster candidate.

This is an 8-byte value that is typically formed using a MAC address

This parameter serves as the tie breaker when the Clock Quality – Class value is greater than 127, which indicates that the quality of the clock is not designed to be a grandmaster. The value is typically a MAC address.

Not configurable

This parameter represents the number of hops from the announced clock and is the last tie breaker when receiving the clock of the same grandmaster from two different ports. If the steps removed is the same for the candidates, the port ID and number is used as a tiebreaker.

These parameters of the grandmaster clock, except for Local Priority, are carried by the PTP Announce messages. Each PTP node compares these values in the order as listed in the table from all Announce messages that the node receives and also the node’s own values. For all parameters, the lower number wins. Each PTP node will then create Announce messages using the parameters of the best clock among the ones the node is aware of, and the node will send the messages from its own master ports to the next client devices.

Note For more information about each parameter, see clause 6.3 in G.8275.1.
PTP Alternate BMCA Examples
In the following example, Clock 1 and Clock 4 are the grandmaster candidates for this PTP domain with the same quality and priority:

Cisco APIC System Management Configuration Guide, Release 6.0(x) 21

PTP Alternate BMCA Examples

Precision Time Protocol

Clock 1 has the following parameter values: Parameter Clock Quality – Class Clock Quality – Accuracy Clock Quality – Variance Priority 2 Steps Removed Clock Identity Clock 4 has the following parameter values:

Value 6 0x21 (< 100ns) 15652 128 * 0000.1111.1111

Cisco APIC System Management Configuration Guide, Release 6.0(x) 22

Precision Time Protocol

PTP Clock Synchronization

Parameter Clock Quality – Class Clock Quality – Accuracy Clock Quality – Variance Priority 2 Steps Removed Clock Identity

Value 6 0x21 (< 100ns) 15652 128 * 0000.1111.2222

Both Clock 1 and Clock 4 send PTP Announce messages, then each PTP node compares the values in the messages. Because the values for the Clock Quality – Class through Priority 2 parameters are the same, Steps Removed decides the active grandmaster for each PTP node.
For Switch 1 and 2, Clock 1 is the grandmaster. For Switch 3, Clock 4 is the grandmaster.

PTP Clock Synchronization
The PTP master ports send PTP Sync and Follow_Up messages to IP address 224.0.1.129 in the case of PTP over IPv4 UDP.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 23

PTP and meanPathDelay

Precision Time Protocol

In One-Step mode, the Sync messages carry the timestamp of when the message was sent out. Follow_Up messages are not required. In Two-Step mode, Sync messages are sent out without a timestamp. Follow_Up messages are sent out immediately after each Sync message with the timestamp of when the Sync message was sent out. Client nodes use the timestamp in the Sync or Follow_Up messages to synchronize their clock along with an offset calculated by meanPathDealy. Sync messages are sent with the interval based on 2logSyncInterval seconds.
PTP and meanPathDelay
meanPathDelay is the mean time that PTP packets take to reach from one end of the PTP path to the other end. In the case of the E2E delay mechanism, this is the time taken to travel between a PTP master port and a client port. PTP needs to calculate meanPathDelay (t in the following illustration) to keep the synchronized time on each of the distributed devices accurate.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 24

Precision Time Protocol

meanPathDelay Measurement

There are two mechanisms to calculate meanPathDelay: · Delay Request-Response (E2E): End-to-end transparent clock nodes can only support this. · Peer Delay Request-Response (P2P): Peer-to-peer transparent clock nodes can only support this.
Boundary clock nodes can support both mechanisms by definition. In IEEE 1588-2008, the delay mechanisms are called “Delay” or “Peer Delay.” However, the Delay Request-Response mechanism is more commonly referred to as the “E2E delay mechanism,” and the Peer Delay mechanism is more commonly referred to as the “P2P delay mechanism.”
meanPathDelay Measurement
Delay Request-Response The delay request-response (E2E) mechanism is initiated by a client port and the meanPathDelay is measured on the client node side. The mechanism uses Sync and Follow_Up messages, which are sent from a master port regardless of the E2E delay mechanism. The meanPathDelay value is calculated based on 4 timestamps from 4 messages.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 25

meanPathDelay Measurement

Precision Time Protocol

t-ms (t2 ­ t1) is the delay for master to client direction. t-sm (t4 ­ t3) is the delay for client to master direction. meanPathDelay is calculated as follows:
(t-ms + t-sm) / 2
Sync is sent with the interval based on 2logSyncInterval sec. Delay_Req is sent with the interval based on 2logMinDelayReqInterval sec.
Note This example focuses on Two-Step mode. See Clause 9.5 from IEEE 1588-2008 for details on the transmission timing.
Peer Delay Request-Response The peer delay request-response (P2P) mechanism is initiated by both master and client port and the meanPathDelay is measured on the requester node side. meanPathDelay is calculated based on 4 timestamps from 3 messages dedicated for this delay mechanism.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 26

Precision Time Protocol

meanPathDelay Measurement

In the two-step mode, t2 and t3 are delivered to the requester in one of the following ways: · As (t3-t2) using Pdelay_Resp_Follow_Up · As t2 using Pdelay_Resp and as t3 using Pdelay_Resp_Follow_Up
meanPathDelay is calculated as follows:
(t4-t1) ­ (t3-t1) / 2
Pdelay_Req is sent with the interval based on 2logMinPDelayReqInterval seconds.
Note Cisco Application Centric Infrastructure (ACI) switches do not support the peer delay request-response (P2P) mechanism. See clause 9.5 from IEEE 1588-2008 for details on the transmission timing.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 27

PTP Multicast, Unicast, and Mixed Mode

Precision Time Protocol

PTP Multicast, Unicast, and Mixed Mode
The following sections describe the different PTP modes using the delay request-response (E2E delay) mechanism.
Multicast Mode
All PTP messages are multicast. Transparent clock or PTP unaware nodes between the master and clients result in inefficient flooding of the Delay messages. However, the flooding is efficient for Announce, Sync, and Follow_Up messages because these messages should be sent toward all client nodes.

Unicast Mode All PTP messages are unicast, which increases the number of messages that the master must generate. Hence, the scale, such as the number of client nodes behind one master port, is impacted.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 28

Precision Time Protocol

PTP Multicast, Unicast, and Mixed Mode

Mixed Mode Only Delay messages are unicast, which resolves the problems that exist in multicast mode and unicast mode.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 29

PTP Transport Protocol

Precision Time Protocol

PTP Transport Protocol
The following illustration provides information about the major transport protocols that PTP supports:
Cisco APIC System Management Configuration Guide, Release 6.0(x) 30

Precision Time Protocol

PTP Signaling and Management Messages

Note Cisco Application Centric Infrastructure (ACI) switches only support IPv4 and Ethernet as a PTP transport protocol.
PTP Signaling and Management Messages
The following illustration shows the Signaling and Management message parameters in the header packet for PTP over IPv4 UDP:
Cisco APIC System Management Configuration Guide, Release 6.0(x) 31

PTP Management Messages

Precision Time Protocol

A Management message is used to configure or collect PTP parameters, such as the current clock and offset from its master. With the message, a single PTP management node can manage and monitor PTP-related parameters without relying on an out-of-band monitoring system. A Signaling message also provides various types of type, length, and value (TLVs) to do additional operations. There are other TLVs that are used by being appended to other messages. For example, the PATH_TRACE TLV as defined in clause 16.2 of IEEE 1588-2008 is appended to Announce messages to trace the path of each boundary clock node in the PTP topology.
Note Cisco Application Centric Infrastructure (ACI) switches do not support management, signal, or other optional TLVs.
PTP Management Messages
PTP Management messages are used to transport management types, lengths, and values (TLVs) toward multiple PTP nodes at once or to a specific node.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 32

Precision Time Protocol

PTP Management Messages

The targets are specified with the targetPortIdentity (clockID and portNumber) parameter. PTP Management messages have the actionField that specify actions such as GET, SET, and COMMAND to inform the targets of what to do with the delivered management TLV. PTP Management messages are forwarded by the PTP boundary clock, and only to the Master, Client, Uncalibrated, or Pre_Master ports. A message is forwarded to those ports only when the message is received on a port in the Master, Client, Uncalibrated, or Pre_Master port. BoundaryHops in the message is decremented by 1 when the message is forwarded. The SMTPE ST2059-2 profile defines that the grandmaster should send PTP Management messages using the action COMMAND with the synchronization metadata TLV that is required for the synchronization of audio/video signals.
Note Cisco Application Centric Infrastructure (ACI) switches do not process Management messages, but forward them to support the SMTPE ST2059-2 PTP profile.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 33

PTP Profiles

Precision Time Protocol

PTP Profiles
The precision time protocol (PTP) has a concept called the PTP profile. A PTP profile is used to define various parameters that are optimized for different use cases of PTP. Some of those parameters include, but not limited to, the appropriate range of PTP message intervals and the PTP transport protocols. A PTP profile is defined by many organizations/standards in different industries. For example:
· IEEE 1588-2008: This standard defines a default PTP profile called the Default Profile.
· AES67-2015: This standard defines a PTP profile for audio requirements. This profile is also called the Media Profile.
· SMPTE ST2059-2: This standard defines a PTP profile for video requirements.
· ITU-T G.8275.1: Also known as the Telecom profile with Full Timing Support. This standard is recommended for telecommunications with Full Timing Support. Full Timing Support is the term defined by ITU to describe a telecommunication network that can provide devices with the PTP G.8275.1 profile on every hop. G.8275.2, which is not supported by Cisco Application Centric Infrastructure (ACI), is for Partial Timing Support that may have devices in the path that do not support PTP.
The telecommunication industry requires both frequency and time/phase synchronization. G.8275.1 is used to synchronize time and phase. The frequency can be synchronized either using PTP through the packet network with another PTP G.8265.1 profile, which is not supported by Cisco ACI, or using the physical layer such as the synchronous digital hierarchy (SDH), synchronous optical networking (SONET) through a dedicated circuit, or synchronous Ethernet (SyncE) through Ethernet. Synchronizing the frequency using SyncE and time/phase using PTP is called the hybrid mode.
The key differences of G.8275.1 compared to the other profiles are as follows:
· G.8275.1 uses the alternate BMCA with the additional parameter Local Priority that does not exist in the other profiles.
· G.8275.1 uses PTP over Ethernet with all PTP messages using the same destination MAC address (forwardable and non-forwardable), which you can choose.
· G.8275.1 expects the telecom boundary clock (T-BC) to follow the accuracy (maximum time error; max|TE|) defined by G.8273.2.
· Class A: 100 ns
· Class B: 70 ns
· Class C: 30 ns

The following table shows some of the parameters defined in each standard for each PTP profile:

Profiles

logAnnounce logSync logMinDelayReq announceReceipt Domain Mode Transport

Interval Interval Interval

Timeout

Number

Protocol

Default Profile 0 to 4 (1)
[= 1 to 16 sec]

-1 to +1 (0) 0 to 5 (0)

2 to 10

[= 0.5 to 2 [= 1 to 32 sec] sec]

announce intervals (3)

0 to 255 Multicast Any/IPv4

(0)

/ Unicast

Cisco APIC System Management Configuration Guide, Release 6.0(x) 34

Precision Time Protocol

Cisco ACI and PTP

Profiles

logAnnounce logSync logMinDelayReq announceReceipt Domain Mode Transport

Interval Interval Interval

Timeout

Number

Protocol

AES67-2015 (Media Profile)

0 to 4 (1)
[= 1 to 16 sec]

-4 to +1 (-3)
[= 1/16 to 2 sec]

-3 to +5 (0)
[= 1/8 to 32 sec] Or

2 to 10 announce intervals (3)

logSyncInterval to logSyncInterval + 5 seconds

0 to 255 Multicast UDP/IPv4

(0)

/ Unicast

SMTPE

-3 to +1 (-2) -7 to -1 logSyncInterval 2 to 10

ST2059-2-2015 [= 1/8 to 2 sec]

(-3)

to

announce

[= 1/128 to 0.5 sec]

logSyncInterval + 5 seconds

intervals (3)

0 to 127 Multicast UDP/IPv4 (127) / Unicast

ITU-T

-3

G.8275.1

-4

-4

2 to 4

24 to 43 Multicast Ethernet (24) Only

Cisco ACI and PTP
In the Cisco Application Centric Infrastructure (ACI) fabric, when the PTP feature is globally enabled in the Cisco Application Policy Infrastructure Controller (APIC), the software automatically enables PTP on specific interfaces of all the supported spine and leaf switches to establish the PTP master-client topology within the fabric. Starting in Cisco APIC release 4.2(5), you can enable PTP on leaf switch front panel ports and extend PTP topology to outside of the fabric. In the absence of an external grandmaster clock, one of the spine switch is chosen as the grandmaster. The master spine switch is given a different PTP priority that is lower by 1 than the other spines and leaf switches.
Implementation in Cisco APIC Release 3.0(1)
Starting in Cisco Application Policy Infrastructure Controller (APIC) release 3.0(1), PTP was partially introduced to synchronize time only within Cisco Application Centric Infrastructure (ACI) fabric switches. PTP was required to provide the latency measurement feature that was also introduced in Cisco APIC release 3.0(1). For this purpose, a single option was introduced to enable or disable PTP globally. When PTP is enabled globally, all leaf and spine switches are configured as PTP boundary clocks. PTP is automatically enabled on all fabric ports that are used by the ftag tree with ID 0 (ftag0 tree), which is one of the internal tree topologies that is automatically built based on Cisco ACI infra ISIS for loop-free multicast connectivity between all leaf and spine switches in each pod. The root spine switch of the ftag 0 tree is automatically configured with PTP priority1 254 to be the grandmaster when there are no external grandmasters in the inter-pod network (IPN). Other spine and leaf switches are configured with PTP priority1 255.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 35

Cisco ACI and PTP

Precision Time Protocol

In a Multi-Pod setup, PTP is automatically enabled on the sub-interfaces configured for IPN connectivity in the tn-infra Multi-Pod L3Out. In Cisco APIC release 3.0(1), this is the only way to enable PTP on external-facing interfaces. With this, it was mandatory to provide the same external grandmaster using IPN to all of the pods for the latency measurement feature to work in the case of Multi-Pod.

In Cisco APIC release 3.0(1), PTP cannot be enabled on any other interfaces on demand, such as the down link (front panel ports) on leaf switches.
Implementation in Cisco APIC Releases 4.2(5) and 5.1(1) Starting in Cisco APIC releases 4.2(5) and 5.1(1), you can enable PTP on a leaf switch’s front panel ports to connect the PTP nodes, clients, or grandmaster. The PTP implementation on fabric ports are still the same as
Cisco APIC System Management Configuration Guide, Release 6.0(x) 36

Precision Time Protocol

Cisco ACI Software and Hardware Requirements

the previous releases, except that the PTP parameters for fabric ports can now be adjusted. With this change, you can use the Cisco ACI fabric to propagate time synchronization using PTP with Cisco ACI switches as PTP boundary clock nodes. Prior to that, the only approach Cisco ACI had was to forward PTP multicast or unicast messages transparently as a PTP unaware switch from one leaf switch to another as a tunnel.

Note The 5.0(x) releases do not support the PTP functionality that was introduced in the 4.2(5) and 5.1(1) releases.
Cisco ACI Software and Hardware Requirements
Supported Software for PTP
The following feature is supported from Cisco Application Policy Infrastructure Controller (APIC) release 3.0(1):
· PTP only within the fabric for the latency measurement feature The following features are supported from Cisco APIC release 4.2(5):
· PTP with external devices by means of the leaf switches
Cisco APIC System Management Configuration Guide, Release 6.0(x) 37

Supported Hardware for PTP

Precision Time Protocol

· PTP on leaf switch front panel ports · Configurable PTP message intervals · Configurable PTP domain number · Configurable PTP priorities · PTP multicast port · PTP unicast master port on leaf switch front panel ports · PTP over IPv4/UDP · PTP profile (Default, AES67, and SMTPE ST2059-2)
The following features are supported from Cisco APIC release 5.2(1): · PTP multicast master-only ports · PTP over Ethernet · PTP Telecom profile with Full Timing Support (ITU-T G.8275.1)
Supported Hardware for PTP
Leaf switches, spine switches, and line cards with -EX or later in the product ID are supported, such as N9K-X9732C-EX or N9K-C93180YC-FX. The PTP Telecom profile (G.8275.1) is supported only on the Cisco N9K-C93180YC-FX3 switch. This switch supports Class B (G.8273.2) accuracy when used along with SyncE. The following leaf switches are not supported:
· N9K-C9332PQ · N9K-C9372PX · N9K-C9372PX-E · N9K-C9372TX · N9K-C9372TX-E · N9K-C9396PX · N9K-C9396TX · N9K-C93120TX · N9K-C93128TX
The following spine box switch is not supported: · N9K-C9336PQ
The following spine switch line card is not supported: · N9K-X9736PQ

Cisco APIC System Management Configuration Guide, Release 6.0(x) 38

Precision Time Protocol

PTP Connectivity

PTP Connectivity
Supported PTP Node Connectivity
External PTP nodes can be connected to the Cisco Application Centric Infrastructure (ACI) fabric using the following methods:
· Inter-pod network · EPG (on a leaf switch) · L3Out (on a leaf switch)

PTP is VRF-agnostic, the same as with a standalone NX-OS switch. All PTP messages are terminated, processed, and generated at the interface level on each Cisco ACI switch node as a PTP boundary clock. Regardless of the VRF, bridge domain, EPG, or VLAN, the Best Master Clock Algorithm (BMCA) is calculated across all of the interfaces on each Cisco ACI switch. There is only one PTP domain for the entire fabric. Any PTP nodes with the E2E delay mechanism (delay req-resp) can be connected to the Cisco ACI switches that are running as a PTP boundary clock.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 39

Supported PTP Interface Connectivity

Precision Time Protocol

Note Cisco ACI switches do not support the Peer Delay (P2P) mechanism. Therefore, a P2P transparent clock node cannot be connected to Cisco ACI switches.

Supported PTP Interface Connectivity

Connection Type

Interface Type

Leaf Switch Supported / Not Type (leaf, Supported remote leaf, (non-Telecom tier-2 leaf) profiles)

Supported / Not Supported (G.8275.1)

Fabric Link (between a leaf Sub-interface

–

and spine switch)

(non-PC)

Supported

Not supported

Fabric Link (between a Sub-interface

–

tier-1 and tier-2 leaf switch) (non-PC)

Supported

Not supported

Spine (toward an IPN)

Sub-interface

–

(non-PC)

Supported

Not supported

Remote leaf (toward an Sub-interface

–

IPN)

(non-PC)

Supported

Not supported

Remote leaf (peer link, back-to-back links)

Physical

–

Supported

Supported

Normal EPG (trunk, access, Physical, port channel, Any

802.1P)

vPC

Supported

Supported

L3Out (routed, routed-sub) Physical, port channel Any

Supported

Supported

L3Out (SVI ­ trunk, access, Physical, port channel, Any

802.1P)

vPC

Not supported

Not supported

L2Out (trunk)

Physical, port channel, Any vPC

Not supported

Not supported

EPG/L3Out in tn-mgmt Service EPG (trunk)1

Physical, port channel, Any vPC
Physical, port channel, Any vPC

Not supported

Not supported

Not supported

Not supported

Any type of FEX interface Any

Any

Not supported

Not supported

Breakout ports

Any

Any

Supported

Supported

Out-of-band management Physical interface

–

Not supported

Not supported

1 The service EPG is an internal EPG created for a Layer 4 to Layer 7 service graph.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 40

Precision Time Protocol

Grandmaster Deployments

Grandmaster Deployments
You can deploy the grandmaster candidates using one of the following methods:
Single Pod In a single pod deployment, grandmaster candidates can be deployed anywhere in the fabric (L3Out, EPG, or both). The Best Master Clock Algorithm (BMCA) picks one active grandmaster from among all of them.

Multipod With BMCA Across Pods Grandmaster candidates can be deployed anywhere in the fabric (an inter-pod network, L3Out, EPG, or all of them). The BMCA picks one active grandmaster from among all of them across pods. We recommend that you place your grandmasters on inter-pod networks (IPNs) so that the PTP clients in any pod have a similar number of hops to the active grandmaster. In addition, the master/client tree topology will not change drastically when the active grandmaster becomes unavailable.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 41

Grandmaster Deployments

Precision Time Protocol

Multipod With BMCA in Each Pod If you must have active grandmasters in each pod because PTP accuracy suffers too much degradation through an IPN domain, PTP messages must not traverse through an IPN across pods. You can accomplish this configuration in one of the following ways:
· Option 1: Ensure sub-interfaces are used between the IPN and spine switches, and disable PTP on the IPN.
· Option 2: If the PTP grandmaster is connected to the IPN in each pod, but the PTP topologies still must be separated, disable PTP on the IPN interfaces that are between the pods.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 42

Precision Time Protocol

Grandmaster Deployments

Remote Leaf Switch Remote leaf switch sites are typically not close to the main data center or to each other, and it is difficult to propagate PTP messages across each location with accurate measurements of delay and correction. Hence, we recommend that you prevent PTP messages from traversing each site (location) so that the PTP topology is established within each site (location). Some remote locations may be close to each other. In such a case, you can enable PTP between those IPNs to form one PTP topology across those locations. You can use the same options mentioned in Multipod With BMCA in Each Pod to prevent PTP message propagation.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 43

Grandmaster Deployments

Precision Time Protocol

Cisco ACI Multi-Site Each site is typically not close to each other, and it is difficult to propagate PTP messages across each site with accurate measurements of delay and correction. Hence, we recommend that you prevent PTP messages from traversing each site so that the PTP topology is established within each site. You can use the same options mentioned in Multipod With BMCA in Each Pod to prevent PTP message propagation. Also, Cisco ACI Multi-Site has no visibility or capability of configuring PTP.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 44

Precision Time Protocol

Grandmaster Deployments

Telecom Profile (G.8275.1) The PTP Telecom profile (G.8275.1) in Cisco Application Centric Infrastructure (ACI) requires SyncE to achieve class B (G.8273.2) accuracy. Also, both the PTP Telecom profile (G.8275.1) and SyncE are supported only on Cisco N9K-C93180YC-FX3 leaf nodes. As a result, spine nodes cannot be used to distribute time, phase, and frequency synchronization for the Telecom profile (G.8275.1). Because of this, fabric links on telecom leaf node (leaf nodes configured for G.8275.1) run in PTP multicast master only mode. This ensures that the telecom leaf nodes not to lock their clock through the spine nodes. This means that the grandmaster deployment for PTP Telecom profile (G.8275.1) in Cisco ACI requires each telecom leaf node to receive timing from the node’s respective down link ports.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 45

PTP Limitations

Precision Time Protocol

PTP Limitations
For general support and implementation information, see Supported Software for PTP, on page 37, Supported Hardware for PTP, on page 38, and PTP Connectivity, on page 39. The following limitations apply to PTP:
· Cisco Application Centric Infrastructure (ACI) leaf and spine switches can work as PTP boundary clocks. The switches cannot work as PTP transparent clocks.
· Only the E2E delay mechanism (delay request/response mechanism) is supported. The P2P delay mechanism is not supported.
· PTP over IPv4/UDP for the default/Media/SMPTE PTP profiles and PTP over Ethernet for the Telecom (G.8275.1) PTP profile are supported. PTP over IPv6 is not supported.
· Only PTPv2 is supported. · Although PTPv1 packets are still redirected to the CPU when PTP is enabled on any of the front panel ports on the leaf switch, the packets will be discarded on the CPU.
Cisco APIC System Management Configuration Guide, Release 6.0(x) 46

Precision Time Protocol

PTP Limitations

· PTP Management TLVs are not recognized by Cisco ACI switches, but they are still forwarded as defined in IEEE1588-2008 to support the SMTPE PTP profile.
· PTP cannot be used as the system clock of the Cisco ACI switches.
· PTP is not supported on Cisco Application Policy Infrastructure Controller (APIC).
· NTP is required for all switches in the fabric.
· PTP offload is not supported. This functionality is the offloading of the PTP packet processing to each line card CPU on a modular spine switch for higher scalability.
· Due to a hardware limitation, interfaces with 1G/100M speed have lower accuracy than 10G interfaces when there are traffic loads. In the 5.2(3) and later releases, this limitation does not apply to the Cisco N9K-C93108TC-FX3P switch for the 1G speed.
· PTP is not fully supported on 100M interfaces due to higher PTP offset corrections.
· The PTP Telecom profile (G.8275.1) is not supported on ports with 1G/10G speed.
· Sync and Delay_Request messages can support up to a -4 interval (1/16 seconds). Interval values of -5 to -7 are not supported.
· For leaf switch front panel ports, PTP can be enabled per interface and VLAN, but PTP is automatically enabled on all appropriate fabric links (interfaces between leaf and spine switches, tier-1 and tier-2 leaf switches, and interfaces toward the IPN/ISN) after PTP is enabled globally. The appropriate fabric links are the interfaces that belong to ftag0 tree.
· PTP must be enabled globally for leaf switch front panel interfaces to use PTP. This means that you cannot enable PTP on leaf switch front panel ports without enabling PTP on the fabric links.
· PTP configuration using tn-mgmt and tn-infra is not supported.
· PTP can be enabled only on one VLAN per interface.
· PTP cannot be enabled on the interface and the VLAN for an L3Out SVI. PTP can be enabled on another VLAN on the same interface using an EPG.
· Only the leaf switch front panel interfaces can be configured as unicast master ports. The interfaces cannot be configured as unicast client ports. Unicast ports are not supported on a spine switch.
· Unicast negotiation is not supported.
· Unicast mode does not work with a PC or vPC when the PC or vPC is connected to a device such as NX-OS, which configures PTP on individual member ports.
· PTP and MACsec should not be configured on the same interface.
· When PTP is globally enabled, to measure the latency of traffic traversing through the fabric, Cisco ACI adds Cisco timestamp tagging (TTag) to traffic going from one ACI switch node to another ACI switch node. This results in an additional 8 bytes for such traffic. Typically, users do not need to take any actions regarding this implementation because the TTag is removed when the packets are sent out to the outside of the ACI fabric. However, when the setup consists of Cisco ACI Multi-Pod, user traffic traversing across pods will keep the TTag in its inner header of the VXLAN. In such a case, increase the MTU size by 8 bytes on the ACI spine switch interfaces facing toward the Inter-Pod Network (IPN) along with all non-ACI devices in the IPN. IPN devices do not need to support nor be aware of the TTag, as the TTag is embedded inside of the VXLAN payload.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 47

Configuring PTP

Precision Time Protocol

· When PTP is globally enabled, ERSPAN traffic traversing through spine nodes to reach to the ERSPAN destination will have Cisco timestamp tagging (TTag) with ethertype 0x8988. There is no impact to the original user traffic.
· In the presence of leaf switches that do not support PTP, you must connect an external grandmaster to all of the spine switches using IPN or using leaf switches that support PTP. If a grandmaster is connected to one or a subset of spine switches, PTP messages from the spine may be blocked by the unsupported leaf switch before they reach other switches depending on the ftag0 tree status. PTP within leaf and spine switches are enabled based on ftag0 tree, which is automatically built based on Cisco ACI infra ISIS for loop free multicast connectivity between all leaf and spine switches in each pod.
· When the PTP Telecom profile is deployed, the Telecom grandmaster clock (T-GM) and Telecom boundary clock (T-BC) timestamps should be within 2 seconds for the T-BC to lock with the T-GM.
· You cannot enable PTP on a VLAN that is deployed on a leaf node interface using VMM domain integration.

Configuring PTP
PTP Configuration Basic Flow
The following steps provide an overview of the PTP configuration process:

Step 1 Step 2
Step 3 Step 4 Step 5

Enable PTP globally and set PTP parameters for all fabric interfaces. For the PTP Telecom profile (G.8275.1) only, create a PTP node policy and apply it to a switch profile through a switch policy group. Create PTP user profile for leaf front panel interfaces under Fabric > Access Policies > Policies > Global. Enable PTP under EPG > Static Ports with the PTP user profile. Enable PTP under L3Out > Logical Interface Profile > Routed or Sub-Interface with the PTP user profile.

Configuring the PTP Policy Globally and For the Fabric Interfaces Using the GUI
This procedure enables the precision time protocol (PTP) globally and for the fabric interfaces using the Cisco Application Policy Infrastructure Controller (APIC) GUI. When PTP is enabled globally, ongoing TEP to TEP latency measurements get enabled automatically.

Step 1 Step 2 Step 3

On the menu bar, choose System > System Settings. In the Navigation pane, choose PTP and Latency Measurement. In the Work pane, set the interface properties as appropriate for your desired configuration. At the least, you must set Precision Time Protocol to Enabled.
See the online help page for information about the fields. If any interval value that you specify is outside of the chosen PTP profile standard range, the configuration is rejected.
The PTP profile, intervals, and timeout fields apply to fabric links. The other fields apply to all of the leaf and spine switches.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 48

Precision Time Protocol

Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy Group Using the GUI

Step 4 Click Submit.
Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy Group Using the GUI
A PTP node policy is required for the leaf nodes to run PTP Telecom profile (G.8275.1) because it uses the Alternate BMCA with additional parameters. Also, the allowed range of the domain number, priority 1, and priority 2 are different from other PTP profiles. You can apply the PTP node policy to a leaf switch using a leaf switch profile and a policy group.

Note For media profile deployment, you do not need to create a node policy.

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9
Step 10
Step 11 Step 12
Step 13 Step 14 Step 15 Step 16

On the menu bar, choose Fabric > Access Policies. In the Navigation pane, choose Switches > Leaf Switches > Profiles. Right-click Profiles and choose Create Leaf Profile. In the Create Leaf Profile dialog, in the Name field, enter a name for the profile. In the Leaf Selectors section, click +. Enter a name, choose the switches, and choose to create a policy group. In the Create Access Switch Policy Group dialog, enter a name for the policy group. In the PTP Node Policy drop-down list, choose Create PTP Node Profile. In the Create PTP Node Profile dialog, set the values as desired for your configuration.
· Node Domain: The value must be between 24 and 43, inclusive. The Telecom leaf nodes that need to be in the same PTP topology should use the same domain number.
· Priority 1: The value must be 128.
· Priority 2: The value must be between 0 and 255, inclusive.
See the online help page for information about the fields.
Click Submit. The Create PTP Node Profile dialog closes.
In the Create Access Switch Policy Group dialog, set any other policies as desired for your configuration. Click Submit. The Create Access Switch Policy Group dialog closes.
In the Leaf Selectors section, click Update. Click Next. In the STEP 2 > Associations screen, associate the interface profiles as desired. Click Finish.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 49

Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the GUI

Precision Time Protocol

Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the GUI
This procedure creates the PTP user profile for leaf switch front panel ports using the Cisco Application Policy Infrastructure Controller (APIC) GUI. A PTP user profile is applied to the leaf switch front panel interfaces using an EPG or L3Out.
Before you begin You must enable PTP globally to use PTP on leaf switch front panel ports that face external devices.

Step 1 Step 2 Step 3 Step 4
Step 5

On the menu bar, choose Fabric > Access Policies. In the Navigation pane, choose Policies > Global > PTP User Profile. Right-click PTP User Profile and choose Create PTP User Profile. In the Create PTP User Profile dialog, set the values as desired for your configuration.
See the online help page for information about the fields. If any interval value that you specify is outside of the chosen PTP profile standard range, the configuration is rejected.
Click Submit.

Enabling PTP on EPG Static Ports Using the GUI
This procedure enables PTP on EPG static ports using the Cisco Application Policy Infrastructure Controller (APIC) GUI. You can enable PTP with multicast dynamic, multicast master, or unicast master mode.
Before you begin You must first create a PTP user profile for the leaf switch front panel ports and enable PTP globally.

Step 1 Step 2 Step 3 Step 4
Step 5

On the menu bar, choose Tenants > All Tenants. In the Work pane, double-click the tenant’s name. In the Navigation pane, choose Tenant tenant_name > Application Profiles > app_profile_name > Application EPGs > app_epg_name > Static Ports > static_port_name. In the Work pane, for the PTP State toggle, choose Enable. You might need to scroll down to see PTP State.
PTP-related fields appear.
Configure the PTP fields as required for your configuration.
· PTP Mode: Choose multicast dynamic, multicast master, or unicast master, as appropriate.
· PTP Source Address: PTP packets from this interface and VLAN are sent with the specified IP address as the source. The leaf switch TEP address is used by default or when you enter “0.0.0.0” as the value. This value is optional for multicast mode. Use the bridge domain SVI or EPG SVI for unicast mode. The source IP address must be reachable by the connected PTP node for unicast mode.
· PTP User Profile: Choose the PTP user profile that you created for the leaf switch front panel ports to specify the message intervals.
See the online help page for additional information about the fields.

Cisco APIC System Management Configuration Guide, Release 6.0(x) 50

Precision Time Protocol

Enabling PTP on L3Out Interfaces Using the GUI

Step 6

A node-level configuration takes precedence over the fabric-level configuration on a node where the PTP Telecom profile (G.8275.1) is deployed.
Click Submit.

Enabling PTP on L3Out Interfaces Using the GUI
This procedure enables PTP on L3Out interfaces using the Cisco Application Policy Infrastructure Controller (APIC) GUI. You can enable PTP with multicast dynamic, multicast master, or unicast master mode.
Before you begin You must first create a PTP user profile for the leaf switch front panel ports and enable PTP globally.

Step 1 Step 2 Step 3 Step 4 Step 5
Step 6

On the menu bar, choose Tenants > All Tenants. In the Work pane, double-click the tenant’s name. In the Navigation pane, choose Tenant tenant_name > Networking > L3Outs > l3out_name > Logical Node Profiles > node_profile_name > Logical Interface Profiles > interface_profile_name. In the Work pane, choose Policy > Routed Sub-Interfaces or Policy > Routed Interfaces, as appropriate. If you want to enable PTP on an existing L3Out, perform the following sub- steps: a) Double-click the desired interface to view its properties. b) Scroll down if necessary to find the PTP properties, set the PTP State to Enable, and enter the same values that you
used for the EPG static ports.
See the online help page for information about the fields.
c) Click Submit.
If you want to enable PTP on a new L3Out, perform the following sub-steps: a) Click + at the upper right of the table. b) In Step 1 > Identity, enter the appropriate values. c) In Step 2 > Configure PTP, set the PTP State to Enable, and enter the same values that you used for the EPG static
ports.
See the online help page for information about the fields.
d) Click Finish.

Configuring the PTP Policy Globally and For the Fabric Interfaces Using the REST API
This procedure enables PTP globally and for the fabric interfaces using the REST API. When PTP is enabled globally, ongoing TEP to TEP latency measurements get enabled automatically.
To configure the PTP policy globally and for the fabric interfaces, send a REST API POST similar to the following example:
POST: /api/mo/uni/fabric/ptpmode.xml

Cisco APIC System Management Configuration Guide, Release 6.0(x) 51

Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy Group Using the REST API

Precision Time Protocol

<latencyPtpMode state=”enabled” systemResolution=”11″ prio1=”255″ prio2=”255″ globalDomain=”0″ fabProfileTemplate=”aes67″ fabAnnounceIntvl=”1″ fabSyncIntvl=”-3″ fabDelayIntvl=”-2″ fabAnnounceTimeout=”3″
/>

PTP admin state # Latency Resolution (can be skipped for PTP) # Global

Priority1 # Global Priority2 # Global Domain # PTP Profile # Announce Interval (2^x sec) # Sync Interval (2^x sec) # Delay Request Interval (2^x sec) # Announce Timeout

Configuring a PTP Node Policy and Applying the Policy to a Switch Profile Using a Switch Policy Group Using the REST API
A PTP node policy is required for the leaf nodes to run PTP Telecom profile (G.8275.1) because it uses the Alternate BMCA with additional parameters. Also, the allowed range of the domain number, priority 1, and priority 2 are different from other PTP profiles. You can apply the PTP node policy to a leaf switch using a leaf switch profile and a policy group.
POST: /api/mo/uni.xml

Cisco APIC System Management Configuration Guide, Release 6.0(x) 52

Precision Time Protocol

Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the REST API

Creating the PTP User Profile for Leaf Switch Front Panel Ports Using the REST API

A PTP user profile is applied to the leaf switch front panel interfaces using an EPG or L3Out. You also must enable PTP globally to use PTP on leaf switch front panel ports that face external devices.

To create the PTP user profile, send a REST API POST similar to the following example:

POST: /api/mo/uni/infra/ptpprofile-Ptelecomprofile.xml

<ptpProfile name=”Ptelecomprofile” profileTemplate=”telecom_full_path” announceIntvl=”-3″ syncIntvl=”-4″ delayIntvl=”-4″ announceTimeout=”3″ annotation=””

PTP user profile name # PTP profile # Announce interval (2^x sec) # Sync

interval (2^x sec) # Delay request interval (2^x sec) # Announce timeout # Annotation key

ptpoeDstMacType=”forwardable” ptpoeDstMacRxNoMatch=”replyWithCfgMac” localPriority=”128″

(Only for Telecom ports) # Destination MAC for PTP messages # Packet handling

Port local priority

nodeProfileOverride=”no” />

(Only for non-Telecom ports on a telecom leaf) # Node profile override

Enabling PTP on EPG Static Ports Using the REST API
Before you can enable PTP on EPG static ports, you must first create a PTP user profile for the leaf switch front panel ports and enable PTP globally. To enable PTP on EPG static ports, send a REST API POST similar to the following example: POST: /api/mo/uni/tn-TK/ap-AP1/epg-EPG1-1.xml

Multicast Mode

PTP mode # PTP user profile

Cisco APIC System Management Configuration Guide, Release 6.0(x) 53

Enabling PTP on L3Out Interfaces Using the REST API

Precision Time Protocol

Unicast Mode

PTP source IP address # PTP mode # PTP user profile # PTP unicast

destination
IP address

If ptpEpgCfg exists, that means that PTP is enabled. If PTP must be disabled on that interface, delete ptpEpgCfg.

Enabling PTP on L3Out Interfaces Using the REST API
This procedure enables PTP on L3Out interfaces using the REST API. Before you can enable PTP on the L3Out interfaces, you must first create a PTP user profile for the leaf switch front panel ports and enable PTP globally.
To enable PTP on L3Out interfaces, send a REST API POST similar to the following example:
POST: /api/node/mo/uni/tn-TK/out-BGP/lnodep-BGP_nodeProfile/lifp- BGP_IfProfile.xml

Multicast Mode
<l3extRsPathL3OutAtt tDn=”topology/pod-1/paths-103/pathep-[eth1/11]” addr=”11.0.0.1/30″ ifInstT=”l3-port”>