verizon Managed Services Change Management User Guide

MANAGED SERVICES CHANGE MANAGEMENT

Standard Change Management

What is a Change Request (CR)?

A Change Request is an electronic document used to initiate and track all changes to the software configuration of a Verizon managed Customer Edge (CE) device. The CR can modify configuration of currently implemented features, routing protocols, addressing etc. that is already applied on the router. It can also be used to request specific technical information or to submit administrative changes
The CR does not cover hardware or design changes or addition of major features or protocols and is not intended for troubleshooting (incident management).

How can I submit a CR?

A CR can only be submitted via the Verizon Enterprise Centre (VEC) portal. Access can be requested via your Account Manager or Service Manager. It is also possible that an authorized Verizon representative submits these CRs on your behalf.
The VEC allows you to submit CRs, schedule CRs and provides reporting capabilities. All CRs should be submitted in the English language (only).
More info: https://enterprise.verizon.com/r3s0u4c3s/global-change-management- user-guide_en_xg.pdf

Who is responsible?

It is your responsibility to provide the technical content of a CR. This may include providing exact configuration commands or descriptions that are easily translatable to configurations commands.
The Change Control team (CC) is responsible for the implementation of the CR. The CC engineers are Cisco certified. The CC team business hours are 08:00-17:00 CET, Monday Friday. CRs with an implementation date outside these hours are handled by the Verizon 24/7 NOC engineers.

CR priorities

There are two different priorities of CRs with the following attributes:

Standard

|

Express

---|---
Verizon will assign an engineer with a maximum of 3 business days to review and schedule.| Verizon will assign an engineer within
1 business day to review and complete.
Can be scheduled on requested date and time (mentioned in change description section).*| Cannot be scheduled on requested date and time
Change can be performed on call (including conference call).| No communication is performed (no conference calls, emails etc.)
Must include all affected devices in the entities section within the same CR| One device per CR only.

Must contain clear and concise specification of the requested change. It is recommended to provide the exact configuration commands to avoid any misconfiguration.

*) The VEC portal will not allow a CR to be submitted with a request completion date and time earlier than 72 hours (3 days). Therefore, please always specify date and time in the CR remarks. The CC team will aim to implement the CR against the requested date/time, although this cannot be guaranteed as it is within the normal lead time.

What are the CR implementation steps?

• Once the CR has been submitted, an auto notification email will be sent, this will contain the CR reference number and the submitted remarks.
• For further queries or an adjustment to the request, completion time and/or date then please reply to ALL on the auto notification email to which the CC team will respond. The CC team only communicates in the English language.
• The Change Control engineer will review, assess the CR to then schedule it towards completion. If readily apparent that the applied change does not work then the change will be rolled back, an appropriate explanation added  to the cancelled CR. If the CR has been completed and the changes must be reverted then the requester must open a new CR for the rollback.

What are the CR statuses?

CRs have several statuses defined as follows:

Initial/Draft

| The CR is in draft form, the requester can edit and add remarks to the description area; the CR will not be reviewed by the CC team until submitted.
---|---

Submitted

| The CR appears in the CC team working queue and waits to be reviewed by an engineer.

Pending

| The CC engineer has requested information from the requester and is waiting for an update. The requester can edit CR and add remarks.

Scheduled

| CR has been reviewed by CC engineer, prepared and scheduled for the requested time. The CR cannot be edited by requester in this status.

Completed

| The CC engineer has completed all work on the request and closed out. The request cannot be reopened. Any rollback or further changes require a new CR.

Rejected

| The CR does not match the criteria or the request is not feasible. A reason has been provided for rejecting the CR. The requester should open a new CR after the reasons for rejection have been addressed.

Cancelled

| The CR has been withdrawn by the requester, the CC engineer has cancelled it at the requester’s request or the change is not possible. Rollback also results in a Cancelled request.

Out of Scope CRs

A CR will be rejected if any of the following applies:

• Price impact (e.g. Port speed change, EF CAR modification).
• Monitoring impact (e.g. shutting down monitored interface).
• Design impact (e.g. implementing new routing protocol).
• Change required on both CE and PE side (e.g. Site of Origin, BFD implementation, BGP AS number change).

For the above out of scope requests a follow-up is required with the Account Manager or Service Manager. The Verizon representative will then assess if the change is billable to then engage the Service Order process.

Change Request Rejection Reasons

The main reasons for Change Request to be rejected or cancelled are:

Wrong CR type| CC team are unable to change CR type. CR has to be rejected once the change falls under another type, due to documentation reasons.
---|---
Information not clear| If the change description is not clearly stated or easily translatable to Cisco commands.
Scheduling Express CR| Express CR cannot be scheduled and is rejected without any prior notification. Only standard CR can be scheduled.
Device is unreachable| If an engineer is not able to reach the device and perform the changes then CR is also rejected because no change can be done. An incident ticket is required to solve connection issue. A new CR can be submitted after reachability to the device is re-established.
Not all affected devices are chosen in CR entity section| CR has to contain all relevant devices (entities) where the change has to be performed. CC engineer cannot make changes on a device which is not selected in entity section in CR.
Change was rolled back| The change was performed but it did not work as expected. The engineer had to roll back the change and reject the CR because the change was not successful.

CR Template

The following template is to be used and applied to the CR on submission; if the information is incomplete then the CR could be rejected by the CC engineer.

Scheduling Requirements (Select one):
– Change to be performed at any time.
– () Change to be performed exactly on DD-MM-YYYY at HH:MM GMT.
Call Requirement (Select one):
– Automated notification upon completing the change is acceptable.
– () Call Mr./Mrs. __ before/after the change on +__ – () Join the conf. call on +__ Using the passcode __
Contact Information (In case a call or further clarification is required):
– Customer/requester name: ____
– Customer/requester email: ____
– Customer/requester phone number: ____
– Customer/requester availability: From HH:MM TO From HH:MM GMT Change Description (Clear, concise and exact terms): ……………………………………………………………………………. CISCO commands to be applied : ……………………………………………………………………………. () Not applicable for EXPRESS priority Change Requests.

Change Request Types

DHCP Configuration ­ Modify

To implement any DHCP-related changes. Changes might include adding or modifying a DHCP pool, adding an IP helper-address command.
Keywords : DHCP, pool, lease, excluded address, IP helper, DHCP relay

Modify Filters/Access-lists

This type covers any update of access-lists, prefix-lists, distribution-lists or route maps. Of course these access-lists might serve different purposes (QoS marking, advertisement restrictions, access restrictions).
Keywords : ACL, ip access-list, ip prefix-list, distribution list, route- map, policy-based routing, af, ef classification

Interface ­ Modify

Interface-related changes – bandwidth statement modification, description, speed & duplex changes. Please note that this “speed” stands for Ethernet access speed (L2), not the ordered bandwidth.
This CR can be also used for shutting down unmonitored LAN interface.
Keywords : CDP, bandwidth statement, interface description, speed, duplex, IP redirects, proxy ARP, shutdown

IP Address/Subnet Mask Changes

Changing IP addresses on interfaces, altering subnet masks, adding secondary IP address, change of WAN IP address on SGSO routers. This CR type will suppress proactive monitoring for the required period of time.

Keywords : LAN, WAN IP address, secondary IP address, subnet mask

IP Network Address Translation ­ Modify

Implementation, modification or removal of Network Address Translation. Changes might include static NAT entries, NAT pool, dynamic NAT configuration or PAT implementation (NAT with the overload option).
Keywords : IP NAT inside, IP NAT outside, static NAT, overload, NAT pool

Privilege Exec Commands ­ Modify Express eligibility

Certain commands may be added to the router to allow users to perform privileged level commands while being in user mode (level 0/1). Requester must have a TACACS account to access the router in order to use these.

We do not provide the “show running-config” command but we can provide the “show configuration” instead. This CR type can be also used to solve the issue when requester can’t use this command even when the appropriate privilege command is applied.

The following commands are generally implemented on the devices:
privilege exec level 0 traceroute
privilege exec level 0 ping
privilege exec level 0 show configuration
privilege exec level 0 show logging

Keywords : Privilege command, exec command, file privilege

Protocol/Feature ­ Modify

Used mainly for modification of protocols and features which do not have their own CR type. This includes modification of LAN routing protocols (EIGRP, OSPF and RIP), adjustments of BGP (not affecting PE side) or Policy Based Routing (PBR). Minor features such as monitoring and statistics tools ­ i.e. IP accounting, Net flow, IP NBAR or IP SLA – can be activated or deactivated via this CR as well.
This also modifies first-hop redundancy protocols, such as HSRP, GLBP and VRRP – standby group number change, HSRP (standby) IP address, priority, timers and pre-empting. Modifications of tracking statements fall also under this category.
Keywords : Netflow, IP accounting, NBAR, EIGRP, OSPF, RIP, BGP, PBR, IP SLA, HSRP, VRRP, GLBP, standby group, virtual IP, HSRP priority, timers, track, decrement value, policy-based routing

QoS ­ Modify

Used for QoS related changes ­ modification of marking or queuing maps on an existing QoS implementation. New QoS template implementation must be done via a service order.
Please note that port speed (shaped bandwidth) can be only adjusted via Dynamic bandwidth feature (Looking Glass) or via service order, not via this change request.
Keywords : QoS, packet marking, af, ef, DiffServ, ETM profile

Request for Copy of Router Configuration

Sanitized configuration of managed CE devices can be requested via this CR type. Request for running configuration and/or for recipient different than the CR requester have to be explicitly specified in the CR remarks.

Keywords : Running configuration, start-up configuration

Request for Information

Used for requesting technical information. This includes parts of router configuration, output of show commands, information about technical parameters of devices (e.g. software and hardware version) etc. This CR also serves for requesting various reports, including customer or Verizon contact reports, notification profile report, site inventory report, Out Of Band and Backup circuit testing results etc.
Keywords : Routing table, show commands, IOS version, report

Software Vulnerability Upgrade

This can be used to upgrade an operating system on CE device if valid reasons exist for the upgrade. Valid reasons include service affecting IOS bugs, faults identified in an incident ticket and missing features in some cases.
The desired IOS version must be specified by the requester and will be validated against list of Verizon approved IOS versions. A change control engineer will load the specified IOS version into the router and schedule the reload of the device. This CR type will suppress proactive monitoring for the required period of time. A working Out Of Band (modem) connection is required for the IOS upgrade.

Keywords : IOS upgrade, IOS downgrade, bug, missing feature

SNMP Community Strings ­ Modify

This can be used for modification of SNMP settings on a managed device. It is structured for the whole customer network (individual entities cannot be selected), but individual routers can be specified in CR Remarks/Notes section. To provide SNMP access to device a SNMP community string and a list of allowed IP addresses are needed. Customers are not allowed to have Read and Write access to managed devices.

Keywords : SNMP, community string, read only

Static Route ­ Add/Delete/Modify Express eligibility

Used for addition, deletion or modification of static routes. This CR requires network IP address, mask and next-hop destination IP address within the local LAN (directly connected LAN IP range). Please always explicitly state in the remarks/notes if the route should be advertised to the rest of the MPLS network.

Keywords : Static route, new network, IP route, next hop, redistribution

Special Change Request Types

Customer maintenance

CR type used for suppressing Priority 1 proactive trouble tickets. Please use this whenever your maintenance actions could lead to proactive alarms – i.e. planned outages. A CR can be set-up to prevent alarms in recurring intervals and is fully automated. This means that CC team does not review the CR.

Keywords : Alarm suppression, ticket suppression, maintenance window, outage

Entity Host Name Change

Both the DNS Entity Name (Identify the device in Verizon systems) and the hostname (the name shown when logged onto the device) can be in sync or the Hostname can have a different naming convention but this is not recommended. For the DNS Entity Name, only the geographical part of the DNS Name can be changed.