Assuring Autonomy V1 AMLAS Tool User Guide

Assuring Autonomy V1 AMLAS Tool

AMLAS Tool overview

The AMLAS Tool supports the implementation of the process described in v1.1 of the AMLAS assurance guidance (available on the AAIP’s guidance website) and this user guide assumes you are familiar with that document. AMLAS Tool support is provided via a Microsoft Visio Add-In, with additional VBA macros and predefined pages with placeholder shapes and argument structures that guide the user through the assurance process. Many of the artifacts are created to act as inputs to several process stages, and one purpose of the tool is to handle this complexity for you so that if you create an artifact, it will automatically appear in the corresponding arguments wherever it is referenced. The tool visually indicates which artifacts have been instantiated and keeps track of the progress of the assurance process so that the user can quickly see which areas require further work.

Installation and setup

The Add-In is supplied as a Microsoft Office Add-In Solution for Visio. Unzip the file and start the installation by double-clicking ‘setup.exe’. In order to use the VBA macros within the AMLAS document, you must enable the macro settings in Visio (File, Options, Trust Center):

The Add-In uses a macro-enabled template file (AMLAS Tool.vstm). The user is advised not to work directly on the template file. Instead, either double- click the AMLAS template file in the file browser to create a new document based on the template or if you open the template from within Visio, use the create button on the AMLAS Process Overview page to create a new document.

Navigating the AMLAS Tool document

The template or derived document should open on the AMLAS Process Overview page. This page acts as a simple navigation aid. Each stage is represented by a button. Clicking on these takes the user to the respective stage:

The buttons down the side of each page also allow the user to navigate within the document. Clicking on the Home button will return you to this page, or if the Artifact Tracker is selected, the Home page is set to the Artifact Tracker. The Back button will return you to the previously visited page .

The toggle button ( ) lets you switch between process activity and argument pages. The Argument Overview button ( ) directs you to the Argument Overview page which shows the entirety of the argument considering all stages of AMLAS.

Navigating from an activity stage to its corresponding argument can also be done by clicking the purple Argument output on the last activity. The argument pattern input to the last activity can also be double-clicked to take you to a reference image of the AMLAS pattern templates.
Warning
You will notice that the data within shapes on the Visio pages are not directly editable except by double-clicking, which allows the user to enter specifically defined data. These restrictions are intended to maintain a consistent data structure for the activities and arguments. Of course, it is possible to get around these restrictions, but we recommend that you don’t attempt to change the document appearance or content (except where enabled) as much of the tool’s functionality depends on shape names and their data definitions within shape sheets. Changing these, and in particular, changing shape sheet or VBA functions may result in your document being irreparably damaged.

Working through the AMLAS assurance process

The AMLAS tool closely follows the process stages described in v1.1 of the AMLAS assurance guidance (available on the AAIP’s guidance website). For this initial version of the tool, it was decided to limit the degree to which users can extend or change arguments or processes. At a later stage, the tool may be extended to permit users to create bespoke processes and argument structures but currently, users can only use the tool within the AMLAS assurance framework (v1.1).

Editing artifact data

Double-clicking an artifact will bring up an edit dialogue box:

Each artifact has a non-editable identifier shown in square brackets (e.g. [A]). Placeholder text describing the artifact is shown within curly brackets (e.g. {System Safety Requirements}). Placeholder descriptions can be edited to reflect specific documents and are intended to be a concise description of the artifact. If more details are required the Details checkbox can be selected and a longer text description can be added. The artifact is instantiated (i.e. given content) by either adding a file reference/path to the artifact (use the ‘Get file path’ button) or by changing the artifact description. If a change is made to either the placeholder text or by adding a file path, the curly brackets from the description will be removed on clicking ‘OK’. An instantiated artifact is shown with a dark green border:

If you instantiated the artifact with a file reference, that will be displayed in the tooltip when the mouse cursor is over the shape. If you want to open the artifact’s referenced file directly from Visio, you can either double- click the shape to bring up the edit dialogue box and click the ‘Open file’ button, or you can right-click on the shape to bring up the side menu. Clicking Open file reference will also open the file. However, if the file reference is a URL, Windows may warn you not to open the link.

Referenced artifacts

Artifacts are referenced in arguments. You can edit the referenced artifact from an argument page by double-clicking on the shape which references the artifact:

A referenced artifact such as the Context C1.2 above has the name of the artifact in square brackets and an element ID (C1.2). Double-clicking on the context will bring up a dialogue box asking if you want to edit the artifact :

Clicking OK will take you to the page containing the artifact and select it for you. Double-clicking on the artifact will allow you to edit it as normal. You can use the back button or the toggle button to return to the argument. An element that references an instantiated artifact will no longer show the small triangle below the shape. The tooltip will show the file path reference of the artifact if one has been added.

Instantiable GSN elements

Although most GSN elements that reference evidence as part of an argument reference an artifact from the activity stages, there are some that need instantiating by directly editing the text. These include all Justifications (ellipses with a J) and Assumptions (ellipses with an A), some Goals, and one context (C5.2). Each of these has a triangle beneath the shape to indicate it can be edited. The text intended to be replaced is again enclosed in curly brackets:

Double-clicking these shapes brings up a dialogue box for editing:

In this example, only the text within the curly brackets is able to be edited and forms part of a longer statement. Adding a file reference or changing the text will remove the curly brackets and hide the triangle beneath the shape on clicking the OK button.

Tracking progress

The two home pages in the AMLAS tool (AMLAS Process Overview and Artifact Tracker) both contain elements to help the user see how far through the process they are and which parts of the process still require work The AMLAS Process Overview contains a simplified version of the Artifact Tracker, with only a textual indication of how many artifacts have been instantiated and a percent bar showing the total progress. The Artifact Tracker gives a detailed breakdown and also shows which artifacts feed into later activities. It can be turned on using the radio button at the top of the home page. The Artifact Tracker shows for example, that one artifact will form part of the activity inputs for several stages (e.g. [H]):

Artifacts arrows highlighted in dark green have been instantiated. Tooltips indicate the file reference used for instantiation. The green percent bars between activities indicate how much of that stage has been completed. The total progress is shown in the purple percent bar at the top. This page can also be used for navigation – double-clicking the activities will take you to that stage, as will clicking the stage buttons along the top. Double-clicking an artifact arrow will take you to the first page the artifact is referenced and where it can be edited.

Updating the number of safety requirements

Having undertaken verification activities, ML verification evidence should be collated and reported in terms that are meaningful to the safety engineer with respect to the ML safety requirements and the operating environment. When developing your safety assurance of ML components using the AMLAS Tool, you may have multiple safety requirements to incorporate (stored in separate documents). The AMLAS Tool includes functionality to specify how many performance safety requirements you have and how many robustness safety requirements you have. This functionality is available on Stage 2: Assurance Argument Pattern for ML Safety Requirements. To edit the number of performance or robustness safety requirements, click on the filled circles shown below.

Performance safety requirements
The ML performance safety requirements should focus on the reduction/elimination of sources of harm while recognizing the need to maintain acceptable overall performance (without which the system, though safe, will not be fit for purpose). Performance requirements may also be driven by constraints on computational power (e.g. the number of objects that can be tracked).
Robustness safety requirements
One useful approach to defining robustness requirements is to consider the dimensions of variation which exist in the input space. These may include, for example:

• variation within the domain (e.g. differences between patients of different ethnicity)
• variation due to external factors (e.g. differences due to limitations of sensing technologies or effects of environmental-phenomenon)
• variation based on a knowledge of the technologies used and their inherent failure modes

Update outcome
Once you have updated the number of performance or robustness safety requirements, the following updates will appear in the Visio document.

NOTE: updating can be slow due to memory limitations in MS Visio and the number of shapes to be processed.

The filled circles in Stage 2: Assurance Argument Pattern for ML Safety Requirements will be replaced by a box showing the number of performance and robustness safety requirements.

Stage 5: Assurance Argument Pattern for ML Verification will be updated to reflect the changes to the number of safety requirements. An example is given below. The example matches the figure above with 2 performance safety requirements and 3 robustness safety requirements.

Multi-Overview will be updated to reflect the changes to the number of safety requirements. An example is given below. The example matches the figure above with 2 performance safety requirements and 3 robustness safety requirements.

Editing artifact data (after updating the number of safety requirements)

Stage 5. Model Verification allows the user to link documents, text descriptions, and notes with particular performance or robustness safety requirements. Double-clicking shape ‘[Z] References to Safety Reqs files’ (shown below) brings up a tabbed user form. The number of tabs reflects the number of safety requirements. The tabs are numbered and labeled ‘P’ for performance and ‘R’ for robustness safety requirements. In the figure below there are 2 performance safety requirements and 3 robustness safety requirements. The ‘Get File Path’ button lets you link to files and the ‘Open File’ button allows you to open the file. You can add descriptions and notes.

On clicking the OK button, Stage 5: Assurance Argument Pattern for ML Verification will be updated. The ‘ML Verification Results’ shapes, shapes Sn5.2 and Sn5.4, will reflect the changes made.

Editing the Assurance Argument Patterns for ML Verification

In order to demonstrate that the ML safety requirement is sufficiently satisfied, the pattern provides a choice over how the claim can be supported. The evidence may come from any combination of testing and formal verification.
By clicking on the filled diamond shown below, you can choose the verification strategy. The choice in the argument should be interpreted as “at least 1”, allowing for multiple legs of argumentation.

• If the verification strategy does not include test-based verification, then the left-hand branch may be removed.
• If the verification strategy does not include formal verification, then the right-hand branch may be removed.

When you click OK, the relevant branch will be removed. The figure below shows an example where the left-hand branch was removed as the verification strategy does not include test-based verification.

Multi-Overview will be updated to reflect the changes to the Assurance Argument Patterns.

Exporting the data in the arguments

A limited export of the instantiable artifacts has been created for the AMLAS tool document. By using the in-built report functionality in Visio (Review→Shape Reports), the following dialogue box will appear. A sample report definition file is included in the installation folder (amlas.vrd) and can be found using the Browse button. Selecting this should make it visible as below:

Clicking Run will let you choose the format of the report. A more extensive exporting facility for Excel and Word is planned in the next release.

• Assuring Autonomy International Programme Department of Computer Science Deramore Lane
• University of York
• York YO10 5GH
• assuring-autonomy@york.ac.uk
• +44 (0) 1904 325345

References

• Assuring Autonomy | AMLAS

Read User Manual Online (PDF format)

Read User Manual Online (PDF format)  >>

Download This Manual (PDF format)

Download this manual  >>