To: MAB Distribution
From: Gary C. Dixon
John W. Gintell
R. C. vonSeeburg
Date: February 13, 1981
Subject: Revised MAB on TR Goals
PLEASE REMOVE THIS COVER LETTER BEFORE FILING THIS MAB
This MAB describes changes to MAB-044, which defines Multics TR
Processing Goals. Most significant changes in definition of
goals include the following:
(1) On page 3 in discussion of response, the following sentence
was removed.
Response time includes the time needed to route the
response back to the submitter.
The TR System does not include such time. Perhaps it should,
but it doesn't.
(2) Removal of concept of installed-in-EXL being a resolution to
a problem. This decision was made some time ago, but the MAB
was never updated to reflect the policy change.
(3) The TR goal is stated in terms of completing processing of
75% of TRs within the expected amount of time (times given in
Table 1). The prior MAB did not state this 75% goal,
implying our goal was to process the average TR within the
expected time.
The goal is measured as percent_completed_on_time for each
type and processing stage, weighted by the number of reports
completed in that stage. The exact algorithm is given in the
MAB.
All other changes are minor wording changes or clarifications.
The intent of this revision is to reflect the current policies
and procedures now in effect.
MULTICS ADMINISTRATIVE BULLETIN MAB-044, Rev 1
To: MAB Distribution
From: Gary C. Dixon
John W. Gintell
R. C. Von Seeburg
Date: February 4, 1981
Subject: Goals for Processing Multics Trouble Reports
INTRODUCTION
This bulletin defines performance goals for processing of
Trouble Reports (TRs) entered by Multics users through their
SiteSAs, and categorized in the TR System as external TRs.
DEFINITION OF TERMS
The criteria for measuring achievement of TR processing
goals are defined using the following terminology.
Types of TRs
There are three kinds of trouble reports.
1. Problem Reports: reports of problems in installed system
programs, info segments, or system documentation.
2. Questions: questions about operation of the system, or
about installed system programs, info segments and
documentation.
3. Suggestions: suggestions for changes/improvements to
installed system programs, info segments, or system
documentation.
_________________________________________________________________
Multics Project internal working documentation. Not to be
reproduced or distributed outside the Multics Project.
_________________________________________________________________
Problem Report Priorities
Problem reports are further divided into Priority groups,
according to the significance of the problem.
1. Problem Reports
A. Critical Priority, a problem which:
- interferes with system operation to an extent
that usefulness of the system is significantly
limited for most users of the system. (1)
_________________________________________________________________
(1) This includes, but is not limited to, situations in which a
site is totally down, or is running in a highly crippled state.
_________________________________________________________________
- irreparably compromises the security or integrity
of data stored in the system. Data storage
problems at system priority must affect many
system files, or files which are difficult to
retrieve or recreate.
B. High Priority, a problem which:
- causes installed system programs to fail to
operate correctly in limited circumstances, with
no known bypass for the problem. Such problems
can be tolerated for short periods of time.
- compromises data security or integrity for a
small number of files in a way which can be
repaired without significant loss of data. Such
problems can be tolerated for short periods of
time.
C. Normal Priority, a problem which:
- causes installed system programs to fail in a
minor way which can be bypassed, or ignored.
- results from incorrect documentation.
TR Processing Operations
Processing of TRs involves the following steps.
1. Submission occurs when a SiteSA or user enters a problem
report, suggestion, or question. A report is entered by
using the enter_problem_report (epr), enter_suggestion
(es), or enter_question (eq) command on System M.
A SiteSA usually reports Critical problems to Multics
Software Support (MSS) by telephone to avoid delay in
processing. For performance tracking purposes, MSS
personnel enter such telephoned TRs into the TR system
after the initial telephone conversation is complete.
2. Verification of a report validates the user's statement
of problem, or initially appraises suggestions, or
answers questions when answer is known. Verification is
performed by the TR Administrator or investigator in
Phoenix.
3. Routing of a report forwards the report to developers,
investigators, and interested parties. Routing is
performed by the TR Administrator in Phoenix.
4. Response to a problem report: acknowledges existence of
a bug, indicates a bug bypass if any, and states that
problem will be fixed in a future release; or it
indicates that the problem cannot or will not be fixed
(a system limitation).
No response is required for questions or suggestions.
(2)
_________________________________________________________________
(2) The answer to a question is provided when the question report
is resolved. Current management policy states that response to,
and resolution of, suggestions by the developer is optional.
_________________________________________________________________
Response is made by the developer, investigator or TR
Administrator, and is forwarded to the submitter by the
TR Administrator. A response is entered by using the
answer_trouble_report (atr) command.
5. Resolution of a problem report occurs when changes to
installed system programs causing the problem are
submitted for installation in the Multics System
Libraries, or when documentation errors are corrected;
or when the nature of a user error causing the problem
is pointed out to the submitter; or when the report is
unclear or unreproducible and additional information
about the problem is required.
Resolution of a question report occurs when the question
is answered.
Software problems are considered resolved when the
programs are submitted for installation in System
Libraries on System M, because at this point, an
emergency fix can be shipped to the site if this is
deemed necessary. Bug fixes are not normally
distributed between releases however, and the time from
submission of installation of software on System M until
distribution of the bug fix is NOT included in the
resolution time.
Documentation is considered fixed when the corrected
manual is sent to the Brighton Print Shop.
A trouble report resolution is entered by using the
answer_trouble_report (atr) command. Developers are
responsible to entering notification that a problem fix
has been submitted for installation into the System
Libraries. The Manager of Multics Technical
Documentation at CISL and the Project Leader for the
Multics Documentation Project in Phoenix are responsible
for entering final resolution notice when manuals
containing corrections are sent to the Brighton Print
Shop.
TR PROCESSING PERFORMANCE GOAL
The goal for TR processing performance states that 75% of
all TR processing operations will be completed within a specified
amount of time, where times are measured from submission of the
TR into the TR system to completion of the TR processing
operation.
The procedures for TR processing state the order in which
TRs are to be processed, and the expected amount of time needed
to perform each TR processing operation.
Expected TR Processing Times
The expected amount of time required to perform each TR
processing operation is shown in the table below. Times are
expressed as the number of calendar weeks which should elapse
from time of submission to the time of operation completion,
averaged over all TRs whose operations have completed during the
reporting period. Because the times are averages, they do not
state the processing performance expected for a particular TR.
They instead address our overall TR processing performance.
Table 1: Average Calendar Weeks from Submission
to Completion of TR Processing Operation
OPERATION Problem Problem
COMPLETED Critical High Query Normal Suggestion
________________________________________________________________
| |
Verification | 0.2 1 1 2 4 |
| |
Response | 1 2 4 13 -- |
| |
Resolution | 4 13 4 52 -- |
|________________________________________________|
Verification and routing operations are performed as a
single processing operation, and are therefore shown as a single
processing goal in the table above.
Also, as the table indicates, no response is required for
questions. No response or resolution is required for suggestion
reports since current management policies state that response and
resolution of suggestions is optional.
Formula For Computing Performance To Goal
_____ _____
\ \ # ops completed
total_ops = > > since 07/01/80
/____ /____
for problems (critical, for verification,
high & normal) and response and
questions resolution ops
_____ _____
\ \ # ops %_on_time
%_on_time = > > completed * for op
/____ /____ since 07/01/80
for problems (critical, for verification,
high & normal) and response and
questions resolution ops
%_performance = %_on_time / total_ops
Our goal is: %_performance >= 75%
Order of Processing TRs
TRs should be processed in the following order of
importance:
1. Problem Reports, Critical Priority
2. Problem Reports, High Priority
3. Questions
4. Problem Reports, Normal Priority
5. Suggestions
Type of report and priority of problem reports are initially
specified by the submitter of the report, but can be adjusted by
investigators, TR Administrator (who routes TRs) and developers
as the report becomes better understood.
REPORTING OF PERFORMANCE
TR Processing performance will be reported to the Director
of Software Development on a quarterly basis. It will be
reported weekly to the Manager of Multics Software Development in
status reports. Reporting will begin with statistics for reports
submitted during the third quarter, 1979.