Section 6.5.  Distributing Questionnaires

6.5. Distributing Questionnaires In the past, software process assessments always began with the distribution of a maturity questionnaire. Recognizing their deficiencies, today's assessment methodologies use them not as the basis of the assessment but rather primarily to identify issues to be explored further during the onsite period and to be corroborated through document review and interviews. (In SCAMPI, questionnaires are optional. If a SCAMPI questionnaire is to be used, that fact needs to be identified in the SCAMPI Data Collection Plan.) The CBA IPI maturity questionnaire is required to be distributed to participants before the start of an assessment [Zubrow 94]. As it includes a glossary of terms and KPA descriptions, the questionnaire partly serves as an aide to CMM terminology. It covers all 18 KPAs of the CMM, addressing each KPA goal in the CMM but not all of the key practices. Limited to six to eight questions per KPA, the questionnaire can usually be completed in one hour. The use of questionnaires to gather written information from members of the organization represents a relatively low-cost way to collect data when done well. Questionnaires typically require yes or no answers. (Respondents are given an opportunity to comment if they wish.) Questionnaire data, however, may be problematic. Because the questionnaire has specific CMM language in it, those who fill out the questionnaires sometimes don't understand the questions. The questionnaire therefore needs to be administered by someone familiar with the model in case explanations are needed. When the team evaluates questionnaire responses about whether a particular process is or is not present, a determination needs to be made as to whether the respondent has understood what he has been asked. The team should also consider whether the respondent has answered yes to something merely because he thinks that a loyal response is better than a truthful one. It is the task of the organization site coordinator to summarize the responses to a given question on the questionnaire. Such summaries give the team a quick overview of what respondents think are the company's strengths and weaknesses, but they should be treated by the team with some skepticism because of the difficulties just described. Answers to questionnaires are most useful in preparing interview questions about specific technical areas. The widespread distribution of questionnaires may be more trouble than it is worth. It is recommended that the questionnaire be filled out by four to ten peopleideally by each participating project manager. In addition, key technical personnel or functional managers for the organization may be asked to fill out the questionnaire as well. Whenever possible, the organization site coordinator should collect the documents mentioned in questionnaire responses so that the team can review them early in the process. 6.5.1. Specimen Section of the CBA IPI Maturity Questionnaire (Section for the Requirements Management KPA) Maturity Questionnaire (version 1.1.0) Page 1 of 42 [Zubrow 94] Software Process Maturity Questionnaire Capability Maturity Model, version 1.1 April 1994 This document contains questions about the implementation of important software practices in your software organization. The questions are organized in groups of key process areas such as software project planning and software configuration management. A short paragraph describing each key process area precedes each group of questions. Unless directed otherwise by the person who administers this questionnaire, please answer the questions based on your knowledge and experience in your current project. To help us better interpret your answers to the questions about software process in your organization, this document begins with questions about your own background in software work. Please read and answer all of the questions. If you wish to comment on any questions or qualify your answers, please use the comment spaces provided. Your answers will be held in strict confidence by the appraisal team. Specific answers will not be identified within your organization, nor in any other manner. Your name will be used for administrative purposes only: to guide the appraisal team during response analysis and help them contact you for any needed clarifications. Thank you for your help. Software Engineering Institute Carnegie Mellon University Pittsburgh, Pennsylvania © Copyright 1994 Carnegie Mellon University This work is sponsored by the U.S. Department of Defense. Page 2 of 42 Maturity Questionnaire (version 1.1.0) 2 CMU/SEI-94-SR-7 Filling in Your Answers Definitions of Terms Respondent Identification (Please specify) YOUR NAME: TODAY'S DATE: PROJECT NAME: WORK TELEPHONE: We will be using optical scanning technology to enter your answers, so please print or write neatly throughout the questionnaire. • Feel free to use the margins if you need more space for your written answers or other comments, but please don't write over the check boxes or crosshair (+) symbols. • Please keep your marks within the check boxes. Any mark will do: • You should use a pen with dark blue or black ink. The Capability Maturity Model on which this Maturity Questionnaire is based uses a number of terms which may be used differently in your organization. • Organizational terms are defined on the blue placard. You may wish to review it now, and refer to it as necessary as you complete the questionnaire. • Technical terms are defined on the pages where they are used. CMU/SEI-94-SR-7 3 Maturity Questionnaire (version 1.1.0) Page 3 of 42 Section I Respondent Background 1 Which best describes your current position? (Please mark as many boxes as apply) PROJECT OR TEAM LEADER MANAGER TECHNICAL MEMBER SOFTWARE ENGINEERING PROCESS GROUP (SEPG) MEMBER OTHER (Please specify) 2 On what activities do you currently work? (Please mark as many boxes as apply) SOFTWARE REQUIREMENTS SOFTWARE QUALITY ASSURANCE SOFTWARE DESIGN CONFIGURATION MANAGEMENT CODE AND UNIT TEST SOFTWARE PROCESS IMPROVEMENT TEST AND INTEGRATION OTHER (Please specify) 3 Have you received any CMM-related training? NO YES (Please describe) 4 What is your software experience in: (Please specify for each category) Your present organization? ......................... YEARS Your overall software experience? ........... YEARS 5 Have you participated in previous forms of Software Process Assessments, Software Capability Evaluations, and/or other forms of software process appraisals? (Please mark one box) NO YES How many? (Please specify for each category) # OF SPAs (Software Process Assessments) # OF SCEs (Software Capability Evaluations) # OF OTHER SEI-BASED METHODS (Please describe briefly: e.g., mini-assessments or instant profiles) BASED ON NON-SEI PROCESS IMPROVEMENT WORK (Please describe briefly: e.g., ISO 9000/9001 audit) Page 4 of 42 Maturity Questionnaire (version 1.1.0) 4 CMU/SEI-94-SR-7 Section II Software Practices Instructions 1 To the right of each question there are boxes for the four possible responses: Yes, No, Does Not Apply, and Don't Know. Check Yes when: The practice is well established and consistently performed. - The practice should be performed nearly always in order to be considered well-established and consistently performed as a standard operating procedure. Check No when: The practice is not well established or is inconsistently performed. - The practice may be performed sometimes, or even frequently, but it is omitted under difficult circumstances. Check Does Not Apply when: You have the required knowledge about the project or organization and the question asked, but you feel the question does not apply to the project. - For example, the entire section on "Software Subcontract Management" may not apply to the project if you don't work with any subcontractors. Check Don't Know when: You are uncertain about how to answer the question. 2 Use the Comments spaces for any elaborations or qualifications about your answers to the questions. 3 Check one of the boxes for each question. Please answer all of the questions. • This Page Intentionally Left Blank • CMU/SEI-94-SR-7 5Maturity Questionnaire (version 1.1.0) Page 5 of 42 Page 6 of 42 Maturity Questionnaire (version 1.1.0) 6 CMU/SEI-94-SR-7 The purpose of Requirements Management is to establish a common understanding between the customer and the software project of the customer's requirements that will be addressed by the software project. Requirements Management involves establishing and maintaining an agreement with the customer on the requirements for the software project. The agreement covers both the technical and non-technical (e.g., delivery dates) requirements. The agreement forms the basis for estimating, planning, performing, and tracking the software project's activities throughout the software life cycle. Whenever the system requirements allocated to software are changed, the affected software plans, work products, and activities are adjusted to remain consistent with the updated requirements. allocated requirements (system requirements allocated to software)The subset of the system requirements that are to be implemented in the software components of the system. The allocated requirements are a primary input to the software development plan. Software requirements analysis elaborates and refines the allocated requirements and results in software requirements that are documented. policy A guiding principle, typically established by senior management, which is adopted by an organization or project to influence and determine decisions. software plans The collection of plans, both formal and informal, used to express how software development and/or maintenance activities will be performed. Examples of plans that could be included: software development plan, software quality assurance plan, software configuration management plan, software test plan, risk management plan, and process improvement plan. software quality assurance (SQA) (1) A planned and systematic pattern of all actions necessary to provide adequate confidence that a software work product conforms to established technical requirements. (2) A set of activities designed to evaluate the process by which software work products are developed and/or maintained. software work product Any artifact created as part of defining, maintaining, or using a software process, including process descriptions, plans, procedures, computer programs, and associated documentation, which may or may not be intended for delivery to a customer or end user. Does Not Apply Don't Know Yes No 1 Are system requirements allocated to software used to establish a baseline for software engineering and management use?............ Comments: 2 As the systems requirements allocated to software change, are the necessary adjustments to software plans, work products, and activities made?.............................................................................. Comments: Does Not Apply Don't Know Yes No CMU/SEI-94-SR-7 7 Maturity Questionnaire (version 1.1.0) Page 7 of 42 3 Does the project follow a written organizational policy for managing the system requirements allocated to software? Comments: 4 Are the people in the project who are charged with managing the allocated requirements trained in the procedures for managing allocated requirements? ................................................................ Comments: 5 Are measurements used to determine the status of the activities performed for managing the allocated requirements (e.g., total number of requirements changes that are proposed, open, approved, and incorporated into the baseline)? ............................ Comments: 6 Are the activities for managing allocated requirements on the project subjected to SQA review? .................................................. Comments: Page 8 of 42 Maturity Questionnaire (version 1.1.0) 8 CMU/SEI-94-SR-7