Section 6.1.  Selecting the Assessment Team

6.1. Selecting the Assessment Team When selecting the assessment team, the organization needs to take into account the knowledge, skills, and abilities of both the assessment team as a whole and of individual team members. Being part of the assessment team is a full-time job during the training and onsite period. The team members must be freed from their daily work to perform these duties. They cannot be expected to attend project meetings, write documents, or travel, for example, during the assessment week. 6.1.1. How Big Should the Team Be? An assessment team is a cohesive group of experienced individuals that assesses the process capability of an organization. The team as a whole must have the collective knowledge, skills, and ability to conduct the assessment for the organization. Team members are selected so that their combined experience and skills match what is required for a particular assessment. (All team members must have successfully completed assessment team training and CMM training for a CBA IPI or the SEI-licensed introduction to the CMMI course, delivered by an instructor who is authorized by the SEI.) Teams usually consist of no fewer than four members including the Lead Assessor. For SCAMPI, the team should have no more than nine members, and for CBA IPI no more than 10. Smaller teams may have difficulty in performing extensive document review and in achieving accuracy in interviewing activities. Larger teams require a great deal of time and coordination in order to come to consensus on collective judgments. A team of seven seems to work well for an average organization for most assessments. If the team is larger, interview sessions can be intimidating to the interviewees. 6.1.2. Insiders Versus Outsiders Generally an assessment team contains members internal to the organization as well as members who are from other organizations in the division or other divisions in the company. It is often useful to have some members from other parts of the division or other divisions in the company so that they can learn how their counterparts are performing their tasks. These "crossings" help break down barriers between divisions and bring different parts of a company together. The most important part of the team, however, will consist of personnel from the organization being assessed who are not directly involved in the representative projects to be interviewed and therefore do not have vested interests in the results. (This means that no manager whose direct reports are to be interviewed in the assessment should be on the team.) For the assessment team to have organization-specific knowledge, local representatives must be on the team. It is also highly desirable for the sake of future developments that local representatives be present to pass on the larger view of the assessment. Later they will help the organization to internalize the assessment results and move forward with the improvement effort. Insiders should represent different engineering disciplines and have knowledge of a number of projects. Both project managers and line managers should be included. If an organization has trained its own Lead Assessors, it is important to have some of them on the team along with the outside Lead Assessor. This gives internal Lead Assessors an opportunity to work with and learn from other leads. It is also useful to have people who aspire to be Lead Assessors on the team. Having insider project managers on the team who are not from one of the projects being assessed has long-term advantages. Although they may not be intimately familiar with the assessment model, being on the team introduces them to best practices and helps them be better managers. Often, they emerge as new improvement advocates when the assessment is over. Insider team members must satisfy certain criteria specified in the assessment method. They also should be highly respected opinion leaders who have experience in the areas being assessed and who can objectively collect assessment data and accurately reflect the results [Dunaway 01a]. When an assessment is over, the assessment team members will know more about the organization than any other individual, sometimes including the president. If team members are not from the organization, they disappear after the assessment, as does their accumulated knowledge. 6.1.3. Official CBA IPI and SCAMPI Team Member Selection Guidelines[1] [1] [Dunaway 01c] and [Members of the AMIT 01]. When selecting team members for assessments, the organization needs to consider their experience: In the engineering field For a CBA IPI, the average experience for individual team members should be at least six years, with no team member having fewer than three years of software field experience. The team should have a minimum of 25 years of combined field experience. For a SCAMPI team, members may also need systems engineering experience. The SCAMPI team members as a group must have an average of at least six years of engineering field experience, and the team as a total must have 25 total years experience in each of the disciplines to be covered in the assessment. In management For both the CBA IPI and SCAMPI, it is expected that the team as a group has a total of at least 10 years of management experience, and at least one team member must have at least six years of experience as a manager. In life-cycle phases and functional activities For a CBA IPI, 75% of the team members must have experience in at least one third of the organization's software life-cycle phases and their associated functional activities. One team member must have experience in a minimum of 80% of the organization's software life-cycle phases and associated activities. At least two team members must have had experience in each life-cycle activity. For SCAMPI, the team should in total have representative experience in the life cycles used within the organization. It is expected that for any given life-cycle phase, at least two members of the team should have experience as a practitioner. In organizational environment, applications, and existing software processes At least one team member must be knowledgeable in both the organization's environment and application domain. Organization team members are important to the team; however, these team members must not have a vested interest in the assessment results. In software process improvement concepts All team members must be knowledgeable in software process improvement concepts. In the CMM and CMMI Each team member must have knowledge that includes the ability to explain the assigned KPAs or PAs and their intent and to provide examples relevant to the organization being assessed. The Lead Assessor will have knowledge of each of the KPAs and PAs in the reference model. The Lead Assessor functions as the assessment team's model authority. In process assessment The Lead Assessor, being authorized as such, will have experience in conducting process assessments. Other team members with previous experience in conducting assessments will provide additional strength to the team. For a SCAMPI, it is desirable that at least half of the assessment team have previous assessment experience. In team skills Each team member must have good written and oral communication skills, the ability to facilitate the free flow of information, and the ability to perform as team players and to negotiate consensus. In credibility Each team member should have credibility with senior management, respect within the organization, and the ability to influence people. In motivation and commitment Each team member must demonstrate the motivation to improve the software process, the commitment to act as change agents, and the willingness to do what it takes to achieve assessment goals. 6.1.4. Selecting Assessment Team Members: Sensitive Areas Selecting assessment team members is a crucial decision that should not be made lightly. The CMM/CMMI assessments specify that team members should not be managers of one of the selected projects or be within the direct supervisory chain of any of the anticipated interviewees. The idea here is that team members should not have a high personal stake in the outcome of the assessment. For an assessment to work properly, this needs to be followed in spirit as well as letter. The point of the exclusion is that people being interviewed may feel uncomfortable talking about problems in front of a manager responsible for their work. People who are intimately involved with managing an activity have a difficult time listening to outside critiques. Team members who are in any way directly impacted by the outcome can be distracted by the potential consequences of their decisions. Objectivity and credibility of assessment results depends on the objectivity and credibility of the team. Biased teams produce biased assessment results. Some organizations put process improvement managers (sometimes known as software engineering process group managers or SEPG managers) on the assessment team because they have been intimately involved in the improvements to date or will be intimately involved in the future. This, however, should be done with caution. Process improvement managers can provide substantial help in organizing an assessment. They may act, for example, as the senior manager's representative in planning the assessment. However, because of their role in the organization, process improvement managers sometimes have a difficult time maintaining objectivity. They have been heavily involved in the improvement process and can be defensive if projected improvements for some reason have not been sufficiently implemented. On the other hand, they are invaluable as interviewees. On one assessment, the process improvement manager who had been on a previous assessment a year before and who was excellent at that time as a team member (he had not been assigned the role of PI manager yet) had asked not to be on the team for the second assessment because he wasn't sure he could be objective. Other team members were selected. However, at the start of the assessment, an assessed division's team member became ill and the PI manager needed to fill in. After the first day of the assessment, the PI manager argued about every point. He said people being interviewed didn't know what they were saying. At the beginning of the second day, the Lead Assessor spoke to the PI manager, explained what his behavior was doing to the assessment, and suggested that he decide if he could continue as a team member. By the end of the second day, the PI manager asked to be removed. On another first-time CMMI assessment, the Lead Assessor allowed the process improvement manager, who incidentally was also the quality assurance manager and the configuration manager, to participate as a team member. It had been explained that this person was the driving force behind the process improvement program and had a realistic view of where the organization was. For most of the assessment, the process improvement manager came to consensus with the team on individual practices. However, on the next to last day of the assessment, after the draft findings but before the team was to rate the organization, the process improvement manager said he would not rate the organization at lower than Level 2. Nothing could be said to change his mind, even though this clearly was not indicated from the draft findings, for all PAs showed that there were a number of weaknesses, of which several were significant. The managing director was called in. Another sensitive area concerns the appropriateness of having someone from the QA organization on the team. These people are sometimes overly invested in the outcome of the assessment, especially a follow-on assessment. A number of QA departments are primarily set up to defend the organization during audits. An assessment is meant to have the organization take a hard and tough look at what the organization is not doing well. In-house auditors are often trained to put the organization in the best possible light, and such instincts are hard to break. During an assessment of Company R, problems occurred early when the quality assurance team members argued vehemently over every statement that, in their opinion, was not "positive." An organization can satisfy a PA/KPA even with weaknesses as long as the team does not consider the weaknesses significant, but the organization's quality assurance people resisted this strenuously, causing the team consolidation period to last at least twice as long as it should have. On the other hand, QA personnel who are not defensive may be extremely useful on an assessment team because they have a deep knowledge of the organization's documents and processes. Two members of a team assessing Company Q were from the QA department. (One was the head of SEPG.) Both of them had been strong internal voices for improvement and were eager to present an accurate picture of the company's prospects to senior management. Not only did they not resist the assessment's analysis, but they also were able to offer invaluable information about not just how the company worked but also why it worked that waynot in a negative spirit but rather in order to facilitate the positive change that many people in the organization desired. A last category of sensitive cases involves the temptation to select people as assessment team members because they are not busy. However, not being busy could mean not being interested, which is a strong indication that such a person should not be on the team.