By Jamie Mulkey, Ed.D.

So what’s all the noise about performance-based testing? Why are companies like Microsoft, Cisco, Lotus, Novell and Hewlett-Packard pushing the testing envelope to include performance-oriented, real-life, problem-solving skills in their certification programs? And, what impact does it have on you, the test-taker?

This article will take a bare bones approach to explaining the process of creating a performance-based test. In essence, we will look at the anatomy we call performance-based testing and dissect its value, place and significance in the realm of certification testing. First, we will take a look at why this type of testing is important. Second, we will explore how these performance tests are created. Finally, we’ll look at the impact on the test-taker and how you can best prepare yourself for these types of tests.

What Is Performance- Based Testing?

Performance-based testing is a methodology for evaluating an individual by having them actually complete a real or simulated task relevant to a particular job.

Performance-based testing differs from multiple-choice testing in that a performance test allows the test-taker to create a solution to solve a problem. Multiple-choice testing allows the test-taker to choose one or more multiple responses from an already prepared list of alternative answers to solve a problem.

Another important distinction between performance-based testing and multiple-choice items is that performance-based testing allows the test-taker to come up with the correct answer using different approaches. Performance-based tests are designed to emulate what a candidate does on the job. As such, because no one solves a problem exactly the same way every time, why should there be only one way to solve a problem when completing a performance test?

Think about it, how many different ways do you solve problems when a system goes down? One time you may make configuration changes. The next time you may need to swap out hardware. The point is, performance-based testing allows for those different problem-solving approaches so that you as the test-taker can use some creativity in finding a resolution.

Putting Muscle Behind What Matters

Why is actual performance important?

Performance-based tests allow individuals to use higher-order thinking skills to solve complex problems. Higher-order thinking skills combine the use of procedural knowledge (how you do something) and declarative knowledge (why you do something).

Setting/changing your system’s IP address, for example, requires the use of higher-order thinking skills. To complete this task one would need to: 1) know why the IP address would need to be changed; 2) know which screen changes the IP configuration; 3) know how to make those changes; and 4) verify the change has taken place.

Higher-order thinking skills help measure exactly what a person does, and help reduce the chance that a candidate will pass a certification exam by book knowledge alone.

“The last thing we want is for someone to pass an exam just through study and without ever having to do the work,” said Jim Adair, Ph.D., worldwide manager of certification/assessment exam development for Lotus Corp. “The basic premise is that we hire someone to do something, not because they know how to do something.”

Performance-based tests also help address the security issues of certification exams. As we know too well, security of items is of great concern to IT certification programs. One estimate that has been thrown around is that a certification test is obsolete within 30 days simply because of test item piracy over the Internet. Performance-based tests actually combat this problem. Even when information about these tests is shared, an individual still must be able to perform the task under testing conditions.

Furthermore, some of the newer performance tests require a combination of performances that make it very difficult for someone who doesn’t know how to do the job to complete the performance successfully.

Lastly, performance-based testing helps build competence and confidence in IT professionals. Many system administrators are excited to perform real-life troubleshooting problems on live systems. They believe these tests have value in truly testing their abilities, and their success in completing these tests shows greater value to their employers.

The Central Nervous System of Performance- Based Testing

How is test performance designed for certification? Not unlike their multiple-choice counterparts, performance-based tests are designed using subject matter experts who help define what they do on the job. One of the first processes used to define these tasks is called a Job Task Analysis, a process used to understand the critical skills an individual needs to perform on the job in order to be successful.

“Job task analysis (JTA) is a way to break the job down into its simpler elements,” said Jim Olsen, Ph.D., vice president and chief measurement scientist for Alpine Media Corp. “The job analysis process also includes job synthesis to integrate the job elements into meaningful, work-related and critical performance tasks. The job analysis and synthesis process identifies meaningful job tasks, evaluates the frequency of task occurrence, level of skill or expertise required and the criticality of successful performance of the tasks.”

“Even the most cynical look at demonstrations of performance testing and say ‘that’s what I’m talking about!’” Paul Jones, Galton Technologies

Once we understand the job and its underlying elements, we can then create performance objectives that support those tasks and sub-tasks, and group them into meaningful performances for the candidate. For example:

• Tasks: Calculate number of network nodes, establish number of subnets, create a default gateway, verify IP address communicates to the network, etc.

• Performance objective: Given a set of company requirements, create an IP addressing scheme including subnetting and gateway configurations.

• Performance(s): Set up the IP addressing scheme for a 1,000-node network, create a default gateway, set up the host server, determine the number of subnets required, communicate between nodes and server.

Paul Jones, Ph.D., senior psychometrician for Galton Technolo-gies, explained what happens next:

“High-level specifications are then written for the assessment tasks. These include needed equipment, system configurations, a draft of the task prompt and a description of the variables that will be important indicators of success or failure on the tasks,” Jones said.

Once specifications are created, pass/fail criteria are calculated and agreed upon by subject matter experts. The performances are then developed and tested in both alpha and beta test environments. Typically, the alpha test will take place with internal candidates. This poses less impact on customers or company partners if the performance tests need significant changes. The beta test is usually conducted with individuals outside of the organization. Benefits to both alpha and beta testers can include free testing/re-testing, product discounts and certification recognition.

The point of both tests is to gather feedback from actual test-takers. It allows the test developer to learn whether or not candidates understand: a) the tasks that need to be performed, b) the defined specification or level of performance or expertise required and c) the intended path(s) for solving the different tasks. Test developers are also timing these tasks to make sure the appropriate amount of time is provided to candidates as they take the performance exam.

Once the process is completed, any additional changes are made and the performance test is released to the general test-taking population.

What Do Performance-Based Test Items Look Like?

Performance-based test items can take the form of simulations, live system tests, live log-in system tests and multiple-item-type tests. Some examples of performance tests for certification include:

• Microsoft’s Certified Solution Developer exam, where candidates are required to create project plans, describe the series of project events and create network maps.

• Novell’s Certified Directory Engineer, where candidates log in to live systems and perform directory optimization and maintenance tasks.

• Cisco’s CCIE performance lab, where candidates install, configure and troubleshoot wide area networks.

• Hewlett-Packard’s advanced system administrator lab, where candidates configure and troubleshoot live HP- UX systems.

• Lotus’ Certified Lotus Professional – Notes Application Developer, where candidates build multiple Notes database applications.

Although these examples all test using different formats, the commonality is in their ability to have candidates perform actual tasks.

Getting a Pulse from the IT Community

What has the response been from IT professionals to performance-based testing? According to Jones, “Even the most cynical look at demonstrations of performance testing and say ‘that’s what I’m talking about!’” IT professionals like the realism of performance-based testing. They feel it’s a truer measure of their abilities and skills.

Adair receives similar reactions from Lotus candidates with favorable feedback. “A look beneath the surface reveals that people who do the job prefer our performance exams. People who have just studied the material prefer multiple-choice exams.”

This response supports the current research in this area. Individuals who are the most confident at attempting performance-based tests are those who have practiced and applied the skills to real situations ahead of time.

The Right Chemistry

IT professionals can expect to see a continued shift toward testing increasingly more realistic job characteristics, skills and problem-solving situations. This means more performance-based testing. As Alpine Media’s Olsen suggested, “We will start to see a more synthesized approach to performance testing.”

While some performance tests of today assess job task elements in isolation, these are not true measures of performance.

“In real-life scenarios, you don’t see job elements happening by themselves, so why should we test that way?” said Olsen. “A good analogy is the Periodic Table of Chemical Elements—the table itself shows each individual element. However, we rarely see these elements work independently. We most often see them acting together—much in the same way that hydrogen and oxygen act together to create water.”

Another role you can expect to see performance tests play is one of diagnosing problem performances and prescribing solutions. This means assessing an individual’s skill level as they learn, not after they learn. As Adair said, “By assessing individuals as they learn a job, we can diagnose error patterns in performance and correct them before they become a problem.”

Let’s Get Physical

How can IT professionals best prepare for performance-based tests?

Practice, practice, practice. The best way to feel the most confident going into one of these types of tests is having performed the skills multiple times in multiple situations before the exam. Here are some ideas for getting the experience you may need:

• Use company-approved preparation materials. Many IT programs now have sample performance items, diagnostic assessments, online labs and online study groups. These materials are your best source for understanding what the company is looking for when they assess performance.

• Locate available systems. See if there are systems that aren’t being used at your place of work that you can “play” on to practice skills without damaging live company data. Or, see if a nearby university or junior college has a lab you can use.

• Try the tag team approach. With a buddy, introduce system problems to each other. Time each other to see how long it takes to solve problems. Identify and discuss the alternative solution strategies that were used and which solution strategies were most successful.

• Get assistance from a mentor. Find someone who has successfully passed the performance test/certification you will be taking. Have him or her set up some sample realistic performance problems for you to solve. Then, have them debrief your performance strengths and areas for improvement.

• Practice while you learn. If you are someone who is learning the ropes of the IT field, try getting an internship in your field of study while you are taking your certification coursework. These internships can be either with a paid or non-paid status. You can usually find them through career placement services in your learning institution. Even if the internship is one where you do not get paid, you will have gained some great experience, practical hands-on application toward your certification and possibly a job lead.

Conclusion

Performance testing is a realistic approach to understanding an individual’s readiness for job performance. Be prepared to start seeing more of these types of tests in the future.

As Jones said, “Certification programs are responding to the perception that exams which require more authentic performance will increase the perceived value of their programs.”

Accordingly, your credential will mean more because of the added value of successfully completing a performance -based test.

Finally, go beyond the book to practice those skills that are defined as critical to the job. Find different ways to prepare, share system time with a partner, practice on school equipment or acquire an internship. Any of these methods will more than likely make you a more confident test-taker and a successful candidate.

Jamie Mulkey, Ed.D. is the worldwide certification program manager for Hewlett-Packard. Jamie has worked with both company-wide and industry-based certification programs over the last several years and serves as adjunct faculty for the University of Southern California and University of Phoenix. Jamie can be reached at jamie_mulkey@hp.com.