Certification testing scrutinized at Performance Testing Council summit

At the two-day summit of the Performance Testing Council in Park City, Utah, representatives of some of the biggest IT certification entities (including Microsoft, CompTIA and Citrix) in the industry gathered to discuss a common concern. Namely, what can test creators and certification organizations do to improve certification testing? GoCertify has a strong interest in testing and certification, so we had a seat at the table on Wednesday for some engaging and illuminating conversations.

One of the presenters was Clyde Seepersad of the nonprofit Linux Foundation, which recently launched its own Linux certification program. Seepersad said that the Linux Foundation wanted to steer clear of offering multiple-choice exams at testing centers. Instead, the group wanted its certification exams to be performance based and distribution-flexible — available to certification candidates at whatever time suits their schedule and aptitude. All that convenience comes at a price, of course: It's much harder to proctor (or monitor) a test candidate who's not sitting in a testing center.

The solution wasn't easy, but the Linux Foundation testing team devised a solution that involves multiple layers of security, including screen sharing, webcam monitoring, and various audio and video recording techniques. There's even a three-day hold on exam results to allow time for security measures taken to be reviewed.

Test security is a core concern area for most people involved in IT certification testing. Wallace Judd of test design firm CertNT said that one way to combat cheating is to do a better job of authoring test questions. When answering a questions involves more than simply selecting A, B or C, a candidate has to undertake more serious preparation just to feel confident taking the test in the first place. Judd explained that authoring test questions is a detailed process that, when done right, involved completing multiple steps.

Judd also discussed the importance of meeting standards set by ASTM International (originally known as the American Society for Testing and Materials). ASTM has a new standard developed specifically to improve cert testing: E2849 — Standard Practice for Professional Certification Testing. If followed, Judd said, E2849 can improve the design, implementation and deployment of performance-based certification tests.

Another means of delivering better exams was touched on by Brian Adams of Alpine Testing Solutions. One of the most important considerations of test creation, Adams said, is validity. Do the questions on a test actually measure what they are intended to measure? If a candidate needs to know how to configure a network switch, then a valid question must show whether the candidate could actually complete that task.

Adams said that security failures often damage the validity of questions because candidates take a shortcut to the answer instead of working it out for themselves. The use of data forensics by test administrators can reveal, in many cases, when the validity of a question has been compromised, alerting testing personnel to an exam security breach. Proactive cheating prevention measures, such as frequently refreshing the content of certification exams can help to ensure that validity remains strong.

The Performance Testing Council holds summit meetings twice a year to promote adoption of performance-based testing across a range of professional and educational sectors. The group has held more than 22 summit events since 2005, and began hosting webinars last year to further its reach.