How a challenge is scored is important. “You need to get to objective scoring to take out any biases and get a really unbiased diversity of solutions,” says Crusan. “Ideation challenges are the most subjective, but we still score them with the help of subject-matter experts. With implementation challenges you can get to very objective scoring, such as error rate, cycle time and power usage.”
ISS Longeron was “extremely objective,” he says, with final rankings based on the raw score of power generated. “I can definitely say that the biggest and most challenging work was done by the people preparing the contest,” says a participant.
Challenges designed to generate concepts and ideas, rather than algorithms, can involve “a lot of back-and-forth with the customer,” Crusan says. It helps to engage the solvers and “a post-challenge post-mortem is very valuable to the community,” adds LaMara.
What becomes of the intellectual property (IP) generated depends on the platform and the challenge. For standard challenges on InnoCentive and TopCoder, IP provisions are part of the submission process. Some winners will grant a government-purpose license while others prefer to transfer their solution to an open-source repository.
When NASA's existing platform contracts come to an end, there will be new awards to continue running challenges and bring in new communities. The goal is to enable mission directorates to “order up a challenge like a laboratory process,” says Crusan. “We are slowly spreading the lessons learned. We have established a working group across the agency to set up the policies and methods. It is in its very early stages.”
NASA programs have shown a willingness to use solutions generated by the challenges, says Crusan. “If it's a sound solution, they will use it. But we do make them commit up front, and that takes time.” Some solutions can be infused directly into programs, while others “help guide our internal investment,” he says. “We can use challenges to seed a program with top-quality ideas, as a technology accelerator.”
Tap on the icon in the digital edition of AW&ST for more about NASA and AFRL challenges on crowd-sourcing and problem-solving platforms, or go to AviationWeek.com/innovation