Sometimes NASA is looking more for ideas than algorithms, as illustrated by series of challenges involving the agency's Planetary Data System (PDS), a 100-terabyte archive of images and data from 30 years of planetary exploration. “We knew we needed better algorithms to search the silos of data, and we knew what we wanted to search on, but what else could you want to search for?” asks Crusan.
“So we ran an ideation [idea-generation] challenge. Anybody can compete, and some like it better than creation,” he says. The challenges were aimed at high school students and teachers, looking for ideas for web-based applications and teaching tools to exploit the planetary data. “After that we ran more detailed challenges focused on implementation.”
A fundamental lesson from NASA's experience with challenges is that problem decomposition is key, says Crusan. “There is value in decomposing a problem to frame it to a community of non-domain experts. That's a valuable skill set, challenge or not,” he says. “Decomposition of the problem allows us to interest a wider audience, but takes longer than you would guess at the beginning.”
NASA understands challenges and platforms better than the other customers, says Hughes, and is bringing people into TopCoder that are new to the community, and to challenges. “In the ISS Challenge, three or four of the top 10 winners are new to the platform,” says Crusan. “We need to go back and ask them why they participate, and how much time they put in,” he says. “This is just as much about getting at the why— that's part of government's role.”
For Carlo Piovesan, a programmer in Italy and first-place finisher in the ISS Longeron Challenge, it was his second NTL problem. “I found them interesting and challenging,” he says, describing ISS Longeron as “very cool.” It took him two weeks to solve, and he says the NASA challenges are worthwhile. “I have tried a few new things, gone back to studying topics which I had forgotten and explored a lot of possible ideas before implementing the best ones.”
Others cited as reasons for participating the prize's size, problem quality, competitor numbers and that it was a NASA competition. “You have to admit it has some ring to it,” says participant Peter Szilagyi. Hou Qiming entered after seeing the problem involved optimizing shadows. “I'm a professional of computing shadows—for the entertainment industry. I'm curious how far computer-game techniques can get you in space,” he says.
“We are definitely tapping into people that would not participate in small business awards, requests for proposals, grants, etc.,” says Crusan. “We are getting ideas that are not from the usual suspects, and the value is quite large. You have to go into a challenge with your eyes wide open, as you do not know where a solution will come from.”
Designing a good challenge takes effort. “If you've seen one prize, you've seen one prize. They are all very different,” says Gustetic, depending on whether the desired outcome is a broad range of ideas, a point solution, or “a skill that is gained by competing in an educational challenge or a network prize.”
How a challenge is scored is important. “You need to get to objective scoring to take out any biases and get a really unbiased diversity of solutions,” says Crusan. “Ideation challenges are the most subjective, but we still score them with the help of subject-matter experts. With implementation challenges you can get to very objective scoring, such as error rate, cycle time and power usage.”
ISS Longeron was “extremely objective,” he says, with final rankings based on the raw score of power generated. “I can definitely say that the biggest and most challenging work was done by the people preparing the contest,” says a participant.