“TopCoder's rating system is central to participation, particularly on the algorithmic side,” says Andy LaMara, program director. “Handles on the forum are color-coded to indicate the individual's proficiency. This has an interesting effect. Red handles are at the very top, several standard deviations above their peers. When they weigh in, others listen,” he says. And if a TopCoder “red handle” applies to Google “they skip five of the nine interview steps.”
Sometimes NASA is looking more for ideas than algorithms, as illustrated by series of challenges involving the agency's Planetary Data System (PDS), a 100-terabyte archive of images and data from 30 years of planetary exploration. “We knew we needed better algorithms to search the silos of data, and we knew what we wanted to search on, but what else could you want to search for?” asks Crusan.
“So we ran an ideation [idea-generation] challenge. Anybody can compete, and some like it better than creation,” he says. The challenges were aimed at high school students and teachers, looking for ideas for web-based applications and teaching tools to exploit the planetary data. “After that we ran more detailed challenges focused on implementation.”
A fundamental lesson from NASA's experience with challenges is that problem decomposition is key, says Crusan. “There is value in decomposing a problem to frame it to a community of non-domain experts. That's a valuable skill set, challenge or not,” he says. “Decomposition of the problem allows us to interest a wider audience, but takes longer than you would guess at the beginning.”
NASA understands challenges and platforms better than the other customers, says Hughes, and is bringing people into TopCoder that are new to the community, and to challenges. “In the ISS Challenge, three or four of the top 10 winners are new to the platform,” says Crusan. “We need to go back and ask them why they participate, and how much time they put in,” he says. “This is just as much about getting at the why— that's part of government's role.”
For Carlo Piovesan, a programmer in Italy and first-place finisher in the ISS Longeron Challenge, it was his second NTL problem. “I found them interesting and challenging,” he says, describing ISS Longeron as “very cool.” It took him two weeks to solve, and he says the NASA challenges are worthwhile. “I have tried a few new things, gone back to studying topics which I had forgotten and explored a lot of possible ideas before implementing the best ones.”
Others cited as reasons for participating the prize's size, problem quality, competitor numbers and that it was a NASA competition. “You have to admit it has some ring to it,” says participant Peter Szilagyi. Hou Qiming entered after seeing the problem involved optimizing shadows. “I'm a professional of computing shadows—for the entertainment industry. I'm curious how far computer-game techniques can get you in space,” he says.
“We are definitely tapping into people that would not participate in small business awards, requests for proposals, grants, etc.,” says Crusan. “We are getting ideas that are not from the usual suspects, and the value is quite large. You have to go into a challenge with your eyes wide open, as you do not know where a solution will come from.”
Designing a good challenge takes effort. “If you've seen one prize, you've seen one prize. They are all very different,” says Gustetic, depending on whether the desired outcome is a broad range of ideas, a point solution, or “a skill that is gained by competing in an educational challenge or a network prize.”