Ex-Baidu Researcher Ren Wu Denies Wrongdoing
Dr. Ren Wu vigorously denies the charges of cheating that led Baidu to reportedly fire the head of its Heterogeneous Computing team after the Chinese search engine developer's supercomputer team was accused of cheating in an artificial intelligence competition.
"We didn't break any rules, and the allegation of cheating is completely baseless," Wu told Enterprise Technology via email. (Wu was a speaker at last year's Enterprise HPC conference in San Diego.)
Wu, a distinguished scientist at Baidu's Institute of Deep Learning, was let go after the company was disqualified from the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), a standardized and independent AI test, where it allegedly created and used multiple accounts to run many more evaluations than its competitors each week. After ILSVRC contacted Baidu to alert the company that it had vastly exceeded the allowable number of weekly submissions to the ImageNet server, Baidu began its own internal inquiry, the company said in a blog post.
"This week, we concluded the inquiry. We found that a team leader had directed junior engineers to submit more than two submissions per week, a breach of the current ImageNet rules," Baidu's Heterogeneous Computing team wrote. "Any action that runs counter to the highest standards of academic and scientific integrity, no matter how large or small, is unacceptable to us and does not reflect the culture of our company. We have zero tolerance for such behavior and have terminated the team leader’s employment."
In addition, Baidu implemented employee training and established a scientific advisory board to prevent similar future incidents, the team said.
Initially, Baidu denied the wrongdoing, claiming it was an error.
"We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more," Wu wrote to ILSVRC. "We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process."
In his letter to Enterprise Technology, Wu shared details from the competition. "There is no official rule specify [sic] how many times one can summit results to ImageNet servers for evaluation. However, when use the webpage to upload results, this is the page.
"The key sentence here is, 'Please note that you cannot make more than 2 submissions per week.' It is our understanding that this is to say that one individual can at most upload twice per week. Otherwise, if the limit was set for a team, the proper text should be ‘your team’ instead," Wu wrote. "Our team has submitted about 200 times total in its lifespan. Our paper have five authors, and so based [on] the rule above, we should be allowed to submit around 260 times. And so, our 200 submissions were well within the 260 limits set by the rule. We obtained the world’s best result on this benchmark, and I am confident we are the best. There are two occasions that we have submitted more than 10 times per week. A mistake in our part, and it was the reason I made a public apology, requested by my management. Of course, this was my biggest mistake. And things have been gone crazy since. [To] state that we are cheating is baseless. Noted that we have obtained the world’s best results on other five benchmarks as well."
The annual ILSVRC event is hosted by Stanford University, the University of North Carolina at Chapel Hill, and the University of Michigan. This year, the sophisticated AI test evaluated supercomputers' ability to classify objects from a set of 100,000 random images into 1,000 categories; according to the rules, each team – which also included Google and Microsoft – could access the database twice a week in order to finesse their image recognition algorithms but Baidu reportedly visited the database more than 200 times in six months. In one week, it was tracked making 40 entries in five days, according to reports.
"The more exercises it did, the higher it scored in the final exam," researcher with Tsinghua University in Beijing who specializes in AI told the South China Morning Post. "But everything was done just to gain a higher score. It doesn’t mean the computer is smarter than other machines. Similarly, Chinese students who learn how to ace tests are not necessarily any better than their foreign peers at solving problems in real life."
Organizers have now banned Baidu, a global Internet company listed on NASDAQ (BIDU), with no government backing, from participating in similar events for 12 months, media reports said.
Baidu announced its entry in this year's ILSVRC competition had an error rate of 5.98 percent. On Feb. 6, 2015, Microsoft Research accomplished an error rate of 4.94 percent, while Google achieved an error rate of 4.8 percent on Feb. 11, 2015. On May 11, Baidu reported a new error rate of 4.58 percent.
Humans have a 5.1 percent error rate on ImageNet tests, HPCwire reported, meaning these latest AI systems have better image recognition capabilities than people.
Wu argues Baidu outperforms its competitors and performs with Top-1 accuracy in six major benchmarks. Deep Image posted 98.7 percent on Oxford Flowers compared with the previous highest performance of 95.3 percent, he said.
The reasons? "High performance computing, which enabled very aggressive data augmentation, working on higher resolution models, and being able to train large models.That is the reason of our success," said Wu. "Our approach is different than others commonly found in computer vision and deep learning field. It is sad that the top scientist in this country ignored the obvious trend where computational power is again the key to advance deep learning/artificial intelligence. Instead, they are trying hard to dishonor other approaches."