So this actually happened, Mark Zuckerberg, CEO of Facebook has attended the annual NIPS conference in Lake Tahoe, Nevada. I didn’t attend, but from what I gathered from the social networks, it really happened. He participated in a panel discussion at the deep learning workshop, was present at a Facebook party (from where the first picture above is), where it was announced that Yann LeCun will head the new AI lab of Facebook.
I think Mark Zuckerberg attending is pretty remarkable. The hype around deep learning has increased this year with stories like Google image search being mostly powered by deep learning, or the acquisition of the Toronto based deep learning startup DNNresearch by Google.
The media has picked up on Deep Learning just the way they have with Big Data and Data Science, occasionally leading to articles like this one claiming that the methods are even too smart for engineers to understand. If you even have a little experience with machine learning, you know that this is no exception, it’s actually the rule. While you will hopefully have understood how the learning algorithms works, “understanding” how an SVM or a deep network performs its predictions in detail are hidden behind thousands of elementary computations.
Don’t get me wrong, I think that any kind of publicity which informs the general public of the importance of machine learning and data analysis is good, even if it is sometimes misinformed.
Still it’s an interesting to see how NIPS has transformed over the years. Originally founded in 1987, it began as a very neuro-centric conference (as was the fashion back then). The name spelled out, Neural Information Processing Systems still hints at that fact. For the first 15 years or so, biological neuroprocessing systems were still a focus, with papers reporting on actual biological findings. The conference was always single track (as opposed to ICML, for example) which meant that scoring a talk would guarantee quite an impact for your work. NIPS has always been very competitive with acceptance rates well below 20%, and its extensive poster sessions are legendary, running well past midnight with frustrated hotel staff turning of the air condition in an attempt to drive the attendants out.
The first time I attended NIPS was in 2001. It really was a different time back then. There was no wifi, I didn’t even have a laptop. NIPS was always pretty huge, but it was also always very academic with mathematically challenging presentations, always trying to push the envelope further. From year to year, the fashion changed, from kernel learning to non-parametric Bayes, and now back to deep learning, doing the full circle, in a way.
It’s probably inaccurate to say that NIPS was only academic, because there was always strong research labs within bigger companies, for example like the AT&T Research Lab (where Yann LeCun used to work, too), or at sites like Microsoft Research in Cambridge. There are also many who worked in the industry but were integral parts of the communtiy, like (and this is really a very short list of people who came to mind), Corrina Cortes, Samy Bengio (Google), Jon Langford (formerly Yahoo), Alex Smola (first Yahoo, then Google, now back to being professor at CMU), or people like Ralf Herbrich who moved from Microsoft Research to Facebook and now to Amazon.
At the same time, I always had the feeling that the Big Data and Data Science hype did certainly not originate at NIPS but was more like an external event to which the NIPS community had to adapt first. Part of the reason is that Big Data is a lot about infrastructure and databases, whereas NIPS always focussed more on algorithms. So while we’ve certainly been doing and thinking about large scale learning, we did it not through massively scaling out, but by finding algorithms which were able to deal with more data first.
Actually, if I recall correctly, back in 2006 when the paper “Map-Reduce for Machine Learning for Multicore” was presented, halfway through the talk exactly the Yann LeCun who will now head the AI lab at Facebook took the microphone to say that he didn’t consider their implementation of neural network learning properly scaled out, because they’re “just doing microbatches”. Of course he was right.
Still, the relevance of what the NIPS community was doing was undeniable, in particular because many of its members became those who engineered complex data analysis systems based on Big Data infrastructure. Just to give an example, Samy Bengio was responsible for an overhaul of Google Image search long before deep learning became mainstream again.
Last year has seen quite an investment into machine learning. I already mentioned the deep learning related acquisitions. Then there is also the new machine learning lab in Berlin headed by Ralf Herbrich and now the new AI lab at Facebook. Then there are smaller companies like Berlin based Zalando (a fashion retailer) who have practically hired a year of fresh Ph.D.s from our lab alone.
So Mark Zuckerberg attending is part logical consequence of what happened the last years, but also crosses a threshold because he is no longer “one of those ML guys in the back” but actually one of the most influential people of Silicon Valley.
What this means, I don’t know, but something is happening. My brain melts.
Edit Dec 11, 2013: Added small paragraph mentioning new Amazon and Facebook labs.
Posted by Mikio L. Braun at 2013-12-10 21:27:00 +0000