Matti Ahlgren

FCAI to design an AI software toolbox to ease transition into utilising AI solutions

Finnish Center for Artificial Intelligence received €1M funding from The Future Makers program of the Technology Industries of Finland Centennial Foundation and the Jane and Aatos Erkko Foundation. They have awarded funding for seven research projects with 3.2 million euros in total. The projects take on issues that are set to shape the future of humankind.

The largest individual funding from the foundations, one million euros, went to the Finnish Center for Artificial Intelligence. FCAI is building a nation-wide competence center that brings together the top artificial intelligence research across fields in Finland.

With the funding, FCAI will build an AI software toolbox to enable companies to have a smoother transition into using artificial intelligence methods. Even though AI has been talked about the world over for quite some time now, it’s full potential still remains largely untapped. The development of new solutions is slowed down by a lack of top experts, of which there’s already a fierce global competition. 

‘We are designing software tools with which companies can develop the AI solutions they need—instead of building AI-assisted software tools from scratch. This means you can apply AI without having extensive in-depth knowledge of AI. Our overall goal is to enable the Finnish technology industry to retain control over the core AI technology they use,’ says Samuel Kaski, Academy Professor at Aalto University.

Lappeenranta University of Technology and Aalto University will jointly a run a group led by Professor Aki Mikkola and FCAI Professor Perttu Hämäläinen (Aalto University) who will will fashion a new way to control machinery with a combination of AI and high-performance computing. The machinery would then be able to comprehend the causality behind different kinds of motion and operate independently even in dynamic surroundings. The project received 230.000 euros from the foundations.

The teams of Aalto University Professors Katja Hölttä-Otto and Mikko Sams received a 200.000-euro funding for their research on making empathy part of technology development. Their goal is to increase designers’ understanding of the needs and behavior of end-users so that the finished products or services meet actual needs.

Press release from Technology Industries in Finland (in Finnish): teknologiateollisuus.fi/fi/ajankohtaista/uutiset/tutkijat-laittavat-tekoalyn-toihin-saatioilta-32-miljoonaa-tekoalyn

Further information:
Laura Juvonen
CEO, Teknologiateollisuuden 100-vuotissäätiö
tel. +358 40 589 6263
laura.juvonen@teknologiateollisuus.fi

Marja Leskinen
Secretary General, Jane and Aatos Erkko Foundation
tel. +358 40 514 6969
mkl@jaes.fi

Share

AI Forum—European Ministerial Conference on AI at Aalto University 8–9 Oct

Political leaders, policy makers, experts, entrepreneurs and industry leaders from all over Europe and beyond will gather in October at Aalto University for the AI Forum to discuss how AI is transforming our world, society and industry.

Renowned speakers and panelists include Jacques Bughin (Director of the McKinsey Global Institute), Pekka Ala-Pietilä (Chairman of the EU High-Level Expert Group on AI and Chairman of AI Finland Programme), Robert Gentz (co-founder and co-CEO at Zalando SE), Ann Mettler (head of the European Political Strategy Centre EPSC), and Risto Siilasmaa (Chairman of Nokia and F-Secure).
See all speakers here: www.tekoalyaika.fi/en/ai-forum-2018/speakers.

FCAI Professors Samuel Kaski (Aalto University) and Teemu Roos (University of Helsinki) will be hosting two round tables at AI Forum. Kaski will lead a discussion about European competitive advantage from AI research, Roos about reskilling and upskilling for the AI era.
See full programme: www.tekoalyaika.fi/en/ai-forum-2018/programme

The AI Forum 2018 is co-hosted by the Ministry of Economic Affairs and Employment of Finland and the European Commission, and organised in partnership with Aalto University.

AI Forum will be held 8–9 October 2018 at Dipoli, Aalto University. There will be a live webcast, see www.tekoalyaika.fi/en/ai-forum-2018 for details.

Share

FCAI–City of Espoo collaboration covered by Yle

The collaboration between FCAI and the City of Espoo to make use of the city’s databases to create new AI-assisted services and solutions gets coverage in Yle (in Finnish): https://yle.fi/uutiset/3-10413353.

Espoo’s recent experiments in child protective services show promise. With AI tools, 280 factors, which anticipate future needs for help and services, were isolated. The results are only experimental for now, but give an indication how cities could improve their social services with AI-assistance.

Many major cities in Finland are looking into how benefit from AI tools and methods and collaboration with research, says FCAI Professor Teemu Roos (University of Helsinki).

Roos says, ‘Cities and municipalities are to gain from AI just as much as companies. AI speeds up data processing and helps in creating projections and policy recommendations, but the greatest benefit is the ability combine data sets in ways that haven’t been feasible without AI.’

You can’t tell whether an online restaurant review is fake—but this AI can

Researchers in the Secure Systems group at Aalto University, led by Professor N. Asokan find AI-generated reviews and comments pose a significant threat to consumers, but machine learning can help detect the fakes.

Sites like TripAdvisor, Yelp and Amazon display user reviews of products and services. Consumers take heed: nine out of ten people read these peer reviews and trust what they see. In fact, up to 40% of users decide to make a purchase based on only a couple of reviews, and great reviews make people spend 30% more on their purchases.

Yet not all reviews are legitimate. Fake reviews written by real people are already common on review sites, but the amount of fakes generated by machines is likely to increase substantially.

According to doctoral student Mika Juuti at Aalto University, fake reviews based on algorithms are nowadays easy, accurate and fast to generate. Most of the time, people are unable to tell the difference between genuine and machine-generated fake reviews.

‘Misbehaving companies can either try to boost their sales by creating a positive brand image artificially or by generating fake negative reviews about a competitor. The motivation is, of course, money: online reviews are a big business for travel destinations, hotels, service providers and consumer products,’ says Mika Juuti.

In 2017, researchers from the University of Chicago described a method for training a machine learning model, a deep neural network, using a dataset of three million real restaurant ratings on Yelp. After the training, the model generated fake restaurant reviews character by character.

There was a slight hiccup in the method, however; it had a hard time staying on topic. For a review of a Japanese restaurant in Las Vegas, the model could make references to an Italian restaurant in Baltimore. These kinds of errors are, of course, easily spotted by readers.

To help the review generator stay on the mark, Juuti and his team used a technique called neural machine translation to give the model a sense of context. Using a text sequence of ‘review rating, restaurant name, city, state, and food tags’, they started to obtain believable results.

‘In the user study we conducted, we showed participants real reviews written by humans and fake machine-generated reviews and asked them to identify the fakes. Up to 60% of the fake reviews were mistakenly thought to be real,’ says Juuti.

Juuti and his colleagues then devised a classifier that would be able to spot the fakes. The classifier turned out to perform well, particularly in cases where human evaluators had the most difficulties in telling whether a review is real or not.

The study was conducted in collaboration with Aalto University’s Secure Systems research group and researchers from Waseda University in Japan. It was presented at the 2018 European Symposium on Research in Computer Security in September.

The work is part of an ongoing project called Deception Detection via Text Analysis in the Secure Systems group at Aalto University.

Research articles:
Mika Juuti, Bo Sun, Tatsuya Mori, N. Asokan:
Stay On-Topic: Generating Context-specific Fake Restaurant Reviewshttps://arxiv.org/abs/1805.02400

Hate speech-detecting AIs are fools for ‘love’

State-of-the-art detectors that screen out online hate speech can be easily duped by humans, shows new study by the Secure Systems group at Aalto University.

Hateful text and comments are an ever-increasing problem in online environments, yet addressing the rampant issue relies on being able to identify toxic content. A new study by the Aalto University Secure Systems research group has discovered weaknesses in many machine learning detectors currently used to recognize and keep hate speech at bay.

Many popular social media and online platforms use hate speech detectors that a team of researchers led by Professor N. Asokan have now shown to be brittle and easy to deceive. Bad grammar and awkward spelling—intentional or not—might make toxic social media comments harder for AI detectors to spot.

The team put seven state-of-the-art hate speech detectors to the test. All of them failed.

Modern natural language processing techniques (NLP) can classify text based on individual characters, words or sentences. When faced with textual data that differs from that used in their training, they begin to fumble.

‘We inserted typos, changed word boundaries or added neutral words to the original hate speech. Removing spaces between words was the most powerful attack, and a combination of these methods was effective even against Google’s comment-ranking system Perspective,’ says Tommi Gröndahl, doctoral student at Aalto University.

Google Perspective ranks the ‘toxicity’ of comments using text analysis methods. In 2017, researchers from the University of Washington showed that Google Perspective can be fooled by introducing simple typos. Gröndahl and his colleagues have now found that Perspective has since become resilient to simple typos yet can still be fooled by other modifications such as removing spaces or adding innocuous words like ‘love’.

A sentence like ‘I hate you’ slipped through the sieve and became non-hateful when modified into ‘Ihateyou love’.

The researchers note that in different contexts the same utterance can be regarded either as hateful or merely offensive. Hate speech is subjective and context-specific, which renders text analysis techniques insufficient as stand-alone solutions.

The researchers recommend that more attention be paid to the quality of data sets used to train machine learning models—rather than refining the model design. The results indicate that character-based detection could be a viable way to improve current applications.

The study was carried out in collaboration with researchers from University of Padua in Italy. The results will be presented at the ACM AISec workshop in October.

The study is part of an ongoing project called Deception Detection via Text Analysis in the Secure Systems group at Aalto University.

Research article:

Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, N.Asokan:
All You Need is "Love": Evading Hate-speech Detection.
https://arxiv.org/abs/1808.09115


Elements of AI becomes the most popular course at University of Helsinki, ever

Image: Tuomas Sauliala / Reaktor

Image: Tuomas Sauliala / Reaktor

The Elements of AI MOOC organised by FCAI and Reaktor awarded diplomas to the first graduates and received endorsement from the President of Finland in the graduation ceremony held 6 September 2018. With approximately 90 000 registered participants, it has become the most popular course ever at the University of Helsinki.

See write-up in the main Finnish daily Helsingin Sanomat (in Finnish): https://www.hs.fi/teknologia/art-2000005817486.html.

Professor Teemu Roos (FCAI, University of Helsinki) emphasised in his speech at the ceremony the societal implications AI technologies will bring—and how we should take them into account by making AI literacy accessible for everyone.

Roos says, ‘AI is not a matter of the future. It is really not a matter of robot uprisings, or transcending humanity. AI is a matter of the present day, every day. AI and algorithms have been woven into the digital fabric that connects us to each other and to the world at large. Communication and access to information has been greatly enhanced by technology.

Because of the great power in AI, we must make sure that the rules that determine how and for what purpose AI can be used are up to date and in line with what we think is right and just. In a democratic society, the power is with the people. This can only be true if the people have access to knowledge, so that they can take part in forming the rules through legislation.’

You can read Roos’s entire speech here.

Share

First Elements of AI graduates receive diplomas—President of Finland to speak at the ceremony

The Elements of AI online course (MOOC) by FCAI (University of Helsinki) and Reaktor has attracted over 90 000 people from 57 countries to sign up. The first graduates will receive their diplomas 6 September 2018, and President of Finland Sauli Niinistö will address the graduates at the ceremony.

A Finnish version of the course will be presented at the ceremony.

Elements of AI graduation ceremony: 6 Sept at 10AM in the Great hall of the University of Helsinki (Fabianinkatu 33).

Read more about the course and sign up!
elementsofai.com

Share

Make-up of superbug MRSA revealed—with prospective methods to prevent inter-species transfer

An international team of researchers, including FCAI Professor Jukka Corander (University of Oslo, University of Helsinki), has mapped the entire genetic make-up of over 800 strains of the common superbug MRSA, or Methicillin-resistant Staphylococcus aureus. The bacteria is known best for its world-wide prevalence in hospital environments.

Superbugs like MRSA are resistant to most antibiotics and can lead to life-threatening or deadly infections in humans. MRSA is common also in live stock and causes, for instance, mastitis in cows and skeletal infections in chickens.

According to the study, humans are the most likely original carrier of the bacteria, but the source for the current strains infecting humans are cows. The researchers now understand the mechanisms of how the bacteria is able to transfer from one species to another thanks to a thorough understanding of its genome. When jumping species, the bacteria is able to acquire new genes that help it thrive in the new environment.

Detailed analysis of the changes in the genetic make-up of the bacteria achieved now could offer a way to develop new anti-bacterial treatments. Knowledge of the transmission can also help devising strategies to prevent the bacteria from developing antibiotic resistance, or to block its access to humans altogether.

The results have been published in Nature Ecology & Evolution.
Link to the article: nature.com/articles/s41559-018-0617-0
Read more on Sanger Institute website: sanger.ac.uk/news/view/gene-study-pinpoints-superbug-link-between-people-and-animals. 

 

Share

StanCon 2018 at Aalto University

StanCon 2018, 29–31 August, introduces cutting-edge methods and applications for statistical modelling—ranging from galaxy clusters to social media, brain research, and anthropology. In Finland, AI research is particularly strong in the field of medicine.

‘Statistical modeling can be used, for example, to improve the safety of drug testing in children. The time it takes for a child’s body to metabolise a drug depends not only on the weight of the child, but also on the ability of the liver to process the drug. The dosage size of the drug should, then, be reduced more than the weight alone would suggest. Modelling methods can be used to evaluate the effects of drugs on an individual level,’ says Aki Vehtari Professor at Aalto University and FCAI, and member of Stan development team.

One of the keynote speakers at the conference, Maggie Lieu, a researcher at the European Space Agency, uses statistical modeling to determine the mass of galaxy clusters.

'Hierarchical modeling has several advantages when there are millions of variables and a lot of noisy data in space. Using modelling, I can get meaningful results in up to ten minutes and study clusters of galaxies in one go instead of a single galaxy group at a time.'

Read more about the StanCon program: 
http://mc-stan.org/events/stancon2018Helsinki.

Share

“The next AI generation will be the big revolution. We haven’t seen anything yet.”

FCAI Professor Petri Myllymäki gave a talk at a conference on innovation in the EU in Brussels organized by Science|Business.

Even though public bodies and large companies are investing heavily on AI technology and development, Myllymäki noted in the conference, according to a story by Science|Business that the current AIs are stil, in fact, in their infancy. Existing AI methods are good with big data sets that are properly annotated, but they are still “black-boxed”: there is little or no way of knowing how the AI came up with a solution it did, or, what happens between the input and the output.

Myllymäki said that ‘[current AIs] work in some narrow environments remarkably well, if you have a lot of data. They have been productised and there are nice tools you can use, so these are the primary reasons for the current AI revolution.’

‘The next generation will be the big revolution. We haven’t seen anything yet.’

What we haven’t seen is Real AI. Making Real AI a reality is very much at the core of what FCAI strives to do: to create AI tools that are transparent, able to explain themselves to the user, use scarce resources efficiently and take not of user privacy and security in all steps.

Read the write-up of the whole conference from Science|Business here: https://sciencebusiness.net/news/not-too-late-europe-ai-race-experts-say

VTT joins FCAI as third founding member

Technical Research Centre of Finland VTT will join Finnish Center for Artificial Intelligence FCAI launched by Aalto University and the University of Helsinki as a third founding member.

VTT will bring their strong industry networks and their know-how in applied technology to the FCAI community. Their help will enforce FCAI’s ability to put the top research in both founding universities into far-ranging and efficient use in companies, public organisations and society at large.

FCAI promotes high-quality research and education on artificial intelligence in Finland and the applicability of AI to benefit companies and society. VTT will expand FCAI’s ability to speed up the necessary renewal and competitiveness of Finnish industry through AI-based innovations.

FCAI strives to make the new generation of AI methods a reality: create AIs that are understandable, trustworthy, and data-efficient. FCAI's goal is to expand into a national network of universities, companies and research institutions who will lay the groundwork for Finland to become a global leader in AI research and AI applications.

Growth in any strand of industry depends on the ability to make use of cutting-edge technology. Artificial intelligence is the key leverage here.

‘Our vision is to bring our high-class research in several strands of artificial intelligence to benefit people's every-day lives, companies and public bodies. FCAI’s impact is a potent mixture of research, a network of startups, doctoral education and competence building in AI, new innovative products and services, and smart experiments in public administration,’ says Head of FCAI, Academy Professor Samuel Kaski.

‘The single most significant growth factor now is applying artificial intelligence and ICT in general. For citizens, new innovations and solutions will bring a change in work content, professional skills, and the services society provides. AI will be able to make, for instance, medical care more efficient and personalised,’ says Tua Huomo, Executive Vice President at VTT.  

FCAI is building a national hub of universities, research institutes, industry and the private sector and public organisations with strong international networks. The FCAI community is constantly expanding with new memberships and projects.

New metagenomics tool mSWEEP accurately characterises mixed bacterial colonies

Determining the composition of bacterial communities at strain level resolution is critical for many applications in infectious disease epidemiology and in bacterial ecology.

Using the latest advances in computational inference and sequence analysis, an international team involving close collaboration with leading institutions on bacterial genomics, including the Wellcome Sanger Institute and University of Oxford, led by professors Jukka Corander and Antti Honkela (both in FCAI) has developed a new metagenomics tool called mSWEEP, which goes significantly beyond the state of the art in this field.

The effectiveness of mSWEEP is demonstrated with infection data from major human pathogens and it is expected to pave the way for entirely new approaches to addressing important biological and clinical questions about inter-strain competition, dissemination of resistance and virulence.

The research article: High-resolution sweep metagenomics using ultrafast read mapping and inference.

Tackling bacteria with statistics – simulator-based inference for drug development

Professor Jukka Corander (FCAI, University of Helsinki, University of Oslo) interviewed with the Academy of Finland about his work on new kinds of artificial intelligence methods for drug and vaccine development and for analysing bacterial populations.

The interview in Finnish here: http://www.aka.fi/fi/akatemia/media/Ajankohtaiset-uutiset/2018/tilastotieteella-bakteerien-kimppuun

In FCAI, Professor Corander is the Responsible Coordinator of the Simulator-Based Inference research group.

Espoo becomes a member of FCAI: researchers to develop artificial intelligence for the services of the city

The City of Espoo has become a member of the Finnish Center for Artificial Intelligence FCAI. FCAI is a research centre launched by Aalto University and University of Helsinki, which gathers together the best artificial intelligence researchers in Finland. FCAI's objective is to make the most advanced methods of artificial intelligence available to enterprises, organisations and society.

The City of Espoo sees that developing artificial intelligence together will be beneficial for the whole innovation community from enterprises to R&D organisations and the inhabitants in Espoo.

“For a researcher, the data in the databases of the city of Espoo and the shared databases of the Helsinki metropolitan area is very interesting. Especially the innovative start-up companies in the area and Espoo's desire to be profiled as a pioneer in the use of intelligent technologies set a good basis for cooperation with researchers developing artificial intelligence. We have all the prerequisites to expand our cooperation to other research centres and other cities as well,” says the Head of FCAI, Academy Professor Samuel Kaski.

“On the one hand, researchers need data for the development of artificial intelligence methods and technology, and public organisations have this data. On the other hand, we as a city get to use the methods, technologies and the latest knowledge of artificial intelligence research in the development of our services,” says Tomas Lehtinen, data analyst consultant for the City of Espoo.

Restoring images without clean data

There are several real-world situations where obtaining clean training data is difficult. For instance, low-light photography – astronomical imaging, for example – physically-based image synhesis and magnetic resonance imaging are such cases.

Aalto University and FCAI professor Jaakko Lehtinen with his team from NVIDIA and MIT postdoctoral researcher Miika Aittala show in their paper accepted to the International Conference on Machine Learning ICML 2018 that it is possible to recover signals under complex corruptions without observing clean signals, at performance levels equal or close to using clean target data.

They have applied basic statistical reasoning to signal reconstruction by machine learning — learning to map corrupted observations to clean signals — with a simple and powerful conclusion: under certain common circumstances, it is possible to learn to restore signals without ever observing clean ones, at performance close or equal to training using clean exemplars.

The team applies their methods to photographic noise removal, denoising of synthetic Monte Carlo images, and reconstruction of MRI scans from under-sampled inputs. All cases are based on only observing corrupted data.

Share

FCAI's and Reaktor's AI MOOC has attracted 30 000 participants

Elements of AI open-for-all online crash course on artificial elements provided the University of Helsinki and Reaktor has already 30 000 registered participants. The course launched 14 May 2018. 

More than 100 organizations have taken a pledge to support their employees in learning about artificial intelligence in #AIChallenge campaign. They include  Finnair, StoraEnso, OP, Nordea, Nokia, Telia, Posti. Read more about the challenge: elementsofai.com/ai-challenge.

Some recent media write-up of the course:
Yle News
Endgadget

elementsofai.com

Blurred lines between search and recommendation: interactive data exploration

Recent work by FCAI researchers Tuukka Ruotsalo, Tung Vuong, Khalil Klouche, Salvatore Andolina and Giulio Jacucci investigate the role of interactive machine learning in exploring data. Their particular emphasis is on efficient user input and transparency of recommendation – the ‘how’ and ‘why’ of search queries, respectively.

In one case, users explore points of interests using available social content and review data from Yelp Phoenix, Arizon (11,000 PoI; 225,000 reviews; 42,000 users): personal preferences, tags combined with personal preferences, and tags and social ratings combined with personal preferences. The transparency (provenance) of recommendation was decisive as the combination of social rating information and personal preference information improves search effectiveness and reduce the need to consult external information.

Klouche, K., Ruotsalo, T., Cabral, D., Andolina, S., Bellucci, A., & Jacucci, G. (2015, April). Designing for exploratory search on touch devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4189-4198). ACM.

In other research in exploring the entire data set of scientific publications over 50 million papers, the FCAI team have been able to show – using similar graph-based machine learning – how to support efficient user input in exploration by allowing users to easily interact with entities such as people, keywords and documents.

The research is a prime example of interactive AI and has important implications for developing system that aid exploration of products, documents, points of interest or people. The examples showcase how online machine learning can make use of user input for interactive AI.

See the research article:
https://www.sciencedirect.com/science/article/pii/S0306457316306045

Share

A new paradigm of ordinary differential equations

Aalto University and FCAI professor Harri Lähdesmäki has with his colleagues introduced a new paradigm of non-parametric ordinary differential equations modeling that can learn the underlying dynamics of arbitrary continuous-time systems without prior knowledge.

For many complex systems it is practically impossible to determine equations or interactions that would govern the underlying dynamics. In these settings, a parametric ODE model cannot be formulated. Lähdesmäki and his team have now overcome this issue. They propose to learn non-linear, unknown differential functions from state observations using Gaussian process vector fields within the exact ODE formalism.

They demonstrate the model’s capabilities to infer dynamics from sparse data and to simulate the system forward into future.

See article by Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, Harri Lähdesmäki, ‘Learning unknown ODE models with Gaussian processes’:
https://arxiv.org/abs/1803.04303

The paper has been accepted to the International Conference on Machine Learning ICML 2018.

 

Making efficient use of sensitive big data and keep it safe and private?

A new method developed by FCAI researchers of University of Helsinki and Aalto University together with Waseda University of Tokyo can use, for example, data distributed on cell phones while guaranteeing data subject privacy.

Modern AI is based on learning from data, and in many applications using data of health and behaviour the data are private and need protection.

Machine learning needs security and privacy: both the data used for learning and the resulting model can leak sensitive information.

Machine learning needs security and privacy: both the data used for learning and the resulting model can leak sensitive information.

Based on the concept of differential privacy, the method guarantees that the published model or result can reveal only limited information on each data subject while avoiding the risks inherent in centralised data.

In the new method, using distributed data avoids the risks of centralized data processing, and the model is learned under strict privacy protection.

In the new method, using distributed data avoids the risks of centralized data processing, and the model is learned under strict privacy protection.

Privacy-aware machine learning is one key in tackling data scarcity and dependability, both identified by FCAI as major bottlenecks for wider adoption of AI. Strong privacy protection encourages people to trust their data with machine learners without having to worry about negative consequences as a result of their participation.

The method was published and presented in December in the annual premiere machine learning conference NIPS: https://papers.nips.cc/paper/6915-differentially-private-bayesian-learning-on-distributed-data.

FCAI researchers involved in the work: Mikko Heikkilä, Eemil Lagerspetz, Sasu Tarkoma, Samuel Kaski, and Antti Honkela.

 

Share

Yes, but did it work? Evaluating Variational Inference

While it’s always possible to compute a variational approximation to a posterior distribution, it can be difficult to discover problems with this approximation. Aalto University and FCAI professor Aki Vehtari proposes with his colleagues two diagnostic algorithms to alleviate this problem.

The Pareto-smoothed importance sampling (PSIS) diagnostic gives a goodness of fit measurement for joint distributions, while simultaneously improving the error in the estimate. The variational simulation-based calibration (VSBC) assesses the average performance of point estimates.

The paper by Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman, ‘Yes, but Did It Work?: Evaluating Variational Inference’ has been accepted to the International Conference on Machine Learning ICML 2018.