How do we know AI is ready to be in the wild? Maybe a critic is needed

Mischief can happen when AI is let loose in the world, just like any technology. The examples of AI gone wrong are numerous, the most vivid in recent memory being the disastrously bad performance of Amazon's facial recognition technology, Rekognition, which had a propensity to erroneously match members of some ethnic groups with criminal mugshots to a disproportionate extent. 

Given the risk, how can society know if a technology has been adequately refined to a level where it is safe to deploy?

"This is a really good question, and one we are actively working on, "Sergey Levine, assistant professor with the University of California at Berkeley's department of electrical engineering and computer science, told ZDNet by email this week. 

Levine and colleagues have been working on an approach to machine learning where the decisions of a software program are subjected to a critique by another algorithm within the same program that acts adversarially. The approach is known as conservative Q-Learning, and it was described in a paper posted on the arXiv preprint server last month. 

ZDNet reached out to Levine this week after he posted an essay on Medium describing the problem of how to safely train AI systems to make real-world decisions. 

Levine has spent years at Berkeley's robotic artificial intelligence and learning lab developing AI software that to direct how a robotic arm moves within carefully designed experiments -- carefully designed because you don't want something to get out of control when a robotic arm can do actual, physical damage.

Robotics often relies on a form of machine learning called reinforcement learning. Reinforcement learning algorithms are trained by testing the effect of decisions and continually revising a policy of action depending on how well the action affects the state of affairs. 

But there's the danger: Do you want a self-driving car to be learning on the road, in real traffic?

In his Medium post, Levine proposes developing "offline" versions of RL. In the offline world, RL could be trained using vast amounts of data, like any conventional supervised learning AI system, to refine the system before it is ever sent out into the world to make decisions. 

Also: A Berkeley mash-up of AI approaches promises continuous learning

"An autonomous vehicle could be trained on millions of videos depicting real-world driving," he writes. "An HVAC controller could be trained using logged data from every single building in which that HVAC system was ever deployed."

Levine uses the analogy of childhood development. Children receive many more signals from the environment than just the immediate results of actions. 

"In the first few years of your life, your brain processed a broad array of sights, sounds, smells, and motor commands that rival the size and diversity of the largest datasets used in machine learning," Levine writes. 

Which comes back to the original question, to wit, after all that offline development, how does one know when an RL program is sufficiently refined to go "online," to be used in the real world?

That's where conservative Q-learning comes in. Conservative Q-learning builds on the widely studied Q-learning, which is itself a form of reinforcement learning. The idea is to "provide theoretical guarantees on the performance of policies learned via offline RL," Levine explained to ZDNet. Those guarantees will block the RL system from carrying out bad decisions.

Imagine you had a long, long history kept in persistent memory of what actions are good actions that prevent chaos. And imagine your AI algorithm had to develop decisions that didn't violate that long collective memory. 

In a typical RL system, a value function is computed based on how much a certain choice of action will contribute to reaching a goal. That informs a policy of actions. 

In the conservative version, the value function places a higher value on that past data in persistent memory about what should be done. In technical terms, everything a policy wants to do is discounted, so that there's an extra burden of proof to say that the policy has achieved its optimal state. 

A struggle ensues, Levine told ZDNet, making an analogy to generative adversarial networks, or GANs, a type of machine learning. 

"The value function (critic) 'fights' the policy (actor), trying to assign the actor low values, but assign the data high values." The interplay of the two functions makes the critic better and better at vetoing bad choices. "The actor tries to maximize the critic," is how Levine puts it. 

Through the struggle, a consensus emerges within the program. "The result is that the actor only does those things for which the critic 'can't deny' that they are good (because there is too much data that supports the goodness of those actions)."

Also: MIT finally gives a name to the sum of all AI fears

There are still some major areas that need refinement, Levine told ZDNet. The program at the moment has some hyperparameters that have to be designed by hand rather than being arrived at from the data, he noted. 

"But so far this seems like a promising path for us toward methods with safety and reliability guarantees in offline RL," said Levine. 

In fact, conservative Q-learning suggests there are ways to incorporate practical considerations into the design of AI from the start, rather than waiting till after such systems are built and deployed. 

Also: To Catch a Fake: Machine learning sniffs out its own machine-written propaganda

The fact that it is Levine carrying out this inquiry should give the approach of conservative Q-learning added significance. With a firm grounding in real-world applications of robotics, Levine and his team are in a position to validate the actor-critic in direct experiments. 

Indeed, the conservative Q-Learning paper, which is lead-authored by Aviral Kumar of Berkeley, and was done with the collaboration of Google Brain, contains numerous examples of robotics tests in which the approach showed improvements over other kinds of offline RL.

There is also a blog post authored by Google if you want to learn more about the effort.

Of course, any system that relies on amassed data offline for its development will be relying on the integrity of that data. A successful critique of the kind Levine envisions will necessarily involve broader questions about where that data comes from, and what parts of it represent good decisions. 

Some aspects of what is good and bad may be a discussion society has to have that cannot be automated.