A home robot that is factory-trained to do housework cannot effectively clean the sink or take out the trash when placed in a user’s kitchen, because this new environment is different from its training ground.
To avoid this, engineers often try to include a simulated training environment that resembles the real world in which the agent will be placed.
However, researchers at MIT and elsewhere have found that the technology is more capable of performing better than the technology used in the past.
Their results show that in some cases, training an artificial intelligence agent in an uncertain or “noisy” world allowed it to outperform a rival artificial intelligence agent trained in the same noisy world, which tests both.
Researchers call this amazing phenomenon the classroom learning effect.
Serena Bono, a research assistant at the MIT Media Lab and lead author of a paper on the effects of indoor exercise, explains: “If we learn to play tennis in a quiet, indoor environment, we can master different strokes more easily. Then, if we move to a noisy environment, like a windy tennis court, we can become better at tennis than if we started learning in a windy environment.”
Play video
The Effect of Intrinsic Learning: The Unexpected Benefits of Changing the Distribution of Transfer Functions
Video: MIT Center for Brains, Minds, and Machines
The researchers studied this phenomenon by training AI agents to play Atari games, and then added some unpredictable elements. They were surprised to find that the home-learning effect was consistent across Atari games and game versions.
They hope that these results will stimulate further research into developing better methods for training artificial intelligence agents.
“This is a whole new axis to think about,” added co-author Spandan Madan, a graduate student at Harvard University. “Instead of matching the training and testing environment, we can create a simulated environment in which the AI agent learns better.”
Bono and Madan were joined on the paper by MIT graduate student Ishan Grover; Yale University graduate student Mao Yasueda; Cynthia Breazeal, professor of media arts and sciences and head of the Personal Robotics Group at MIT Media Lab; Hanspeter Pfister, the An Wang Professor of Computer Science at Harvard; and Gabriel Kreiman, a professor at Harvard Medical School. The research will be presented at the Society for the Advancement of Artificial Intelligence meeting.
Training issues
Researchers are beginning to understand why reinforcement learning agents perform so poorly when tested in environments other than the training room.
Reinforcement learning is a trial-and-error method where the agent learns by exploring the learning space and taking actions to increase rewards.
The team developed a technique for adding a certain amount of noise to an element of the learning problem, called a transition function. The transition function describes the probability that an agent will move from one state to another based on its chosen action.
If an agent is playing Pac-Man, the transfer function might determine the probability that the ghosts on the game board will move up, down, left, or right. In standard reinforcement learning, artificial intelligence is trained and tested using the same transfer function.
Using this traditional method, the researchers added noise to the transfer function, and as expected, this degraded the agent’s Pac-Man performance.
But when the researchers trained the agent on Pac-Man without noise and then tested it in an environment where noise was introduced into the transfer function, it performed better than the agent trained on the noisy game.
“The general rule is that to get the best performance, you should try to get the transfer function with the best placement conditions during training. We tested this concept to death because we couldn’t believe it ourselves,” Madan said.
Introducing different amounts of noise into the transition function allowed the researchers to experiment in different environments, but they couldn’t create realistic games. The more noise you put into Pac-Man, the more likely it is that the ghost will teleport to different squares at random.
To see if there was a training effect in regular Pac-Man games, they adjusted the baseline probabilities so that the ghosts moved normally, but up and down rather than left and right. AI agents trained in a quiet environment performed better in these real-world games.
“This is not just a way of adding noise to create an arbitrary environment. It seems to be a feature of the reinforcement learning problem. It’s even more surprising to see,” Bono said.
Opening description
The researchers dug deeper to find an explanation and found some correlations with how AI agents explore learning spaces.
When both AI agents explored the same terrain, the agent trained in a quiet environment performed better, perhaps because the agent found it easier to learn the rules of the game when there was no noise.
An agent trained in a noisy environment tends to perform better if their exploration patterns are different. This can happen because the agent needs to understand patterns that it cannot learn in a quiet environment.
“If I just learn to play tennis with my forehand in a quiet environment, and then I have to play with my backhand in that quiet environment too, I won’t play well in a quiet environment,” Bono explained.
In the future, researchers hope to explore how the effect of implicit learning can be applied to more complex learning environments or other methods, such as computer vision and natural language processing. They also want to create learning environments designed to harness the effect of implicit learning to help AI agents perform better in uncertain environments.
