Sensing to Intelligence

When you think about sensing, you might imagine a fairly passive process in which information is picked up by a sensor and dutifully passed on to a centralized system. Examples abound: sensors in your home trigger an alarm when smoke is detected, an accelerometer provides movement data to your smartwatch which is then processed to indicate how many steps you've taken, and your car can sense and give warnings as you get too close to nearby objects.

In recent years, an exponential increase in sensors everywhere is generating massive amounts of data that will soon surpass what we can currently handle; yet artificial intelligence, machine learning, and algorithmic sophistication are accompanying this proliferation of sensors. For decades, sensors (hardware) and intelligence (software) have been treated as largely separate worlds, connected only by passive flow of information in one direction. This separation is endemic in the design of systems all around us, and in the organization of engineering research itself. Fortunately, a radically different approach is possible, and this is the goal of the EAS Division's newest research thrust: Sensing to Intelligence (S2I).

"Simultaneous advances in sensor technologies and artificial intelligence are converging to enable a fundamental shift away from centralized systems, toward distributed networks of sensors that are fully intelligent on their own," says Guruswami Ravichandran, Chair of the Division of Engineering and Applied Science. He notes this cannot be accomplished within the traditional constraints of single disciplines. "Only by taking a holistic approach that bridges the hardware-software divide can we truly harness the power of sensing to intelligence—to unlock a world that, in the words of Theodore von Kármán, has never been."

Given Caltech's deeply collaborative culture and leading expertise in both hardware and software, the Institute is an ideal incubator for advancing S2I. To learn more, ENGenious sat down with five EAS faculty working at the forefront of this emerging field: Azita Emami, Anima Anandkumar, Alireza Marandi, Changhuei Yang, and Katie Bouman.

"The goal is to bring two isolated fields together—the field of sensing and imaging, and the field of computation and algorithms—in order to develop new systems that are more efficient and intelligent. This will help create a world that is more connected, resilient, and healthier," says Azita Emami, Andrew and Peggy Cherng Professor of Electrical Engineering and Medical Engineering. The key is not just that software and hardware are closely connected but that the flow of information goes both ways. As Changhuei Yang, Thomas G. Myers Professor of Electrical Engineering, Bioengineering, and Medical Engineering, explains, "Traditionally, data is just taken from sensors and processed. But now we're trying to send data back in the other direction to optimize the sensors themselves."

"Closing the loop" in this way is critical because it allows a system to adapt and learn. Katie Bouman, Assistant Professor of Computing and Mathematical Sciences and Electrical Engineering, observes, "Often, when people build these systems, they just sit down with a piece of paper and think, ‘Ideally this is how the world works.' But that can only get you so far, because the real world is messy and imperfect." An S2I approach, on the other hand, could more efficiently deal with the realities of a physical environment.

Emami has been working closely with Anima Anandkumar, Bren Professor of Computing and Mathematical Sciences, on optimal ‘hardware plus software' co-design. This can be a daunting task, explains Anandkumar, "but if we con- sider our strong hardware expertise, then ask what noise models for hardware are reasonable and which machine learning approaches would mesh well, it's a great starting point—and we can grow the complexity from there."

How far to grow that complexity depends on the type of problem researchers are trying to solve. There is theoretically no limit to the number of sensors that might operate together as a vast, intelligent system with powerful emergent properties. "Each sensor has only a limited field of view, but by coordinating, they can build a much bigger picture and potentially solve extremely complex tasks," says Anandkumar. "This is just the tip of the iceberg," she adds.

Alireza Marandi, Assistant Professor of Electrical Engineering and Applied Physics, encourages us to reimagine what's possible. "When we think about sensors," he says, "we have to ask: What things are we not able to sense efficiently with existing methods? Similarly, on the information-processing side, what types of processes are not easy to do with standard computer hardware?" His research group is using photonic technologies to enable entirely new types of hardware for capturing and processing information.

Yang's group, on the other hand, seeks to combine multiple types of sensor information (optical, electrical, and ultrasound, for example). The catch, he notes, is that such "multimodal" systems are very complicated to design. "It actually may not be possible for us as humans to design systems that optimally integrate multimodal data, but a machine learning approach could be used to find a solution—and ultimately give us better information."

Anandkumar is especially drawn to machine learning challenges in S2I. "The systems we're interested in operate at the boundaries of our knowledge, which means we have to invent completely new algorithms," she says. Bouman, who played a key role in producing the first-ever image of a black hole earlier this year, agrees and observed, "We had to develop entirely new techniques to generate an image from the data. With these novel techniques, we were able to make the black hole image within minutes—but we still went through months of additional tests and developed new methods to gain confidence in the results."

With the right sensor technologies and machine learning algorithms combined, potential applications are truly limitless. Medicine is a "perfect fit," according to Emami, in part because S2I can deal with the limited energy capacity of biomedical devices that "often need to be wearable or implantable and sometimes don't even have batteries." S2I can also help avoid information overload for patients and doctors, she says. "Intelligent sensor systems could process and make sense of the data, do a better job of avoiding false positives, and decrease cost."

Noise is another major challenge in biomedical data, particularly neural recordings. Emami is collaborating with Caltech neuroscientists to apply S2I to brain-machine interfaces. "Neural data is extremely messy, and there are lots of day-to-day variations as well as gradual changes as electrodes move and some become less efficient over time," she explains. Machine learning algorithms optimized for noisy environments could enable long-term recording of neural data, which, for example, could be used to predict desired movements of people with spinal cord injury.

Yang is also interested in neuroscience applications, specifically improving the efficiency of functional imaging techniques. As he explains, neural activity has sparsity characteristics—that is, groups of neurons tend to fire in a coordinated way for any particular task or process. Therefore, he points out, "if we're clever in our approach and optimize sensors to capture just the relevant subsets of neurons, we should be able to efficiently capture data in low resolution and make good sense of it."

Marandi sees broad potential for S2I medical applications, which he likens to the evolution of cameras. "Years ago, cameras were a luxury item. Now they're everywhere—including the phone in your pocket—and everyone has access to this powerful tool." He envisions a future where new sensor technologies in your smartphone could provide early warnings of disease based on your breath or other simple tests. "Being able to process sensor information in a more efficient way could allow us to bring these technologies out of the lab and into day-to-day life."

Beyond the field of medicine are countless industrial applications. For example, smart networks of molecular sensors could monitor for the presence of hazardous materials in a factory or plant. To accomplish this, Emami states that efficiency is crucial. "The devices need to operate continuously with almost zero energy, and then ‘wake up' when something happens," she says. "The network of sensors needs to be distributed and needs to have some level of intelligence to be able to operate efficiently." Such systems could significantly improve safety for industrial workers in agriculture, chemical plants, and other settings.

Researchers are also looking to use S2I beyond our own terrestrial boundaries. For example, Bouman is already starting to collaborate with researchers at the Jet Propulsion Laboratory ( JPL) and elsewhere to use S2I for optimization of satellite arrays. As Emami remarks, "S2I systems are very promising for space because they can deal with extreme conditions and optimize themselves to find things that we don't even know we're looking for." She describes "smart instruments" that could design experiments on their own—an invaluable capability in unfamiliar worlds.

S2I will change the way we explore alien and unmapped terrain. "When we're using machines like a Mars rover to explore new regions, they can't go everywhere," explains Anandkumar. "So we need a way to optimize this process of exploration—where to go, what types of measurements to take, etc." Such approaches might be used back on Earth as well—for instance, in mapping inhospitable regions like networks of caves or the ocean floor.

While S2I has great potential to improve our lives, it is not without hazards. Anandkumar points to the example of monitoring and surveillance. "We need to wield these tools responsibly and balance the technology with individual rights and privacy," she says. "These are hard questions, and it shouldn't just be scientists making the decisions. This is where policymakers and people from all communities should come together to ask how we as a society will deal with these new capabilities."

She also describes facial recognition algorithms that perform in troubling ways when analyzing people with darker skin. "We know these systems aren't perfect, so we need to understand the errors they make and have mechanisms to deal with those errors." Anandkumar isn't the only one concerned— she and several dozen other researchers recently signed a letter asking law enforcement to stop using facial recognition software until it can be certified to meet agreed-upon standards.

In pursuing the potential and also addressing the challenges of S2I, Caltech will undoubtedly play a leading role in shaping the future. "This work requires collaborations across computation, algorithms, and hardware, as well as other areas of science and engineering," says Emami. "Caltech is the perfect place to do this because it's a highly collaborative environment and we're very used to multidisciplinary research."

The Institute's collaborative culture permeates not only the faculty level but the student level as well, Yang explains. "We have really good students who think in creative and unconventional ways. And because the barriers to working together are so low here, the students are generating new ideas and fostering new collaborations." Anandkumar adds, "As long as we equip students with the fundamentals, they can connect the dots." She teaches a class in which students across disciplines work together to apply machine learning to real projects. Her group also hosts AI4Science open office hours in which anybody can drop by with machine learning questions. She notes that some students have already put what they've learned into practice, resulting in publishable discoveries.

Asked to sum up the potential of S2I, Bouman says, "This new research area has the potential to make what you can do cheaper, and to make what you can't do possible." Emami agrees, adding that we can take great strides if we "truly take a holistic approach instead of the isolated approaches we've seen so far."

Azita Emami is Andrew and Peggy Cherng Professor of Electrical Engineering and Medical Engineering; Executive Officer for the Department of Electrical Engineering. Alireza Marandi is Assistant Professor of Electrical Engineering and Applied Physics. Changhuei Yang is Thomas G. Myers Professor of Electrical Engineering, Bioengineering, and Medical Engineering. Ani- mashree (Anima) Anandkumar is Bren Professor of Computing and Mathematical Sciences. Katherine (Katie) L. Bouman is Assistant Professor of Computing and Mathematical Sciences and Electrical Engineering, as well as a Rosenberg Scholar.