Why artificial intelligence at the edge matters

AI at the edge computing solutions

A few years ago, the concept of having personal assistant in your home, automobile or even phone that recognized your voice and could play your favorite music, shop, monitor your health, control home automation and handle a range of life’s chores was the work of science fiction. Since Apple unveiled Siri in 2011, we’ve seen strong growth in AI connected devices with digital assistants that now include Alexa from Amazon, Cortana from Microsoft and Bixby from Samsung, among several others.

Our lives can quickly be disrupted when our connected devices lose their internet connection. The AI used for voice recognition runs in the cloud, so no cloud access means no assistance. To safeguard our connected devices, couldn’t the device handle the task locally, without an internet connection? It could indeed, with Artificial Intelligence (AI) at the edge.

While the application of this technology to digital assistants represents one possible use, AI at the edge will have profound effects on many industries, including transportation, defense, manufacturing and healthcare to name a few. How will AI at the edge change these industries, and why does it matter?

What is AI at the edge?

AI at the edge means that AI algorithms can be processed locally on a hardware device. The processing algorithms use data (signals or sensor data) that originates from the device. A device using AI at the edge does not need to be continuously connected to the cloud in order to function properly — it can process data and make decisions independently without an internet connection.

AI at the edge moves intelligence closer to where it is needed: in the device

In order to employ AI at the edge, a device must include a microprocessor(s), sensor(s) and AI algorithm(s) in addition to the components with which it was originally designed to perform its set task.

One example is a battery-operated device that can connect to the cloud but has an AI at the edge software application running on a microprocessor processing data in real time. The data being generated is analyzed on the device and also stored locally. When needed, the device can connect to the cloud and transmit its data via the internet for redundant storage and long-term analytics. If the device were to continuously stream data to the cloud, battery life would be negatively impacted. In this simple example, AI at the edge can help extend the battery life of the device.

Machine-learning basics: training and inference

The two main phases of any machine-learning based solution are Training and Inference.

• Training is a phase where a very large amount of known data is given to a machine-learning algorithm, for it to “learn” what to do. With this data, the algorithm can output a "model", containing the results of its learning. This step is extremely demanding in terms of processing power.

• Inference is a device using the learned model with new data to infer what it should recognize. In short, the device interprets what is being asked and then completes the requested task.

The “known data” in the training phase is called labeled data. This means that each piece of data (e.g., sound, image, etc.) has a tag that describes what it is. A speech-recognition AI is trained with thousands of hours of labeled voice data in order to extract the text from a spoken sentence. Natural language recognition can then be used to convert the text into commands that a computer can understand.

Once trained, a device requires a fraction of the processing power to perform the inference phase, mainly because inference uses a single set of input data, while training typically requires a huge number of samples. The production model used for inference is also “frozen” — it cannot learn anymore— and may have less-relevant features trimmed out, and it is carefully optimized for the target environment. The net result is that it can run directly on an embedded device. Decision-making capability lies within the device, thus allowing it to be autonomous. This is AI at the edge.

Hardware challenges of AI at the edge on devices

Like a lot of new concepts, the technology behind AI at the edge has been around for some time: machine-learning algorithms are common in computers and smartphones, where they work just fine. But what about embedded devices? Well, the tools and hardware are now coming together to form a solution that makes sense, mainly thanks to:

• Increases in processing power of devices, and the availability of modules providing hardware acceleration for AI graphic processing units (GPUs) and application-specific integrated circuits (ASICs).

• Constant improvement of AI algorithm models and their performance

• The quality of the tools and resources that make the journey easier for data scientists, AI specialists and developers

Now we can integrate AI capabilities not only in supercomputers, but in cars, smartphones, web pages, Wi-Fi routers, factory robots or any number of applications ranging from small to large, simple to complex.

What does it take to efficiently embed inference on a device?

AI at the edge is first a matter of choosing the right hardware. AI inference on edge devices can be implemented on several hardware pieces.

• CPU: On smartphones and embedded devices, any recent Arm CPU (Cortex-A7 and higher) is more than capable of handling AI at the edge. It may not be the fastest or most efficient solution, but it is usually the easiest. TensorFlow Lite software is commonly used and provides key features from the large TensorFlow framework.

• GPU: Out-of-the box support for GPUs will vary, but they typically provide a large data throughput, allowing superior inference frequency and lower latency. The GPU also removes a large workload from the CPU which improves overall operating speed and efficiency.

• AI specialized hardware: ASICs and tensor processing units (TPUs). These hardware components provide the most efficient AI solutions, but can be pricey and more difficult to design.

Let's further analyze GPUs and AI specialized hardware.

One possibility is to leverage the processing power and paralleling capabilities of GPUs. An AI is like a virtual brain with hundreds of neurons: It looks complex but is made of a large number of simple elements like neurons. This is where GPUs come into focus. Simple independent operations are applied to every single point (pixel or vertex) on your screen. Most machine learning frameworks (e.g., TensorFlow, Caffe, AML, etc.) are designed to take advantage of the proper hardware when it is present. Almost any GPU can be leveraged for these tasks.

Another solution is to integrate specialized hardware. Machine learning can be accelerated through custom hardware, and the two contenders are AI-specific chips and AI ASICs. Google has unveiled its Edge TPU, Arm has unveiled its Machine Learning and Object Detection processors, with Intel, Microsoft and Amazon all working on their own solutions as well. Currently, the best option is to have a GPU supported by the AI toolset you are using.

While AI at the edge might be a hot topic at the moment, it is not a passing fad, and it delivers real value.

Benefits of employing AI at the edge

1. Offline availability/latency: If an application needs to be available no matter the conditions and connectivity, intelligence must be handled locally. Latency or the complete loss of connectivity can happen due to unstable cellular data in a remote location, a DDoS attack or any number of factors. This is a huge challenge for cloud-based solutions. But if the intelligence is maintained locally on a device, there is nothing to worry about.

2. Lower cloud-service costs: Cloud services are convenient because of their scalability and availability, but represent a considerable recurrent cost that usually increases as a solution is more widely adopted. These costs are incurred throughout the life of a product. But if you were to sell a standalone device running the AI, you could significantly reduce its recurring costs and infrastructure needs.

3. Limit connectivity costs: Bandwidth and cellular data are expensive, too. Processing the information locally can greatly save on costs, by sending only the result of the AI’s computation for use in long-term analytics in the cloud. For a video security solution, megabytes of video data would transform into a few bytes.

4. Compliance and confidential information: Why send critical information over the internet when it can be gathered and processed locally? This does not mean that less concern should be placed on the security of devices, but it is one less thing to worry about, bringing peace of mind to customers.

5. Response time is critical: Gathering and processing data locally achieves faster response times which improves safety, performance and the user experience. This is only true if the device can process the data fast enough.

6. Think green: Processing data locally definitely makes sense in designing an efficient AI device. A small to medium IoT device will transmit about 1MB of data daily, which roughly equates to 20g of CO2. Compounded annually, 10,000 devices are responsible for up to 80 tons of CO2 pollution. Processing data locally could shrink this to under 1 ton, which is much better for the planet. Video or image-based solutions could realize a much greater impact.

What are the limits of AI at the edge?

AI at the edge is newer than cloud AI integration and intrinsically has some limits. A few of these limitations and risks are listed here.

1. Edge-capable devices: The AI will need a place to reside. It can be a mobile device like a smartphone or one that is as large as an autonomous vehicle. Whatever it is, it must be capable of running an AI solution efficiently.

2. Performance is limited by the device: An edge device will always have less processing power than an army of virtual servers. For this reason, it is important to identify from the start the response time and the complexity of a solution in order to properly design it.

3. Connectivity is needed to update the model: While inference is fast, training is usually a very long process, even on powerful, dedicated hardware. This is why it is performed on dedicated machines and cannot be done on the device level. After some time, a model will need to be updated to recognize new sentences, support new scenarios, perform new capabilities, etc., and the most efficient way to deploy this update is with an over the air (OTA) mechanism.

Making the move to AI at the edge can bring new efficiencies and opportunities

A few examples of how AI at the edge is effecting change.

Industrial applications:

• Greatly reduced equipment failure and downtime, which costs an estimated $50 billion annually.

• AI energy management with an average 20% reduction in power consumption during manufacturing.

• Intelligent oversight that uses local AI to quickly identify and resolve production problems.

Transportation applications:

• 24/7/365 functionality, maximizing equipment ROI.

• Reduction in labor costs.

• An estimated 60% drop in harmful emissions.

AI at the edge is the next step for many AI-based services. Maximum availability, data security, reduced latency and cost savings are all key strengths of tomorrow’s AI systems. As a company or an individual, now is a good time to make the move to this technology. While some might argue that it has not yet reached maturity, technologies are quickly advancing and converging. Experimenting and implementing a proof of concept is possible with limited effort, granting a competitive advantage before it becomes globally adopted. Given the exponential growth of AI and IoT, AI at the edge growth is inevitable, making it an excellent opportunity for investment.

Learn how we can make AI at the edge work for you. Contact us today.

Follow Avnet Silica on LinkedIn