100 TOPS, Zero Production: Why Edge AI Projects Die Between Demo and Deployment

Ruth: I'm really happy to have you, Amir and Michaël. Would you like to tell us a little bit about yourself, Amir? What is it that you do at DeepX?

Amir: Yeah, so thank you. Thank you for the opportunity being here. I have a long career in the semiconductor industry. I worked for Avnet Electronics almost 20 years, six years in Germany, where I've been able to explore the culture and work with many companies in Europe and globally. And in the last five years, when I decided to leave the company, I opened my own business development company and joined multiple companies. And this is where the first time I entered the AI market. I joined a company named Edge Impulse that was having a platform for machine learning. Last year, they've been acquired by Qualcomm, so it was a really great journey. And then after that, I joined DeepX as the head of sales and business development in Europe. And it's a very unique company that we will talk more about them today.

Ruth: Great. Thank you. And Michaël, you are a frequent guest on the show, but maybe for the listeners who don't know you yet, I give you the opportunity to introduce yourself as well.

Michaël: Okay, thank you, Ruth. Yeah, indeed. I joined a few of the podcast series. So I'm responsible at Avnet Silica—so we are a distributor of electronic components. I'm responsible for artificial intelligence, so we support customers here in Europe on the topic AI, which is now a very hot topic. And that's also one of the reasons why we have the podcast today, to discuss more about what we can do with AI on the edge, and specifically then with DeepX, which we believe that this is the right choice for us.

Ruth: Exciting. Thank you. So maybe we should start with an overview of what the edge ML market looks like right now. Can you give us a quick overview, if that's possible?

Amir: You know, if you have like, I don't know, eight hours, we can start. But yeah, so, you know, AI is not new in the market. People were doing AI for many, many years. I did AI 20 years ago. It just didn't call it AI. It was DSPs and algorithms and mathematic equations that were used in embedded applications. The vibe of AI started a couple of years ago, and for sure it exploded with the cloud AI that was highlighted by ChatGPT. But we are in edge AI, meaning that everything what we are doing will be on the edge. You are not connected to the cloud. Now, this market is split in many ways, and everybody can look at it in different types of perspectives. But you have what some people call Tiny ML, where it's a low-end endpoint AI, normally based on microcontrollers with AI capabilities, DSPs, and low-end NPUs, normally stopped at the one TOPS, 600 GOPS type of range. And then you move to edge AI, normally driven by microprocessors and AI accelerators, and you have SoCs with built-in GPUs, NPUs, DSPs that are able to drive AI for cameras and sound and sensors. And then you had the high-end AI, where in the past it was mainly dominated by GPGPUs from NVIDIA. And now with the latest advanced technologies, NPUs—unique NPUs—are able to do that type of work. And there is no real limitation: 50 TOPS, 100 TOPS, 200 TOPS, even being driven in low-power applications. This is where the edge AI is being set, in my perspective. Michaël, what is your thinking?

Michaël: Yeah, indeed. Well, you are right. So I think what is also important to mention is that if we take a look at the market, there is a big focus on the edge today for several reasons. If you take a look, first of all, to power consumption—as you see that even in some countries, they open or reopen power plants, they start building new power plants. There is a very strong demand for power, and this is mainly because it's driven by training of AI. And of course, everyone knows what NVIDIA is doing here, so they deliver the hardware or the GPUs for training AI. This is not what we talk about on the edge. What we talk about on the edge is that this has lower power consumption. Also, we talk here about AI inference, and that means this is really like running AI or using AI that was trained.

Ruth: Mm-hmm.

Michaël: For example, if you take a look at NVIDIA, they believe that inference is also the future because they acquired the assets of Groq. So Groq is a company that delivers chips that can run AI inference. So this is one of the topics: if you have power consumption and the inference running it on the edge, the key thing here is to have this at lower current consumption, but also at the best performance. And that's what we try to do with neural processing units, like from DeepX, that we will discuss then more in detail.

Ruth: And what are the biggest challenges customers face when deploying edge AI? I understood from our previous conversations that some people overlook things, especially in industries like manufacturing and robotics. Can you elaborate on that?

Amir: Yeah, sure. So the market is split in many ways. But the majority of the customers will start their AI journey with NVIDIA. There is a huge ecosystem. They will buy the NVIDIA hardware development kits online. You have the Jetson, the CUDA, the ecosystem, and you will start—you will train your models, you will use all of the machine learning platforms that are easy to use today. You will have your dataset and your model being deployed under the NVIDIA GPU, and it's working. And then when they realise that they need to take this into production, there is a long way to go. It's not easy. It's big, it has a huge heat sink, it's expensive. Not everybody understands how you will be able later on to productionise it. There is an operating system running on that machine. I need to think about security. I need to think about OTA updates. I need to think about so many things. And for sure, price, size—if you want to put it on a low-end camera. And everybody remembers, NVIDIA is this big—how can I put it in a small type of camera application in factories, industry? So this is where, for the first time, people are looking around, and then they see multiple options. They see SoCs with AI, they see accelerators. Now they don't know what to do because today I'm using 100 TOPS GPGPU. And one important message: we all know that the GPGPU TOPS is not equal to an NPU TOPS, that is not equal to a SoC with AI TOPS. So there is a lot of trial and error that you need to do. And how do I move my AI model and everything what I did to an NPU? So there are many roadblocks here, and customers need help. And this is why they need to go and talk to a super—you know, a subject matter expert—to somebody who can help them and guide them. And this is why Avnet Silica has this dedicated AI team that is able to answer those questions, both on the hardware side and also on the software side. And in our example, the move from the NVIDIA to the DeepX AI accelerator—and then we see that the results are even greater and better than we expected. So this is what we are doing, and this is where we are now with our collaboration.

Ruth: Michaël, from your experience, what are the most common pain points your customers encounter when integrating vision-based AI solutions?
Michaël: There are several points. Indeed, a struggle, like how to implement it. Software is a key thing. If the software is not good enough for customers to do the implementation and firmware—there are a lot of parts on this firmware. For example, a customer is spoiled today by the NVIDIA tools, which are very good. So they need to find a way that they can do the implementation in an understandable integration. And in fact, this is also what we did when we investigated DeepX, to check the benchmarks. Because we saw, like in a marketing slide, that they compared with competitors. And we said, this is something we need to double-check ourselves because you don't know if this is true or not. And that is what we did. We did an in-depth technical due diligence where we compared not only how the software is working, but also what is the performance of the chip, based on TOPS per second per watt. For example, it's not purely the TOPS—that's the computing performance for an AI chip. It is also how it performs in comparison with how much you consume. So TOPS per second per watt. Also, for example, TOPS per second that you have as the implementation that it works on a high level with the same kind of accuracy. So all those kinds of testing we did—we compared it with several competitive solutions, and we found out that this was really the best performance we had. So there are several key points or pain points for a customer, which is like how to integrate it. In addition, also like power consumption. So this is something that you can run under five watts, but with very high performance. The compilation time is very short. This is also something that a customer—if they need to run something for many hours to do the implementation, this is in minutes. The accuracy—for example, if they compare an AI model, like a vision AI model, and they do quantisation, you will lose some accuracy. But with this kind of implementation we have with DeepX, you don't lose a lot, even if you go to INT8. So there are many, many parts which are important for a customer, besides the standard pain points of software-hardware integration. And this is exactly what we tried to solve here with this solution with DeepX, and also to provide complete solutions for that.

Ruth: And you mentioned that customers often struggle with choosing the right hardware for edge AI. Why is this so complex?

Michaël: Well, you cannot see this like a standard PC that you just start up and everything is running and you install an application. This is really on embedded devices. With embedded devices, you work, for example, on Yocto Linux or Ubuntu. It can also run on Windows, but it's not exactly the same kind of implementation. So that means you need to have some knowledge on embedded hardware. You need to have some knowledge on embedded software. If a customer wants to develop their application, for example, like in Python, they need to understand the different parts to make it working together. And this can be quite complex. And the way to ease this, or to help a customer, is to work on those solutions as a company, like what we try to do, and to provide the tools to the customer, which are the hardware tools, but also then the software tools, example applications, so that they can start easily with their design. And especially on embedded devices, Amir, this is a pain point. This is something that they need help. Because the core of a chip, for example, this is not something that we have in our hands. This is typically the supplier that builds the chip that has it in their hands.

Amir: I will highlight another thing. You're absolutely right. We need also to be honest to ourselves—the edge AI market is starting now, or started really in 2024. What does it mean? It means that companies are getting requests from their marketing department saying that we need to add AI to solve problems, to get better results, to do a lot of things. But not all the products will have edge AI—they don't need it. Think about the retail application with an HMI kiosk. Not all kiosks need to do face recognition or something that is related to AI. It's maybe 10% of them, 20%, and it'll grow because they want to see how the market reacts. And specifically in embedded applications, you cannot now choose the biggest chip in the market that will cost you 200, 300 dollars as a chip—not even hardware. So your bill of materials will be 2,000 dollars, and then I will use only 10% of the AI for it. So I need to be able to build it now. The good thing in the embedded market is that the embedded processors today have the peripherals that, in our case, is the PCIe, that is normally related in an M.2 connector. Now they can add an M.2 module like this, having the AI accelerator on it. And then when they need the AI, they will use it. So then we believe that in the coming years—one, two, three, even to five years—companies will start to add it, you know, in a step-by-step way, and then understand how to do the hardware and the software integration. And maybe in the near future, when they will decide to do new hardware, they already maybe are going to choose a processor that has enough NPU inside, or they will continue to use that way. But even integrate the embedded module inside their hardware, and then they will mount it or don't mount it, depending on the offering to the customers. Not everything is going to AI, and they're able to do it in this type of way. That is a very easy way of helping the customers. I can do it in steps. And again, nobody needs 100 TOPS tomorrow. And as Michaël said, it's TOPS per second per watt. I still want that my product will be this small without heating too much. I want to do something but still get the AI capabilities. And this is what we are trying to do, and we are very successful in this case.

Ruth: Can you share some use cases? DeepX technology, I think we've mentioned it before, is used especially in vision applications, like I'm thinking predictive maintenance and robotics. Do you have an example of how your solutions have transformed a customer's operation?

Amir: Yeah, so I will give a couple of examples, and Michaël, you can chime in. Today, DeepX is winning in robotics applications in Asia. They won several projects with Hyundai Robotics and POSCO and Baidu in different types of applications. Now I will share two examples in a very fast way. One robotics application—so many robotics applications you're thinking about, you know, delivery robots that are delivering stuff. They've been used by NVIDIA solutions, and then it works perfectly. You know, the NVIDIA was able to do double models. One to support the robots being able to, you know, move from one way to another. And then the second one that will be able to recognise the face. Because when you use an app and you order something, you want to confirm that whomever ordered—when he recognises the face, he will open the hatch, the door, to take the shipment. Now the problem is that after one or two deliveries, these robots need to charge. And then if they charge 20 minutes, 30 minutes, one hour, they're losing money. So the idea is to do the same type of task but not to charge. And then this is a good example where you're moving from a GPGPU to a low-end embedded, less than five watts type of application, still being able to do both AIs—the ability to drive, you know, to move in an office building, not to step on the toes. So you have one AI model that sees the human body, and you can see the legs, and then you can, you know, move around. And then at the same time do face recognition, still doing the same type of job. This is a good example. A different type of example that was a little bit surprising for me was the winning from DeepX with Baidu, where this was for an IT application. So OCR—to recognise Chinese letters is super complex. Now, when you use a normal machine, either a desktop or a PC, to do that with a camera, or even NVIDIA, you are not using the right AI functionality because it's not all about that. It's an IT machine that needs to do other things. And what we saw is that when you combine an x86 machine—and in Avnet Silica's case, it'll be AMD that they are very powerful—with a DeepX AI accelerator, the processor will still run only at 25% of its power, and the AI will do all the job. Versus having the processor doing what it was not normally meant to do—AI—and then you jump to 90% of overhead of the processor. These two examples—into the IT market or industrial robotics—is the one that they're winning in Asia. In Europe, we see a lot of things. We see industrial applications for factory automation, putting cameras in places where you would like to define: is it the right place to go or not? So then there is an alarm. No, everybody knows about the hard hat type of application. There are many others. Michaël, what do you see?

Michaël: Yeah, I agree. For example, like in Europe—typically in Europe, or if you compare it with Asia, where they focus a lot also on robotics, humanoid robots—in Europe, it's more focused on industrial applications where you have like industrial automation, smart cameras, street cameras, for example. Not purely for face recognition, but for traffic analysis. But also, and I think that's interesting to mention also, like multi-camera solutions where you have AGVs—like Automatic Guided Vehicles. So from these smaller robots that drive inside warehouses, where you need, for example, like four or six cameras. In fact, this chip—the DX M1 from DeepX, or as an M.2 module, so that's that you put that in a socket, like what Amir just mentioned. So this can be used for the implementation, but it can run several multi-camera streams, and even more than six, only on one chip or only one module. So this is a super strong implementation that a customer can have. So it's not purely focused on one vision camera. It can be with several input streams, output streams. And in addition, one of the things that will be as then as the next step—because we talk now about vision—is, of course, generative AI. And this is a chip that is in development where you can run even a 20-billion-parameter model on a chip, with the same kind of performance, in less than five watts. So this is crucial. This is not available yet. This will be the next step. But in fact, it proves that based on a company that uses the latest kind of technology, like DeepX, for example, even with two-nanometre process, where they want to implement the generative AI models—if you implement it this way, then you can have the performance with low current consumption, high accuracy, high performance output. And so that's in fact the kind of implementation or the kind of applications we can build with this kind of solutions.

Ruth: And what would be an example of an application with this new chip?

Michaël: Well, you need to consider the first application you can think about is like you have ChatGPT, but on a chip level. So that means you avoid privacy issues. You have no latency issues. Because if you go through the cloud, you need to consider like ChatGPT, for example. This works very well, but it's over the cloud. You share data with a third party. This is running locally. If you take a look at the kind of application you can use it, then it can be, for example, a kind of assistant you have on a terminal. If you are in a shop, for example, it can analyse what you do, it can support you. I think in one of the previous sessions we talked about conference systems, so where you have like a device in the middle of the room, where you have this chip inside. It can analyse what people are saying. It can summarise. It can translate, for example, because that's what you can do with chatbots, generative AI. And it can even send, for example, summaries as an email to the participants you have in the room. So if you have this, for example, installed in, I don't know, like a parliament where sensitive information is shared, they do not want to use third-party companies where that can run this kind of intelligence. So this is an example of what you can do then with a generative AI chip, but it is also capable of running vision applications.

Ruth: We could also use it for this podcast, I guess, and then it could make translation into different languages on the go.

Michaël: Exactly, exactly right. Cool. Yeah.

Ruth: Okay.

Michaël: In fact, those applications already exist, eh? And it starts now—even companies that implement this on embedded devices. But of course, the best example is that you have it already today on the cloud services. There are companies doing that, but this will move also to the edge because of the reasons that I just mentioned: privacy, latency, lower power consumption, for example.

Ruth: And I think in one of our previous discussions, you also mentioned use cases around the big topic of smart cities. I suppose that would be a use case as well. Can you give an example?

Amir: So specifically on smart cities, DeepX collaborated with companies that are able to support, you know, enormous amounts of AI models. Some of the names in my mind, like a company called Videonetics, Network Optix, and many others. Now think about it—that in smart city applications, surveillance, you know, urban living, you will be able to use the vision application in many ways. And again, we see this already in Asia now moving more to the Western countries, where specifically in crowded crossroads, in some cases even, you know, being able to save two seconds in a traffic light can create something very unique on one side or the other. So for sure, you cannot put a person in every crossroad. So if you have real AI with the unique models that are literally being able to count, in zero latency, the number of cars and predict all the traffic, it can change the traffic light and then really be able to solve any type of traffic issues. Now this is not science fiction. It's stuff that is now being really implemented. Now in the past, this has been done with very complex cameras, so then normally the cities will be able to put it in only specific places. Now with low-cost AI cameras, you literally can put it in any crossroad, and then every traffic light can become a smart traffic light. And then you really—and all are connected together—and you can, you know, be able to support this type of application. And we see this a lot. This is something that we already have been engaging with customers. Michaël, any other idea for this type of smart city application?

Michaël: Well, a key one is indeed like traffic analysis, street cameras. What we also see is that, for example, like security—if you have this like in train stations to find out if something is going wrong, I don't know, a fight or a fire or those kinds of things, to analyse this. It's mainly analysing what is happening around, or the situation that people are in, to find out what is going on. Those are the main trends and the main applications we see in smart city.

Amir: And I can highlight another thing. You know, there is a unique type of models called VLMs. So it's Vision Language Models. What does it mean? It means that it's not only creating a bounding box or telling me something—it tells exactly the situation because the models are big and in a matter of—something is happening in the situation. So if I will put now a VLM here, they will not say that there are three people in this room. It'll literally say there is a conversation, because they see headphones. So it may be a video podcast that's being done and people are talking. So it analyses the situation. Now, if you think about smart city applications, when you put that type of models, if there is an accident, you already know that it's an accident, and it would call 911 or any type of security officer to support it because it knows the situation. And VLMs are becoming more and more popular.

Ruth: Terrific. Amir, what is one thing you wish more people understood about the potential of edge AI?

Amir: Oh, wow. This is a wonderful question. So when I'm talking to customers, normally they're coming with: I have a problem. Can you help me with your experience? Can you solve this? This is one way to approach a discussion, but I normally want to stop and tell them: What is the problem we want to solve here? Let's go for a second to the roots. Most of the companies will say, I want to sell more of my product. This is obvious. But in the end, and you mentioned this when you started talking about this—predictive maintenance. In some cases, you can add predictive maintenance or preventive maintenance to your application even if you didn't think about it, so you can help your customers. I'm really trying to tell my customers: Let's stop for one second, and then—what is the bigger problem we want to solve, or how we want to help the customers? Because edge AI will not be able to do one thing. It can do a lot of things. And you can think about it, and in many cases, you will have people thinking about: Yeah, okay, we can do this, we can do that. You mentioned the predictive maintenance. We can do other stuff. And then I'm saying: Okay, now when we have more understanding, let's solve the problem why you called me, but add more stuff because you have that. And it creates a lot of ideas to the customers about—I didn't think about it. Because, you know, we need to be happy. We are now in days where we have literally supercomputers in our hands that are this small and can be low power. It's amazing. Let's use them in the best way and the correct way and in the nice way. And edge AI can do that. And it's amazing. It's amazing.

Ruth: Terrific. Michaël, same question to you. What's the one thing you'd like our listeners to take away about edge AI?

Michaël: Yeah, it's indeed a great question. I think what is most important is when people start with AI—so because many customers do that today, or they consider to implement AI—they need to review what is available. But it's really a lot that you find everywhere and in many kinds of directions. So what we try to do is to support them in this journey by providing tools, by doing the pre-testing and preparing the advice for them with the kind of tools that they need. For example, we talk here today about an AI accelerator, but it doesn't work alone, so we need to have also the processor. So that's then it becomes the complexity behind. So where we prepare it with tools in combination with AMD or with NXP or with Renesas. And so the advice is that they need to look around and test different tools—not only from us, not only what we provide. Many customers already started with, for example, NVIDIA. But then if it goes to a lower-end device in combination with an accelerator, then it can be quite complex. So my advice is more like: look around, we can give advice. We are here to support the customers. But the complexity behind, it's something that we can do together building their application.

Ruth: Terrific. Thank you. Is there anything I haven't asked you that you wish I had asked you?

Amir: I think that going back to the basics is that today engineers in the market—whether you are a hardware engineer, you are a software engineer—so a hardware engineer today can use a tool, you know, being generated by a ChatGPT type of application to create his block diagram, even his BOM. There are tools today that are really easy for the hardware mindset. And go to the software guys—you know, you have tools that can do code for you. We all know the names of the companies that are able to write code. I will not use those tools alone without talking to a subject matter expert. We cannot—with all due respect to ChatGPT—I take only me and Michaël here. We are talking about more than 50 years of embedded experience in this call, and I'm literally just saying me and Michaël. In this case, it's priceless. Meaning we walked in the time where there was something called books, and you look at datasheets in books, and we really go to the details of the power of your application, and everything counts. And we saw the changes up to now, so we know how to utilise those tools. I'm not saying not to use them—utilise any type of tools. But then think really at the system level: choosing the processors, the AI accelerator, how your system is being managed, everything in the hardware perspective. And then on the software side, using people that have the experience to support you. And again, I'm just summarising that there was a reason why DeepX chose Avnet Silica. They have great application engineers that are able to support from a hardware side, great engineers that are able to support on the software side. And by the way, it's unique for distribution. And then customers can be in good hands, being able to get that type of support, so then helping them to go into production. This is a very important message.

Ruth: Okay. For our listeners on YouTube, I must ask you, Amir, I can see a drum set in your background. Are you playing?

Amir: Yeah, yeah, yeah. You know, it was something that I wanted to do for many, many years. And I was only dealing with embedded applications and working hard. So a couple of years ago when I celebrated 40, I said, I'm doing it. And then I bought this drum set. And it was really in the COVID time, so I needed to stay with YouTube and learn. When I need a one-hour or 30-minute break, I just, you know, turn on the camera, turn on the phone, go there, make some, you know, big beats and love it. And then go back to the emails and the work and supporting customers. It's very good.

Ruth: Okay, that's terrific. If I had known that beforehand, I would've asked you to play a little bit for us. Usually at this point, Michaël, you know what's coming. I ask my guests: if you had to put together a soundtrack for this episode, what song would you put on it?

Amir: There is one specific song that in my life, everybody thinks that it's really connected to me. Because on one hand, I'm a crazy guy that did a lot of things and being able to support multiple projects, application topics. I play drums, and I'm also dancing Latin at night. I'm dancing salsa and bachata. Me and my wife, and I have, you know, a very unique life. So 'Bohemian Rhapsody' is normally my pick where somebody wants to describe me—going ups and downs and crazy, not the beginning. The mid, you know, from the mid side, with the screaming, the opera. This is me. This is how my life soundtrack looks like.

Ruth: Ah, terrific. And Michaël, I'm not sure if all our listeners must have seen it by now. You actually, for the last episode, composed your own song with AI itself, right?

Michaël: Yeah, exactly. Well, first of all, I would say I follow Amir in his selection because I like also that song. But indeed, one of the previous sessions was also the question: what kind of song? And that time, we talked about a tool that is used in AI—it's called llama.cpp. So we were looking for a song related to llamas. We didn't find it immediately. So we decided then to create an AI song based on llama.cpp. And so this is in fact a song that was generated with AI tools, but also the video behind, because there is a video where you have a llama dancing and making music. And, well, it proves again how fast this kind of technology is moving forward. Because like five years ago, it was absolutely not possible to do it this way, or if you had the experience and tools and those kinds of things. But now, those days, as we see what can be generated—in a good and a bad way—it's quite incredible, and it's only moving faster and faster. But indeed, to come back to that song, it was fully AI-generated. It's slightly modified to make it a bit more compact and, I would say, a bit better. But in general, it was an AI-generated video song.

Ruth: Yeah. Amir, you must listen to it. But it's also quite catchy. It gets stuck in your head really easily. I kept humming that song for days on end, so be warned.

Amir: Okay. I will check it out for sure.

Ruth: Thank you, Amir and Michaël, for sharing your insights and your wealth of knowledge on this topic of edge AI. It was a pleasure having you on the show.

Michaël: Thank you, Ruth, for having us.

Amir: Thank you. Thank you very much.

Ruth: And as always, stay curious and keep innovating. This was Avnet Silica's We Talk IoT. If you enjoyed this episode, please subscribe and leave a rating. Talk to you soon.