From Hollywood to highways: How zero-latency video compression powers autonomous

Ruth: Can you explain what JPEG XS is and why a broadcasting technology suddenly is relevant for automotive and industrial applications?

Siegfried: So JPEG XS is a, let's say, tailored codec, especially for the production of video data. And as you know today, most codecs are used for distribution. That means it is used to transfer the final data to the end customer. With this, you need a lot of compression or you have less bandwidth available. But in the production itself, of course, you have a much higher bandwidth available and you want to have the best quality for the contribution from the stadium to the studio so that you can do overlays, that you can mix the content together with other content. And for this JPEG XS is developed. So that means the main goal was really visual lossless compression so that you cannot see any artefact or any loss in the transmission of this video data. And for this, we have, as I said, created a tailored codec, which has some specific features like low latency, low complexity, so that it can be implemented in different platforms, in different operating systems, and that's the main goal for this.

Ruth: And what makes JPEG XS different from other video codecs?

Siegfried: One is of course the latency. So typically a video codec has a latency of, let's say, 10 to 20 frames or even in the second range because you have group of pictures which are correlated to each other. For JPEG XS you only compress part of one frame, means typically 16 lines. So that's the reason why you can reach a latency end-to-end up to 32 lines. That's really a big benefit, especially if you want to have, let's say, ultra-low analysis of the video data or you want to contribute it over different video hubs.

Ruth: And what made you dive into this research project? What was the need for this codec to compress the video data even more?

Siegfried: Originally, we developed a codec for KVM extender, keyboard, video, mouse, so that you can have your keyboard and monitor away from the computer. So most high-end computers are very loud and noisy, and that's the reason why for keyboard video mouse, we developed one predecessor of the codec. But at some point we said, okay, it's good to have this proprietary codec, but if we want to reach a higher market penetration, we need a standardised one. And that's the reason why we went to the ISO IEC standardisation body and said, hey, there's a need for such kind of standardised codec, and especially in the broadcast arena. Today we have only hardwired SDI interfaces, so serial data interfaces over coax cable. And for this, we also see a transfer to IP-related interfaces. Then we started with this standardisation.

Siegfried: And at the end, of course, we developed this JPEG XS together with other companies as well. And I think we were very successful also with reaching a high quality and a very good codec.

Ruth: And when you compress things, there's always something you lose. So what's the trade-off here?

Siegfried: The trade-off is of course that the compression ratio is not as high as for an H.264 or H.265 coding. So that means for H.264, H.265, you want to reach a compression ratio of about a hundred to one or even higher, so that you can transmit an HD signal over, let's say, two megabit. This is not possible with JPEG XS. With JPEG XS, a typical compression ratio is 10 to one, to 15 to one. But this is sufficient to have, let's say, transmission over standard Ethernet, for example. So we go below one gigabit for HD, we go below 10 gigabit for Ultra HD or even for 8K. And that is in most cases sufficient as a compression measure.

Ruth: And did you start this first with broadcast and television and movie scenarios in mind, or did you have other use cases in mind as well?

Siegfried: In the requirements discussion we had in the standardisation body, of course we had several use cases in mind. Not only the broadcast, but the broadcast seems to us the lowest-hanging fruit, I would say. So that means there was a need to go from this SDI interface to IP interface. And that's the reason why we started first with this market. And in the meantime, many, especially sports transmissions, are done with JPEG XS because of this visual lossless compression and these benefits for IP transport. It's also usable for other markets.

Ruth: You have also mentioned automotive applications, correct?

Siegfried: That's right. I mean, here we see a really big market in the future because we see more and more cameras integrated into cars. For autonomous driving, you want to have the highest quality. And we made also some tests and experiments with tier-one companies who tested if the compression has any effect on the analysis. And the result was that it does not change the positive results of the analysis.

Ruth: Oh, that's terrific because for autonomous vehicles, there's a massive amount of data these cameras are gathering and having to transfer them, and then analysing all that data.

Siegfried: Especially in the cars. Of course, you want to have also low-cost transmission lines. You do not want to have expensive fibre or specialised cables. And you want to have, of course, a reduced number of cables as well. So that's the reason why IP interfaces, of course, are also relevant for this market.

Ruth: And beyond self-driving cars, are there any other applications in automotive that could apply?

Siegfried: In automotive, of course, you can use also electronic mirrors, for example. That's one part or any system where you need a fast response. Of course, if you have entertainment in your car, it might not matter if you have a one-second delay, but for really ultra-low latency systems, then JPEG XS is the right choice.

Ruth: Okay. What do you mean with electronic mirrors?

Siegfried: Yeah. Today you have these regular mirrors. So you'll have your silver-plated mirrors. But in the future, you will have a camera which looks to your back and a display not only from your backside, but also from your side looking back.

Ruth: Oh, okay.

Siegfried: So that's—I mean, with electronic mirror, in principle you have a display instead of a mirror.

Ruth: Yeah. I think there was also, I think I remember correctly, there was a Fraunhofer project where the mirror could actually see if the person starts to fall asleep. Wasn't that also the Fraunhofer IIS?

Siegfried: Yeah. Of course. This is also a possible application if you want to analyse, of course, the reaction of the driver. It falls to sleep or not, and this can also be used for this kind of applications.

Ruth: I think you also mentioned industrial use cases like machine vision or automation. They need to process images in real time for quality control, defect detection, and I suppose even robot guidance. How does JPEG XS change what's possible in these applications?

Siegfried: Exactly. Yeah. So typical is the biggest issue for such kind of quality control system is you need big computer cluster to analyse this high-speed data. And of course either you have to build in the factory a specific room which is very close to the line and typically this is below 10 metres away from the manufacturing line. But of course you want, in many cases, that these computers are set away in a separate computer room, about a hundred metres away or something like this. And then of course it would be good to transmit the data from the cameras, from the inspection cameras to the computer server room. For this JPEG XS, of course, it's also a good solution. And for robotics, of course, it's the same, especially for robotic. You want to have a real fast reaction of the robotic arm.

Ruth: Yeah. Ideally, yeah. It's like, stop cutting. Stop cutting. Please stop cutting.

Siegfried: Exactly. Yeah.

Ruth: Yeah. You mentioned low-cost equipment and low-bandwidth transmission channels earlier. Can you give us a concrete example of how this plays out in a factory setting?

Siegfried: Typically, of course, you want to integrate your compression codec or your encoder inside the camera. For this reason, a typical application is integration as an FPGA IP core. Many cameras today already have FPGAs, of course, especially for the control of the sensor and the interface. But you can also integrate this into the FPGA. And for this, you need of course a very low-footprint codec, I would call it. So that means the codec does not have to need external memory or some high amount of gates or whatever. And JPEG XS can do this. So because it's not making the rate control on a complete frame, it makes a rate control on a flying window. It can use the RAMs inside the FPGA and also needs less computational power. For this reason, JPEG XS can be easily integrated directly into the camera. And it's not only for the automotive or the industrial vision systems. It's meant for all embedded cameras which need transmission out of the camera.

Ruth: As I understand it, it's not a proprietary solution, right? It works on all the boards, all the cameras, and I think you even provide a software development kit to make it as accessible as possible.

Siegfried: Yeah, of course it's an ISO standard, so in principle everyone can implement it. Besides that, it has to be licensed, of course, from us, but the implementation itself can be done in principle by reading the public standard and implemented. But of course, as we have the experience in optimisation of the coding and the implementation, we have created some software development kits. The software development kits are available more or less on all platforms, on x86, on ARM processors, on every operating system. In the moment, we are also working together with partners who have already created FPGA IP cores for embedding of the codec. But we also work on an own FPGA IP core as well.

Ruth: And when we think about the healthcare use case, healthcare is moving towards remote diagnostics and telemedicine. Medical imaging requires absolute precision, as do self-driving cars and autonomous robotics. How does JPEG XS perform in a healthcare scenario? Do you have some use cases you can talk about?

Siegfried: Yeah, that's a special use case. So think about having a remote display in your surgery room. Then of course you want to have the highest quality. Each codec, it doesn't matter if it's H.264, even JPEG XS normally has some defects. However, with a new Edition 3 of JPEG XS, we developed some new idea where we said, okay, we only update the part of the image which has some changes. So that means we save more bandwidth. This allows us to reach a real mathematically lossless image after two or three frames. No person can realise this, but after two to three frames, you have this lossless image on the screen. And that's a big benefit of JPEG XS. And we believe that this might be useful in many scenarios where the doctor has to look to the image and check if everything is okay or not.

Ruth: That's clever. So I'm thinking remote surgery could be—well, surgery, it might be a bit drastic, but special. Let's start with specialist consultations, diagnostic imaging transmissions.

Siegfried: Even if you have a remote transmission of an image over a far distance, so it's successively updates the image so that at the end it's lossless. And this is, I think, a good scenario where the doctor can then analyse what he sees on the screen.

Ruth: That's incredible. And in the future, would it even be possible to do a remote surgery or a robotic surgery?

Siegfried: It might be possible, yeah. Why not? So it depends on the, of course, available bandwidth. But if you look what is available on video broadcasting side, if you have a similar bandwidth, it should be not a big deal to use a remote surgery for this.

Ruth: That sounds really incredible. Can we look maybe into the backstage area of your lab? What obstacles did you and your team have to overcome when working on JPEG XS?

Siegfried: Let's say we have to build a system out of partly already available modules, I have to say. Of course, we see some entropy coders, we see some transformations. All of this are available, but we had to fit this together in the right way. And we have to select the right wavelet transform, for example, so that it does not create a higher latency. So that's the reason why it's an asymmetric, where we have more in horizontal direction and less in vertical direction, which allows us to reduce the number of lines of latency. This is one example, or the other one is that we have some parallelisation on multiple steps. Typically, for example, JPEG 2000 also is a very good codec, but it has more a sequential processing, and that means you cannot parallelise the data even if you have more computing power. That was changed for JPEG XS, so we have, for example, parallelisation on a pixel level. So we process four coefficients in parallel. Then we have these multiple lines which we process and can separate from each other so that dependencies are, let's say, reduced so that this allows us to have this parallelisation on many steps. This allows us then if we have 1, 2, 3, 4, 5 cores, for example, to linearly increase the amount of speed processing.

Ruth: Okay. Was there something that surprised you that you didn't anticipate happening when working on this?

Siegfried: Not really. It was more a design criteria, I would say. In the early days of the computers, you had one processor and the frequency increases year by year. But at some point, of course, the industry detected that it cannot increase the frequency. So that's the reason why they made these multiple cores. I think with the codec development we have to go the same direction, so we have to take care of the developments in the industry and jump onto this road. That's the reason why this parallelisation helps us now in CPUs, in FPGAs, in also GPUs, of course, with multiple calls on GPUs.

Ruth: For our listeners who are now interested in building something on their own or starting with high-quality video in real time, what is your advice? What should they consider when evaluating whether JPEG XS is the right choice for them?

Siegfried: The first one is of course, what are the requirements they have in the use case? If they want to have the highest compression ratio, then of course something like H.264, H.265 is the right one. If they want to have low latency and low complexity and even integrated into embedded systems which do not have an H.264, H.265 encoder, then JPEG XS is a good solution. Of course, on many devices you have already an H.264 encoder, but then in many cases it's only a 4:2:0 encoder. Or in best case, maybe 4:2:2 with 8-bit. But with JPEG XS, you can really have the original image, 4:4:4, means RGB with up to 12 or even 16 bit per component, per colour component. And I think this is a big advantage to the other codecs so that you can really preserve the highest quality.

Ruth: We did have several episodes on the show in the past where we have now talked about Vision AI and embedded systems, remote cameras that work not in the cloud, but on the device itself. So I think maybe I should go back to the guests and connect you so that everybody's profiting from this.

Siegfried: Yeah, sure. Why not?

Ruth: Looking ahead, what new applications are you most excited about? Or is there anything new in the pipeline?

Siegfried: The issue is that bringing a codec to the market needs really, let's say, eight to 10 years. So really a long period. Of course, we have now the success in the broadcast market and the adoption in the automotive or machine vision is our next goal. But of course, in our lab, we already worked on the next generation video codec, and this is of course an AI-based video codec. Here, our first step is now video communication with about 10 kilobit per second, ultra-low bitrate. So for, let's say, direct-to-satellite communication, this is an example and we can reach really a good thing. So something like we have here on the show can be done with this new video codec, really with around 10 kilobit.

Ruth: That is terrific, and I would like it also, if I can have a wishlist, please. It needs to be a little bit more compressed because the data—it is really a lot of data now that we are shuffling around, downloading, uploading, putting it back together, uploading it to all the platforms.

Siegfried: Yeah, this is decided also in our labs that we work more now on video coding. I mean, we are more famous for the audio coding, of course, but now the decision was done that video coding is a good market and there's still a lot of things to do for the video.

Ruth: Absolutely. And I guess even if it's a very trivial example, everything is video these days. Everything is a video chat. Everything is a little video recorded on the phone and sent to someone instead of writing a text. And all the other platforms, they all go to video. So yeah, I guess it is the perfect time to make it good.

Siegfried: Yeah, exactly.

Ruth: Before we close, is there anything I haven't asked you that you wish I had asked you?

Siegfried: Yeah, maybe one thing which might be important, especially for transmission over IP, is that JPEG XS is generating a constant bitrate. So most codecs today have a very variable bitrate where you define the quantisation and then you check what's coming out from the codec as a bitrate. In our case, we can really define this is an exact number of bits which should come out of the codec, and then it generates the bitstream directly.

Ruth: How do you start writing a codec, like in real life? It is coding, it is programming.

Siegfried: Typically, first we start with some workshops, I would call it. So that means the scientists come together, think what is the next codec which we want to develop? What are the requirements? The collection of the requirements is an important step in each process. So first to think, what is a use case you want to address, what is a requirement? And then of course in the workshops, ideas come up. So how to generate a codec, which elements or blocks are necessary to reach this? And of course then there's a kind of trial-and-error phase where you test, is this idea good? Is it not good? And so you try to improve it step by step more and more.

Ruth: Yeah, it seems the industry agrees that it is good. Congratulations again on your Emmy Award. I think you've also won the Fraunhofer Research Prize this year, so I guess everybody likes it and is really impressed by the work you and your team did, right?

Siegfried: Of course, the team likes it very much, and it's, as I said, a big honour and it's seen as acceptance of the market for this big research and development.

Ruth: That is terrific. If you had to pick a song for the soundtrack of this episode, what would it be? What song would best capture the mood of a scientist writing a codec? Or did you have a playlist when working on this?

Siegfried: Maybe this will not everyone like in our institute, but I would pick 'Video Killed the Radio Star.'

Ruth: Very nice. Thank you very much. That is a perfect addition to our playlist. Thank you very much, Siegfried. It has been very interesting to get a sneak peek into the lab and to hear the use cases and the work you have done with JPEG XS.

Siegfried: Yeah, thank you also from my side. Thank you for having me here.

Ruth: Yeah. Thank you for being on the show.

Ruth: This was Avnet Silica's We Talk IoT. If you enjoyed this episode, please subscribe and leave a rating. Talk to you soon.