Five Significant Transformations in the AV Space

                     Listen Now:      

Autonomy is here. While still a few years away from deploying full autonomy across all transportation, autonomy is currently robust — particularly across industrial use cases where Cyngn operates. In recent years, we have seen the development of autonomous vehicles continue. This can be attributed to the notable changes in the self-driving space that either didn’t exist five years ago, or have improved by many multiples in that time. These technological advancements in various domains have allowed the industry to increase capabilities for AV with higher levels of quality, quantity, locality, and efficiency.

In this article, we’ll explore the five massive transformations that have come to autonomous vehicles, and their positive impacts on the autonomy landscape. These transformations include new computational methods, improved sensors, advanced computing power, an open ecosystem, and 5G.

1. New Computational Methods:

Autonomy can be described as a super-set of hardware, software platforms, and various other tools. Accordingly, in order to discuss the recent improvements to computational methods, we must first look at how AI-based learning methods are being used in different aspects of autonomy today.

AI-based learning methods have significantly developed and optimized in the past five years. Autonomy begins with perception. A vehicle’s perception stack allows it to perceive the world around it. These improvements go beyond just the perception stack and are present in the entire system. The ability to perceive, classify, track, and predict objects is the place where learning-based messages shine, opening a door that didn’t exist several years ago. These advanced algorithms have enabled greater accuracy by improving awareness of the surrounding environment, which renders enhanced safety.  

According to Cyngn’s Head of Autonomy, Biao Ma, many algorithms have recently been developed in autonomy across the stack. These algorithms form AI models that “replicate a decision process to enable automation and understanding”. This includes the architecture for a perception stack, as discussed, or say, the object direction classification stack. “If you compare these state of the art models today to not even five or three years ago,” says Ma, “there’s a significant improvement of accuracy, latency, and the efficient utilization of the compute unit.” 

Data sets for these AI models have also transformed. There are not only a big variety of open data sets today, but those in 2021 are better registered, calibrated, and synchronized. By having an advanced way to synchronize sensors, it allows for a set of data that is a hundred times bigger at a much higher quality. Consider, Kitti Dataset, which was popular around ten years ago. This data set had around 7,000 frames with each labeled with what each object it is. Now, a popular dataset has about a hundred times that many frames. 

Improved computational methods have also helped us combat the challenge of corner cases that vehicles may experience while driving in the autonomous space. Corner cases will exist for many years, so advanced computation methods are very important in autonomy. If computation is not done in an effective and scalable way, solving these long-tail problems becomes very difficult. Therefore, researchers are developing new computational methods and platforms to do three key things in an effective way (a) capture and create scenarios, (b) require and provide automatic grading, and (c) manage data pipelines to trigger computation and effectively store data.    

2. Improved Sensors:

There has also been a significant improvement in the distance resolution of sensors in recent years, particularly lidar, a light detection and ranging technology. The number of beams in these units have improved from about 40 lines to more than 300 lines today. Furthermore, the range of these units has gone from about 25 meters of perception to more than 100 meters. This plays a vital role in AV because sensors help vehicles collect information that allow them to capture and react to the given environment. With more beams and greater distances, lidar can better create 3D representations of detected objects and surroundings, and expand a vehicle’s previously narrow field of view. 

Lastly, the signal-to-noise ratio (SNR) has seen significant development by reducing the signal’s noise. SNR is the ratio of signal power to noise power, meaning as noise decreases, receivers can better decipher the signal. The SNR has reduced by almost 50% in the past five years. “This changes the nature of the work of perception in lidar,” says Ma. This advancement of sensor technologies in the last five years can also be seen in radar, cameras, and other sensors beyond lidar. 

3. Advanced Computing Power:

Improved GPU computation power and video memory size has also benefited developers of autonomous driving. GPU computational power has increased threefold. “If you look at a learning-based model, its usefulness is bounded by how fast the model can go'' and an increase in speed facilitates the ability for different structures of the learning-based model, says Ma. We typically expect a learning-based model to go beyond 10 or 20 hertz, but if it is running less than this at two or five hertz, then it is too slow. With the latest generation of perception, models are focusing more on using the latest generation of GPU to increase efficiency and speed. This speed allows for greater opportunity for different structures of the learning-based model. 

Another aspect is the VRAM or “video random access memory”. If a model is too big, it means that it cannot fit in the GPU. A RAM’s size changes the size of the model you can fit into a GPU, and these recent improvements have led to faster and bigger models. This changes the variety and richness of the learning-based model engineers can choose from. Specifically, later generations have more CUDA cores, which are parallel processors that help process that data that comes in and out of the GPU. They could also utilize Tensor cores, which can be leveraged from an open platform known as Pytroch or Tensorflow to create optimization. While before things were either too slow or unable to fit the model in the GPU, this is no longer the case. 

4. Open Ecosystem:

The rise of the open-source ecosystem has also altered the AV landscape and allowed engineers to work cross-functionally. In the last five years there has been a list of open-source projects that have given those in the autonomous space the opportunity to learn from each other by being exposed to different angles and ways of implementations. Several years ago there weren't open platforms, open-source projects, or open tools like Pytroch or Tensorflow for developers to develop.  Ma believes that “these are great tools and methods for young engineers looking into autonomy, who can utilize them as a starting point for their autonomous vehicle careers.” This has resulted in more and more data sets that are bigger in size and better quality.

5. 5G - New Communication Technology: 

Lastly, the difference between 5G and 4G greatly affects AV development. While 4G is already thousands of times faster than previous generations, 5G presents a network that is a hundred times faster than 4G. The 4G network is not enough for these advanced technologies. The fifth-generation wireless technology allows for more than just higher speed and also results in greater locality and low latency. This expanded network improves reliability and generates greater capacity by providing more space at faster speeds.

Ma says that 5G changes the locality of how the architecture of autonomous systems could be designed. Simply stated, it changes where things can happen. Putting hardware in a vehicle, for example, has limits in terms of how much energy it can consume and how well it can compute. In the past, it’s been hard for developers to make edge devices 2X or 10X more powerful because there simply isn’t enough space or energy in a vehicle. However, 5G will allow the ability to put certain key components in places that are not bounded by power usage, bringing in many more opportunities for next steps in autonomy. 

Ma further breaks this down into two key numbers related to 5G: the reduction of latency (how fast something is sent and delivered) and throughput (how fast you can get data and how big a load is sent). First, 5G is 1 or 2 milliseconds in terms of latency, which is more than ten times faster than 4G; this changes where these essential and time-sensitive computations happen. Second, 5G gets us beyond a gigabit per second of data transfer, meaning a significant load of data can get to the vehicle. These two components combined can lead to “a significant change in where things could happen and where it is more optimized”.   

Consider the example of a light pole in regards to the computation of autonomy. With 5G, there are reduced bandwidth limitations and developers do not need to consider whether it’s too late to deliver that information to a given vehicle or not. By increasing speed, 5G can make it so various AV computations don’t have to live right inside of the vehicle.   

Just in these past five years, the autonomous vehicle sector has seen these five massive transformations- new computational methods, improved sensors, advanced computing power, an open ecosystem, and 5G. Yet the AV sector will only continue to advance as we expect to see 5-or-10x improvements to autonomy in the following years. This is just the starting point. It begs the question, what will five years from now look like?

Podcast Episode Transcript:

Luke Renner: This is advanced autonomy. I’m Luke Renner. My guest today is a colleague of mine here at Cyngn, Biao Ma. Biao is the VP of engineering and the head of autonomy. He spent his entire career in this space, bringing engineering expertise to the entire technology stack. This includes perception, planning and control, mapping and localization, and simulation. He was a software architect for Baidu’s autonomous driving division and has a master’s degree in software engineering from Carnegie Mellon University.

In this conversation, we’ll be talking about the five significant changes that have come to the autonomous vehicle sector in the last five years. Biao defines /significant/ as changes that either (a) didn’t exist five years ago or (b) have improved by many many multiples.

Hi Biao, welcome.

Biao Ma: Hey, Luke.

Luke Renner: You are here to walk us through the changes that have come to the self-driving space in recent years. So to kick us off, I wonder if you could just tell us what those changes are.

Biao Ma: Autonomy is a super-set of hardware, software platforms, tools, and a lot of others. So, first of all, I think the biggest significant advancement is the new computational methods in the autonomy space across the key subsystems.

Number two is the sensors. Take lidar, for example, the distance resolution and SNR (the signal-noise ratio) has increased at least two-to-three times.

The third is about the compute GPU. For example, comparing to the 97, 5 years ago, now the 3090 is 3x faster and, I remember, 3x bigger of the VRAM.

Also, five years ago, we didn’t have open platforms, open-source projects. open tools like Pytorch or Tensorflow for developers to develop. [Also, we have] way bigger and more data sets [of a] bigger size and better quality.

Last but not least is new communication technology such as 5G that has significantly changed the game. It will change the locality of how the architecture could be designed.

Luke Renner: So, we have new computational methods, more sensors, advances in computing power, the open ecosystem, and 5G. Is that right?

Biao Ma: Yep, yep.

Luke Renner: Okay.  Alright so let’s dive into those. We’ll start with the computational methods. You know you can’t really talk about new advances in computational methods without talking about artificial intelligence so how are AI-based learning methods being used in different aspects of autonomy?

Biao Ma: So learning-based methods have significantly drawn attention and developed a lot in these five years. If you look at the autonomy stack, perception is the starting point. Being able to perceive objects, classify them, track, and predict objects is the place that really learning-based messages shine. At the end of the day, having learning-based methods being developed and optimized in these several years, we see significant change. It’s really an open door that we several years ago we didn’t have.

Luke Renner: Understood. So tell me a little bit about some of the new algorithms that have been developed?

Biao Ma: There are many algorithms developed in autonomy across the stack. For example, the architecture for a perception stack or specifically the object detection classification tracking stack. There we see generations of models. If you compare these state of art models today to not even five years ago or three years ago, there’s a significant improvement of accuracy, latency, and the efficient utilization of the compute unit — and we’re going to cover how the computer also getting better.

Luke Renner: One of the things that I know that has changed a lot in the last few years is the idea that the data sets for AI models have really changed. Can you talk about some of the transformations that have happened there?

Biao Ma: More than five years ago — well it’s about five to ten years ago —there’s one of the popular data set called the Kitti Dataset. If I remember correctly, the Kitty Dataset had about 7000 frames. Each of the frames was labeled with what object it is. It’s a combination of camera and lidar and they had a different variance.

Now, today in 2021, first of all, there is a big variety of open data sets. But take one of the popular datasets and you have about five-to-seven hundred thousand frames. So, you see a significant difference in terms of size. It’s bigger by several hundred times and also the quality is better.

If we know more about the perception stack, it’s not only about the label itself but also the registration or the calibration of different sensor combinations of lidar, RADAR. The quality also matters. We also see a significant change, not a small improvement, but uh is aligned way better. 

To put it in simple terms, right? These datasets are registered, calibrated, and synchronized way better, right? And us an advanced way to synchronize the sensors so it’s a way bigger set of the data and much higher quality.

Luke Renner: I know that one of the promises of bringing artificial intelligence to the autonomous vehicle space is particularly with dealing with corner cases or the long-tail scenarios that a vehicle may have to deal with while driving autonomously. So, can you tell me a little bit about what are the challenges of corner cases and what are people doing to solve those problems?

Corner case challenges will be with us for a while. If we don’t do computation in an effective and scalable way, solving long-tail problems is extremely hard. That’s why methods and platforms are being built. Some are proprietary and some are open and do, at least three things in a very effective way.

One is to capture and create scenarios. So scenarios are really one key to the development of autonomous driving. Number two is a lot of them require automatic grading because, again, if you need a human to look at the scenario that manual effort will be bounded by human time, and the third is the big pipeline to trigger the computation in an effective way that comes with not only the data pipe but also the analytics, and effectively store the data that matter to us.

Luke Renner: So you mentioned sensors, I want to talk a little bit more about that. Can tell us what the major changes that have come to sensors in the last five years are?

Biao Ma: Let me give you one specific example: lidar. There’s been significant improvement in the number of how many beams of the lidar unit. Five years ago, there were about 16 lines — or up to 64 lines. As of today,  there are more than 300 lines in the market. So you can see it’s several times more, right? And that is just one aspect.

The second is the distance. In the 16 line example, we can effectively do perception using that for about 25 to 30 meters. As of today, a lot of the popular units support a perception stack that goes beyond 100 meters so you can see it’s actually multiple-X improvement.

The third one is really about the SNR, the signal-to-noise ratio. It’s not only about how much you get but also it’s important how much noise you get, right? So based on the experiment that the data we have, there’s a significant improvement, almost cut half and actually more than that in terms of reduced noise of the signal. So that really changes the nature of the work of perception in lidar. Radar, cameras, and other sensors are enjoying the same significant advancement in the last five years but I’m just using lidar as one example.

Luke Renner: Yeah, yeah, I understand so in the last five years GPU computational power has increased threefold and the video memory size has also significantly increased. What would you say has been the impact for developers in autonomous driving?

Biao Ma: If you look at a learning-based model, its usefulness is bounded by how fast the model can go, right? Typically, we expect a learning-based model to go beyond 10 or 20 hertz to make it usable, right? If a model is running at two hertz or five hertz, it is too slow, and that is not only how fast they go. Think about the doors this increase in speed opens for different structures of the learning-based model. So now, with the latest generation of perception, models are really focusing more on using the latest generation GPU to make it efficient and fast.

So that is one side, the other aspect is we typically say the model is too big in the sense that it cannot fit in the GPU. We’re talking about the VRAM here. The ram's size really changes the size of the model you can fit into the GPU. If you cannot fit the model onto the chip, it’s not usable, right? So the fact that it’s way faster and much bigger really changes the variety and the richness of the learning-based model you can pick. And, of course, there are other details.

The latest generations, for example, have way more Cuda cores. There are also Tensor cores that we can leverage that are already optimized with platforms like Tensorflow or Pytorch. We use the software plus hardware optimization for that so this is the key difference. Before, things were either too slow or we just couldn’t fit the model in the GPU. Now everything has changed.

Luke Renner: I know that the rise of the open-source ecosystem has really made it easier for engineers to work cross-functionally. You know, you mentioned Tensorflow as one example. I’m wondering if you could talk about that for a little bit and tell us how open source projects have really changed the AV landscape.

Biao Ma: In the last five years, there is a list of open-source projects, big and small, coming out that have really given us, number one, the opportunity to learn from each other because we’re exposed to different angles and different way of implementations. We can see beyond the architectural decisions that they are looking at and/or we can infer the challenges that they’re having.

The second point is these are great tools and a great method for young engineers looking into autonomy as a starting point for their careers. 

Luke Renner: Bringing us to the end of our five things — the last thing you mentioned that’s really transformed in the last few years is 5G. Now, I know a lot of people really assume that 5G is going to play a big role in this space and have a big impact on how we think about autonomous vehicles and their ecosystems but I don’t think people really understand what that change is going to look like, right? Like we know it’s going to be faster but maybe that is sort of the end of people’s understanding so I’m wondering if you could really explain it to us. How is 5G that much different than 4G and how’s it going to be that much different for AV development?

Biao Ma: 5G changes where things happen. Technically, this is called locality. It’s in the educational view of complicated architectures like autonomous systems. So, the reason it is so hard to make edge devices 2X or 10X more powerful is bounded by power itself. Think about an edge device in a car. There’s a limit both in terms of power and but also there’s a physical limit of how big and how power-hungry the device can be. So having the opportunity to put certain key components in a different place that is not bounded by the physical dimension and not bounded by power really opens doors for next steps.

Let’s look at two key numbers related to 5G to help us understand. Number one is the significant reduction of the latency. 4G is typically 50, 100, or even more milliseconds in terms of latency, which is basically how do I get something delivered to your hand. By comparison, 5G is 1 or 2 milliseconds or even less so it’s significantly faster. Again, that is not 10% faster that is 10X, many-X faster, right? So, this has really changed where things happen. And that is not the only change.

The second change is throughput. Throughput is not only how fast I can get data to you but also how big a load I’m getting to you. So in that sense, 5G is getting beyond a gigabit per second and to the scale of a gigabyte. So in that one second, a significant load of data is getting to your side.

By putting the latency and the throughput together, we’re about to see a significant change in where things could happen and where it is more optimized.  One example is maybe part of the computation of autonomy could be from the light pole because we don’t need to worry about whether it is too late to deliver that information to the vehicle because of our bandwidth limitations.

Luke Renner: OK, 5G is going to make things so fast that, like, some of these AV computations, some of this decision-making and processing — doesn’t necessarily need to live right in the car. Alright, well, all of this is really fascinating. I really appreciate you walking us through all of that. 

We’re a bit out of time here so I thought we would end with a question I’ve been asking everyone so far. This is our second podcast so two people. My question for you is what are you excited about in 2021? What are the breakthroughs that you predict might come to pass this year?

Oh, this is a great question. Uh, I feel very lucky to be able to study and spend my career in this amazing industry and this is just a starting point. I see autonomy from today towards um a full extent is we still have a long way to go. A 5-or-10-X improvement will continue happening.

Yeah, absolutely. So is there anything else you want to tell us?

Biao Ma: So we are actually hiring across the stack, okay? If you are interested in an industry that doesn’t show 3-or-5-percent improvement, rather each area has several-X improvement, this is really a good time to look into this space and join us.

Luke Renner: So, if someone’s interested in working for us where should they go?

Biao Ma Cyngn.com/careers.

Luke Renner: Alright, sounds good. Well, I really want to thank you for your time. It’s been super interesting. I look forward to speaking with you again. Talk soon.

Biao Ma: Alright, see you next time.