Aron Kirschen: Analog AI Chips Deserve Better than RRAM*
What would it take to run powerful AI models like GPT-4 continuously and with low latency on edge devices like your smartphone, earbuds, or even smart glasses?
Today, machine learning models are transforming industries, automating grunt work, and solving complex problems, but there’s a significant challenge: running these models on digital chips is energy-intensive and inefficient.
SEMRON pioneers analog AI chips based on memcapacitors, encoding the weights and biases of machine learning models by hardware design to reach unparalleled energy efficiency. Not only is this inherently good, but it also avoids heat dissipation, enabling the stacking of compute layers and, thus, truly three-dimensional AI chips with the highest parameter density possible, bringing powerful AI models to the edge.
Since our previous interview in the summer of 2022, SEMRON went through the Intel Ignite startup program and raised a €7.3 million seed round by Join Capital, SquareOne, OTB Ventures, Onsight Ventures, Hermann Hauser, and others.
We had the pleasure of speaking again with the co-founder and CEO, Aron Kirschen, about what it takes to develop analog AI chips, from the foundry process to a demo board, and how Intel Ignite accelerated their journey.
What’s On Your Roadmap Currently?
Since the last time we spoke, we raised our seed round in December last year, allowing us to work with many new people, partners, and investors and to scale our team. We have done two successful tape-outs, and we’re now working on a demo board that you can place on a table to benchmark machine learning models.
The challenge with developing analog AI chips is that many subsystems need to work together. There are digital components, e.g., for input and output and interfacing with existing digital compute infrastructure, which are relatively easy to simulate, and we can license externally developed IP for them.
The analog parts of our chips are where our core innovation is and where our chip’s USP comes from. They are much more difficult to simulate, and it is important to allow for some buffer to ensure that they work reliably without sacrificing performance.
Our goal in building a demo board is not to show everything that our AI chips will be able to do, like what parameter density, speed, and energy efficiency we’ll reach in the future. It takes a lot of effort to get that right.
If you use a standard manufacturing process for a standard system-on-chip and industry-standard IP, everyone in the industry knows it will work. Instead, our goal is to show things that are not easy to explain and where it’s not obvious that it will work.
What I like to stress at this point is that there’s a huge difference between a technology and a great product. For AI chips, one should consider four aspects:
- The power consumption and overall energy efficiency of a chip,
- The size of the biggest AI models you can run on a chip, like how many million parameters per square centimeter,
- The cost of the chip, and more crucially, the cost per performance: no one will simply pay 10x more for an edge AI chip—you have to show that it makes sense given its performance and demonstrate how you’ll drive down the cost per performance,
- And the software integration, i.e., that it can run not just one machine learning model, which might be outdated tomorrow, but a broad range of models and architectures.
Now, the question for us was, which ones do we have to show with our prototype, and which ones can we demonstrate later? Let’s go through it one by one:
Regarding costs, we have a roadmap and plausible strategy, so we don’t have to show it right now—our costs will come down when we’re mass-manufacturing our chips with a proprietary technology that allows a massive reduction in cost per performance by at least two orders of magnitude.
Next, energy efficiency is important, and we’re about to demonstrate excellent TOPs/W (tera-operations per second per watt) metrics. However, at this point, we don’t overly optimize our chips to push for maximum energy efficiency.
More important is what kind of AI models we’ll be able to run and that we develop the software stack to deploy those models. We need to show that the hardware works, that the accuracy doesn’t decrease when we run AI models in an analog way, and that we can still achieve great energy efficiency and the highest parameter density in the industry.
To get here, we not only had to design our AI chip but also develop a completely new foundry process, which took several years, and we didn’t even use exotic or special materials (which makes it a venture case in the first place if you think about it).
How Did You Develop a Foundry Process?
First, it’s not just a technical challenge—you have to convince the foundry of the business opportunity and get a foot in the door. Only then do you start developing the actual process and its steps and parameters. You back it up with simulations and present it in a concise and compelling way to the foundry.
Having team members, or at least advisors, with strong semiconductor experience becomes crucial, as some process steps you have envisioned may be impractical, or by tweaking the manufacturing process a bit, you can save a lot of costs.
Ultimately, you need to give clear guidelines to the foundry, e.g., how many semiconductor layers and what kind of parameters you’ll need. Then it’s time for the first tape-out and testing everything along the way, which means don’t let all wafers go through all steps of the fabrication process. For example, let six wafers go only until certain points, while six wafers go all the way through. Then, you can measure all wafers to understand if something went wrong at a certain step and iterate quickly to adjust the foundry process.
If you need exotic materials, there are huge upfront costs for implementing the material at the foundry, making it difficult to change foundries. Since we don’t need exotic materials, we’re almost completely foundry-agnostic. All the IP of the foundry process we developed belongs to us.
How Do Approach Developing Things In-House Versus Working With Suppliers?
Generally, we’re trying to do as much as we can in-house, even though we sacrifice some speed with that (and this is a tough decision—almost nothing is as important as speed is!).
We believe the best products need end-to-end control, and that’s why we’re thinking about the whole stack, from quantizing machine learning models to implementing them at the lowest level on our analog hardware. We see that trend in the industry as well. Creativity is required to make that work with very limited resources as a startup.
Also, we learned a lot over the last six months by having analog chip designers talk to AI engineers quantizing machine learning models, which is another benefit of having all the expertise in-house.
We think developing software as a hardware company is crucial—just think of Nvidia. It wouldn’t be this huge success story, which gave them a superb moat against competitors. We can achieve a lot more if we develop software and hardware together.
Finally, software is the thing that stands between the computing hardware and the end customers. As a hardware company, you risk developing a cool technology but not a great product if you don’t develop software for your customers as well—and I can’t stress enough how important a great product rather than just technology is.
It doesn’t make sense to cross software out of the equation. Every hardware company relies on software skills. If managed well, software development is not a distraction, and you only develop what is needed to interface with existing higher-level software.
What Applications Are You Going to Target?
Our initial goal is to run a ~10B-parameter AI model, like a small language model, on our chip with less than 50 mm2 footprint—so it fits into a smartphone—with less than 500mW of sustained power consumption and a price point that matches the boundaries of the smartphone industry.
As mentioned before, we won’t demonstrate all our capabilities at once with our demo board, but as we’re moving to mass manufacturing our chips, we have a precise roadmap for how to reach those numbers and make language models run locally on end-user devices.
The history of consumer electronics has shown that the hardware you have available will be used and opens up new possibilities. So, we don’t worry too much about individual applications in the first place; there will be more than enough applications for OEMs to use our chips. It’s like being worried that there might be no reason to put a more powerful chip into a 2010 smartphone.
If you can’t run AI models locally, you also face additional challenges and overhead through cloud computing subscriptions, infrastructure requirements, the last-mile problem, and privacy. We nevertheless focus a lot on getting a detailed understanding of potential use cases because we need to learn which direction in a trade-off we should follow.
We’re close to the point where cloud subscriptions are sold with smartphones, be it for data storage or AI compute. But clouds aren’t cheap, and if you can avoid the costs of sending data to the cloud and instead process information locally, you will do it.
Why Did You Decide to Join Intel Ignite?
We got in touch with Intel Ignite very early when they were launching the first Europe batch. We were a lot too early back then, but we stayed in contact. After we secured our seed round last year and received lots of positive references from other startups, we were convinced to apply for the program.
We had to wait out one batch as we were finalizing the details of our funding round, but even during that time, being in touch with the Intel Ignite team was incredibly helpful, as we got lots of help and fundraising advice. This was not just marketing! We had experienced and been convinced of their value-add before the batch even started.
How Did Intel Ignite Accelerate Your Journey?
The program was intense, and it came at the perfect time when we were thinking about scaling our team and structuring our startup better.
We weren’t ten people anymore, and as a first-time founder, you always have blind spots, often simple stuff like setting your human resources operations properly: what’s your code of conduct, policies, or automation to manage a growing team?
We learned a big deal from one Intel Ignite advisor, who gave us a set of tools for responding to certain situations—nothing you can easily find since you don’t know what you are looking for. When we were running into problems, we had a clearer picture of our options on how to react and the downstream implications, which saved us a lot of time. And we continued working with some of them even after the program.
The Intel Ignite program gave us hope for Germany as a place for deep tech startups pushing the frontiers of technology.
What Was One of Your Key Learnings from Intel Ignite?
One very memorable session was when the CEOs of all the start-ups in our batch sat together and discussed when to fire someone. It’s a tricky question, where different people had more strict or tolerant views of how they’d handle it.
Yet, everyone agreed that you should fire a person as fast as possible if trust is gone. If the wrong person is in the wrong spot, there’s no other remedy; being too soft doesn’t help you or the company. You have to stay connected to what matters to the company and not be too afraid to hurt someone’s feelings.
*Sponsored post—we greatly appreciate the support from Intel Ignite