Machine learning at the edge: A hardware and software ecosystem
Edge computing is booming. The idea of taking compute out of the data center, and bringing it as close as possible to where data is generated, is seeing lots of traction. Estimates for edge computing growth are in the 40% CAGR, $50 billion area.
Whether it's stand-alone IoT sensors, devices of all kinds, drones, or autonomous vehicles, there's one thing in common. Increasingly, data generated at the edge are used to feed applications powered by machine learning models.
Robotics
TinyML is a fast-growing field of machine learning technologies and applications that enable machine learning to work at the edge. It includes hardware, algorithms and software capable of performing on-device sensor data analytics at extremely low power, hence enabling a variety of always-on use-cases.
In order for TinyML to work, a confluence of hardware and software is needed, creating an ecosystem built around the notion of frugal energy needs. This is a prerequisite for applications at the edge.
Today Arm, a global semiconductor IP provider known for its focus on ecosystem creation and frugal energy needs for its processors, is announcing a partnership with Neuton, a provider of an automated TinyML platform. Earlier in September, Alif Semiconductors, another Arm partner building AI chips for the edge, released new product lines.
ZDNet caught up with Henrik Flodell, Sr. Marketing Manager at Alif, Philip Lewer, Director of Ecosystem and Developer Relations, Machine Learning at Arm, and Blair Newman, CTO at Neuton, all seasoned experts in the embedded space. We discussed their respective offerings and what it takes to get an ecosystem for machine learning at the edge going.
Arm, building an AI ecosystem
Arm is an ecosystem champion with more than 1000 partners. According to Lewer, that is a key reason why the company was able to ship more than 190 billion chips based on its technology worldwide.
Arm chips are used everywhere, from cloud data centers to laptops and from wearables to drones. Lewer described Arm's AI platform as "a collection of technologies and partnerships that enable AI to happen".
For Arm, the foundation is at the hardware level, and its AI platformincludes everything from Arm Cortex CPUs to Mali GPUs and Ethos NPUs and microNPUs. The Cortex-M family is a very popular choice, often included in microcontrollers as well as other chips. Arm's Ethos-U processor series focuses specifically on machine learning inference for low-power devices.
Alif's recently unveiled Ensemble™ and Crescendo™ product families utilize Arm's Cortex-M too. Ensemble chips are aimed at smart home products, appliances, point-of-sale, robotics, and other applications at the edge.
Alif was founded in 2019, and Flodell noted that the motivation was to "develop a new platform from the ground up, based on the very latest technology that really enables functionality like ubiquitous wireless connectivity and edge processing with AI and machine learning capabilities".
Machine learning acceleration and multi-layered security are the key features that both Ensemble and Crescendo share. The Crescendo family also offers connectivity and positioning features, which Alif notes make them suitable for smart city, connected infrastructure, asset tracking, healthcare devices, and wearables applications.
Back in 2018, Neuton caused a splash by announcing a neural network framework claiming to be far more effective than any other framework and non-neural algorithm available on the market.
Newman noted that although Bell Integrator, the vendor behind the Neuton framework, has been around for a little over 17 years, it was about 6 years ago that they focused their attention on building zero-code SaaS solutions.
Historically, Newman added that Bell Integrator has been leveraging all "traditional" machine learning frameworks available. However, the issue of resource scarcity has always been hard to navigate. It's one thing to build a machine learning model and another thing to deploy it in production, especially at the edge.
Neuton, building no-code machine learning models from the ground up
Newman emphasized two aspects of Neuton's approach, which went against the grain of established machine learning frameworks. First, the no-code aspect, enabling non-data scientists to build models. Second, the custom architecture that Neuton's machine learning models employ:
"As soon as those models are produced, they can immediately, without any interaction, be integrated into microcontrollers. Our customers are really empowered to go through the entire lifecycle of bringing machine learning to the edge without any technical skills", said Newman.
Neuton was invited to join Arm's partner ecosystem after presenting its approach to building compact and accurate models at the TinyML EMEA Technical Forum 2021. The whole purpose of Arm's partner ecosystem is to bring together companies that enable different capabilities, said Lewer.
Building an ecosystem for AI at the edge takes collaboration and working across hardware and software
Case in point, Alif and Neuton, Lewer went on to add. Alif is utilizing Arm's Cortex design in innovative ways, but the real question for users is how to deploy machine learning models on Alif's chips:
"It's really important, especially for developers that are closer to conventional programming backgrounds, to bridge that gap into the world of machine learning. Then you have someone like Neuton who comes in and says, well, that's where we fit in. If we have customers who are satisfied and partners who are satisfied, that's how we measure success".
Arm was a natural partner for Alif; Flodell concurred because they have great IP and a focus on ecosystem enablement: "We know that people are going to be able to be productive with these devices as soon as they get their hands on them". Flodell said.
Special attention has been paid to the power characteristics of Alif's product line. Naturally, Flodell explained, chips with integrated connectivity features such as the Crescendo line will have higher power requirements compared to something like the Ensemble.
It all comes down to how much power you're consuming just to participate in a network, and this is the part Alif has focused on optimizing. In that respect, he added that Alif's benchmarks show Crescendo to be 2 to 3 times more frugal than chips with similar characteristics, which simply means applications will be able to run for longer.
Alif, building embedded controllers for constrained environments
For applications deployed at the edge, battery consumption really is the currency, Newman concluded. Neuton's approach is to "build [models] from the ground up, neuron by neuron. You only have to build your models once, and they come out extremely compact without compromising accuracy".
Neuton is relatively new to the partnerships game. However, Newman identified the partnership with Arm as a strategic one for Neuton's goal to democratize machine learning. For Lewer and Arm, partnerships are a key part of their strategy, which they will continue to develop. Even though Arm has many partners, it's not all about numbers, Lewer said:
"It's really about making those partnerships effective, and that means leaning in. We spend a considerable amount of time with each partner trying to understand where they're trying to go so that we can find common ground".
When we talk about machine learning at the edge, being able to work in a resource-constrained environment is key, Flodell noted. AI's pedigree is tied to data centers, but this has to change for real-world AI applications at the edge:
"When you want to scale down and run in something that is configured as a microcontroller, sometimes with less than even a megabyte a memory, that in itself becomes a challenge.
Then adding to that, AI is in some ways very different from the traditional development that embedded systems designers go through. It's still more in the domain of data scientists to understand how to tune the models to produce the right results.
Being able to leverage Arm's partnership activities to connect to companies like Neuton should be able to bridge that gap to make the data scientists and the embedded developer's expertise merge and to make the models and the technology fit a constrained system.
That is really the challenge. If we can overcome it, it's going to open the floodgates for adoption of this technology".