Neuro-Symbolic Models are Making a Comeback

Next Week in The Sequence:

• Edge 387: Our series about autonomous agents continues with an overview of tool learning. We review UC Berkeley’s Gorilla LLM which is fine-tuned for tool learning and the Microsoft TaskWeaver framework.

• Edge 388: We deep dive into SIMA, Google DeepMind’s agent that can follow instructions to interact with any 3D environment.

You can subscribed to The Sequence below:

 Editorial: Neuro-Symbolic Models are Making a Comeback

Large language models (LLMs) have dominated the AI narrative in recent years to the point that we almost need to wonder about the future of other areas of machine learning. Despite the remarkable capabilities of LLMs, they remain quite impractical in many business scenarios. High computation costs, hallucinations, lack of interpretability, and the constant vulnerabilities associated with relying on stochastic machines are some of the known limitations of LLMs, with no clear solutions in the current state of technology. One of the most interesting options to address these limitations comes from a pretty old ML school: neuro-symbolic models.

As the name indicates, neuro-symbolic models combine neural networks, such as LLMs, with smaller, easier-to-interpret symbolic models to adapt LLMs to specific domains. One of the recent breakthroughs in neuro-symbolic models came from Google DeepMind with their work on AlphaGeometry, a model that was able to solve geometry problems at the level of math Olympiad gold medalists. AlphaGeometry combines a geometry symbolic model with an LLM used mostly for exploring possible solutions to a given problem. Can we expand the AlphaGeometry approach to mainstream use cases?

Last week, a new AI startup called Symbolica emerged from stealth mode with $33 million in new funding from iconic VCs such as Khosla Ventures. Symbolica uses mathematical techniques such as category theory to build simpler models that are easier to manage and interpret. In their press release, Symbolica cited AlphaGeometry as one of the inspirations for their work.

As every company transitions to AI-first business processes, there is no doubt that we need smaller, easier-to-manage and understand models. Neuro-symbolic architectures are making a comeback as one of the possible architectures to play a role in this movement. Symbolica certainly brings a different value proposition from the common mantra of building bigger and more complex models. Neuro-symbolic models are back!

ML Research

Persuasiveness in LLMs

Anthropic Research published a paper proposing a method to measure persuasiveness in LLMs. The method uses LLMs to match claims and the corresponding explanations and evaluate its effectiveness convincing human evaluators of their validity —> Read more.

SambaLingo

Samba Nova published the paper behind SambaLingo, their method for teaching a pretrained LLMs new languages. The method was applied to several LLMs and covers aspects such as vocabulary extension, direct preference optimization and the data scarcity problem —> Read more.

Ferret-UI

Apple Research published a paper introducing Ferret-UI, a multimodal LLM optimized for understanding mobile UI screens. Ferret-UI could be one of the foundations for building LLM agents in IPhones —> Read more.

SOAR

Google Research published a paper proposing pilling with Orthogonality-Amplified Residuals (SOAR), a new method for vector search. The method is included in the ScaNN library which is actively used in embedding search in LLMs and RAG applications —> Read more.

Cool AI Tech Releases

Gemini in Google Cloud

Google announced integrations of its Gemini model across many products in Google Cloud —> Read more.

Google Gen AI Products

Google announced several generative AI products at its Cloud Next conference —> Read more.

Mistral 8x22B

Mistral released a new MoE model via torrent link —> Read more.

Stable LM 12B

Stability AI open sourced Stable LM 12B, a 12 billion parameter LLM trained on multiple languages such as English, Spanish, German, Italian, French, Portuguese, and Dutch —> Read more.

JetStream

Google open sourced JetStream, an engine for high performance LLM inference and optimization.

Real World ML

Airbnb ML Platform

Airbnb open sourced Chronon, its platform for ML observability and management —> Read more.

AIOps at Salesforce

Salesforce shared some details about its AI operations(AIOps) practices including the implementation of a similarity model —> Read more.

AI Radar

• Symbolica AI raised $33 million to bet on neurosymbolic models.

• Meta unveiled details of its Meta Training and Inference Accelerator(MTIA).

• Intel introduced its Gaudi 3 AI accelerator with impressive performance and a more open ecosystem.

• TMSC was awarded $6.6 billion in grants and $5 billion in potential loans from the U.S goverment to open a factory in Arizona.

• AI legend Andrew Ng was appointed to Amazon’s board of directors.

• Microsoft AI announced the launch of a London hub.

• Collaborative Robotics announced a $100 million Series B.

• Former DeepMind scientists raised $10 million to launch Udio, an AI music studio.

• Betaworks announced a new cohort of companies focused on AI agents.

• AI data security platform Cyera announced a $300 million Series C.

• Showflake announced its SQL Copilot.

• Vals.ai raised an undisclosed sum to work on AI benchmarks.

• The widely expected Human AI Pin wearable device is out with mixed reviews.