Google IO 2021 Sundar Pichai talks LaMDA

Luka Mlinar / Android Authority

If you’ve read anything about state-of-the-art AI chatbots like ChatGPT and Google Bard, you’ve probably come across the term large language models (LLMs). OpenAI’s GPT family of LLMs power ChatGPT, while Google uses LaMDA for its Bard chatbot. Under the hood, these are powerful machine learning models that can generate natural-sounding text. However, as is usually the case with new technologies, not all large language models are equal.

So in this article, let’s take a closer look at LaMDA — the large language model that powers Google’s Bard chatbot.

What is Google LaMDA?

Google IO 2022 Sundar talks LaMDA 2

LaMDA is a conversational language model developed entirely in-house at Google. You can think of it as a direct rival to GPT-4 — OpenAI’s cutting-edge language model. The term LaMDA stands for Language Model for Dialogue Applications. As you may have guessed, that signal the model has been specifically designed to mimic human dialogue.

When Google first unveiled its large language model in 2020, it wasn’t named LaMDA. At the time, we knew it was Meena — a conversational AI trained on some 40 billion words. An early demo showed the model as capable of telling jokes entirely on its own, without referencing a database or pre-programmed list.

Google would go on to introduce its language model as LaMDA to a broader audience at its annual I/O keynote in 2021. The company says that LaMDA has been trained on human conversations and stories. This allowed it to sound more natural and even take on various personas — for example, LaMDA could pretend to speak on behalf of Pluto or even a paper airplane.

LaMDA can generate human-like text, just like ChatGPT.

Besides generating human-like dialogue, LaMDA differed from existing chatbots as it could prioritize sensible and interesting replies. For example, it avoids generic responses like “Okay” or “I’m not sure”. Instead, LaMDA prioritizes helpful suggestions and witty retorts.

According to a Google blog post on LaMDA, factual accuracy was a big concern as existing chatbots would generate contradicting or outright fictional texts when asked about a new subject. So to prevent its language model from sprouting misinformation, the company allowed it to source facts from third-party information sources. This so-called second-generation LaMDA could search the Internet for information just like a human.

How was LaMDA trained?

Google Lambda

Before we talk about LaMDA specifically, it’s worth talking about how modern language models work in general. LaMDA and OpenAI’s GPT models both rely on Google’s transformer deep learning architecture from 2017. Transformers essentially enable the model to “read” multiple words at once and analyze how they relate to each other. Armed with this knowledge, a trained model can make predictions to combine words and form brand-new sentences.

As for LaMDA specifically, its training took place in two stages:

  1. Pre-training: In the first stage, LaMDA was trained on a dataset of 1.56 trillion words, sourced from “public dialog data and web text”. According to Google, LaMDA used a dataset 40 times larger than the company’s previous language models.
  2. Fine-tuning: It’s tempting to think that language models like LaMDA will perform better if you simply feed it with more data. However, that’s not necessarily the case. According to Google researchers, fine-tuning was much more effective at improving the model’s safety and factual accuracy. Safety measures how often the model generates potentially harmful text, including slurs and polarizing opinions.

For the fine-tuning stage, Google recruited humans to have conversations with LaMDA and evaluate its performance. If it was replied in a potentially harmful way, the human worker would annotate the conversation and rate the response. Eventually, this fine-tuning improved LaMDA’s response quality far beyond its initial pre-trained state.

google bard safety fine tuning

You can see how fine-tuning improved Google’s language model in the screenshot above. The middle column shows how the basic model would respond, while the right is indicative of modern LaMDA after fine-tuning.

LaMDA vs GPT-3 and ChatGPT: Is Google’s language model better?

ChatGPT stock photo 2

Edgar Cervantes / Android Authority

On paper, LaMDA competes with OpenAI’s GPT-3 and GPT-4 language models. However, Google hasn’t given us a way to access LaMDA directly — you can only use it through Bard, which is primarily a search companion and not a general-purpose text generator. On the other hand, anyone can access GPT-3 via OpenAI’s API.

Likewise, ChatGPT isn’t the same thing as GPT-3 or OpenAI’s newer models. ChatGPT is indeed based on GPT-3.5, but it was further fine-tuned to mimic human conversations. It also came along several years after GPT-3’s initial developer-only debut.

So how does LaMDA compare vs. GPT-3? Here’s a quick rundown of the key differences:

  1. Knowledge and accuracy: LaMDA can access the internet for the latest information, while both GPT-3 and even GPT-4 have knowledge cut-off dates of September 2021. If asked about more up-to-date events, these models could generate fictional responses.
  2. Training data: LaMDA’s training dataset consisted primarily of dialogue, while GPT-3 used everything from Wikipedia entries to traditional books. That makes GPT-3 more general-purpose and adaptable for applications like ChatGPT.
  3. Human training: In the previous section, we talked about how Google hired human workers to fine-tune its model for safety and quality. By contrast, OpenAI’s GPT-3 didn’t accept any human oversight or fine-tuning. That task is left up to developers or creators of apps like ChatGPT and Bing Chat.

Can I talk to LaMDA?

Google IO 2022 LaMDA 2 example

At this point in time, you cannot talk to LaMDA directly. Unlike GPT-3 and GPT-4, Google doesn’t offer an API that you can use to interact with its language model. As a workaround, you can talk to Bard — Google’s AI chatbot built on top of LaMDA.

There’s a catch, however. You cannot see everything LaMDA has to offer through Bard. It has been sanitized and further fine-tuned to serve solely as a search companion. For example, while Google’s own research paper showed that the model could respond in several languages, Bard only supports English at the moment. This limitation is likely because Google hired US-based, English-speaking “crowdworkers” to fine-tune LaMDA for safety.

Once the company gets around to fine-tuning its language model in other languages, we’ll likely see the English-only restriction dropped. Likewise, as Google becomes more confident in the technology, we’ll see LaMDA show up in Gmail, Drive, Search, and other apps.


FAQs

LaMDA made headlines when a Google engineer claimed that the model was sentient because it could emulate a human better than any previous chatbot. However, the company maintains that its language model does not possess sentiment.

Yes, many experts believe that LaMDA can pass the Turing Test. The test is used to check if a computer system has human-like intelligence. However, some argue that LaMDA only has the ability to make people believe it is intelligent, rather than possessing actual intelligence.

LaMDA is short for Language Model for Dialogue Applications. It’s a large language model developed by Google.

By

Leave a Reply

Your email address will not be published. Required fields are marked *