logo-footer

GPT3: The New Era Of AI

Mariel Lettier

Table of Content

What is GPT-3?
​​How does GPT-3 work?
GPT-3 Common Questions
How was GPT-3 trained?
What is GPT-3 used for?
The Dark Side of GPT-3
Conclusion

Artificial Intelligence (AI) and Machine Learning (ML) have taken the world by storm over the last few decades. You can find these technologies anywhere you look, from speech recognition and image search to self-driving cars. In this article, we’ll focus on one of the most impressive breakthroughs in the field yet: GPT-3. We will explain what the GPT-3 model is, how it works, what can it do, and what its limitations are. Are you ready to dive in?

What is GPT-3?

GPT-3 is a third-generation, machine-learning system and text-generating neural network created by OpenAI. Its name stands for Generative Pre-Trained Transformer 3. This system uses an algorithm based on 45TB of data. As a result, GPT-3 uses 175 billion ML parameters to produce human-like text. What makes its model rather impressive is that it’s 10 times larger than any other model created before. GPT-3 is a considerable step up from the previous version, GPT-2, which used "only" 1.5 billion parameters. You can access GPT-3 through OpenAI’s API by creating an account here.

What does GPT-3 mean?

We now know what the GPT-3 model is. But, what does “generative pre-trained transformer” actually mean? In this section, we’ll go over each of the terms that make up the name of this machine-learning system.

There are two types of models in machine learning: discriminative and generative. These are often used to perform classification tasks. The discriminative, or conditional, model learns the boundaries between classes in a dataset. As this model focuses on the differences between classes, it can't produce new data points. Meanwhile, the Generative model does not only focus on the differences. Actually, it learns from the data fed to it. Thus, it can then generate new data like what it has received. You’ll find more detailed information on the discriminative and generative models here.

The fact that GPT-3 is Pre-trained means that it has previous training, so it can then form its own parameters. In consequence, these are able for usage in different tasks. Like it happens with us humans, a pre-trained model doesn’t need to learn everything from scratch. It can use old knowledge and apply it to new tasks. If you want to learn more about pre-training ML models, you can take a look at this article.

A Transformer is a type of neural network that first appeared in 2017. These rose to solve some problems related to machine translation. Since then, transformers have evolved and got used for various natural language processing tasks. Also, they've expanded beyond language processing into tasks such as time series forecasting. You can learn more about transformers here.

What is GPT-3?

What about GPT-1 and GPT-2?

OpenAI launched GPT, the first generative pre-trained transformer, back in 2018. This generative language model had 117 million parameters. The greatest breakthrough of GPT-1 was its ability to carry out zero-shot performance in different tasks. The model had its limitations, which is why OpenAI moved on to the next stage. GPT-2 was developed in 2019 using a larger dataset, with 10 times more parameters (1.5 billion) and data than GPT-1. GPT-2 can translate text, summarize passages and answer questions. It can also generate text, but the results can often be repetitive and non-sensical. The largest difference between GPT-1 and GPT-2 is that the latter can multitask. GPT-3 was the obvious next step and, as you will see in this article, it has brought on major improvements.

​​How does GPT-3 work?

GPT-3 uses the sample texts it’s been fed to calculate how likely it is that a specific word appears in a text, considering another word in the text. Given the vast number of parameters, GPT-3 can meta-learn. This means the system can perform tasks without training when given a single example. You can take a look at the GPT-3 documentation here and a more detailed explanation of how the model works here.

GPT-3 FAQs

Now that we know the basics of GPT-3, let's take a look at the most frequent topics about the model.

Who created GPT-3?

As we’ve mentioned, GPT-3 is a product of OpenAI. This AI research and dev laboratory was —among others—founded by Elon Musk and Sam Altman in 2015. Its final goal is to create artificial intelligence that benefits humanity. In 2016, OpenAI developed the OpenAI Gym, “a toolkit for developing and comparing Reinforcement Learning (RL) algorithms”. It also encompasses multimodal neurons in AI networks, and Dall-E, both from 2021.

What can GPT-3 do?

GPT-3 only requires textual-interactional demonstration to work. Once it has it, it can perform the following tasks and more:

• Translate common languages
• Write stories, poems, and music
• Write news articles from just a title
• Predict the ending of a sentence from the context
• Write technical documentation
• Answer questions correctly
• Write software code
• Create PR material
• Create SQL queries

What can GPT-3 do?

Can GPT-3 translate?

As we mentioned above, GPT-3 can translate. Yet, to get good results, it needs to be first fed previous translation memory data. Due to GPT-3’s size, this can be quite a challenge. There are current machine translation tools that are comparable to or even better than GPT-3. This is possible because they're fully optimized for this single task.

Can GPT-3 code?

Yes, GPT-3 can generate code in various programming languages. This does not mean that programmers will get replaced, though. GPT-3 and AI, in general, will most likely take over mundane tasks. For example, by helping cut bottlenecks in software production. This means programmers will be able to focus on more creative tasks.

Will GPT-3 replace programmers?

As we’ve mentioned, it is not likely that the GPT-3 model will replace programmers. Instead, the system will become a handy tool that will take care of the “boring” tasks. This will give programmers more time to focus on interesting and important assignments.

How was GPT-3 trained?

OpenAI used almost all the data on the internet to pre-train GPT-3.

It did this through the following four approaches:

Fine-Tuning

GPT-3 was fine-tuned by providing it with a vast number of datasets for unsupervised learning and then adapting the model for different tasks through supervised training using smaller datasets. You can learn more about the fine-tuning process here.

Few-Shot

This type of learning, also called low-shot, entails providing the model with several examples. These are about how to complete a specific task. It then enables GTP-3 to intuit the task someone is trying to perform and generate a possible completion.

One-Shot

The one-shot learning model is like the few-shot one, except only one example is given.

Zero-shot

In this case, there are no examples. The only thing provided is the task description.

The GPT-3 model performs wonders in all these cases!

How was GPT-3 trained?

What is GPT-3 used for?

Above, we saw quite a few potential uses of GPT-3. Now it’s time to take a look at some real-life examples of the GPT-3 model in action.

GitHub Copilot

GitHub Copilot is an AI pair programmer developed by GitHub and OpenAI. The tool uses the OpenAI Codex to suggest lines or functions inside editors. For instance, JetBrains and Visual Studio Code. The AI has achieved excellent results with languages like Python, TypeScript, and Java.

​​Project December

Project December used the GPT-3 model to create extremely realistic chatbots you can talk to online. It was created by Jason Rohrer and you can buy it for $5. You can feed the bot algorithm text to train it and it will also learn from your input as you talk to it.

AI Writer

Andrew Mayne created AI Writer so people can interact with historical figures by email. Mayne has also used OpenAI to create simple versions of popular games such as Wordle and Zelda. You can find more info about this here.

AI Dungeon

AI Dungeon is an adventure-story game that uses GPT-3 to generate its content. The text-based single-player game is free to play online and users can establish their own custom settings.

The Guardian

The Guardian used GPT-3 to write an essay aimed to convince us that humans that robots come in peace. You can read it here.

As you can see, the GPT-3 model has proven its value in the AI field. But now we’d like to focus on its most impressive product yet. Below, we’ll go over an AI system also created by OpenAI from GPT-3.

What is GPT-3 used for?

DALL-E

DALL-E is an AI program released in January of 2021. It's used to create images using only text captions in natural language. The system is a 12-billion parameter version of GPT-3 trained for this purpose. This is possible by feeding it millions of images tied to their captions. In April 2022, OpenAI announced the release of DALL-E 2. This program creates realistic art and images from a description in natural language. DALL-E 2 produces more realistic and accurate images. These have four times the resolution of the previous version. Moreover, it can add or remove elements from existing images. For instance, it takes shadows, reflections, and textures into consideration. The results from DALL-E 2 are quite impressive. The program is a research project for now, as OpenAI is still focused on exploring its capabilities and limitations. Yet, you can try the first version, also known as DALL-E Mini, here. We should warn you the program does not do great with faces and you might expect some gory results if your prompt includes human beings.

The Dark Side of GPT-3

We can agree on the fact that GPT-3 shows amazing potential. Also, it's an enormous step forward in the field of artificial intelligence. But like every new tool, it is not without its shortcomings. One of the issues GPT-3 faces it’s an ongoing attempt to remove bias from the system. The biases that can be found in GPT-3 include gender, race, and religion. The GPT-3 model is also prone to spread fake news as it has an outstanding ability to produce human-like articles. Moreover, there is much debate about the carbon footprint of GPT-3. The resources needed to train AI are not only huge but are also constantly increasing. The system is environmentally problematic at a time when we should be focusing on reducing our impact on the environment. You can learn more about this and other limitations of the GPT-3 model here.

Conclusion

GPT-3 is an exciting machine-learning system. From what we discussed above, one can see that the model has a lot of potential. Yet, it still needs some adjusting before it is optimal for widespread use. We are looking forward to what the next stage brings and how its shortcomings are handled. Are you excited to see more of GPT-3 in action? What would you use it for?


Technologies like Artificial Intelligence and Machine Learning never cease to surprise us. Imagine what these amazing tools could do for your business! Let's have a chat to start imaging your project endless possibilities.

We are ready
to make your project happen

Let’s talk