A brief introduction to AI
A quick summary of modern AI and some ethical issues it poses
Introduction
When people talk about AI nowadays, they are mostly referring to “neural networks”. At the moment there is a lot of discussion in and out of academia on whether or not these algorithms can be intelligent, including an ex Googler making the media rounds arguing that Google’s language models may be conscious.
In this article I give a quick overview of how modern AI works, and share my thoughts on why these algorithms aren’t intelligent now, and will probably not be in the future bar some unforeseen innovation. More practically, I also outline three immediate ethical problems that arise from this technology.
What are Neural Networks?
Neural networks are algorithms loosely inspired by the brain. A neural network is a program where a given input is passed through a set of layers of artificial neurons. Each neuron is connected to the neurons in the next layer through weights, which control how much the neurons effect each other. This is supposed to mirror how neurons work in the brain - certain neurons are more strongly wired to other neurons which is parametrised by the “weights”.
You can think of a neural network as a set of functions. Each layer is a function (f_i) that is parameterised by its weights to the next layer (W_i), and its input. So the first layer may be f_1(W_1,x) where x is the input (i.e an image we want to classify). If we have two layers, we can write the neural network as a nested function f_2(W_2, f_1(W_1, x)). The functions have a nonlinear component (called an “activation function”) so that the neural network can learn nonlinear patterns in the data.
It turns out that neural networks are universal function approximators: given an infinitely wide or deep neural network (a large enough number of layers), we can learn a function that maps any data input to whatever output we want. A common example of this would be a function that takes as input a picture of an animal (represented as an array of RBG pixels) and then maps this set of numbers to an output label: 0 (cat) or 1 (dog).
Neural networks learn these functions that recognise patterns in data by making guesses about the data, i.e you give it an input image and it guesses cat or dog. They improve their guesses by adjusting their weights every time they get a guess wrong. This changes the function to better map the input you give it to the correct output. At the start these guesses are random but, in the end, they learn the correct function to map the data input to the right output. An example could be an input picture of an animal, which would be mapped to an output label: 0 (cat) or 1 (dog).
This process of improving the weights to make better guesses is called back-propagation, because the error from the output layer is “propagated backwards” from the last layer to the first layer to determine how each weight is contributing to the error in the guess. This allows us to adjust each weight a tiny amount to reduce their contribution to the error and improve the guess overall. The math educator 3blue1brown has a good explainer of the theory behind back-propagation here. The math is relatively accessible for someone with a couple of college math classes, although the notation used is quite dense and takes a bit of effort to understand.
Are Neural Networks Intelligent?
As described above, in theory neural networks are universal machines capable of learning any function if you make them big enough. In practice however, they are quite limited. The universal function approximation theorem is more a mathematical proof based on a limiting case: in the real world we are not able to create an infinitely large neural network and give it an infinite amount of data. To train a neural network to do even a simple task requires a tremendous amount of data: they may have to see thousands of images before they can perform the cat/dog labelling I was describing earlier. In contrast to this, human beings have an internal model of the world and can learn new things after a few examples by comparing it with concepts they already understand.
Neural networks require a lot of data because of the type of reasoning they employ. In contrast to earlier logic based systems that performed deduction, neural networks reason via induction. In inductive reasoning, you infer a general rule from a pattern of observations. Whenever I see a swan, it is white. Therefore all swans are white. More formally we can say if we always observe A then B, we can say A implies B (or A=>B). When we show neural networks many example so that it can learn patterns in data, we are training it to make inductive inferences. This form of statistical inference neural networks use has been legitimately useful in science, engineering, etc. However it seems to fail when we try to use it to solve common sense problems. There is therefore a lot of scepticism among non “Big-Tech” affiliated academics that these systems can be intelligent.
For example: right now there is a lot of hype around “large language models” (LLM) like GPT3. The architecture for these is complicated, but it relies on a type of neural network called a Transformer. Transformers are next word predictors: they learn efficiently which words relate to each other and use this statistical knowledge to predict the next word when you give them a sentence. If they have a big enough training set, LLMs can build up a decent statistical model of human language. They can then coherently mimic it. Ask them a couple of probing questions however and you will see they have no understanding of what they are saying: they have no world model or concept of causality. They fail when you ask them something outside of their training set. This is a more general problem that all neural networks have: their knowledge is brittle and narrow, and they cannot transfer it to new tasks.
Three Ethical Concerns
Firstly, these methods are data hungry which means that companies and governments are incentivised to collect massive amounts of personal data which is used to violate people’s privacy and manipulate them, like in the case of the Cambridge Analytica scandal. For a good popular treatment of this problem, I recommend the Netflix documentary “The Social Dilemma”.
Secondly, induction does not account for events outside the dataset, such as the proverbial black swan. This means these algorithms will have rare but catastrophic failures. Think of all the recent stories of self-driving cars crashing. When human drivers crash, they do so because of inattention, inebriation, or small mechanical errors. In contrast to this, self-driving cars often crash for no real reason, from a standstill, or while cruising down an empty street. Wikipedia has a great section of its page on Tesla criticism that has cartoon schematic drawings of the worst crashes showing the car’s trajectory. In these crashes you again and again see the consequences of the unreliability of neural networks: the Teslas accelerates into a tree of the side of the road, drives over pedestrians that have pulled over from car trouble, or just plain drives into a wall. Why the AI crashed in a given instance is inscrutable: maybe it saw a plastic bag flying by that it thought was a street sign.
Lastly the way these algorithms make decisions is opaque. They are a complete black box. You cannot ask the algorithm “why did you come up with this decision” because it is the result of the summation of millions of tiny calculations. This means that we cannot trust that the algorithms are making decisions for reasons we would be ok with. This is especially important when they are used to determine things like who should get a bank loan or who should be suspected of a crime, etc. This gets into the issue of algorithmic bias: harmful social biases that we do not want to reproduce may be present in the data we feed the AI which can make it discriminatory.
The truth of the matter is that AI as a field is probably a long way away from general intelligence. In the meantime, these algorithms have become more integrated with our lives, in ways most people aren't aware of. The aim of my newsletter is to write essays that help people understand what these algorithms are and what they are being used for. The application of AI won’t be democratic until the average person understands AI. Once this is the case, we can maximise the benefits and minimise the harm to the average person.