Floating Point: The Origin Story
Where does floating point arithmetic come from?
Floating point arithmetic is a crucial part of modern computing. A large proportion of the silicon currently being installed consists of ‘floating point units’ and floating point standards continue to emerge to meet the needs of modern machine learning.
But where did floating point arithmetic come from and where was it first implemented? Perhaps surprisingly it can trace its beginnings to the origins of the development of the computing universe as we know it today ….
Space is big. You just won’t believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it’s a long way down the road to the chemist’s, but that’s just peanuts to space.
Douglas Adams: The Hitchhikers Guide to the Galaxy
Douglas Adams was right. The distance to the nearest star - roughly equivalent in astronomical terms to the chemist down the road - is roughly 40,200,000,000,000 kilometres. We’d normally say 40.2 trillion, which is a lot easier to write than a long list of digits, but has the drawback that you have to remind yourself what a trillion is.
Of course we have a better alternative to either of these in the form:
This notation has the immediate advantage of giving the reader a sense of scale without having to count lots of digits.
It also has the advantage of working for both much bigger and much smaller numbers. The number of atoms in the known universe? Not a problem: it’s roughly:
In words that’s roughly ten quadrillion vigintillion, which is really not a useful description at all.
The mass of an electron? That’s:
The notation has three key elements:
Mantissa: 9.109 in our last example
Exponent: -31
Base: 10
This notation dates back as far as René Descartes and his text La Géométrie of 1637. The term exponent, from the Latin exponere meaning “to put out”, dates back even earlier to German mathematician Michael Stifel in 1544, and mantissa again has a Latin source meaning “makeweight” or something of minor value.
We’ve boiled the scale of objects in the universe, big and small, down to a simple, compact notation.
That’s terrific and we could leave it there but, from a computing perspective we can make this even more convenient.
In these examples we’ve chosen different numbers of digits for the mantissa and exponent but we could easily have fixed these to a predetermined number. Let’s say we’ll use sixteen for the mantissa and two for the exponent plus a sign digit in each case.
It’s clear that the number of digits used for the mantissa determines the accuracy of the representation and the number used for the exponent determines how big, and how small, the numbers are that can be represented.
In the real world how many times do we need to represent a physical object with more than sixteen digits of accuracy? Not that many. Equally, we’ve seen that we can represent the number of atoms in the known universe with two digits in the exponent so that should be more than enough for our purposes.
Much of the motivation behind the development of modern computing came from a desire to automate the accurate calculation of values that relate to the physical world. A Royal Society report on Charles Babbage’s Difference Engine had ‘Astronomy’ and ‘Navigation’ as reasons for supporting Babbage’s work:
… they regard as obvious to every one who considers the immense advantage of accurate numerical Tables in all matters of calculation, especially in those which relate to Astronomy and Navigation, and the great variety and extent of those which it is the object and within the compass of Mr. Babbage’s Engine to calculate and print with perfect accuracy.”—Report of Committee of Royal Society, 12th Feb., 1829.
Charles Babbage’s: Passages from the Life of a Philosopher.
To be sufficiently accurate for those purposes both the Difference Engine and the Analytical Engine needed lots of digits.
How many digits?
Both the Difference Engine and the Analytical Engine used ‘wheels’ a rotating brass mechanism used to store and manipulate decimal digits. There were 31 wheels in the Difference Engine and 50 in the Analytical Engine. The numbers held in these engines were a simple string of decimal digits: in effect an integer (e.g. 1234567890123456789012345678901 in the 31-wheeled Difference Engine) or a real number with the decimal at a fixed point.
Those 31 digits came at a cost. Building the engines using the technology of the mid 19th century was very expensive. By the time Babbage had abandoned the Difference Engine in 1842 he had been given, and spent, £17,000 by the British government. As a proportion of British GDP scaled up to 2025 values this is around £850 million or roughly a billion dollars!
These costs in turn doomed the projects. In the end the Babbage’s backers ran out of patience. A less costly approach might have enabled him to turn the Difference Engine into a viable ‘commercial’ proposition and history could have been very different.
Leonardo Torres y Quevedo and “Essays on Automatics”
If Babbage’s work on the Difference Engine and Analytical Engine proved to be a practical and commercial dead end, the ideas behind his work were not forgotten.
Leonardo Torres y Quevedo (1852-1936) was a polymathic engineer whose career started on railways but expanded to include cable-cars, radio control, airships - the Astra-Torres airship was used by French, British and US armed forces - and more.
Torres Quevedo was also interested in mechanical computation initially focusing on analogue computers. Google has a terrific website showcasing his devices as exhibited at the Torres Quevedo museum in Madrid.
s
Later in his career he switched to the study and creation of electro-mechanical digital computers. He published Essays on Automatics in 1915 when he was 63. In it he built on Charles Babbage ideas developed in the course of the design of the Analytical Engine.
Torres Quevedo correctly identified that the number of ‘wheels’ required by Babbage’s Engine was a problem:
Very large numbers are as embarrassing in mechanical calculations as in usual calculations (Babbage planned 50 wheels to represent each variable, and even then they would not be sufficient if one does not have recourse to the means that I will indicate later, or to another analogue).
He then went on to propose a solution:
In these, they are usually avoided by representing each quantity by a small number of significant figures (six to eight at the most, except in exceptional cases) and by indicating by a comma or zeros, if necessary, the order of magnitude of the units represented by each digit.
Sometimes also, so as not to have to write a lot of zeros, we write the quantities in the form n x 10 ^ m
We could greatly simplify this writing by arbitrarily establishing these three simple rules:
1. n will always have the same number of digits (six for example).
2. The first digit of n will be of order of tenths, the second of hundredths, etc.
3. One will write each quantity in the form: n; m.
Thus, instead of 2435.27 and 0.00000341682, they will be respectively, 243527; 4 and 341862; −5.
And the end result of this is a dramatic reduction in the amount of hardware needed to store numbers:
I have not indicated a limit for the value of the exponent, but it is obvious that, in all the usual calculations, it will be less than one hundred, so that, in this system, one will write all the quantities which intervene in calculations with eight or ten digits only.
Torres Quevedo was a practical engineer with the expertise to actually create the technology that he describes. As Brian Rendell sets out in his paper “From Analytical Engine to Electronic Digital Computer: The Contributions of Ludgate, Torres, and Bush”:
The paper [Essays on Automatics] ends with a comparison of the advantages of electromechanical devices over the sort of mechanical devices that were all that were available to Babbage. It establishes, to my mind at least, that Torres y Quevedo would have been quite capable of building a general-purpose electromechanical computer, more than 20 years ahead of its time, had the practical need, motivation, and financing been present. This opinion need not rest solely on the fact that Torres documented a plausible theoretical design, however, because it turns out that he went ahead to prove his point with a series of working prototypes.
Essays in Automatics contains descriptions of devices that would implement many of the building blocks of the modern computer:
[the paper] demonstrates cunning electromechanical gadgets for storing decimal digits, for performing arithmetic operations using built-in function tables, and for comparing the values of two quantities. The whole machine was to be controlled from a read only program (complete with provisions for conditional branching), represented by a pattern of conducting areas mounted around the surface of a rotating cylinder.
Torres Quevedo did go on to create and exhibit machines that implemented some of the ideas in Essays on Automatics including an electromechanical ‘arithmometer’ which combined a typewriter keyboard for input with an arithmetic unit and the ability to print the results.
It’s debatable how much weight to give Torres Quevedo’s work though. The ‘n;m’ idea was perhaps not especially novel and although he probably could have gone on to create a working computer using the idea he didn’t do so.
It would be another two decades though before a programmable machine implementing floating point arithmetic would be created by an engineer who, remarkably, had no knowledge of either Babbage’s or Torres Quevedo’s earlier work.
Konrad Zuse and the Z1
Konrad Zuse (1910-1995) trained as a civil engineer in Berlin before working for the Ford Motor company after graduation and then moving to work as a aircraft design engineer at Henschel und Sohn.
Zuse’s interest in computation started whilst still training as an civil engineer:
At the beginning of the 1930s, while studying civil engineering in Berlin, I decided to develop and build bigger calculating machines, more suitable for engineering purposes.
As Professor Raúl Rojas sets out in Konrad Zuse’s Legacy: The Architecture of the Z1 and Z3:
He realized that he could construct an automaton capable of executing a sequence of arithmetical operations like those needed to compute mathematical tables. Coming from a civil engineering background, he had no formal training in electronics and was not acquainted with the technology used in conventional mechanical calculators. This nominal deficit worked to his advantage, however, because he had to rethink the whole problem of arithmetic computation and thus hit on new and original solutions.
Working from a room in his parent’s flat in Berlin Zuse created the Z1 in 1936.
The Z1 was a mechanical computer, with both switching and memory storage implemented purely mechanically, like Babbage’s Engines, but unlike the Engines, it used binary storage and was powered by an electric motor. The program was read from paper tape and never fully loaded into memory.
To understand how the Z1 works, we are indebted to Professor Raul Rojas who sets out his research in the paper The Z1: Architecture and Algorithms of Konrad Zuse’s First Computer. Most of what follows is sourced from that paper.
The architecture looks remarkably similar to a modern computer. Instructions included ‘load / store’ operations and four arithmetic arithmetic operations. The latter were implemented by way of what we would now call microcode and operated on floating point numbers with a sign bit, a 7-bit exponent, and a 16-bit mantissa.
Multiplication was implemented using a sequence of microinstructions described by professor Rojas as follows:
Division was also implemented in microcode:
The division algorithm takes 21 cycles and is based on so-called “non-restoring floating-point division”. The bits of the quotient are computed one by one, starting from the highest order bit and moving to the lower-order bits successively.
After creating the Z1 and finding it unreliable Zuse soon became convinced of the advantages of electromagnetic relays. The Z2 acted as an intermediate test machine before converting the design of the Z1:
All in all, I was able to gather enough experience with the Z2 in order to convert the complete Z1 design for relay operation. What emerged was the Z3, which is now considered to have been the first properly functioning computer in the world. In order to make fast progress the memory was also given a 64-word capacity, making use of relays.
What happened to the Z1? It was completely destroyed by bombing in the second. world war. Zuse himself recreated it from memory in the 1980s. Whether his recreation is a precise reproduction is not certain. The Z1 recreation is still exhibited in Berlin but will likely never be operated again.
When the machine was delivered to the Berlin museum in 1989 it was shown running and computing some arithmetical operations on several occasions. However, the reconstructed Z1 was, like the original, never reliable enough to run unattended for long stretches of time. It even failed at the inauguration and Zuse spent months repairing the machine. After Konrad Zuse passed away in 1995, the machine was never restarted again.
If I feel a little equivocal about Torres Quevedo’s work on floating point I have no such reluctance to give Zuse his full due. Working independently he not only built working computers using two distinct underlying technologies but both recognised the need for floating point operations and successfully implemented floating point hardware in those machines, a staggering achievement.
Z1 Resources and Simulation
Professor Raúl Rojas has a website with an extensive set of resources on the Z1.
Of particular interest is a terrific online simulation of the Z1’s floating point arithmetic unit created by Oleg Sergin.
A full set of instructions are accessible by clicking the ‘?’ in the top right corner:
Choose an example from Examples. Keep clicking ▷ or pressing a on your keyboard until Lz lights up and nothing changes anymore. Depending on the example, the result will appear in the (Ae,Be) register pair or in the decimal output panel. … To reset the simulation, use Reset or simply your browser's refresh button. To see more at once, switch your browser to full-screen mode or zoom out a little bit.
The source code for the project is hosted here on Gitlab.
Afterword
Neither Torres Quevedo, Zuse nor any of their contemporaries would have recognised the term ‘floating point’. We’ll trace the development of the technology and the early use of the term ‘floating point’ in a future post.
Nice article!
At first sight, Babbage's 31 digits in the Difference Engine seems quite ludicrous. But I think he was not just trying to compensate for the "lack of floating point" with more digits. As I understand it, the Difference Engine was built to calculate tables through polynomial extrapolation, a process which is notoriously ill conditioned except for very low polynomial degrees. That means that you need to have much more precision to represent the intermediate results than whatever you get in the final result.
Babbage was overreaching, but I wonder how many digits he would have needed for the Difference Engine to be practically useful.
four arithmetic arithmetic operations