Everything You Ever Wanted to Know About Floating Point but Were Afraid to Ask

This post was written by Nathan Whitehead

serial summationA few days ago a friend came to me with a question about floating point.  Let me start by saying that my friend knows his stuff, he doesn’t ask stupid questions.  So he had my attention.  He was working on some biosciences simulation code and was getting answers of a different precision than he expected on the GPU  and wanted to know what was up.

Even expert CUDA programmers don’t always know all the intricacies of floating point.  It’s a tricky topic.  Even my friend, who is so cool he wears sunglasses indoors, needed some help.  If you look at the NVIDIA CUDA forums, questions and concerns about floating point come up regularly.  [1] [2] [3] [4] [5] [6] [7]  Getting a handle on how to effectively use floating point is obviously very important if you are doing numeric computations in CUDA.

In an attempt to help out, Alex and I have written a short whitepaper about floating point on NVIDIA GPUs.

In the paper we talk about various issues related to floating point in CUDA.  You will learn:

  • How the IEEE 754 standard fits in with NVIDIA GPUs
  • How fused multiply-add improves accuracy
  • There’s more than one way to compute a dot product (we present three)
  • How to make sense of different numerical results between CPU and GPU
∥∀

About Mark Harris

Mark is Chief Technologist for GPU Computing Software at NVIDIA. Mark has fifteen years of experience developing software for GPUs, ranging from graphics and games, to physically-based simulation, to parallel algorithms and high-performance computing. Mark has been using GPUs for general-purpose computing since before they even supported floating point arithmetic. While a Ph.D. student at UNC he recognized this nascent trend and coined a name for it: GPGPU (General-Purpose computing on Graphics Processing Units), and started GPGPU.org to provide a forum for those working in the field to share and discuss their work. Follow @harrism on Twitter