What is Precision: Definition and 125 Discussions

Extended precision refers to floating-point number formats that provide greater precision than the basic floating-point formats. Extended precision formats support a basic format by minimizing roundoff and overflow errors in intermediate values of expressions on the base format. In contrast to extended precision, arbitrary-precision arithmetic refers to implementations of much larger numeric types (with a storage count that usually is not a power of two) using special software (or, rarely, hardware).

View More On Wikipedia.org
Recent contents Top users
  1. B

    Why do some integrals require correction for extreme precision?

    I have read that even the simplest integrals (like y=x2) might need some correction if we want to reach an extreme precision. Is that really so? Can you explain why or give me some useful links? Thanks
  2. M

    Change the precision in Ipython

    I want to plot an exponential function in Ipython, something like y=exp(x^0.53). But for small values of "x" it plots a straight line. Actually it plots the first terms of taylor expansion. How can I make it to give me curve for all values of x in my defined range? Should I change the precision...
  3. A

    Improved Precision Rectifier Circuit Benefits

    I'm having trouble understanding the following sentence, regarding an improved precision rectifier circuit: This circuit has the benefit that the op-amp never goes into saturation, but its output must change by two diode voltage drops (about 1.2 V) each time the input signal crosses zero...
  4. N

    Engineer a precision positioning setup for a paper on a roll

    There are several parts to this effort. Bond paper roll being stretched between 2 paper feeders on wheel bearings Precision positioning of the roll to a coordinate system devised for that roll That is if the paper roll is 35 meters long or 35'000 mm and 914.4mm wide I could choose a point on...
  5. I

    How can I build a precise voltage regulator for my Nokia N8 car charger?

    I have a nokia N8 cell phone and it is very precise about what it will allow to charge it. Mostly it will only accept wall chargers, not car power in any way. There is no way it knows the car charger isn't in the wall so all I can guess is the power is "dirty" and therefore it denies it. My...
  6. D

    Precision of the values of gravitational constant G

    Homework Statement What are the magnitud of G (gravitational constant) and the age of the solar system in CGS system of units and in seconds respectively? and what is the precision of these values and why? Homework Equations The Attempt at a Solution I found that the value of G in CGS system is...
  7. M

    Best way to go for precision aluminum parts?

    I might need to go to a metal shop/fabricator/what have you soon to have work done on a conceptually simple part that has some elements with measurements on the microscopic scale. Think cylinders with a diameter of .2 microns. Lots of them. I would want it done in aluminum. What would be the...
  8. F

    Calculating gravity with high precision ?

    Calculating gravity with high precision ? Few questions / concepts I am trying to get clarified...adding all the mass and all? types of energy ? Q Does a hot brick have a slightly stronger gravitational field than a cold brick ? Q Does a spinning brick have a slightly stronger gravitational...
  9. T

    Precision particle delivery via laser for 3D printing

    Greetings PFers, I was thinking about methods of delivering mediums used in 3D printing process, specifically 'selective laser sintering'. Currently SLS uses layers of fine metal powder which are fused together, layer by layer. A new coating of powder is applied every layer. Afterwards, the...
  10. O

    Uncertainty-what is the rate with 0.59% precision?

    Uncertainty--what is the rate with 0.59% precision?? Homework Statement Assume that negligible uncertainty in timing of a clock, calculate the number of counts required to measure a rate with 0.50% precision. Homework Equations The least accuracy is 0.50%? The Attempt at a Solution...
  11. G

    Fortran Problem setting integer precision in Fortran

    I'm trying to work with big integers but for some reason this program won't compile: program prob003 implicit none integer, parameter :: k32 = selected_int_kind(32) integer(kind=k32) :: num = 600851475143 end program prob003 The file name is prob003.f90 and I'm trying to...
  12. Daaavde

    How precise can Doppler shift be measured in detecting exo-planets?

    As the title says, I'm interested in the precision a Doppler shift can be revealed. That's why I read that it's possible to detect exo-planets by measuring the Doppler shift of the star's spectra. Works like this: the stars moves, since it moves back and forth is spectrum suffer from Doppler...
  13. haruspex

    Ranking the pairs in a bridge match - what precision?

    I've come across a bug in a program that calculates rankings from the results of a bridge tournament. It should be fixable by increasing the precision, but it's not obvious what the precision needs to be. (I don't have access to the source and anticipate push back from the coder, so I need it...
  14. M

    Calculator Vs computer precision

    Calculator Vs computer...precision Im doing a simple research to find out which device is better for calculations Valueof π On scientific calculator = 3.141592654 on computer calculator = 3.14159265359 How do I find percent error? And would this difference have any impact on...
  15. Q

    Arbitrary precision calculations

    I decided to write an arbitrary precision calculator but I'm having problems with division. It will multiply a thousand digit number by another thousand digit number in about a second but to get a division result to about 250 decimal places is taking around 20 seconds, I cannot figure out how...
  16. R

    Maximum precision of deterministic measurements

    Okay, I tend to have some weird thoughts, so bear with my odd question here. Suppose you have a collection of particles that obey Newtonian mechanics. For simplification, all particles are identical and can be assumed to be hard spheres that collide elastically. Each particle has a position...
  17. J

    The precision of the argument function error message, graph not plotted

    "The precision of the argument function" error message, graph not plotted I've got a function, integratedadvthirdaltb, that I'm trying to use in plotting some graphs: thirdaltb[KP_, Ps_, C_, M_] := NSolve[Sqrt[2*M]*b + InverseCDF[NormalDistribution[0, 1], Ps]*...
  18. D

    Precision of a sphere impact and velocity forces

    I am a police officer specializing in less lethal weapons. This question has come up many times and is beyond my understanding of physics to explain it. Please help: We use a tool called a PepperBall. It is the same shape and diameter of a paintball. This sphere however is rigid plastic...
  19. S

    Single and double precision matlab

    Homework Statement MATLAB uses double precision as default. How can I change this to single? so that all computations will be made using single precision. Homework Equations The Attempt at a Solution I went to: file preferences virtual reality precision: floating point...
  20. S

    MATLAB precision for variables

    Homework Statement The precision in MATLAB is approximately 2*10^(-16), is there a way that my computations can be calculated using a smaller precision? Homework Equations The Attempt at a Solution if i want to calculate the sum of pi^2*j where j=1:20 but i want it to be...
  21. D

    Comp Sci Fortran help: high precision intrinsic functions (kind = 16)

    Fortran help: high precision intrinsic functions (kind = 16)... Homework Statement Dear Everyone, I am using Fortran G95 compiler in windows 7 with SciTE as text editor. I have successfully compiled many programs (so there is no problem with my installations). I am having problems with...
  22. B

    Calculating Exponent Using Taylor Series To Given Precision

    Homework Statement The course is Computational Physics, but in a sense this is a pretty straight computer science or even mathematical challenge. The first part of the assignment - the relatively easy part - was to write a Fortran program to take two variables - the number to which e...
  23. A

    Accuracy, Precision and Sensitivity for measurement instruments

    My book defines accuracy as the closeness of a measurement to the real/actual value, precision as when a set of values are consistent with each other (i.e. close to each other) and sensitivity as the smallest possible reading of an instrument (i.e. to how many decimal places it can measure)...
  24. _maxim_

    High precision, good accuracy Current Source Schematic

    I am looking for a schematic diagram of a compact, temperature compensated, stabilized, high precision (accuracy within 0.5% or less) current source. The target current should be easily set to 100/150 mA, no matter if discrete or integrated solutions must be used. This project will be...
  25. C

    Launching a Ball with Precision: A Projectile Motion Project

    I'm currently working on a project for projectile motion and my teacher wants to see us launch a round ball about marble(maybe smaller) sized into a ring clamp on a retort stand somewhere around the room. She suggest we all use "gravity" (a simple ramp that let's a ball slide down and shoot up)...
  26. A

    Wavelength and precision of observation

    I don't understand why when you make an observation, you need a wavelength shorter than the wanted precision. i.e. you can't make clear pictures of golf balls with radio waves, you can't observe things smaller than a photon with an optical microscope.
  27. Y

    Measuring classical object to arbitrary precision

    This has been nagging me in the background of my mind for many years and I've decided to get it sorted. I note that I'm not sure if this should be in the classical or quantum forum. I have heard it mentioned by many and often when referring to the inherent uncertainty when measuring...
  28. B

    Op-Amp Precision Diode: Eliminating Voltage Drop

    [PLAIN]http://pokit.org/get/3deef3b185bf6b18007a25ddc18ce7d5.jpg So I have another thread running on op-amps, and while waiting for mr. Jiggy to come back online, I want to ask another op-amp question. This precision diode. I understand what is the point, and why it is so useful...
  29. M

    Call the same subroutine/function with different precision

    Hello everyone, As a part of my computational physics course i have been assigned a homework which is pretty simple. My problem is in optimising my source code. So the task is this: Write programs that calculate the machine round-off error for a real type variable with precision levels ...
  30. L

    Finding Time Given Number Of Trials And Precision

    Homework Statement We want to measure the specific activity (number of decays per second) of a radioactive source so that we can use it to calibrate the equipment of the gamma-ray experiment. We use an electronic counter and a timer to measure the number of decays in a given time interval...
  31. M

    Single precision floating point

    Homework Statement Convert 80/9 to single-precision floating-point. Homework Equations The Attempt at a Solution Right, I don't understand this at all but I'm trying to follow my notes, not very successfully.. 80/9 is positive so the sign bit is 0. Exponent is adjusted so...
  32. H

    Fortran Real vs. double precision in Fortran 90?

    Hey, I'm trying to write my very first Fortran program. I'm a physics student and I don't know a lot about computer science and since I'm new to Fortran I'm still having problems with finding answers to my questions on my own. So..please bear with me. I'm doing heavy numerical calculations...
  33. J

    Equivalence Principle precision

    I found 2 tables describing raising precision in experiments performed to investigate Equivalence Principle: http://en.wikipedia.org/wiki/Equivalence_principle#Tests_of_the_weak_equivalence_principle http://en.wikipedia.org/wiki/Hughes%E2%80%93Drever_experiment#Modern_experiments I can't...
  34. L

    Precision Rectifier Circuit w/ Opamp: Solving the Mystery

    I am bit confused about precision rectifier/superdiode circuit using opamp. The first ckt in the wikipedia link - http://en.wikipedia.org/wiki/Precision_rectifier When the input is positive, say 1v, the opamp tries to make the -ve terminal equal to its +ve terminal. So to do that, the...
  35. X

    Precision Rectifier: Get Answers to Your Questions

    hey,dude,i m confusing about the precision rectifier...why the Vout =E,my question is why the Vout=E - V(diode) ?why the Vdiode does not take into account when finding Vout?here the attachement of the precision circuit diagram..
  36. P

    5V to 5V precision voltage reference

    Is there any IC that can output a precision 5V from a 5V supply?
  37. P

    Precision 5V Reference: LT1085 vs LM317

    Anyone can suggest a very precise and constant 5V reference? I Google and see many are recommend LT1085 and LM317? Which one is better? Any better recommendation?
  38. L

    What is the Optimal Current Source for Extremely Low Current Applications?

    an application I'm working on requires 1.35 yoctoamps with a tolerance of + or - 0.10yoctoamps. Any current-source reccomendations?
  39. A

    How do you count Uncertainty Measurement using Precision Error and Bias Error?

    Homework Statement Find the Uncertainty Measurement using Precision Error and Bias Error? The Length of line is 5.625 in. Precision Error is 0.13 in. Bias Error is -0.115 in. What is the UNCERTAINTY measurement? Homework Equations None. The Attempt at a Solution Please help! I...
  40. G

    Building a Precision CNC Router

    I am going to build a CNC router. Although that's misleading since I don't nessecarily plan on actually using a router as the main tool. I want to see how precise I can get and what kind of quality I can achieve without spending lots of money. So I have some basic questions. The plans I've seen...
  41. Y

    Mathematica Precision issues in Mathematica (i think)

    Hi everyone. It's my first time posting in here. I have a mathematica notebook that defines a four-parameter prob density function (PDF). I define a likelihood function, import some data, and run a Nelder-Mead optimization strategy on the likelihood function using the data. My likelihood...
  42. T

    High precision tests of Maxwell equations

    Can anybody point me to some high precision tests of Maxwell's equations. I've tried hard to find some. Skepticism is a curse, I know.
  43. M

    Warning: Matrix is singular to working precision.

    Hello I am generating a matrix which is coming up with this warning. I know the matrix elements are correct though, as I put them in. But when I work with the matrix, the results don't make sense. I presume this warning is the reason. Can anyone help? I don't understand why MATLAB can't...
  44. A

    How to get first digit of a double precision number?

    EDIT: Sorry, forgot to mention 'FORTRAN' in the subject line... Hi, I'm new to fortran but I'm trying to generate the first 1476 terms of the fibonacci sequence, then examine the first digit of each term and store the number of 1s, 2s, 3s, ..., 9s that occur into an array. I have it to where...
  45. E

    Mathematica Precision Issue in Mathematica

    Hello, I have an equation that when I wrote using Mathematica to get results, the results were wrong in some places. This has nothing to do with the code, which is a 100% correct code. It is a presicion issue. But when I increased the precision, say up to 20, even to 10, the code takes a lot...
  46. R

    Precision during free-fall experiments.

    I'm not sure how to answer this question, so I'm turning to you guys for help. The question, specifically, is: "When timing the fall of a ball, how would you achieve greater precision, with larger or smaller heights?" Any help would you appreciated.
  47. R

    Are Physics Formulas Always Approximations?

    Are more complex physics formulas, like those involving exponents, roots, logs, trig functions, etc., approximations to a certain degree? Are even simple formulas approximations? If so, what kinda precision: deci-, centi-, milli-, or better? Is there a regulated precision for new physics...
  48. Topher925

    Design of precision circuitry?

    Design of precision circuitry? I'm currently working on a circuit that is so sensitive that if I even so much as breath on it for just a moment, it becomes completely useless as my signal will become completely saturated. While the circuit works very well (when I'm not breathing on it), even...
  49. F

    Clock precision as a function of temperature and height above sea level

    Homework Statement A grandfather pendulum clock is to be moved from a location at sea level to a new location approximately 200 m up in height. We assume that the Earth's radius is 6400 km. We know Newton's gravitational law and the oscillation equation for a pendulum. (*) How much...
  50. K

    Understanding Precision and Accuracy: Demystifying Uncertainty in Measurements

    Please explain uncertainty and give examples like i know it is 'half the smallest degree' just not suree.
Back
Top