accuracy/reproducability of regression tests

[Axel]

hi everybody!

i'm currently in the process of trying to automate
building cp2k on a number of platforms so that i
have a way to supply people with up-to-date executables
and a list of tested features.

however it seems that quite a number of the test results
could only be reproduced on the exact same machine
with the exact same compiler/library parallel etc. settings.

or more precisely asked: why do i get a 'WRONG' result with
a relative error of less than 1.e-10 when you have only 14 digits
absolute precision in a real*8 floating point number to begin with?
especially, when the SCF convergence of an input is not set very
tightly. just moving to a different platform, using a different
compiler,
a different optimization level, a different BLAS/LAPACK or running
a serial instead of parallel executable can induce changes of that
magnitude while still being accurate within the boundaries of
floating
point arithmetik, considering how many FLOPS are involved into
computing the properties the regression tester is comparing.

what about tests of properties, that simply cannot be computed
to that high accuracy at all?

do i have to make a 'leap of faith' and say that machine X is
executing
cp2k correctly when all regtests are flagged ok on berlios and then
use that output as my internal standard for that machine?

cheers,
   axel.