report/reflection.tools.tex

   1 % How useful were code analysis tools?
   2 The usefulness of the Fortify Static Code Analysis tool turned out to be very limited.
   3 Since we had most verdicts ready before a license was provided we could not use
   4 the tool as an initial guide through the code. This forced us to manually check
   5 the application source which took quite some time. After the tool became available we
   6 did not get any new insights regarding potential security risks, just more examples
   7 of problems we already detected. An example would be the use of the \code{crypt()} \PHP{}
   8 function which will, on some platforms, use the outdated \code{DES} algorithm in order to encrypt data. The
   9 use of this function would pose a security risk and results in a failed check. Since
  10 we where only interested in providing a verdict for each check a single occurrence of
  11 this function allowed us to back-up our verdict. Fortify provides a full list of all
  12 occurrences which could be used to fix the application, our verdict on the other hand
  13 does not provide the developer with any information on how to fix the problem. We just
  14 state that there is a problem.
  15
  16 % How could they be improved? (niet echt een antwoord maar we hebben de tool ook niet echt gebruikt?)
  17 In our opinion the tool could have proved very useful in pointing out certain security
  18 flaws in the initial stage of this project since we spent a lot of time scanning the
  19 application code-base. Since Fortify located relatively low-level problems we could
  20 have used these to locate potential hot-spots.
  21 Saving us from going through every source file and trying to determine if they are part of the
  22 applications external access points. In order to improve upon the tool we suggest a larger
  23 focus on determining which parts of a application need to be secure and less on pointing
  24 out actual security flaws.
  25
  26 % How did you experience the rates and amounts of false and true positives?
  27 As far we where able to verify the tool did not produce any false positives.
  28 However, Fortify was not able to detect all problems we found.
  29 Fortify concluded the application passed all checks in the
  30 Error reporting and Logging (\ref{sec:V8}) section, however we detected a number of severe
  31 problems in this area.
  32
  33 % How might that be improved?
  34 Since some problems occur multiple times it might be nice if Fortify was able to
  35 generate a clear overview of which components of the application contain detected
  36 problems. This could be very useful in combination with the information about
  37 components and functions which do pass the given security check. This would allow
  38 developers to determine if they suffer from chronically malformed code
  39 (e.g. all relevant code fails the check, indicating a very serious problem throughout the entire code base)
  40 or a single error (e.g. most relevant code passes the check except for a few isolated cases).
  41 In the tested code base there is a clean distinction between an installer component and the
  42 actual web application. If the installer suffers from problems not present in
  43 the web application and Fortify would be able to point out the specific check is
  44 relevant to both components the company would know which team or developer needs some major
  45 reeducation and who would be the best person to teach them.