report/reflection.tools.tex

   1
   2 % How useful were code analysis tools?
   3 The usefulness of the Fortify Static Code Analysis tool turned out to be very limited.
   4 Since we had most verdicts ready before a license was provided we couldn't use
   5 the tool as an initial guide trough the code. This forced us to manually check
   6 the application source which took quite some time. After the tool became available we
   7 didn't get any new insights regarding potential security risks, just more examples
   8 of problems we already detected. An example would be the use of the \emph{crypt()} \PHP
   9 function which uses the outdated \emph{DES} algorithm in order to encrypt data. The
  10 use of this function would pose a security risk and results in a failed check. Since
  11 we where only interested in providing a verdict for each check a single occurrence of
  12 this function allowed us to back-up our verdict. Fortify provides a full list of all
  13 occurrences which could be used to fix the application, our verdict on the other hand
  14 doesn't provide the developer with any information on how to fix the problem. We just
  15 state that there is a problem.
  16
  17 % How could they be improved? (niet echt een antwoord maar we hebben de tool ook niet echt gebruikt?)
  18 In our opinion the tool could have proved very useful in pointing out certain security
  19 flaws in the initial stage of this project since we spent a lot of time scanning the
  20 application code-base. Since Fortify located relatively low-level problems we could
  21 have used these to locate potential hot-spots.
  22 Saving us from going trough every source file and trying to determine if they are part of the
  23 applications external access points. In order to improve upon the tool we suggest a larger
  24 focus on determining which parts of a application need to be secure and less on pointing
  25 out actual security flaws.
  26
  27 % How did you experience the rates and amounts of false and true positives?
  28 As far we where able to verify the tool didn't produce any false positives.
  29 However Fortify was not able to detect all problems we found.
  30 Fortify concluded the application passed all checks in the
  31 Error reporting and Logging (V8) section, however we detected a number of severe
  32 problems in this area.
  33
  34 % How might that be improved?
  35 Since some problems occur multiple times it might be nice if Fortify was able to
  36 generate a clear overview of which components of the application contain detected
  37 problems. This could be very useful in combination with the information about
  38 components/functions which do pass the given security check. This would allow
  39 developers to determine if they suffer from chronically malformed code
  40 (eg. all relevant code fails the check, indicating a very serious problem throughout the entire code-base)
  41 or a single error (eg. most relevant code passes the check except for a few isolated cases).
  42 In the tested code-base there is a clean distinction between an installer component and the
  43 actual web application. If the installer suffers from problems not present in
  44 the web application and Fortify would be able to point out the specific check is
  45 relevant to both components the company would know which team/developer needs some major
  46 reeducation and who would be the best person to teach them.