Software Security - Application Security Verification Project
OWASP ASVS Souce Code Review Project
In groups of about 4 students, you do a security code review of a small
web application, using the
OWASP ASVS (Application Security Verification Standard),
ASVS 3.0.1 (PDF)
and using static analysis tools to support the manual code review.
See here the current list of groups!
Goals of the project include
We realise that this is throwing you in at the deep end. Some of
you will have more experience with web applications, PHP, etc.
than others. Finding security vulnerabilities in the
application is less important than having a sensible &
well-argued opinion about the ASVS, static code analysis tools,
and the quality and design of the code in the end.
- Experiencing the process of doing a code review, where you struggle with limited time, people, and knowledge - and too much code without good design documentation;
- Finding out about the capabilities and limitations of code analysis tools;
- Finding out how good current best practices for doing security code reviews are, in particular the OWASP, are; the ASVS is one of the mature documents about (web-)application security.
- Getting some idea of how security design decisions and choices should be performed and documented to facilitate security reviews.
Whether you find all or indeed any security vulnerabilities is
not so important, so resist the temptation of getting ideas from
other groups. Indeed, we are hoping to use this experiment to get
some empirical data on code reviews -- how many eyeballs make for
As a guide, the goal is that everyone spends around 30 hours in
total for this project. You still have around 10 weeks till
Xmas, so spend a morning or afternoon a week.
The first time you get together with your group, fill in the questionnaire
together and email it to Erik Poll.
What to do
For the given application, check Verification Requirements V2 up to and incl. V9 and V11 in the
OWASP ASVS 3.0.1 [PDF]
Begin with a Level 1 verification (i.e. Opportunistic);
if you have time, move on to a Level 2 verification (i.e. Standard).
Some things you may run into:
- The need for sampling?
Some verification requirements will be too labour intensive
to check exhaustively for the entire codebase. In such cases
you might resort to sampling, i.e. only check the requirements
in a few places, and then base your verdict on that sample.
Different strategies are possible for this:
9i) you could choose a random sample (e.g. randomly pick some
files of source code, or pick some part of the interface of the
application to investigate in detail), or (ii) you could decide to
focus on the most security-critical functionality parts
of the code, or (iii) use a combination of these approaches.
- code vs configuration?
At some places you may run into issues which depend not so much
on the code but rather on the configuration of a particular
installation of the code. The border between code and
configuration is sometimes hard to draw; moreover, one can debate
about how defensive code should be when it comes to
facilitating/enforcing good and secure configuration.
If you come across such issues, better to concentrate on the code
first and not to get distracted into considering configuration
issues. It is then good to document where you drew the line
in doing the review. If you note that there are or may be important
configuration issues that impact security, it is of course useful
to flag these in your report.
- Finding one vs all security flaws?
A single security flaw of a certain kind can already show
that some verification condition is broken. So if you find one
flaw, it is not
necessary to then go hunting for more flaws of the same type.
Of course, it may be tempting...
If you notice that some problem occurs multiple times,
then it is good to record this, as it suggest the verification
condition is broken consistently, and you have not stumbled
on the one place the problem occurs.
Some of the security requirements may not be applicable or
relevant for a given application. One would probably say that
in these cases the
requirement is 'trivially' passed, albeit for a special reason.
For some security requirements you maybe unable to provide
a judgement, not only because of the issues above, but also
because the seecurity requirement may not be clear to you
or because it is unclear to you if the code meets the
requirement as you understand it, or both.
What to report
At the end, you have to provide
- a report which gives a motivated
verdict for all the verification requirements, and which
provides some reflection on
the whole process, including the ASVS and the use of static code
analysis tools. This report MUST be in PDF and mention your names
and group number on the front page. For TRU/e students from the
TU/e, please list both your RU and TU/e student nr.
a summary of your verdict in this
to allow is to quickly compare results when we discuss them
A suggested organisation of the report would be in the sections
as listed below, but feel free to diverge of this if you think
makes sense. In the organisation below the section giving the
verdict would be the longest, simply because it has to list all
the ASVS security requirements. Describing the organisation or
process for the review might only take half a page or so, and the
reflection would probably be longer than that.
At the end of the semester we'll have a discussion in class to
Briefly describe the way you organised the review.
E.g. did you split the work by files in the code, or by
category of security requirements? Did you double-check important
findings? Or did you use several pairs of eyeballs on the same
code/ security requirement, in the hope that more eyeballs spot
more problems? (How) did you integrate using the static code
analysis tools into this process? Did you use other tools and
Have you tried to run the application? (If so, was this
useful, and did you find than running the application was helpful
to then review the code, understanding its functionality better?
But you might want to dicuss this in the Reflection section.)
Give your judgement for each of the verification
requirements, with a short motivation.
The judgement could be
If you need addition judgments besides these, feel free to
introduce them, but -- as always -- do make sure your use
of terminology is consistent.
- DON'T KNOW,
if you are unsure of whether the code meets the requirement,
if you are unsure what the ASVS requirement means, or
or if you did not have time to look at it.
- NOT RELEVANT,
if a given requirements does not apply, and therefore you don't
need to look at it. So it is effectively a PASS, for the
trivial reason that it does not apply.
With respect to motivation: ideally you'd like to give a brief
and concise justification, of say one short sentence, for your
verdict of a verification requirement. But if a verification
requirement is very clear and it is straightforward how one would
check it, it might not be worthwhile to write up anything.
Conversely, sometimes it may be quite hard to argue that some
verification requirement is met: the violation of a requirement
is often easier to motivate (namely with a counterexample) than
If you resorted to sampling to judge some requirement (or group of
requirements) or if you considered some aspects out
of scope (e.g. because it is related to the configuration rather
than the code), that would be something to mention too. You could
also say that in Section 1, if that makes more sense.
Reflect on the whole process, including
For example, questions to consider are:
- the ASVS,
- the use of static code analysis tools,
- the way you organised the process,
- and possibly also the TestCMS code.
- How good (useful, clear, ...) is the ASVS? How could it be
- How useful were code analysis tools? How could they be
How did you experience the rates and amounts of false and true
positives? How might that be improved?
- What were the bottlenecks in doing the security review in your
- Maybe in the points above you can distinguish different
(types of) security flaws or verification requirements.
E.g., are some (categories of) verification requirements
easier to check than others?
- If you would have to do something like this again, what would you do
Eg. about organising things within the group: i.e.,
in retrospect, what do you think the
best approach is to organise and divide the work in a team? (Dividing the verification requirements over the
team members? Or by dividing the code? Or letting everyone
look at everything, because different people will spot different
things? Or work in pairs where one person confirms the findings
of the other? ...)
- About the TestCMS code: are there important aspects that could (or
should) be changed to improve security? Or aspects that could be
changed to facilitate doing a security review?
This last question might be generalised, to (web) applications in
general, rather than this specific example of the TestCMS: ie.
if you were to develop an application that will need to be
subjected to a security review, would you do anything
differently, given your experience in doing this, and if so, what
- (optional) Appendix: vulnerabilities
Instead of describing vulnerabilities that you came across as
part of the motivation in your verdict in Section 2, you could
also move the details of these vulnerabilities to an appendix,
say in a numbered list, that you can then refer to in the `Verdict'
Tool support for the ASVS
Security Knowledge Framework, also available here, is a tool that supports using the ASVS in development, with pointers to more background knowledge, code samples, etc. It's more geared towards development rather than code reviews, but the pointers to background info on the ASVS maybe useful. For more info, there a screencast walking you through the online demo.
Pointers to static code scanning tools
Pointers to commercial code scanning tool will be emailed separately.
- RATS, which stands for "Rough Auditing Tool for Security" is a extremely basic static checker, which essentially uses grep or CTRL-F to look for dangerous functionality.
- Unfortunately, all open source efforts at source code
analyzers I am aware of, such as
have died, gone commercial (which is good news in a way), or are simply too immature to use.
Misc sources of information about PHP
Documentation generation tools
The tools below automatically generate some documention and API
information from source code, which might be useful to browse the
code. Of course, your favourite IDE for PHP might also have
support for this.