We have decided to introduce testing for security vulnerabilities into web application development. Previous part introduced cross site scripting, our web application and expectations we have for this project.
In this part, we go through all penetration testing tools we could find. Our goal is to find a suitable open source scanner. We wrote mini review of each found tool and picked up two scanners we will use.
We created a simple web application and run all scanners against it. It requires log in and contains one form. Each user can fill his 'name', 'last name' and 'about' fields on his personal account page. Anyone can see submitted data on 'all accounts' page.
The application has no cross site scripting protection. There is no tricky situation or gotcha such as partially escaped field. Such detailed quality comparison is out of scope of this article.
The framework had to be able to pass log in screen and find an obvious cross site scripting vulnerabilities. We evaluated also documentation, user interface and general hustle-free ease of use.
Long reviews are available for following scanners:
Short reviews are available for following scanners:
It is customizable via extensions, the same way as Firefox.
WebSecurify is a GUI-only application. They have some plans for server version (whatever it means).
Weak points:
Introduction article is available on creators blog and documentation on Google Code wiki. The project produced also 'Browser Security Handbook' which is definitely worth reading.
SkipFish has slightly different design goals than other scanners. They value speed and memory footprint over anything else. They also promise to use well crafted high quality security checks. Unfortunately, SkipFish found neither reflected nor stored XSS vulnerability in our sample application.
Our test application relies on correct hidden parameters. If the hidden parameter is not among allowed set of values, application crashes. As a results, SkipFishs attempts to find a vulnerability caused application crashes and no vulnerability was found. Moreover, it was not possible to find about the issue from SkipFish output alone.
SkipFish found reflected XSS vulnerabilities only after we changed the demo application. Unfortunately, such solution is not feasible for any real application.
Neutral points:
Weak points:
The scanner has both console and graphical GUI. Moreover, it is possible to write script for the console version and run it from command line.
Which is needed. Our sample application turned out to be challenging for W3af. We redirect all un-logged users to the log in screen. W3af came into conclusion that it must be a 404 page and all links in our application are broken. We had to remove the authentication to make it test our demo application.
On the other hand, W3af is the only scanner we evaluated that found stored XSS vulnerability. Moreover, the source code is easily readable, so we could quickly identify the problem.
The most important phases are discovery, audit and attack:
Weak points:
Project goal was to create free, easy to use and well documented penetration testing tool. GUI is designed with an inexperienced penetration tester in mind. It is extremely easy to use. ZAP proxy is the only penetration testing tool we found with detailed vulnerabilities documentation accessible to anyone.
Command line version is not available, but you can programmatically interact with a running instance. Fortunately, they accepted an open issue for command line enhancement. It is not possible to add own attack vectors, but they have some plans for it. As their list of open issues is full of cool ideas, the project is worth of following.
Exploring phase: You are supposed to show ZAP all pages you wish to test during this phase.
Start ZAP and configure your browser to use it as a proxy. Open application you wish to test and visit as many pages as possible. ZAP will passively analyze all visited pages and show warnings and alerts as you browse.
The list of visited pages is available in 'Sites' view. You may manually edit it or save for later use. Automated spider is available too.
Active phase: ZAP actively attacks all pages you choose to. Unless you explicitly use spider, only visited pages are analyzed. In other words, it completely ignores all links and buttons not used during previous phase.
ZAP sends various malformed/malicious requests to all target pages. Dangerous XSS, XSRF and SQL vulnerabilities are found during this phase.
Weak points:
The tool is designed for those 'who can write code themselves, or at least have a pretty good understanding of the HTTP protocol'. WebScarab is a GUI-only application and has two modes:
Notable features as found in manual:
As it requires an active experience user, we did not consider it further. However, it may be a good tool for an experienced professional.
It is designed as passive scanner and thus may be employed against production systems. Human interaction is required. The documentation is available and we found one tutorial.
It does not satisfy our requirements, so we did not consider it further. However, the project may be worth evaluation if you are looking for a passive tool designed for an experienced user.
It works under Linux or Cygwin environment. From all evaluated scanners, Arachni is the most difficult to install on Windows 7. We spend more time trying to install it, then analyzing all other scanners. It worked at one moment, but we could not reliably reproduce the success.
Arachni may be solution for someone with linux or 32 bit windows, but we gave up on it for now.
We decided to use W3af and Zed Attack Proxy. They have almost all required features. Both are open source and pluggable. As they have easily readable source code, we can extend them as we need.
The former one will become part of our automated tests. It will run from unit tests on continuous integration server or IDE. The latter one will be used by our test team.
Last note: Browser Security Handbook from SkipFish deserves special mention. The book is the best reading found while writing this post.
In this part, we go through all penetration testing tools we could find. Our goal is to find a suitable open source scanner. We wrote mini review of each found tool and picked up two scanners we will use.
Methodology
Long version of our requirements is written in previous post. In short, we are looking for a penetration testing tool that:- is easy to use so we can teach it to testers,
- requires no human interaction so we can automate test suite.
We created a simple web application and run all scanners against it. It requires log in and contains one form. Each user can fill his 'name', 'last name' and 'about' fields on his personal account page. Anyone can see submitted data on 'all accounts' page.
The application has no cross site scripting protection. There is no tricky situation or gotcha such as partially escaped field. Such detailed quality comparison is out of scope of this article.
The framework had to be able to pass log in screen and find an obvious cross site scripting vulnerabilities. We evaluated also documentation, user interface and general hustle-free ease of use.
Reviews
We wrote reviews on all scanners we found. Long review was written if the scanner looked promising or was simple enough to learn all about it in short time. Short review was made otherwise.Long reviews are available for following scanners:
Short reviews are available for following scanners:
- WebScarab
- WebScarab Next Generation
- Ratproxy
- Arachni
- Exploit Me
- Nikto
- ParosProxy
- ProxMon
- Nessus
- Burp Suite
- Netsparker
WebSecurify
WebSecurify is currently in version 0.8. Source code is available under 'GNU GPL v2' license. The project has a blog, a discussion group and basically no documentation at all. The documenation is planned for 1.0.It is customizable via extensions, the same way as Firefox.
WebSecurify is a GUI-only application. They have some plans for server version (whatever it means).
Experiences in Points
Strong points:- WebSecurify is extremely easy to use. User literally needs to know nothing about security, web applications or penetration testing.
- It is configurable through Firefox extensions.
- Found reflected XSS vulnerability.
Weak points:
- It is configurable
only through Firefox extensions. - Documentation is not available.
- Command line is not available - requires active user.
- URL whitelist is not available, only blacklist.
- Did not found stored XSS vulnerability.
- We have no idea which and how many attack vectors have been used.
- Impossible to export reports.
- We do not know whether it is able to pass through JavaScript of flash.
Conclusion
Not suitable for original requirements, but good enough for first quick and dirty test. Not bad if you do not have time at all, have absolutely no experience and want to do at least something. Which is not our case.SkipFish
SkipFish is a command line only tool written in c. It is distributed as packed source code under 'Apache License 2.0'. We did not found an online version control repository.Introduction article is available on creators blog and documentation on Google Code wiki. The project produced also 'Browser Security Handbook' which is definitely worth reading.
SkipFish has slightly different design goals than other scanners. They value speed and memory footprint over anything else. They also promise to use well crafted high quality security checks. Unfortunately, SkipFish found neither reflected nor stored XSS vulnerability in our sample application.
Problem Explanation
All scanners have to change request parameters to find out XSS vulnerabilities. Usually, they change them one at a time. Unfortunately, SkipFish changes all parameters at once.Our test application relies on correct hidden parameters. If the hidden parameter is not among allowed set of values, application crashes. As a results, SkipFishs attempts to find a vulnerability caused application crashes and no vulnerability was found. Moreover, it was not possible to find about the issue from SkipFish output alone.
SkipFish found reflected XSS vulnerabilities only after we changed the demo application. Unfortunately, such solution is not feasible for any real application.
Experiences in Points
Strong points:- HTTPS is handled with no problems.
- Reported XSRF vulnerability.
- Generates nice HTML report.
- Easy to learn and use.
- Browser Security Handbook
Neutral points:
- Session cookie must be acquired elsewhere and passed as a parameter to bypass login.
Weak points:
- Does not support plugins.
- Did not found neither reflected not stored XSS.
- We had to change the application to use it - it would miss vulnerabilities otherwise.
Conclusion
Being a command line utility with nice HTML report, SkipFish was a strong candidate. Unfortunately, we can not use it due to the way it handles request parameters.W3af
W3af is currently in version 1.0. Python source code is available under 'GNU GPL v2' license. Getting started manual is downloadable in pdf format.The scanner has both console and graphical GUI. Moreover, it is possible to write script for the console version and run it from command line.
Overview
W3af has extremely broad list of features. The default configuration contains almost everything including yahoo, google and bing searches. Each plugin (feature) can be enabled or disabled and comes with a description. Of course, you can create your own plugins.Which is needed. Our sample application turned out to be challenging for W3af. We redirect all un-logged users to the log in screen. W3af came into conclusion that it must be a 404 page and all links in our application are broken. We had to remove the authentication to make it test our demo application.
On the other hand, W3af is the only scanner we evaluated that found stored XSS vulnerability. Moreover, the source code is easily readable, so we could quickly identify the problem.
Plugins and Phases
W3af plugins are grouped into eight phases: audit, bruteforce, grep, evasion, attack, output, mangle and discovery. All plugins that belongs to the same phase run at the same time. Complete list of all plugins is quite long.The most important phases are discovery, audit and attack:
- Discovery plugins search for new URLs, forms and other "injection points".
- Audit plugins sends specially crafted data to all known "injection points". XSS and SQL injection vulnerabilities are found during this phase.
- Attack plugins exploit vulnerabilities found during previous phase.
Experiences in Points
Strong points:- huge number of features,
- found reflected XSS vulnerability,
- found stored XSS,
- generates HTML, XML and text reports
- pluggable.
Weak points:
- white list and black list are available only for web spider plugin,
- does not support form based authentication.
Conclusion
W3af is a very strong candidate. It is the only scanner able to find stored XSS vulnerability. It does not require human interaction, so it will be possible to run it from a continuous integration tool or test suite.Zed Attack Proxy (ZAP)
Zed Attack Proxy (ZAP) is currently in 1.3.0. It is written in java and available under 'Apache License 2.0' license. ZAP is fork of ParosProxy. Tutorials are available on founders blog.Project goal was to create free, easy to use and well documented penetration testing tool. GUI is designed with an inexperienced penetration tester in mind. It is extremely easy to use. ZAP proxy is the only penetration testing tool we found with detailed vulnerabilities documentation accessible to anyone.
Command line version is not available, but you can programmatically interact with a running instance. Fortunately, they accepted an open issue for command line enhancement. It is not possible to add own attack vectors, but they have some plans for it. As their list of open issues is full of cool ideas, the project is worth of following.
Overview
Zed Attack Proxy evaluation runs in two phases:Exploring phase: You are supposed to show ZAP all pages you wish to test during this phase.
Start ZAP and configure your browser to use it as a proxy. Open application you wish to test and visit as many pages as possible. ZAP will passively analyze all visited pages and show warnings and alerts as you browse.
The list of visited pages is available in 'Sites' view. You may manually edit it or save for later use. Automated spider is available too.
Active phase: ZAP actively attacks all pages you choose to. Unless you explicitly use spider, only visited pages are analyzed. In other words, it completely ignores all links and buttons not used during previous phase.
ZAP sends various malformed/malicious requests to all target pages. Dangerous XSS, XSRF and SQL vulnerabilities are found during this phase.
Vulnerabilities Documentation
Found vulnerabilities are available on Alerts page. Double click on found vulnerability shows risk level (high, low, ...), comprehensible description and more importantly solution and links to further reading. This view alone makes ZAP great educational tool.Plugins
Create jar file with subclass of AbstractPlugin inside and copy it to plugin directory. We did not tried to write new plugin, but judging from what we have seen in the source code, it should be easy.Experiences in Points
Strong points:- best GUI we found,
- great education tool,
- plugginable,
- documentation,
- highly recommended for inexperienced penetration testers,
- found reflected XSS vulnerability,
- generates both XML and HTML reports,
- HTTPS is handled with no problems.
Weak points:
- command line not available - requires active user,
- did not found stored XSS vulnerability.
Conclusion
Our test team will use Zed Attack Proxy.WebScarab
WebScarab is a proxy sitting in between browser and targeted website. It is written in java and have pluggable architecture.The tool is designed for those 'who can write code themselves, or at least have a pretty good understanding of the HTTP protocol'. WebScarab is a GUI-only application and has two modes:
Notable features as found in manual:
- HTTPS is supported,
- scripting console,
- reveal hidden fields - turns all hidden field on the page into text fields,
- spider does not identify URLS from HTML forms.
As it requires an active experience user, we did not consider it further. However, it may be a good tool for an experienced professional.
WebScarab Next Generation
WebScarab Next Generation is a GUI redesign of WebScarab. It will have nicer GUI and all original features. The project is currently under development. We did not consider it further.Ratproxy
Ratproxy is distributed as a packed source code. We could not find an online version control repository. The project is written in c.It is designed as passive scanner and thus may be employed against production systems. Human interaction is required. The documentation is available and we found one tutorial.
It does not satisfy our requirements, so we did not consider it further. However, the project may be worth evaluation if you are looking for a passive tool designed for an experienced user.
Arachni
Arachni is in version 0.2.3. It is written in ruby and available under 'GNU GPL v2' license. It is a command line utility with web interface available.It works under Linux or Cygwin environment. From all evaluated scanners, Arachni is the most difficult to install on Windows 7. We spend more time trying to install it, then analyzing all other scanners. It worked at one moment, but we could not reliably reproduce the success.
Arachni may be solution for someone with linux or 32 bit windows, but we gave up on it for now.
Exploit Me
Exploit Me is a suite of plugins compatible with Firefox 2.0.0.8 - 3.7a1pre. They are designed to be lightweight and easy to use. The suite has positive reviews, but does not satisfy our requirements. We did not consider it further.Nikto
Nikto is an Open Source (GPL) web server scanner which performs comprehensive tests against web servers, not web applications. We did not consider it further.ParosProxy
ParosProxy project is abandoned. According to SourceForge cvs statistics, it was alive in 2003. Little activity took place in summer 2007. We did not consider it further.ProxMon
ProxMon project activity page is empty. The project was announced in 2007 but probably never finished. Their blog was updated last time in 2009. It is written in python and available under 'GNU GPL v2' license. We did not consider it further.Nessus
Nessus is a paid tool. If you wish to use it for work, you have to purchase a ProfessionalFeed subscription. We did not consider it further.Burp Suite
Burp Suite is a paid tool. We did not consider it further.Netsparker
Netsparker is a paid tool. We did not consider it further.Conclusion
We found no scanner that would fully satisfy our requirements. For instance, no scanner covers first half of Crawler Limitations section. Fortunately, multiple scanners come close.We decided to use W3af and Zed Attack Proxy. They have almost all required features. Both are open source and pluggable. As they have easily readable source code, we can extend them as we need.
The former one will become part of our automated tests. It will run from unit tests on continuous integration server or IDE. The latter one will be used by our test team.
Last note: Browser Security Handbook from SkipFish deserves special mention. The book is the best reading found while writing this post.
7 comments:
Hi!
Nice post! I conducted similar test recently and I agree about W3AF. I decided to use it too. Seems like I underestimated ZAP a little bit, I'll give it a second look...
However I am surprised that you missed Wapiti in your test! Although it's known as SQLi scanner, it does pretty decent job with XSS too. And Netsparker - although it's a commercial tool, they have a free Community version.
Seems like we had similar experience with Arachni. ;) Took me 2 days to install and run it + 1 day to run web interface. :D But it's running now, and it's stable... However you cannot install it on Windows, you will need Linux.
Thank you for hints Anonymous. Wapity seems like nice command line tool. I did not had time for Netsparker Community Edition yet, but I will definitely check it out.
Proxystrike can be given some attention
need your help can u pls post this page ... on your page ... need some likes so that i can remove confirm button and get my website up and running ...
www.facebook.com/Hackerspoint00
OWASP Xenotix is a new entry. it's a dedicated XSS scanner claiming some unique features.what do you think about it?
No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject!
A very relevant post which is discussed on a introduction testing for security vulnerabilities into web application development. I got a lot of info about this new thing over here. I got how to refill canon ink cartridges is relevant for those who are using scanner. So you can check it out.
Post a Comment