AppSensor - Intrusion Detection

Imagine that you have created a nice web application and secured it to your best. Users came, used it and everything was OK until someone stumbled upon vulnerability in your application and used it. Of course, you analyzed logs and found that the bad guy was looking for the vulnerability for weeks until he found one.

Creators of AppSensor intrusion detection framework believe that the above situation should not happen. The application should not just lie there and let itself beat with SQL injections, XSS attacks and whatever else. It should take active measures to protect itself. As the average attacker has to make several attempts to find the vulnerability in the application, it should by possible to detect hacking attempts.

The idea of in-application intrusion detection framework seems to be quite original. While intrusion detection frameworks are common in network security world, we found no alternative to AppSensor. If you know about one, please let us know.

AppSensor Project

AppSensor is part of Open Web Application Security Project (OWASP) project. It started as a conceptual framework and evolved into a real library. The project is currently in beta.

AppSensor is located in two places:

• page on OWASP (Open Web Application Security Project) wiki,
• google code page.

The creator wrote short and easily readable book with project overview, recommendations and best practices. The book is worth reading even if you decide to implement your own solution. If you decide to use AppSensor, getting started guide for developers is available on Google code.

Overview

High level design is nice and simple. AppSensor has detection points, response actions and intrusion detector. Detection points are able to detect ongoing or probable attacks. They are spread through the application and raise app sensor exceptions whenever intrusion condition is met.

For example, "GET When Expecting POST" detection point triggers warning whenever page expecting only POST requests, is requested by HTTP method GET. Or "Cross Site Scripting Attempt" detection point raises an exception if the request contains common XSS attack pattern.

Intrusion detector analyses all raised exceptions. If it detects an ongoing attack, it decides which response action is needed. Response action may do anything from raising up log level to shutting down whole application. Which action is selected depends on both suspicious activity and application nature. Where trading application would immediately disable user account, discussion forum would only raise log level and warn user to stop the activity.

AppSensor is part of Enterprise Security API (ESAPI) project. Out of the box version supports only ESAPI secured projects. However, our example application is secured with Shiro framework. The integration between both frameworks is described in another blog post.

Do no Harm

While the whole point of intrusion detection system is to detect and stop an ongoing hack attack, it is equally important not to harm innocent users. AppSensor book suggests to divide detection points into two categories:

• attack events,
• suspicious events.

If a hidden form variable suddenly contains "' OR 1=1"", there is no doubt that someone is hacking the application. The event is considered an attack event and the system may take strong action (log out, disable account) against the user.

On the other hand, unexpected ";'" in request parameter is considered only suspicious. Maybe it was a typo. Maybe it was malicious hacker trying to keep low profile. As we do not know, we can conclude that the system is under attack only if the event occurs multiple times on multiple fields. In this case, the system should record the event and raise log level for current user. Stronger action will be taken only after repeated events.

Of course, application context does matter too. Unexpected <img> html tag submitted into trading applications 'last name' field must be taken seriously. The same prohibited <img> tag in discussion forum can be taken lightly. The user may be trying to add an animation or whatever to his username. It is annoying, but hardly an attack.

Example Application

We have been testing AppSensor on SimpleShiroSecuredApplication created as part of Apache Shiro series. You can download example from intrusion_detection branch on Github. Use test class RunTestWait test case to run a web server with application deployed on https://localhost:8443/simpleshirosecuredapplication/ URL.

The application has seven users: administrator, friendlyrepairman, unfriendlyrepairman, mathematician, physicien, productsales and servicessales. They all share the same password: 'heslo'.

Add AppSensor

First, we have to add AppSensor dependency into the application and enable the framework as intrusion detection provider in ESAPI configuration. We have to do this even if we are not using ESAPI in the project.

Add AppSensor dependency to pom.xml:

  org.owasp.appsensor
  AppSensor
  0.1.3.2
  jar
  compile

Create '.esapi' folder inside src/main/resources folder. Hint for Eclipse users: eclipse package explorer does not show files and folders whose names begin with a . (period). To show them, go to the package explorer view and press little down arrow in the upper right corner of the view. Click Filters and uncheck '.* resources' entry.

Unpack AppSensor jar file and copy three configuration files

• .esapi/appsensor.properties - AppSensor configuration,
• .esapi/validation.properties - ESAPI input validation configuration,
• .esapi/ESAPI.properties - ESAPI configuration

into resource/.esapi folder. Alternatively, you can download them from Google code repository. Change ESAPI.IntrusionDetector entry in ESAPI.properties file to use AppSensor as intrusion detector:

ESAPI.IntrusionDetector = org.owasp.appsensor.intrusiondetection.AppSensorIntrusionDetector

Integration

Some user actions are considered to be threat only if they are repeated by the same user too many times. Moreover, intrusion detector may decide that 'log out user' or 'disable account' response actions are needed. All that requires access to underlying application data and APIs.

Default AppSensor configuration assumes that application supports ESAPI interfaces. However, SimpleShiroSecuredApplication is secured by Shiro framework which has different set of features. As the integration between two frameworks is more about Shiro than about AppSensor, we moved it to separate post.

Detection Points

OWASP page hosts extensive list of suggested detection points. It contains almost everything possible. Detection points are divided into several categories:

• request and session - manipulation with URL, request, cookies, ...
• input - user input contains common XSS attack, unusual character, entered text is longer than input field size, ...
• honey trap - otherwise unused hidden field or request variable with tempting name is_admin is modified, url available only in html comment is requested, ...
• user or system trend - too many log outs across the site, user clicks faster than is humanly possible, user changes his first name field too often, ...
• ...

A lot of effort went to finding out detection points that could be triggered by an innocent activity. Most of these detection points are not implemented yet. The class AttackDetectorUtils contains the majority of implemented detection points:

• unexpected request method,
• xss attack,
• sql injection,
• null byte in request parameter,
• cookies presence,
• carriage return or line feed presence.

Additionally, two servlet filters are available:

• validate whether user agent changed mid session (UserAgentChangeDetectionFilter),
• validate whether IP address changed mid session (IpAddressChangeDetectionFilter).

Beware: be careful, xss and sql injection detection points need to be reconfigured before use (more about it later).

Adding Detection Points

Each detection point has unique code. If the application detects an intrusion or suspect event, it can use the code to raise an AppSensorException:

if (intrusionDetected()) {
  new AppSensorException("CODE", "Message to user.", "Log message.");
};

Note: there is no 'throw' clause. Exception constructor automatically triggers it.

Alternatively, prepared detection points are available in AttackDetectorUtils class:

AttackDetectorUtils.verifyXSSAttack(inputValue)

or you can configure servlet filters in web.xml:

  IpChangedDetectionPoint
   org.owasp.appsensor.filters.IpAddressChangeDetectionFilter 
 

 
  IpChangedDetectionPoint
  /*
 

Each detection point has intrusion threshold and associated response actions. Threshold is a number of events needed within time period before an associated action is taken.

For example, intrusion exception CODE is considered an attack if it is raised four times within ten minutes. If the threshold is reached first time, the event is only logged. User is signed out after second offense. Third offense (e.g. the exception was raised twelve times within ten minutes) will completely disable user account:

# number of intrusions in a specified segment of time that constitutes the upper threshold - once crossed, it's considered an "attack"
IntrusionDetector.CODE.count=4
# time period (in seconds)
IntrusionDetector.CODE.interval=600
# list of actions you want executed in the specified order
IntrusionDetector.CODE.actions=log, logout, disable

XSS Attack and SQL Injection Detection

We start with detection points detecting two popular attacks on web applications: SQL injection and XSS attack. Whenever application reads value from request parameter, the value is passed to detection points:

/**
 * Reads sanitized request parameter value from request.
 */
protected String getField(HttpServletRequest request, String parameter) {
  String dirtyValue = request.getParameter(parameter);
  
  // detection point: XSS, SQL injection
  AttackDetectorUtils.verifySQLInjectionAttack(dirtyValue);
  AttackDetectorUtils.verifyXSSAttack(dirtyValue);
  
  return sanitizer.sanitize(dirtyValue);
}

According to AppSensor book, these detection points detect intrusion whenever an attacker tries to put in SQL injection or XSS attack. If that would be a case, system should take strong action against user. We configured it in appsensor.properties file:

# XSS attack -- be careful about these settings
# clear attack event, taking action immediately 
IntrusionDetector.IE1.count=1
# clear log after 20 minutes
IntrusionDetector.IE1.interval=1200
# first offense log user out, second disable account  
IntrusionDetector.IE1.actions=logout, disable

# SQL injection -- be careful about these settings
# clear attack event, taking action immediately 
IntrusionDetector.CIE1.count=1
# clear log after 20 minutes
IntrusionDetector.CIE1.interval=1200
# first offense log user out, second disable account  
IntrusionDetector.CIE1.actions=logout, disable

However, we suggest to be very careful about it. Open 'personal account page' of SimpleShiroSecuredApplication and put one of these strings into any field:

The dog should fetch the stick as soon as possible.

please delete all unused configuration files

Any text with ; in it.

SQL injection detection point triggers an exception after any of them.

The problem is caused by the word fetch in the first message, by the word delete in the second message and by the symbol ';' in the last message. They are quite common in standard English texts, but all of them raise the SQL injection exception.

XSS attack detection point may cause false positives too (however less likely):

Hi, 
thank you for installation scripts and requirements document.cookies has been delicious, please bring them again :)).
See you soon,
Andy

Default attack patterns are configured in appsensor.properties file. Use standard java regular expressions:

# This collection of strings is the XSS attack pattern list
xss.attack.patterns=\"><script>,script.*document\\.cookie,<script>,<IMG.*SRC.*=.*script,<iframe>.*</iframe>

# This collection of strings is the SQL Injection attack pattern list
sql.injection.attack.patterns=\\-\\-,\\;,\\/\\*,\\*\\/,\\@\\@,\\@,nchar,varchar,nvarchar,alter,cursor,delete,drop,exec,fetch,insert,kill,sysobjects,syscolumns

We strongly suggest to change these lists to something less strict.

Response Actions

The project contains also list of possible response actions. Suggested actions are divided into following categories:

• silent - logging change, administrator notification, ...
• passive - show warning, slow down application for the user, ...
• active - log out, disable module, require additional verification (captcha), ...
• intrusive - be careful to stay on the legal side

AppSensor implements only seven response actions. Five are available in our application:

• log event,
• logout user,
• disable user account,
• send sms to administrator,
• email admin.
• disable component,
• disable component for user.

Both disable component response actions lock access to last called URL. They both require additional configuration.

Keep in mind, that response action may significantly change code behavior. For example, it may cause seemingly random unexpected user log outs, which may cause unexpected null pointer exceptions or other similar problems. Your design have to be robust enough to handle such situations correctly.

Disable Component

If you wish to use 'disable component' or 'disable component for user' response actions, you have to create page that will show instead of blocked page:

<title>Access Denied</title>
</head>
<body>
Sorry, you do not have access rights to that area.
</body>
</html>

Only URLs that ends with suffixes configured in appsensor.properties are blockable. We will use only jsp and html pages. All our servlets will have suffix servlet:

# This is the list of extensions to check for disabling, ie. jsp (for jsp's), do (for struts 1), UpdateProfile (for the UpdateProfile servlet) 
disable.component.extensionsToCheck=jsp,html,servlet,

It is not possible to configure 'all pages suffix does not matter'. You have to list all possible suffixes. However, you can configure exceptions that will never be blocked. As we do not use it, we will leave there default configuration:

# This is the list of exceptions to disabling components, ie this list should never be disabled
disable.component.exceptions=/AppSensorDemo/appsensor_locked.jsp,/AppSensorDemo/login.jsp,/AppSensorDemo/updateProfile.jsp

Enable disable component actions in web.xml:

 AppSensorBlockComponent
  org.owasp.appsensor.filters.AppSensorRequestBlockingFilter 
 
  redirectURL/simpleshirosecuredapplication/account/accessdenied.jsp 



 AppSensorBlockComponent
 /*

AppSensor redirects blocked requests to redirectURL.

Finally, disable component response actions are configured inside detection point configuration in appsensor.properties file. Configuration of 'disable component for user' looks like this:

# some integer - duration of time to disable
IntrusionDetector.ACTION_DOES_NOT_EXISTS.disableComponentForUser.duration=30
# some measure of time, currently supported are s,m,h,d (second, minute, hour, day)
IntrusionDetector.ACTION_DOES_NOT_EXISTS.disableComponentForUser.timeScale=m

Configuration of 'disable component' is similar. Default duration is 0, so the action does nothing if you do not configure it. Use constant -1 to disable component forever.

Custom Response Action

If you plan to use AppSensor in your application, you will probably want to add a new response action or modify a default one.

Default response actions are handled by DefaultResponseAction class. If you need different set of response actions, you have to implement ResponseAction interface. Following class adds a 'warn user' action:

public class CustomResponseAction implements ResponseAction {
 
 private final ResponseAction delegee=DefaultResponseAction.getInstance();

  @Override
  public boolean handleResponse(..., AppSensorIntrusion currentIntrusion) {
  if ("warn".equals(action)) {
   Exception securityException = currentIntrusion.getSecurityException();
   String localizedMessage = securityException.getLocalizedMessage();
   
   ASUtilities asUtilities = APPSENSOR.asUtilities();
   HttpServletRequest request = asUtilities.getCurrentRequest();
   request.setAttribute("securityWarning", localizedMessage);
   
   return true;
  }
   
   return delegee.handleResponse(action, currentIntrusion);
  }

}

Enable new class in appsensor.properties file:

# This is the class that handles the response actions
AppSensor.responseAction = org.meri.simpleshirosecuredapplication.intrusiondetection.responseaction.CustomResponseAction

Note: If you want to extend DefaultResponseAction, you have to override static method getInstance(). The class responsible for initialization would otherwise create an instance of DefaultResponseAction.

Intrusion Store

All detected intrusions are stored in intrusion store object. Default intrusion store provides very simple implementation: it stores all intrusions in static in memory map. It is not suitable for clustered environment and all stored data are lost after system restart.

If you have special needs, you can replace it with own solution. Implement IntrusionStore interface and configure it in appsensor.properties file:

# This is the class that handles the intrusion store
AppSensor.intrusionStore = org.owasp.appsensor.intrusiondetection.reference.DefaultIntrusionStore

Overall Impressions

AppSensor is a great idea and the only intrusion detection framework I know about. The project produced easy to read free e-book and is worth spending some time playing with. Framework design, e.g. division to detection points, intrusion detector and response action are solid and practical.

The project is still in beta and it occasionally shows. Default configuration of XSS attack and SQL injection detection points is too strict and we found some small bugs while writing this article. However, project committers response to submitted issues was always fast.

Multiple expected small features are missing. It is not possible to say 'all URLs are blockable'. User has to add all suffixes into extensionsToCheck manually. As default response actions are configured in detection point blocks in appsensor.properties file, we would also expect support for same configuration for custom actions. Or, some classes (DefaultResponseAction) are singletons for no good reason.

If you decide to use intrusion detection in your project, your design have to be robust enough to handle all those random log outs and account disables.