Apache Shiro Part 2 - Realms, Database and PGP Certificates

This is second part of series dedicated to Apache Shiro. We started previous part with simple unsecured web application. When we finished, the application had basic authentication and authorization. Users could log in and log out. All web pages and buttons had access rights assigned and enforced. Both authorization and authentication data have been stored in static configuration file.

As we promised in the end of last part, we will move user account data to database. In addition, we will give users an option to authenticate themselves via PGP certificates. As a result, our application will have multiple alternative log in options: log in with user name/password and log in with certificate. We will finish by turning alternative log in options mandatory.

In other words, we will show how to create custom realm and how to handle multi-realm scenario. We will create three different versions of SimpleShiroSecuredApplication:

• version with all account information moved to database,
• version that allows PGP Certificates as alternative authentication mechanism,
• version that requires both user name/passwords and PGP Certificates for log-in.

Each version has test class RunWaitTest. The class starts web server with application deployed at http://localhost:9180/simpleshirosecuredapplication/ url.

Note: We updated previous part since first release. The most notable change is new section which shows how to add error message to login page. Thanks everybody for feedback.

Realms

First, we explain what realms are and how to create them. If you are not interested in theory, proceed to next chapter.

Realms are responsible for authentication and authorization. Any time user wants to log in to the application, authentication information is collected and passed to realm. Realm verifies supplied data and decides whether user should be allowed to log in, have access to resource or own specific role.

Authentication information has two parts:

• principal - represents account unique identifier e.g. user name, account id, PGP certificate, ...
• credential - proves users identity e.g. password, PGP certificate, fingerprint, ... .

Shiro provides realms able to read authorization data from active directory, ldap, ini file, properties file and database. Realms are configured in main section of Shiro.ini file:

realmName=org.apache.shiro.realm.jdbc.JdbcRealm

Authentication

All realms implement Realm interface. Two interface methods are important: supports and getAuthenticationInfo. Both receive principal and credentials inside authentication token object.

Supports method decides whether realm is able to authenticate user based on supplied authentication token. For example, if my realm checks user name and password, it rejects authentication token with X509 certificate only.

Method getAuthenticationInfo performs authentication itself. If principal and credential from authentication token represents valid log in information, the method returns authentication info object. otherwise, the realm returns null.

Authorization

If the realm wishes to do also authorization, it has to implement Authorizer interface. Each Authorizer method takes principal as parameter and checks either role(s) or permission(s). It is important to understand, that the realm obtains all authorization requests, even if they came from user authenticated by another realm. Of course, realm may decide to ignore any authorization request.

Permissions are supplied either as strings or as permission objects. Unless you have strong reason to do otherwise, use WildcardPermissionResolver to convert strings into permission objects.

Other Options

Shiro framework investigates realms at run time for additional interfaces. If the realm implements them, it can use:

• information about users log outs,
• information about system start up,
• global cache,
• name configured in configuration file,
• configured converter between permission strings and permission objects.

These features are available to any realm that implements additional interface. No other configuration is necessary.

Custom Realms

The easiest way to create new realm is to extend either AuthenticatingRealm or AuthorizingRealm class. They have reasonable implementation of all useful interfaces mentioned in previous section. If they are not usable for your needs, you can extend CachingRealm or create new realm from scratch.

Move to Database

Current version of SimpleShiroSecuredApplication uses default realm for both authentication and authorization. Default realm - IniRealm reads user account information from configuration file. Such storage is acceptable only for simplest applications. Anything slightly more complex needs to have credentials stored in better persistent storage.

New requirements: account credentials and access rights are stored in database. Stored passwords are hashed and salted.

In this chapter, we will connect application to database and create tables to store all user account data. Then, we will replace IniRealm with realm able to read from database and salt passwords.

Database Infrastructure

The section describes sample application infrastructure. It contains no information about Shiro, so you can freely skip it.

Example application uses Apache Derby database in embedded mode.

We use Liquibase for database deployment and upgrades. It is open source library for tracking, managing and applying database changes. Database changes (new tables, new columns, foreign keys) are stored in database changelog file. Upon start up, Liquibase investigates database and apply all new changes. As a result, database is always consistent and up to date with no real effort on our part.

Add dependency to Derby and Liquibase into SimpleShiroSecuredApplication pom.xml:

<dependency>
    <groupid>org.apache.derby</groupid>
    <artifactid>derby</artifactid>
    <version>10.7.1.1</version>
</dependency>
<dependency>
    <groupid>org.liquibase</groupid>
    <artifactid>liquibase-core</artifactid>
    <version>2.0.1</version>
</dependency>

Add jndi to jetty:

<dependency>
   <groupid>org.mortbay.jetty</groupid>
   <artifactid>jetty-naming</artifactid>
   <version>${jetty.version}</version>
   <scope>test</scope>
</dependency>  
<dependency>
   <groupid>org.mortbay.jetty</groupid>
   <artifactid>jetty-plus</artifactid>
   <version>${jetty.version}</version>
   <scope>test</scope>
</dependency>  

Create db.changelog.xml file with database structure description. It creates tables where users, roles and permissions are stored. It also fills those tables with initial data. We used random_salt_value_username as salt and following method to create hashed salted passwords:

public static String simpleSaltedHash(String username, String password) {
   Sha256Hash sha256Hash = new Sha256Hash(password, (new SimpleByteSource("random_salt_value_" + username)).getBytes());
  String result = sha256Hash.toHex();

   System.out.println(username + " simple salted hash: " + result);
   return result;
}

Create data source pointing to derby in WEB-INF/jetty-web.xml file:

<configure class="org.mortbay.jetty.webapp.WebAppContext" id="SimpleShiroSecuredApplication">
 <new class="org.mortbay.jetty.plus.naming.Resource" id="SimpleShiroSecuredApplication">
  <arg>jdbc/SimpleShiroSecuredApplicationDB</arg>
  <arg>
   <new class="org.apache.derby.jdbc.EmbeddedDataSource">
    <set name="DatabaseName">../SimpleShiroSecuredApplicationDatabase</set>
    <set name="createDatabase">create</set>
   </new>
  </arg>
 </new>
</configure>

Configure datasource and liquibase in web.xml file:

<resource-ref>
  <description>Derby Connection</description>
  <res-ref-name>jdbc/SimpleShiroSecuredApplicationDB</res-ref-name>
  <res-type>javax.sql.DataSource</res-type>
  <res-auth>Container</res-auth>
</resource-ref>
 
<context-param>
  <param-name>liquibase.changelog</param-name>
  <param-value>src/main/resources/db.changelog.xml</param-value>
</context-param>

<context-param>
  <param-name>liquibase.datasource</param-name>
  <param-value>jdbc/SimpleShiroSecuredApplicationDB</param-value>
</context-param>

<listener>
  <listener-class>
    liquibase.integration.servlet.LiquibaseServletListener
  </listener-class>
</listener>

Finally, jetty configured to read jetty-web.xml with enabled jndi is in AbstractContainerTest class.

Create New Realm

Shiro provided JDBCRealm is able to do both authentication and authorization. It uses configurable SQL queries to read user names, passwords, permissions and roles from database. Unfortunately, the realm has two shortcomings:

• It is not able to load data source from JNDI (open issue).
• It is not able to salt passwords (open issue).

We extend it and create new class JNDIAndSaltAwareJdbcRealm. As all properties are configurable in ini file, new property jndiDataSourceName will be automatically configurable too. The realm looks up data source in JNDI whenever new property is set:

protected String jndiDataSourceName;

public String getJndiDataSourceName() {
 return jndiDataSourceName;
}

public void setJndiDataSourceName(String jndiDataSourceName) {
 this.jndiDataSourceName = jndiDataSourceName;
 this.dataSource = getDataSourceFromJNDI(jndiDataSourceName);
}

private DataSource getDataSourceFromJNDI(String jndiDataSourceName) {
 try {
  InitialContext ic = new InitialContext();
  return (DataSource) ic.lookup(jndiDataSourceName);
 } catch (NamingException e) {
  log.error("JNDI error while retrieving " + jndiDataSourceName, e);
  throw new AuthorizationException(e);
 }
}

Method doGetAuthenticationInfo reads account authentication information from the database and converts it into authentication info object. It returns null if no account information is found. Parent class AuthenticatingRealm compares authentication info object with original user supplied data.

We override doGetAuthenticationInfo to read both password hash and salt from database and store them in authentication info object:

doGetAuthenticationInfo(AuthenticationToken token) throws AuthenticationException {
 ...
 // read password hash and salt from db 
 PasswdSalt passwdSalt = getPasswordForUser(username);
 ...
 // return salted credentials
 SimpleAuthenticationInfo info = new SimpleAuthenticationInfo(username, passwdSalt.password, getName());
 info.setCredentialsSalt(new SimpleByteSource(passwdSalt.salt));

 return info;
}

The example here contains only most important piece of code. Full class is available on Github.

Configure New Realm

Configure realm and jndi name in Shiro.ini file:

[main] 
# realm to be used
saltedJdbcRealm=org.meri.simpleshirosecuredapplication.realm.JNDIAndSaltAwareJdbcRealm
# any object property is automatically configurable in Shiro.ini file
saltedJdbcRealm.jndiDataSourceName=jdbc/SimpleShiroSecuredApplicationDB 
# the realm should handle also authorization
saltedJdbcRealm.permissionsLookupEnabled=true

Configure SQL queries:

# If not filled, subclasses of JdbcRealm assume "select password from users where username = ?"
# first result column is password, second result column is salt 
saltedJdbcRealm.authenticationQuery = select password, salt from sec_users where name = ?
# If not filled, subclasses of JdbcRealm assume "select role_name from user_roles where username = ?"
saltedJdbcRealm.userRolesQuery = select role_name from sec_users_roles where user_name = ?
# If not filled, subclasses of JdbcRealm assume "select permission from roles_permissions where role_name = ?"
saltedJdbcRealm.permissionsQuery = select permission from sec_roles_permissions where role_name = ?

JdbcRealm uses credetials matcher exactly the same way as IniRealm was:

# password hashing specification
sha256Matcher = org.apache.shiro.authc.credential.HashedCredentialsMatcher
sha256Matcher.hashAlgorithmName=SHA-256
saltedJdbcRealm.credentialsMatcher = $sha256Matcher

NOTE: we removed sections [users] and [roles] from configuration file. Otherwise, Shiro would use both IniRealm and JdbcRealm. This would create multi-realm scenario which is out of scope of this chapter.

From user point of view, application works exactly the same way as before. He can log in to the same user accounts as before. However, user names, passwords, salts, permissions and roles are now stored in database.

Full source code is available in 'authentication_stored_in_database' branch on Github.

Alternative Login - Certificates

Some systems allow multiple authentication means for user log in. For example, user may supply user name/password, log in with Google account, Facebook account or anything else. We will add something similar to our simple application. We will give our users option to authenticate themselves with PGP certificates.

New requirements: application supports PGP certificates as alternative authentication mechanism. Login screen shows up only if user does not possess valid certificate associated with application account. If the user has valid known PGP certificate, he is logged automatically.

When user tries to log in to the application, he has to supply authentication data. Those data are captured by servlet filter. Filter converts data to authentication token and pass the token to realms. If any realm wish to authenticate user, it converts authentication token to authentication info object. If the realm does not wish to do so, then it returns null.

Out of the box Shiro framework filters ignores PGP certificates in request. Available authentication tokens are not able to hold them and realms are not aware of PGP certificates at all. Therefore, we have to create:

• authentication token to move certificate around,
• servlet filter able to read certificates,
• realm to validate certificates and match them to user accounts.

Our application will have two different realms. One uses names to identify accounts and passwords to authenticate users, other uses PGP certificates to do both.

Before we start coding, we have to deal with PGP certificates and infrastructure that surrounds our application. If you are not interested in PGP certificates set up, jump over next section.

Infrastructure

When user visits web application, his web browser may send copy of PGP certificate to web server. The certificate is signed either by some certificate authority or by the certificate itself (self-signed certificate). Web server keeps list of certificates it trusts to in storage called truststore. If the truststore contains either users certificate or certificate of authority that signed it, then web server trusts users certificate. Trusted certificates are passed to the application.

We will:

• create certificate for each user,
• create truststore,
• configure web server,
• associate certificates with user accounts.

Create and manage certificates in portecle. Sample certificates for SimpleShiroSecuredApplication are located in src\test\resources\clients directory. All stores and certificates have common password 'secret'.

Create Certificate

Create self-signed certificate for each user in portecle:

• Create new jks keystore: File -> New Keystore, select jks.
• Generate new certificate: Tools -> Generate Key Pair. Leave password fields empty, certificate will inherit password from keystore.
• Export public certificate: select new certificate -> right click -> Export, select Head Certificate. This creates .cer file.
• Export private key and certificate: select new certificate -> right click -> Export, select Private Key and Certificates. This creates .p12 file.

.cer file contains only public certificate, so you can give it to anybody. On the other hand, .p12 file contains users private key and must be kept secret. Distribute it only to user (e.g. import it to your browser for testing).

Create Truststore

Create new truststore and import public certificate .cer files to it:

• File -> New Keystore, select jks.
• Tools -> Import Trusted Certificate.

Configure Web Server

Web server has to ask for certificates and validate them against truststore. It is not possible to ask for certificate from Java. Each web server is configured differently. Jetty configuration is available in Look at AbstractContainerTest class on Github.

Associate Certificates with Accounts

Each certificate is uniquely identified by serial number and name of certificate authority that signed it. We store them together with user names and passwords in database table. Database changes are in db.changelog.xml file, see changeset 3 for new columns and changeset 4 for data initialization.

Authentication Token

Authentication token represents user data and credentials during an authentication attempt. It must implement authentication token interface and holds whatever data we wish to pass between servlet filter and realm.

As we wish to use both user names/passwords and certificates for authentication, we extend UsernamePasswordToken class and add certificate property to it. New authentication token X509CertificateUsernamePasswordToken implements new interface X509CertificateAuthenticationToken and both are available on Github:

public class X509CertificateUsernamePasswordToken extends UsernamePasswordToken implements X509CertificateAuthenticationToken {

    private X509Certificate certificate;

    @Override
    public X509Certificate getCertificate() {
      return certificate;
    }

    public void setCertificate(X509Certificate certificate) {
      this.certificate = certificate;
    }

}

Servlet Filter

Shiro filter converts user data to authentication tokens. Up to now, we used FormAuthenticationFilter. If incoming request comes from logged user, the filter lets user in. If the user is trying to authenticate himself, the filter creates authentication token and pass it to the framework. Otherwise it redirects user to the login screen.

Our filter CertificateOrFormAuthenticationFilter extends FormAuthenticationFilter.

First, we have to convince it that not only requests with user name and passwords, but also any request with PGP certificate can be considered a log in attempt. Second, we have to modify the filter to send PGP certificate along with user name and password in an authentication token.

The method isLoginSubmission determines whether request represents authentication attempt:

@Override
    protected boolean isLoginSubmission(ServletRequest request, ServletResponse response) {
        return super.isLoginSubmission(request, response) || isCertificateLogInAttempt(request, response);
    }
    
    private boolean isCertificateLogInAttempt(ServletRequest request, ServletResponse response) {
        return hasCertificate(request) && !getSubject(request, response).isAuthenticated();
    }
    
    private boolean hasCertificate(ServletRequest request) {
        return null != getCertificate(request);
    }
    
    private X509Certificate getCertificate(ServletRequest request) {
        X509Certificate[] attribute = (X509Certificate[]) request.getAttribute("javax.servlet.request.X509Certificate");
        return attribute==null? null : attribute[0];
    }

The method createToken creates authentication token:

@Override
    protected AuthenticationToken createToken(String username, String password, ServletRequest request, ServletResponse response) {
        boolean rememberMe = isRememberMe(request);
        String host = getHost(request);
        X509Certificate certificate = getCertificate(request);
        return createToken(username, password, rememberMe, host, certificate);
    }
    
    protected AuthenticationToken createToken(String username, String password, boolean rememberMe, String host, X509Certificate certificate) {
        return new X509CertificateUsernamePasswordToken(username, password, rememberMe, host, certificate);
    }

Replace FormAuthenticationFilter with CertificateOrFormAuthenticationFilter filter in configuration file:

[main]
# filter configuration
certificateFilter = org.meri.simpleshirosecuredapplication.servlet.CertificateOrFormAuthenticationFilter
# specify login page
certificateFilter.loginUrl = /simpleshirosecuredapplication/account/login.jsp
# name of request parameter with username; if not present filter assumes 'username'
certificateFilter.usernameParam = user
# name of request parameter with password; if not present filter assumes 'password'
certificateFilter.passwordParam = pass
# does the user wish to be remembered?; if not present filter assumes 'rememberMe'
certificateFilter.rememberMeParam = remember
# redirect after successful login
certificateFilter.successUrl  = /simpleshirosecuredapplication/account/personalaccountpage.jsp

Redirect all URLs to new filter:

[urls]
# force ssl for login page 
/simpleshirosecuredapplication/account/login.jsp=ssl[8443], certificateFilter

# only users with some roles are allowed to use role-specific pages 
/simpleshirosecuredapplication/repairmen/**=certificateFilter, roles[repairman]
/simpleshirosecuredapplication/sales/**=certificateFilter, roles[sales]
/simpleshirosecuredapplication/scientists/**=certificateFilter, roles[scientist]
/simpleshirosecuredapplication/adminarea/**=certificateFilter, roles[Administrator]

# enable certificateFilter filter for all application pages
/simpleshirosecuredapplication/**=certificateFilter

Custom Realm

Our new realm will be responsible only for authentication. Authorization (access rights) will be handled by JNDIAndSaltAwareJdbcRealm. Such configuration works as long as PGP certificate authenticates user into the same account as user name/password would. Otherwise said, primary principal returned by new realm must be the same as primary principal returned by JNDIAndSaltAwareJdbcRealm.

Our realm does not need caching nor any other service provided by optional interfaces. Therefore, we have to implement only two interfaces: Realm and Nameable.

X509CertificateRealm supports only authentication tokens with PGP certificate:

@Override
 public boolean supports(AuthenticationToken token) {
  if (token!=null)
   return  token instanceof X509CertificateAuthenticationToken;

  return false;
 }

The method getAuthentcationInfo is responsible for authentication. If supplied certificate is valid and associated with an user account, the realm creates authentication info object. Remember that primary principal must be the same as the one returned by JNDIAndSaltAwareJdbcRealm:

@Override
public AuthenticationInfo getAuthenticationInfo(AuthenticationToken token) throws AuthenticationException {
    // the cast is legal, since Shiro will let in only X509CertificateAuthenticationToken tokens
    X509CertificateAuthenticationToken certificateToken = (X509CertificateAuthenticationToken) token;
    X509Certificate certificate = certificateToken.getCertificate();
    
    // verify certificate
    if (!certificateOK(certificate)) {
        return null;
    }
    
    // the issuer name and serial number uniquely identifies certificate
    BigInteger serialNumber = certificate.getSerialNumber();
    String issuerName = certificate.getIssuerDN().getName();
    
    // find account associated with certificate
    String username = findUsernameToCertificate(issuerName, serialNumber);
    if (username == null) {
        // return null as no account was found
        return null;
    }
    
    // sucesfull verification, return authentication info
    return new SimpleAuthenticationInfo(username, certificate, getName());
}

Note that the realm has two new properties: trustStore and trustStorePassword. Both are needed for PGP certificate validation. As any other property, both are configurable in configuration file.

Add new realm to Shiro.ini file:

[main]
certificateRealm = org.meri.simpleshirosecuredapplication.realm.X509CertificateRealm
certificateRealm.trustStore=src/main/resources/truststore
certificateRealm.trustStorePassword=secret

It is now possible to log in to the application with PGP certificate. If the certificate is not available, user name and password works too.

Application source code is available in 'certificates_as_alternative_log_in_method' branch on Github.

Multiple Realms

If configuration file contains more than one realm, all of them are used. In such case, Shiro tries to authenticate user with all configured realms and authentication results are merged together. The object responsible for merging is called authentication strategy. Framework provides three authentication strategies:

• all successful strategy,
• at least one successful strategy,
• first successful strategy.

By default, 'at least one successful strategy' is used, which suits our purposes well. Again, it is possible to create custom authentication strategy. For example, we may require user to present both PGP certificate and user name/password credentials to log in.

New requirements: user has to present both PGP certificate and user name/password credentials to log in.

In other words, we need strategy that:

• fails if some realm does not support token,
• fails if some realm does not authenticate user,
• fails if two realms authenticate different principals.

Authentication strategy is an object that implements authentication strategy interface. The interface methods are called after and before authentication attempts. We create 'primary principal same authentication strategy' from 'all successful strategy', closest strategy available. We compare principals after each realm authentication attempt:

@Override
public AuthenticationInfo afterAttempt(...) {
    validatePrimaryPrincipals(info, aggregate, realm);
    return super.afterAttempt(realm, token, info, aggregate, t);
}

private void validatePrimaryPrincipals(...) {
     ...

    Object aggregPrincipal = aggregPrincipals.getPrimaryPrincipal();
    Object infoPrincipal = infoPrincipals.getPrimaryPrincipal();
    if (!aggregPrincipal.equals(infoPrincipal)) {
        String message = "All realms are required to return the same primary principal. Offending realm: " + realm.getName();
        log.debug(message);
        throw new AuthenticationException(message);
    }
}

Authentication strategy is configured in Shiro.ini file:

# multi-realms strategy
authenticationStrategy=org.meri.simpleshirosecuredapplication.authc.PrimaryPrincipalSameAuthenticationStrategy
securityManager.authenticator.authenticationStrategy = $authenticationStrategy

Finaly, we have to change isLoginSubmission method of CertificateOrFormAuthenticationFilter back. Only requests with user name and password are now considered log in attempts. Certificate is not sufficient anymore:

@Override
protected boolean isLoginSubmission(ServletRequest request, ServletResponse response) {
  return super.isLoginSubmission(request, response);
}

If you run the application now, both certificate and user name/password log in methods are mandatory.

This version is available in 'certificates_as_mandatory_log_in_method' branch on Github.

End

This part was dedicated to Shiro realms. We created three different application versions all of them are available on Github. They covered basic and probably most important realm features.

If you need to know more, start with classes linked here and read their javadocs. They are well written and extensive.

The next part is dedicated to Apache Shiro cryptography package.