Kerberos Authentication For kdb+ – Some Assembly Required

Wherever I’ve worked on kdb+ installations the approach to authentication has been that there is no authentication. Generally, everyone’s just relied on the fact that you’d have to port-scan a host to find the kdb+ instances. To be fair, these have been extremely high-trust environments.

However, if you have a requirement for strong authentication, you can use Kerberos not only for IPC but also for HTTP authentication. When implementing an authetication step in the past, I really didn’t want to deal with (a) the aggro of integrating with the enterprise SSO product du jour or (b) the grief of managing passwords. Kerberos (or rather, “SPNEGO”) authentication fixes this for you.

Since this is a bit of a sprawling topic, I’ll start with getting the Kerberos environment set up on your developer machine (if you’re at home, for which you’ll need root privileges; if you’re at work, speak to the guys looking after the AD infrastructure). I’ll then describe IPC authentication, before moving on to the more gnarly HTTP implementation.

Installing a local KDC

The KDC is the Key Distribution Centre. Wikipedia has a section on the protocol. Essentially, your client contacts the KDC’s Ticket Granting Service (TGS) to ask for a ticket you can offer some other service to prove your identity. The TGS generates the ticket (since it knows you are who you say you are), and encrypts it with the target service’s key. You send this ticket to the remote service, which decrypts it if it has the right credentials. If you were to send it to the wrong service, it wouldn’t be able to read it. The two of you then know who each other are and can, if you like, protect subsequent message exchanges.

I only really intend to deal with the first part, namely the authentication. kdb+ supports TLS, if you’re worried about snooping, but if someone’s sniffing your q-SQL within the corporate data-centre, you’ve got bigger problems.

As an aside, since Kerberos can protect against message replay attacks, using it to secure the authentication step frees you from using TLS and encrypting every subsequent application request/response with TLS.

Install packages

I’m currently using Fedora 42, so please adapt the instructions as needed.

$ sudo dnf install krb5-devel krb5-libs krb5-server krb5-workstation

Edit `krb5.conf`

As root, edit /etc/krb5.conf, and update the default Realm. Apparently the convention is to use lowercase for host-names and uppercase for realms. I’ve called mine MINDFRUIT.KRB.

I’m just showing the parts I’ve changed:

[libdefautlts]
  default_realm = MINDFRUIT.KRB

[realms]
MINDFRUIT.KRB = {
    kdc = kerberos.mindfruit.krb
    admin_server = kerberos.mindfruit.krb
}

[domain_realm]
.mindfruit.krb = MINDFRUIT.KRB
mindfruit.krb = MINDFRUIT.KRB

Edit `kdc.conf`

As root, edit /var/kerberos/krb5kdc/kdc.conf, I added the Mindfruit realm information:

[realms]
MINDFRUIT.KRB = {
     master_key_type = aes256-cts-hmac-sha384-192
     acl_file = /var/kerberos/krb5kdc/kadm5.acl
     dict_file = /usr/share/dict/words
     default_principal_flags = +preauth
     admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
     supported_enctypes = aes256-cts-hmac-sha384-192:normal aes128-cts-hmac-sha256-128:normal aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal camellia256-cts-cmac:normal camellia128-cts-cmac:normal arcfour-hmac-md5:normal
}

Nothing sensitive in there (I hope).

Edit `/etc/hosts`

I added to the localhost 127.0.0.1 line:

127.0.0.1  [.. existing entries] mindfruit.krb kerberos.mindfruit.krb

Set up your Kerberos Realm

I found it easiest to drop into a root shell. Unless I mention otherwise, all the following command require root.

> Create the Realm

kdb5_util -r MINDFRUIT.KRB create -s

You will be prompted for the database master password, which you should store safely.

> Create an Admin principal

kadmin.local -r MINDFRUIT.KRB -q "addprinc admin/fedora"

Again, it’ll ask you for a new password, so write it down somewhere.

> Create the Service principal for HTTP

When browsers receive a SPNEGO challenge from the server they’re talking to (which essentially means they look up a Kerberos token), instead of freaking out, they just decide to ask the TGS for a ticket for HTTP/<server host>. I think you can customise this behaviour in the Chromium policies area, and am not sure about Firefox. But since the answer for me to the query:

hostname

is fedora, I’m going to create the principal HTTP/fedora. We’ll use this identity in the kdb+ server instance.

In this case I’m going to use the REPL for kadmin.local:

kadmin.local -r MINDFRUIT.KRB

The following commands create a new principal with a random key, and then the norandkey option is passed when writing a keytab with the credentials for the HTTP/fedora principal. The following are typed at the kadmin console:

addprinc -randkey HTTP/fedora
ktadd -k HTTP_fedora.keytab -norandkey HTTP/fedora@MINDFRUIT.KRB
quit

Copy the new keytab file to somewhere convenient for your own user. Since mine is michaelg, I copied it to /home/michaelg/.krb5/

cp HTTP_fedora.keytab /home/michaelg/.krb5/
chown michaelg:michaelg /home/michaelg/.krb5/HTTP_fedora.keytab

> Create the user’s principal

kadmin.local -r MINDFRUIT.Kerberos

addprinc michaelg

Enter a new password, and write it down somewhere.

Start the Kerberos services

systemctl start krb5kdc.service
systemctl start kadmin.service

Initialise the user-account’s cache and keytab

Create a credentials cache

In my case, as michaelg once more, I create a credentials cache as follows:

kinit -C -c ~/.krb5/krb5cc michaelg@MINDFRUIT.KRB

Enter the password you used when creating your user principal.

We then find the “key version number” using kvno:

kvno -c ~/.krb5/krb5cc michaelg@MINDFRUIT.KRB

Example output could be

michaelg@MINDFRUIT.KRB: kvno = 1

Create a keytab

Run ktutil and at its REPL, enter the information as follows, using your password when prompted:

ktutil

Note the argument to the -k argument matches the interger response given above by kvno.

addent -password -p michaelg@MINDFRUIT.KRB -k 1 -f

Then write the keytab, adjusting the path for your user:

write_kt /home/michaelg/.krb5/michaelg.keytab
quit

Set up your environment

We need to set the respective client and server keytabs. We use KRB5_KTNAME for the (default) keytab, typically set on the server-side, and KRB5_CLIENT_KTNAME for the client. To avoid any potential cross-pollination, I’ve been setting these on the command line before running q.

If you wanted to export these, they would be:

export KRB5_CLIENT_KTNAME=~/.krb5/michaelg.keytab
export KRB5_KTNAME=~/.krb5/HTTP_fedora.keytab

IPC Authentication Overview

Kerberos authentication is typically a multi-step exchange of messages. What is handy is that typically “you’re done” after the client has sent the server its token: at this point, the server should know who the client is and can make an access decision based on some list or other.

The client may, however, ask for “mutual authentication”, in which case the server needs to send something back. Unfortunately, that exchange of data doesn’t fit into the .z.pw process, so any reply from the server, after granting access, needs to be an application rather than system message.

There are a couple of contortions I can think of to keep the handshake data away from application code:

We open a socket in native code and
1. listen for and respond to handshake requests away from kdb+
2. This has the serious downside that the connection is forever more a “native code” connection, so we’d have to evaluate every message and things like .z.u wouldn’t be set. Ugh.
We open a socket in native code and
1. listen for handshake data from a remote
2. we close the connection
3. we respond using a new, kdb+ IPC hopen connection, passing our unique connection identifier as our username and further handshake data as our password
4. when all is complete (at the cost of at least two TCP connections), we have a kdb-owned IPC connection

I don’t really like either of these, and have therefore coded-up the initial solution, where the server response is sent using an async application IPC message. Here’s roughly what happens:

The client uses the GSSAPI to initialise a security context and request a ticket from the Ticket Granting Service. The client specifies “mutual authentication”.
The TGS encrypts the ticket with the server-principal’s key, and returns it to the client.
The client kdb+ instance connects to the server and passes the base64-encoded ticket as the password, e.g.: `::30097:michaelg:YIIDmAYGKwY--8<--1j7SPE
The server is passed the two authentication tokens as arguments to .z.pw; we pass the base-64 encoded password to the GSSAPI and establish the identity of the client, and can also determine which service the client was targeting.
Since mutual authentication was requested, the server encrypts a reply for the client; this is briefly stored, and we set up a timer to send it as an async-message almost immediately. Of course, we first need to return a value from .z.pw.
q responds to the handshake as expected and the IPC connection is established
The timer fires and the server now sends the message to the client identifying itself.
Upon receipt of the message, the client passes it to the GSSAPI which validates the peer identity.

The process is quite nice apart from the out-of-band response from the server. The client has to hang around once the IPC connection is established to wait for the identity confirmation. Unfortunately, I can’t think of another way of doing it, except as briefly described above.

I’ve got a working implementation in my Github mgkdb project, which could serve as a springboard for you implementing something of higher quality ;)

HTTP Authentication Overview

Modern browsers, and here I’m including Chromium and Firefox, will (with a bit of a nudge and some additional configuration) engage in something called SPNEGO authentication. Roughly speaking, the browser can be prompted to cough-up a Kerberos token and include it as an HTTP header; the server can use the token to authenticate the client. So far as I can tell, there’s no facility for the browser to request mutual authentication from the server.

In order to get kdb+ to participate in this workflow, we’ll use the .z.ac callback and reply with different response-types to elicit the required behaviour. There are some complexities, though.

HTTP is a stateless protocol (via SO). However, the SPNEGO handshake is not stateless, leading to this remark on the IANA website (via SO):

This authentication scheme violates both HTTP semantics (being connection-oriented) and syntax (use of syntax incompatible with the WWW-Authenticate and Authorization header field syntax).

Browsers will typically open two TCP connections to a server in order to load static assets (don’t quote me on that, it’s my recollection). Even if we could lean on the HTTP keep-alive header not closing the TCP connection between requests (from kdb+ 4.1 onwards), kdb+ has no HTTP equivalent of .z.pc, so you can’t be sure that the browser hasn’t closed the connection. In short: you can’t use the TCP connection lifetime as a way of ensuring a private, in-sequence exchange of messages.

What we can do is the following:

The browser client requests a resource
The server notes the lack of an authentication cookie, and redirects the client to the same resource but with an unique “auth-ID” URL parameter (via a 302 Found status code).
The client re-requests the resource using the auth-ID parameter
The server responds with the special SPNEGO 401 Unauthorized’s WWW-Authenticate: Negotiate response header
The client generates the SPNEGO Kerberos token and replies to the server (perhaps on a new TCP connection, but, crucially, with the unique auth-ID URL parameter), appending the token to the Authorization: Negotiate request header.
The server accepts the security context and decodes the token, establishing the client identity, and the target principal the client was hoping to reach; the server stores pointers to the long-lived context it in a kdb+ table against the unique auth-ID parameter.
If the server needs to send further data back to the client, it does so via another 401 Unauthorized response, and again awaits the client reply
If the handshake is complete, we have to do something a bit special, as from .z.ac we can’t both set an identifying cookie and bless the connection by replying (1;"username"). We simply short-circuit the process by sending a custom 200 OK response, which not only returns the requested resource but also sets the auth-complete cookie.
On subsequent requests, the client presents and we check the cookie against our username cache (just a tabl). No further Kerberos token exchange takes place. Websocket upgrades proceed as normal.

I tried merging the initial redirect and subsequent WWW-Authenticate: Negotiate responses, but it doesn’t do what I’d hoped it would do. Upon reflection, this should have been obvious: the SPNEGO authentication routine is initiated by a 401 Unauthorized response.

There’s a fair old amount of C code that goes with this post which I’ve posted on my Github mgkdb project.

The C-GSSAPI (sibling to the GSSAPI RPC) is a bit old-school in its API, and example code really helps understand what’s going on. Funny that oracle.com would have the documentation, but then, this goes back to Unix and Solaris days.

Cross-Origin requests

This post doesn’t really touch on the issues associated with cross-origin requests. This arises, for example, where you might want to serve your site from something like Nginx but direct websocket requests to kdb+ listening elsewhere.

This can be made to work but you need to implement handlers to service the CORS (Cross Origin Resource Sharing) negotiation. I’ve done this before with kdb+, and all it takes are the MDN docs and a bit of trial and error with your browser’s debug console.

IPC Authentication Walkthrough

Running the IPC server

In order to ensure a clean-slate, I have taken to running kdestroy then kinit before starting a new test:

$ kdestroy
$ kinit -R -k -t ~/.krb5/michaelg.keytab michaelg@MINDFRUIT.KRB

We then start kdb+:

$ KRB5_KTNAME=~/.krb5/HTTP_fedora.keytab KRB5_TRACE=/dev/stdout qq q/boot.q -p 30097 -krb.server 1

Running the IPC client

I then run the client instance:

$ KRB5_CLIENT_KTNAME=~/.krb5/michaelg.keytab KRB5_TRACE=/dev/stdout qq q/boot.q

We then try to establish a connection:

q)svc:.krb.kopen[`;30097;"michaelg";"HTTP/fedora"]

And see the following output at the client’s console. I’ve separated the KRB5_TRACE output from my own logging:

DEBUG: resolving michaelg@MINDFRUIT.KRB to internal format

[158684] 1759391071.880138: Matching michaelg@MINDFRUIT.KRB in collection with result: 0/Success

DEBUG: acquired credentials for michaelg@MINDFRUIT.KRB
DEBUG: resolving HTTP/fedora@MINDFRUIT.KRB to internal format
DEBUG: resolved target service HTTP/fedora@MINDFRUIT.KRB

[158684] 1759391071.880139: Getting credentials michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB using ccache KCM:1000
[158684] 1759391071.880140: Retrieving michaelg@MINDFRUIT.KRB -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[158684] 1759391071.880141: Retrieving michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB from KCM:1000 with result: -1765328243/Matching credential not found
[158684] 1759391071.880142: Retrieving michaelg@MINDFRUIT.KRB -> krbtgt/MINDFRUIT.KRB@MINDFRUIT.KRB from KCM:1000 with result: 0/Success
[158684] 1759391071.880143: Starting with TGT for client realm: michaelg@MINDFRUIT.KRB -> krbtgt/MINDFRUIT.KRB@MINDFRUIT.KRB
[158684] 1759391071.880144: Requesting tickets for HTTP/fedora@MINDFRUIT.KRB, referrals on
[158684] 1759391071.880145: Generated subkey for TGS request: aes256-sha2/3474
[158684] 1759391071.880146: etypes requested in TGS request: aes256-sha2, aes128-sha2, aes256-cts, aes128-cts, camellia256-cts, camellia128-cts
[158684] 1759391071.880148: Encoding request body and padata into FAST request
[158684] 1759391071.880149: Sending request (1126 bytes) to MINDFRUIT.KRB
[158684] 1759391071.880150: Resolving hostname kerberos.mindfruit.krb
[158684] 1759391071.880151: Sending initial UDP request to dgram 127.0.0.1:88
[158684] 1759391071.880152: Received answer (1199 bytes) from dgram 127.0.0.1:88
[158684] 1759391071.880153: Sending DNS URI query for _kerberos.MINDFRUIT.KRB.
[158684] 1759391071.880154: No URI records found
[158684] 1759391071.880155: Sending DNS SRV query for _kerberos-master._udp.MINDFRUIT.KRB.
[158684] 1759391071.880156: Sending DNS SRV query for _kerberos-master._tcp.MINDFRUIT.KRB.
[158684] 1759391071.880157: No SRV records found
[158684] 1759391071.880158: Response was not from primary KDC
[158684] 1759391071.880159: Decoding FAST response
[158684] 1759391071.880160: FAST reply key: aes256-sha2/E5B5
[158684] 1759391071.880161: TGS reply is for michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB with session key aes256-sha2/B8EE
[158684] 1759391071.880162: TGS request result: 0/Success
[158684] 1759391071.880163: Received creds for desired service HTTP/fedora@MINDFRUIT.KRB
[158684] 1759391071.880164: Storing michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB in KCM:1000
[158684] 1759391071.880165: Creating authenticator for michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB, seqnum 73992062, subkey aes256-sha2/D102, session key aes256-sha2/B8EE

DEBUG: token includes the GSS_C_MUTUAL_FLAG flag
DEBUG: token includes the GSS_C_REPLAY_FLAG flag
DEBUG: token includes the GSS_C_CONF_FLAG flag
DEBUG: token includes the GSS_C_INTEG_FLAG flag
DEBUG: token includes the GSS_C_TRANS_FLAG flag
DEBUG: sec-context requires response from peer

We then get the async IPC reply from the remote peer:

DEBUG: received reply-token on FD 4i from remote YIGnBgkqhkiG9xIBAgICAG+BlzCBlKADAgEFoQMCAQ+igYcwgY..

[158684] 1759391071.880167: Read AP-REP, time 1759391071.880166, subkey aes256-sha2/D38C, seqnum 118018605

DEBUG: token includes the GSS_C_MUTUAL_FLAG flag
DEBUG: token includes the GSS_C_REPLAY_FLAG flag
DEBUG: token includes the GSS_C_CONF_FLAG flag
DEBUG: token includes the GSS_C_INTEG_FLAG flag
DEBUG: token includes the GSS_C_TRANS_FLAG flag
DEBUG: sec-context initialised
 WARN: for some reason, INTEG and CONF checks are apparently not possible
DEBUG: Have mutual-auth reply from HTTP/fedora@MINDFRUIT.KRB

We do, oddly, see that the GSS_C_PROT_READY_FLAG is still not set in the ret_flags value of gss_init_sec_context. When this flag is set, it indicates that the protections afforded by the GSS_C_CONF_FLAG and GSS_C_INTEG_FLAG can be tested and verified. I had expected this to be set once all the handshaking had completed. Perhaps it’s because the messages so far are not application messages, but context-establishment messages, and thus have not been “wrapped” (and therefore do not need either gss_unwrap or gss_verify_mic). Anyway, I’ve left the warning in the code, which might be helpful for anyone investigating GSSAPI.

We can then check the result of the .krb.kopen call in the svc variable:

q)svc
3i
q)svc(-1;"Hello, World!")

… which duly prints the expect text to the server’s console.

The output at the server console

We see that we still get the .z.u username separately from the Kerberos password (see .krb.kopen in krb5.q):

DEBUG: authenticating IPC connetion for user `michaelg with password of length 1193
DEBUG: calling .krb.acceptSecCtx, no existing context found

[160025] 1759391720.766821: Decrypted AP-REQ with server principal HTTP/fedora@MINDFRUIT.KRB: aes256-sha2/3F59
[160025] 1759391720.766822: AP-REQ ticket: michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB, session key aes256-sha2/42E5
[160025] 1759391720.766823: Negotiated enctype based on authenticator: aes256-sha2
[160025] 1759391720.766824: Authenticator contains subkey: aes256-sha2/9648
[160025] 1759391720.766825: Creating AP-REP, time 1759391720.735456, subkey aes256-sha2/1957, seqnum 219862365

DEBUG: gss_accept_sec_context success; context is valid for 86696 seconds
DEBUG: Client is michaelg@MINDFRUIT.KRB
DEBUG: token includes the GSS_C_MUTUAL_FLAG flag
DEBUG: token includes the GSS_C_REPLAY_FLAG flag
DEBUG: token includes the GSS_C_CONF_FLAG flag
DEBUG: token includes the GSS_C_INTEG_FLAG flag
DEBUG: token includes the GSS_C_PRGSS_C_PROT_READY_FLAGOT_READY_FLAG flag
DEBUG: token includes the GSS_C_TRANS_FLAG flag
DEBUG: PROT_READY not set, not validating
DEBUG: accept-context published reply of length 170 for the caller
DEBUG: Service target is HTTP/fedora@MINDFRUIT.KRB
DEBUG: krb5 authentication response is (1i;"michaelg@MINDFRUIT.KRB";"HTTP/fedora@MINDFRUIT.KRB";86696i;"AAAACSqGSIb3EgE...")
 INFO: authenticated user michaelg@MINDFRUIT.KRB for service HTTP/fedora@MINDFRUIT.KRB on FD 5i
DEBUG: Have socket-open event for FD 5i
DEBUG: Cleared timer with id 1

Exporting Security Contexts

I originally implemented a solution to this using the two functions gss_export_sec_context and gss_import_sec_context as a way of storing the context between handshake messages, which looked as though it would have the happy side effects of avoiding resource leaks and having to store pointers to opaque, in-memory objects.

This didn’t work out, though: I got a segfault in the kdb+ client when calling gss_inquire_context. I suspect the problem lay in the fact that the other arguments to gss_init_sec_context, such as the credentials-handle and target-name objects needed to be the identical, and it wasn’t enough to re-resolve these on each new invocation.

As a result, I had to store pointers all dynamically resolved objects (and request-flags) in kdb+ against the IPC file descriptor between messages. Once the security-context is fully initialised, the shared library disposes of the resources and kdb+ deletes the dictionary pointing to them. It’s not wildly pretty, and I’m sure it can be made more robust, but it enables mutual authentication.

HTTP Authentication

We start the server in the same way as above, except we now also pass the -web.server 1 switch to install the .z.ac handler. The logging produced when a simple GET request is made looks like this:

DEBUG: HTTP auth required for FD 5i
DEBUG: sending auth_id redirect
DEBUG: HTTP auth required for FD 5i
DEBUG: sending initial WWW-Authenticate response
DEBUG: HTTP auth required for FD 5i
DEBUG: sending further WWW-Authenticate challenge
DEBUG: calling .krb.acceptSecCtxCont on FD 5i; token begins YIIDmAYGKwYBBQUCoIIDjDCCA4igDTALBgkqhkiG9xIBAgKigg..

[189709] 1759408146.666971: Decrypted AP-REQ with server principal HTTP/fedora@MINDFRUIT.KRB: aes256-sha2/3F59
[189709] 1759408146.666972: AP-REQ ticket: michaelg@MINDFRUIT.KRB -> HTTP/fedora@MINDFRUIT.KRB, session key aes256-sha2/E354
[189709] 1759408146.666973: Negotiated enctype based on authenticator: aes256-sha2
[189709] 1759408146.666974: Authenticator contains subkey: aes256-sha2/9C5D

I’ve only got the one laptop to play with here, so it’s no surprise that the browser is using the Kerberos identity associated with my Linux account. On Fedora 42, this is evident from the Settings->Online Accounts applet.

DEBUG: Client is michaelg@MINDFRUIT.KRB
DEBUG: token includes the GSS_C_CONF_FLAG flag
DEBUG: token includes the GSS_C_INTEG_FLAG flag
DEBUG: token includes the GSS_C_TRANS_FLAG flag
DEBUG: PROT_READY not set, not validating
DEBUG: accept-sec-context created reply of length 22
DEBUG: Service target is HTTP/fedora@MINDFRUIT.KRB
DEBUG: accept-sec-context complete; context is valid for 86697 seconds
DEBUG: Client authenticated, sending 200 OK for index.html?auth_id=10743abf89fa8617c306abec1c01fcc40460d38efee6fc794ffb412ea08d56fc
DEBUG: Serving file `:/home/michaelg/dev/projects/github.com/mgkdb/krb5/html/index.html

At this point the Kerberos/SPNEGO authentication is complete and the browser client has a new client_id cookie whose value is a key into the .web.cookies table in web.q. We then see requests for the ipc.js file, and then the ugprade of a subsequent connection to a websocket.

DEBUG: HTTP auth required for FD 5i
DEBUG: Have GET request for ipc.js
DEBUG: Serving file `:/home/michaelg/dev/projects/github.com/mgkdb/krb5/html/ipc.js
DEBUG: HTTP auth required for FD 5i
DEBUG: Have websocket-open on FD 5i