Too many tgs_req from jdbc (for kerberos)

[lurnagao-dahua]

Hi !

my trino version is 355.
I found about five tgs_req from jdbc per second in one query.

Thank you very much for the answer！

[hashhar]

Each worker needs to authenticate to Kerberos so it's entirely possible for there to be at-least as many requests to TGS as number of workers participating in query.

[lurnagao-dahua]

Question: are you talking about Trino's JDBC driver?
answer: yes

My trino jdbc(java) in maven:

io.trino
trino-jdbc
355

In function StatementClientV1#advance, each call to the JsonResponse.execute(QUERY_RESULTS_CODEC, httpClient, request) will sends a tgs_req to kdc :

public boolean advance()
{
    ...
    while (true) {
        if (isClientAborted()) {
            return false;
        }
        ...
        JsonResponse<QueryResults> response;
        try {
            response = JsonResponse.execute(QUERY_RESULTS_CODEC, httpClient, request);
        }
        catch (RuntimeException e) {
            cause = e;
            continue;
        }

        if ((response.getStatusCode() == HTTP_OK) && response.hasValue()) {
            processResponse(response.getHeaders(), response.getValue());
            return true;
        }
    }
}

[lurnagao-dahua]

The function statementClientV1#advance is getting results from coordinator.

[hashhar]

cc: @wendigo (for kerberos)/ @electrum (For the JDBC driver)

Sounds like there is no ticket cache so the client does the auth everytime.

[lurnagao-dahua]

I think so too.
The client ticket(TGT) is cache, but service ticket is not cache.

[lurnagao-dahua]

thank you very much for your answer！

[lurnagao-dahua]

Hi, Is it my configuration that is wrong or the current situation?

[overbum]

This is classic problem with all Java apps. The reason is wrong application architecture. JDK for Java has 2 auth modules: 1) krb5LoginModule (jaas) and 2) Sun GSSAPI Native. I have no idea why developers advertise jaas that supports a short part of MIT Kerberos but full support is Native module only because it uses OS (win, unix/linux, macos, solaris, etc) krb5 libraries. MIT Kerberos KDC (Active Directory or FreeIPA project) provide SSO by credentical cache. In spite of comments about TGS many of developers don't know difference between TGT and TGS therefore they copypaste they have found on Oracle website. However no one read Oracle arcticle about Native module. You know this oldest module changes Java auth mechanism to C/C++ application behavior that uses system krb5 from the box. The only way you should get TGT by kinit and automatically store to ccache, when you connect any service (SPN) you get TGS is stored to ccache too. You can read MSLSA cache in Windows every time you try to open spnego website or to send email. As fact in user session (process owner) any your application instance or application thread re-use ccache for a long time (for example 24h by default). Of course if you have specific infrastructure where you cannot save a keytab on file system you can use renewal Kerberos extension for TGT. This feature help you to move lifetime for the next 24h once without keytab. Unfotunately developers create many of isolated contexts (because jaas doesn't support MIT TGS format and there is no way to store) for an incoming user so you can see infinite tickets on KDC side. You can compare it such as many of user relogins to Windows Desktop that is redundant of course and stupidly. Accoding trino server it requires a refactoring to use Native else TGS-flood will fail your KDC sooner or later. Btw trino jdbc for Windows (client app) should be refactored too because it doesn't support MSLSA Windows cache but it is impossible to define file ticket cache without kinit or MIT Kerberos for Windows.
Official native doc https://docs.oracle.com/en/java/javase/17/security/accessing-native-gss-api.html
Redhat, Microsoft, IBM, VMware and Apache community.

S4U extension
https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-sfu/1fb9caca-449f-4183-8f7a-1a5fc7e7290a
https://www.freeipa.org/page/V4/Service_Constraint_Delegation

How it must work but really doesn't:

1. User logins to Windows or Linux (ssh) and execute any Client App with trino jdbc setup.
2. From user session MSLSA or Linux KEYRING cache the user can extract TGS for trino/hostame@REALM (the 1st connection generates a TGS to store to the cache)
3. Using TGS user reaches trino server (gateway)
4. trino server works in native GSSAPI mode only so it has KRB5CCNAME and KRB5_KTNAME variables in environment.
5. Trino server or a worker has own SPN, keytab and of course ccache for own tickets and transit tickets. All ticket in cache - it's important and you can check with klist -c $KRB5CCNAME
6. Cron or systemd service does kinit by keytab for trino SPN or renew kinit -R every N hours (ticket_lifetime in /etc/krb5.conf for POSIX). krb5 library stores TGT to KRB5CCNAME and then any unique TGS.
7. The user, who wants a negotiation with trino server, sends user encrypted TGS then trino server decrypts user TGS by SPN keytab. If it is ok users passthrough.
8. If trino works in cluster mode so each member of cluster ask each other using Kerberos (trino/host1@REALM for trino/host2@REALM)
9. Therefore any trino SPN need to go KDC for specific S4U2Proxy ticket for user to connect behalf of user. Btw S4U can work only in native GSSAPI. jaas doesn't support this mechanism because it is Kerberos extension for Microsot and MIT
10. The final step to get S4U2Proxy TGS for target service for example PostgresSQL
11. All transit tickets must be stored to trino server ccache. You can emulate by kvno on Linux system (of course S4U must be on by default for SPNs)
12. PostgreSQL decrypts transit TGS and user can pass and then must be authorized.