Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle Custom Domain with SSL on Fabio #925

Open
maddy023 opened this issue Feb 27, 2023 · 5 comments
Open

Handle Custom Domain with SSL on Fabio #925

maddy023 opened this issue Feb 27, 2023 · 5 comments

Comments

@maddy023
Copy link

maddy023 commented Feb 27, 2023

Problem
Currently Nomad Cluster is deployed along with fabio -> AWS Application Load Balancer.
Where Load Balancer is configured with domain1.com SSL cert with ACM , where all the job deployed in nomad cluster with *.domain1.com works as expected., where route53 records are created for respective jobs.

When moving on with custom domain where as helo.domain2.com , i did map the route53 records to reroute the traffic but i am facing SSL cert issue here as load balancer is configure with different ACM.

Test Cases:
Deployed NLB and mapped nomad cluster to handle the traffic , but this case how to handle multiple custom domains? Need to deploy fabio for all the custom domains which are setup ? also how to handle the traffic with SSL Certs (lets say used letsencrypt to generate CA certs )

Is there any feature or work around in fabio to handle multiple custom certs to serve traffic with SSL Certs?

@tristanmorgan
Copy link
Member

To handle multiple domain names on Fabio you can use tags on the consul services like"urlprefix-helo.domain2.com/"and then you just need to work out how you want to setup your certificate store (proxy.cs configuration).

@maddy023
Copy link
Author

fabio.properties
proxy.cs = cs=consul;type=consul;cert=http://localhost:8500/v1/kv/ssl
proxy.addr = :9999;proto=http,:9995;proto=https;cs=consul

Used letsencrypt to generate certs for domain and placed in consul kv :
cat fullchain.pem privkey.pem > cert01.pem

as per : #621

Cluster as nomad + Consul Deployment - deployed loki
`

job "loki" {
  datacenters = ["us-west-1"]
  type        = "service"

  constraint {
    attribute = meta.cluster
    operator  = "="
    value     = "proxy"
  }

  group "loki" {
    count = 1

    network {

      port "http" {
        to = 3100
      }
    }

    restart {
      attempts = 3
      delay    = "20s"
      mode     = "delay"
    }

    task "loki" {
      driver = "docker"

      config {
        image = "grafana/loki:2.6.1"
        ports = ["http"]

      resources {
        cpu    = 100
        memory = 100
      }

      service {
        name = "loki"
        port = "http"
        tags = ["urlprefix-devops.helper.co/"]
      }
    }
  }
}

Fabio Logs:

`

sh-4.2$ ./fabioex -cfg ./fabio/fabio.properties
2023/02/28 09:53:07 [INFO] Setting log level to INFO
2023/02/28 09:53:07 [INFO] Runtime config
{
    "Proxy": {
        "Strategy": "rnd",
        "Matcher": "prefix",
        "NoRouteStatus": 404,
        "MaxConn": 10000,
        "ShutdownWait": 0,
        "DeregisterGracePeriod": 0,
        "DialTimeout": 30000000000,
        "ResponseHeaderTimeout": 0,
        "KeepAliveTimeout": 0,
        "IdleConnTimeout": 15000000000,
        "FlushInterval": 1000000000,
        "GlobalFlushInterval": 0,
        "LocalIP": "198.54.12.11",
        "ClientIPHeader": "",
        "TLSHeader": "",
        "TLSHeaderValue": "",
        "GZIPContentTypes": null,
        "RequestID": "",
        "STSHeader": {
            "MaxAge": 0,
            "Subdomains": false,
            "Preload": false
        },
        "AuthSchemes": {},
        "GRPCMaxRxMsgSize": 4194304,
        "GRPCMaxTxMsgSize": 4194304
    },
    "Registry": {
        "Backend": "consul",
        "Static": {
            "NoRouteHTML": "",
            "Routes": ""
        },
        "File": {
            "NoRouteHTMLPath": "",
            "RoutesPath": ""
        },
        "Consul": {
            "Addr": "localhost:8500",
            "Scheme": "http",
            "Token": "",
            "KVPath": "/fabio/config",
            "NoRouteHTMLPath": "/fabio/noroute.html",
            "TagPrefix": "urlprefix-",
            "Register": true,
            "ServiceAddr": ":9998",
            "ServiceName": "fabio",
            "ServiceTags": null,
            "ServiceStatus": [
                "passing"
            ],
            "CheckInterval": 1000000000,
            "CheckTimeout": 3000000000,
            "CheckScheme": "http",
            "CheckTLSSkipVerify": false,
            "ChecksRequired": "one",
            "ServiceMonitors": 1,
            "TLS": {
                "KeyFile": "",
                "CertFile": "",
                "CAFile": "",
                "CAPath": "",
                "InsecureSkipVerify": false
            },
            "PollInterval": 0,
            "RequireConsistent": true,
            "AllowStale": false
        },
        "Custom": {
            "Host": "",
            "Path": "",
            "QueryParams": "",
            "Scheme": "https",
            "CheckTLSSkipVerify": false,
            "PollInterval": 5,
            "NoRouteHTML": "",
            "Timeout": 10
        },
        "Timeout": 10000000000,
        "Retry": 500000000
    },
    "Listen": [
        {
            "Addr": ":9999",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        },
        {
            "Addr": ":9995",
            "Proto": "https",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "consul",
                "Type": "consul",
                "CertPath": "http://localhost:8500/v1/kv/ssl",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        }
    ],
    "Log": {
        "AccessFormat": "common",
        "AccessTarget": "",
        "RoutesFormat": "delta",
        "Level": "INFO"
    },
    "Metrics": {
        "Target": "",
        "Prefix": "{{clean .Hostname}}.{{clean .Exec}}",
        "Names": "{{clean .Service}}.{{clean .Host}}.{{clean .Path}}.{{clean .TargetURL.Host}}",
        "Interval": 30000000000,
        "Timeout": 10000000000,
        "Retry": 500000000,
        "GraphiteAddr": "",
        "StatsDAddr": "",
        "DogstatsdAddr": "",
        "Circonus": {
            "APIKey": "",
            "APIApp": "fabio",
            "APIURL": "",
            "CheckID": "",
            "BrokerID": "",
            "SubmissionURL": ""
        },
        "Prometheus": {
            "Subsystem": "",
            "Path": "/metrics",
            "Buckets": [
                0.005,
                0.01,
                0.025,
                0.05,
                0.1,
                0.25,
                0.5,
                1,
                2.5,
                5,
                10
            ]
        }
    },
    "UI": {
        "Listen": {
            "Addr": ":9998",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        },
        "Color": "light-green",
        "Title": "",
        "Access": "rw",
        "RoutingTable": {
            "Source": {
                "LinkEnabled": false,
                "NewTab": true,
                "Scheme": "http",
                "Host": "",
                "Port": ""
            }
        }
    },
    "Runtime": {
        "GOGC": 100,
        "GOMAXPROCS": 2
    },
    "Tracing": {
        "TracingEnabled": false,
        "CollectorType": "http",
        "ConnectString": "http://localhost:9411/api/v1/spans",
        "ServiceName": "Fabiolb",
        "Topic": "Fabiolb-Kafka-Topic",
        "SamplerRate": -1,
        "SpanHost": "localhost:9998",
        "SpanName": "",
        "TraceID128Bit": true
    },
    "ProfileMode": "",
    "ProfilePath": "/tmp",
    "Insecure": false,
    "GlobMatchingDisabled": false,
    "GlobCacheSize": 1000,
    "BGP": {
        "BGPEnabled": false,
        "Asn": 65000,
        "AnycastAddresses": null,
        "RouterID": "",
        "ListenPort": 179,
        "ListenAddresses": [
            "0.0.0.0"
        ],
        "Peers": null,
        "EnableGRPC": false,
        "GRPCListenAddress": "127.0.0.1:50051",
        "GRPCTLS": false,
        "CertFile": "",
        "KeyFile": "",
        "GOBGPDCfgFile": "",
        "NextHop": ""
    }
}
2023/02/28 09:53:07 [INFO] Version 1.6.3 starting
2023/02/28 09:53:07 [INFO] Go runtime is go1.19.4
2023/02/28 09:53:07 [INFO] Registering metrics provider ""
2023/02/28 09:53:07 [INFO] Metrics disabled
2023/02/28 09:53:07 [INFO] Setting GOGC=100
2023/02/28 09:53:07 [INFO] Setting GOMAXPROCS=2
2023/02/28 09:53:07 [INFO] consul: Connecting to "localhost:8500" in datacenter "us-west-1"
2023/02/28 09:53:07 [INFO] Admin server access mode "rw"
2023/02/28 09:53:07 [INFO] Admin server listening on ":9998"
2023/02/28 09:53:07 [INFO] Waiting for first routing table
2023/02/28 09:53:07 [INFO] consul: Using dynamic routes
2023/02/28 09:53:07 [INFO] consul: Using tag prefix "urlprefix-"
2023/02/28 09:53:07 [INFO] consul: Watching KV path "/fabio/config"
2023/02/28 09:53:07 [INFO] consul: Watching KV path "/fabio/noroute.html"
2023/02/28 09:53:07 [INFO] HTTP proxy listening on :9999
2023/02/28 09:53:07 [INFO] HTTPS proxy listening on :9995
2023/02/28 09:53:07 [INFO] Access logging disabled
2023/02/28 09:53:07 [INFO] Using routing strategy "rnd"
2023/02/28 09:53:07 [INFO] Using route matching "prefix"
2023/02/28 09:53:07 [INFO] Access logging disabled
2023/02/28 09:53:07 [INFO] Using routing strategy "rnd"
2023/02/28 09:53:07 [INFO] Using route matching "prefix"
2023/02/28 09:53:07 [INFO] cert: Store has certificates for ["devops.helper.co"]
2023/02/28 09:53:07 [INFO] Config updates
+ route add loki devops.helper.co/ http://198.54.12.11:31454/
2023/02/28 09:53:07 [INFO] consul: Registered fabio as "fabio"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with id "fabio-ip-198.54.12.11.us-west-1.compute.internal-9998"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with address "198.54.12.11"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with tags ""
2023/02/28 09:53:07 [INFO] consul: Registered fabio with check &{CheckID:fabio-ip-198.54.12.11.us-west-1.compute.internal-9998-ttl Name: Args:[] DockerContainerID: Shell: Interval: Timeout: TTL:15s HTTP: Header:map[] Method: Body: TCP: Status: Notes: TLSServerName: TLSSkipVerify:false GRPC: GRPCUseTLS:false H2PING: H2PingUseTLS:false AliasNode: AliasService: SuccessBeforePassing:0 FailuresBeforeWarning:0 FailuresBeforeCritical:0 DeregisterCriticalServiceAfter:1m0s}
2023/02/28 09:53:07 [INFO] consul: Registered fabio with check &{CheckID: Name: Args:[] DockerContainerID: Shell: Interval:1s Timeout:3s TTL: HTTP:http://[198.54.12.11]:9998/health Header:map[] Method: Body: TCP: Status: Notes: TLSServerName: TLSSkipVerify:false GRPC: GRPCUseTLS:false H2PING: H2PingUseTLS:false AliasNode: AliasService: SuccessBeforePassing:0 FailuresBeforeWarning:0 FailuresBeforeCritical:0 DeregisterCriticalServiceAfter:}
`

When i hit the Server IP or domain : https://devops.helper.co

image

@tristanmorgan
Copy link
Member

Have you seen any errors in the Fabio logs saying the connection failed? I think it might be the NLB or the SG's on the machine not letting traffic in. (with NLB's you need to add rules on the ec2 instances to accept the internet based traffic).

@maddy023
Copy link
Author

maddy023 commented Mar 1, 2023

Yeah , there is no logs in fabio.
also i am trying to connect via machine public ip, removed NLB out of picture for now.

mapped machine ip to the domain , but still same.

@tristanmorgan
Copy link
Member

When you have the Fabio listener is on 9998 and you try to directly connect then you'll need to add the port to the urlprefix "urlprefix-devops.helper.co:9998/". alternatives are to listen on 443 with CAP_NET_BIND_SERVICE or maybe use some Iptables magic to nat the traffic.
essentially if its a Fabio config error I'd expect to see some sort of error message. maybe check the traffic is arriving with tcpdump...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants