blog Caddy, Golang, and Changed Headers

Tags:
golang deployment caddy

Usually you can run an application without knowing what language it is written in. Sometimes when debugging unexpected behavior, that becomes very important. While debuging some headers changing, knowing that Caddy was written in [golang] explained the bug.

Basic Overview

For a lot of my personal projects , I am hosting code on forgejo and managing events in kafka . Forgejo supports webhooks at the project, organization, or system level. For my instance, I have a system level webhook to send everything into Kafka. The flow looks like this.

Forgejo --> Caddy --> Faust --> Kafka

While this looks like many parts, it is a fairly simple setup.

  1. Forgejo posts a webhook to hooks.example.com which is handled by Caddy
  2. Caddy handles http/https and reverse proxies to a Faust script
  3. Faust takes the payload body and submits it to Kafka

Once I have my webhooks in Kafka, I can use other Faust scripts to process the payloads in interesting ways, or send the data to other systems for logging.

Detecting a Bug

I use sentry for a lot of my projects due to it great support for handling Python exceptions. I was surprised when I started getting errors for KeyError('X-GitHub-Event-Type') from my processing system.

In my processing script, I am using some basic code to split the various event types into their own streams

match headers["X-GitHub-Event-Type"]:
    case "push":
        ...
    case "package":
        ...

This code hadn’t changed recently so it was time to start digging. Pulling up https://forgejo.example.com/admin/hooks/:id to look at Recent deliveries, and everything looked fine. A quick search and "X-GitHub-Event-Type" existed as expected. Checking the Kafka stream, each webhook was being written and a quick find and we see the header in the log there "X-Github-Event-Type".

Case Sensitivity

One might have to read the previous paragraph several times before they realize the issue. It also took me a long time to get my first hint that something was off. It does not help that by default, browser search is case-insensitive by default and I so rarely think about case it did not occur to me to enable the match case checkbox. At some point in the pipeline, GitHub became Github so my header match in Python was now wrong. What had changed in the meantime.

Even though nginx has recently announced native ACME support when I started some server migration I migrated to Caddy to simplify the ACME side of things. Now that I knew that the header casing had changed, I realized that the error started showing up at the same time I had migrated nginx to caddy. This was my first major clue.

Searching on the internet has become far more challenging recently, and I was having poor luck searching for combinations of Caddy and Headers and case sensitivity and other similar terms. Time to jump into the code.

Even though in the completed system, it is the forge server sending data to Kafka, I started by looking at Caddy logging. If we look at how Caddy logs, we can search for the resp_headers value.

https://github.com/caddyserver/caddy/blob/v2.10.0/modules/caddyhttp/server.go#L829-L832 

zap.Object("resp_headers", LoggableHTTPHeader{
    Header:               wrec.Header(),
    ShouldLogCredentials: shouldLogCredentials,
}),

From there if we look at LoggableHTTPHeader we find this

https://github.com/caddyserver/caddy/blob/v2.10.0/modules/caddyhttp/marshalers.go#L63-L70 

type LoggableHTTPHeader struct {
	http.Header

	ShouldLogCredentials bool
}

From there, if we look at http.Header in the net.http package, we get this definition.

// A Header represents the key-value pairs in an HTTP header.
//
// The keys should be in canonical form, as returned by
// [CanonicalHeaderKey].
type Header map[string][]string

Looking up CanonicalHeaderKey we have this definition.

// CanonicalHeaderKey returns the canonical format of the
// header key s. The canonicalization converts the first
// letter and any letter following a hyphen to upper case;
// the rest are converted to lowercase. For example, the
// canonical key for "accept-encoding" is "Accept-Encoding".
// If s contains a space or invalid header field bytes, it is
// returned without modifications.
func CanonicalHeaderKey(s string) string { return textproto.CanonicalMIMEHeaderKey(s) }

This is where I’ve stopped debugging, but I have the following theory.

When Forgejo POSTs to Caddy, Caddy loads the request, ultimately using some of the golang net.http library. The net.http types change into this CanonicalHeaderKey form as part of processing the request. This representation is likely used both by the logging system and also when reverseProxying to the upstream server.

The underlying Header type is ultimately just map[string][]string so in theory it could keep the headers in the same format as the original ones, but I guess the defaults do not make that as easy.

Even though field names are case-insensitive as indicated by rfc2616 , it still feels a bit like a Caddy bug. If I POST GitHub as a header, it is somewhat unexpected behavior for Caddy to send it as Github to the upstream proxy. I may post an issue upstream for documentation purposes though I am not sure it’s a bug they would likely fix.