Introduction

When developing an application, it’s common to integrate it with third-party services, APIs, and external systems.

Testing the integration is a crucial step in ensuring the reliability and correctness of your application.

In the Go programming language, you can utilize the built-in net/http/httptest package to test HTTP client/server interactions.

While this approach is straightforward, it may not accurately capture real-world interactions or handle complex scenarios involving external services.

In this article, I am introducing you to a powerful testing strategy for interacting with third-party servers.

This strategy is commonly referred to as “VCR” (Video Cassette Recorder) or “recording/replaying”, and it is especially beneficial in scenarios such as:

Testing an integration with a payment gateway to validate credit card payment processing.
Testing an integration with a third-party API for sending push notifications to users.

The Recording/Replaying Strategy

The testing strategy involves two fundamental phases: recording and replaying.

In the initial phase, known as “recording”, the test captures and stores the requests and responses exchanged with external HTTP/gRPC servers in a replay file.

This recording process ensures that the data sent via the requests and the responses received from the external server are preserved for future needs.

The main advantage of this method lies in the fact that it allows the HTTP/gRPC requests to be directed to a real production system, ensuring that the client’s test remains unbiased regarding how the server responds or behaves in similar situations.

In the second phase, referred to as “replaying”, the test configures the HTTP/gRPC client to compare a given request to a previously stored one as a recording file.

When a request matches a recorded interaction, the recorded response is retrieved and returned, simulating a real response from the external service.

As a result of these two phases, the tests are now equipped to make assertions based real interactions, providing a powerful testing mechanism for your application’s integration with third-party services.

To demonstrate the “recording/replaying” approach in action, I will leverage an external package provided by Google called go-replayers.

However, that there are several others similar packages available in the Go ecosystem.

Use case

Before jumping into go-replayers, it’s essential to understand the context where we intend to use it.

Let’s imagine a scenario where an external system manages the user’s data, and your team needs to build a client that facilitates this integration.

The external server exposes two HTTP APIs

POST /users
GET /users/{id}

This type represents the user data exchanged by those APIs:

type User struct {
    ID        string
    Name      string
    CreatedAt time.Time
}

Your team ends up writing a quite simple client, looking like this:

package user

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"net/http"
)

var (
	ErrGet      = errors.New("could not get user")
	ErrSave     = errors.New("could not save user")
	ErrNotFound = errors.New("could not find user")
)

type Client struct {
	BaseClient *http.Client
	BaseURL    string
}

func (c *Client) Get(ctx context.Context, id string) (*User, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, fmt.Sprintf("%s/users/%s", c.BaseURL, id), http.NoBody)
	if err != nil {
		return nil, fmt.Errorf("%w: could not create request: %s", ErrGet, err)
	}

	res, err := c.BaseClient.Do(req)
	if err != nil {
		return nil, fmt.Errorf("%w: could not do request: %s", ErrGet, err)
	}
	defer res.Body.Close()

	switch {
	case res.StatusCode == http.StatusNotFound:
		return nil, fmt.Errorf("%w: %s", ErrNotFound, err)
	case res.StatusCode != http.StatusOK:
		return nil, fmt.Errorf("%w: could not handle status code: %s", ErrGet, err)
	}

	var u User
	if err := json.NewDecoder(res.Body).Decode(&u); err != nil {
		return nil, fmt.Errorf("%w: could not decode data: %s", ErrGet, err)
	}

	return &u, nil
}

func (c *Client) Save(ctx context.Context, u *User) error {
	body := &bytes.Buffer{}
	if err := json.NewEncoder(body).Encode(u); err != nil {
		return fmt.Errorf("%w: could not encode data: %s", ErrSave, err)

	}
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, fmt.Sprintf("%s/users", c.BaseURL), body)
	if err != nil {
		return fmt.Errorf("%w: could not create request: %s", ErrSave, err)
	}

	res, err := c.BaseClient.Do(req)
	if err != nil {
		return fmt.Errorf("%w: could not do request: %s", ErrSave, err)
	}
	defer res.Body.Close()

	if res.StatusCode != http.StatusOK {
		return fmt.Errorf("%w: could not handle status code: %d", ErrSave, res.StatusCode)
	}

	return nil
}

Recording

When considering how to test the client, your team must ensure that errors and successes are appropriately handled and returned.

Focusing on reading user data, the tests might look like this:

func TestClient_Get(t *testing.T) {
	t.Run("found", func(t *testing.T) {
		ctx := context.Background()

		cl := getClient(t) // returns *user.Client

		id := "an-existing-id"
		name := "Luke"

		u, err := cl.Get(ctx, id)
		if err != nil {
			t.Errorf("could not get existing user: %s", err)
		}

		if u.ID != id {
			t.Errorf("could not match existing user id: %s", u.ID)
		}

		if u.Name != name {
			t.Errorf("could not match existing user name: %s", u.Name)
		}
	})

	t.Run("not found", func(t *testing.T) {
		ctx := context.Background()

		cl := getClient(t) // returns *user.Client

		id := "a-non-existing-id"

		u, err := cl.Get(ctx, id)
		if !errors.Is(err, user.ErrNotFound) {
			t.Errorf("could not match non existing user error: %s", err)
		}

		if u != nil {
			t.Errorf("could get non existing user: %v", u)
		}
	})
}

The test itself appears quite standard as a client test.

However, the real magic happens within the getClient helper function.

To enable the recording of requests and responses, the tests can rely on a specialized *http.Client configured by the httpreplay.NewRecorderWithOpts function.

func getClient(t *testing.T) *user.Client {
	t.Helper()

	// Where the recorded file need to be stored
	filename := fmt.Sprintf("testdata/%s", t.Name())

    // Creates the recorder
	rec, err := httpreplay.NewRecorderWithOpts(filename)
	if err != nil {
		t.Fatalf("could not create recorder: %s", err)
	}

	// In the test cleanup, it ensures the recorder is closed, so the record file is written down to the disk as well.
	t.Cleanup(func() {
		if err := rec.Close(); err != nil {
			t.Errorf("could not close recorder: %s", err)
		}
	})

	// The *http.Client which records the request and response
	cl := rec.Client()

	// Returns the *user.Client with the recorder client
	const url = "http://localhost:8080" // or wherever your external service lives
	return &user.Client{BaseClient: cl, BaseURL: url}
}

With this configuration, the HTTP client captures the data sent to the server and the corresponding responses, storing them as a file, named after the test name itself, in a designated testdata folder.

Now, when it comes to testing the Save method, the code can continue to rely on the getClient helper function for convenience:

func TestClient_Save(t *testing.T) {
	t.Run("saved", func(t *testing.T) {
		cl := getClient(t)

		u := &user.User{
			ID:        "an-existing-id",
			Name:      "Luke",
			CreatedAt: time.Now(),
		}

		ctx := context.Background()

		if err := cl.Save(ctx, u); err != nil {
			t.Errorf("could not save existing user: %s", err)
		}
	})
}

Despite its apparent simplicity and conciseness, there’s a hidden difference when comparing it to the previous test.

In this scenario, the user type instantiated by the test sets the CreatedAt field using time.Now() each time it runs.

As a result, the recorded data will differ with each test execution.

To address this, the test must provide the recorder with instructions to scrub specific details from the request.

Note: a similar approach can be applied to headers, especially when dealing with scenarios involving authentication tokens.


func TestClient_Save(t *testing.T) {
	t.Run("saved", func(t *testing.T) {
		cl := getClient(
			t, 
			func(rec *httpreplay.Recorder) { rec.ScrubBody(`"CreatedAt":\s*".*?"`) }, // Scrubs the "CreatedAt" field from the JSON body 
			func(rec *httpreplay.Recorder) { rec.RemoveRequestHeaders("Content-Length") }, // Remove the "Content-Length" header since the size of the body will change.
		)
		
	// ...
}

// ...

func getClient(t *testing.T, opts ...func(*httpreplay.Recorder)) *user.Client {
	// ...
	
    // Creates the recorder
	rec, err := httpreplay.NewRecorderWithOpts(filename)
	if err != nil {
		t.Fatalf("could not create recorder: %s", err)
	}

	// Applies the options
	for i := range opts {
opts[i](rec)
	}

	// ...
}

Replaying

Now that the tests are recorded with real request and response data, let’s delve into the aspect of replaying.

In the replaying phase, the tests must leverage the recorded interactions to simulate real-world scenarios and test our application’s behavior.

To allow this, the test needs to have a different implementation of the getClient function, which returns a replayer instead of a recorder, and the rest of the tests do not need to change:

func getClient(t *testing.T, _ ...func(*httpreplay.Recorder)) *user.Client {
	t.Helper()

	// Where the recorded file need to be read
	filename := fmt.Sprintf("testdata/%s", t.Name())

    // Creates the replayer
	rep, err := httpreplay.NewReplayer(filename)
    if err != nil {
        t.Fatalf("could not create replayer: %s", err)
    }

    t.Cleanup(func() {
        if err := rep.Close(); err != nil {
            t.Errorf("could not close replayer: %s", err)
        }
    })

	// The *http.Client which replays the request and response
	cl := rep.Client()

	// Returns the *user.Client with the recorder client
	const url = "http://localhost:8080" // or wherever your external service lives
	return &user.Client{BaseClient: cl, BaseURL: url}
}

Switch from replayer to recorder using build tags

For switching between the recording and replaying modes, I recommend adopting the golden file philosophy.

In essence, our recorders are creating what is known as “golden files”, which serve as reference points for validating the outcomes of our tests later on.

To facilitate switching between replayer and recorder modes, the tests can rely on a build tag called “golden” (for example) to conditionally control which of the getClient functions the tests are using.

Example of the norecorder_test.go file:

//go:build !golden
// +build !golden

// ... your package name and imports

// place here the getClient function used in the replayer example
func getClient(t *testing.T, _ ...func(*httpreplay.Recorder)) *user.Client {
	// ...
}

Example of the recorder_test.go file:

//go:build golden
// +build golden

// ... your package name and imports

// place here the getClient function used in the recorder example
func getClient(t *testing.T, opts ...func(*httpreplay.Recorder)) *user.Client {
	// ...
}

What about gRPC

The just-discussed testing strategy can also be applied to gRPC.

In fact, go-replayers provides a dedicated module for gRPC, which mirrors the same APIs and behaviors we’ve explored for HTTP.

Conclusion

In conclusion, the recorder/replayer strategy makes testing third-party connections more trustworthy, faster, and more reliable as they mimic how the end production client interacts with real external services.

You can find the showcased example from the article by visiting this repository.

If you have any questions, feel free to send me a DM on Twitter, LinkedIn or add a comment in the section below!

Introduction#

The Recording/Replaying Strategy#

Use case#

Recording#

Replaying#

Switch from replayer to recorder using build tags#

What about gRPC#

Conclusion#