Fuzzin' in Go
Go Blog recently announced a beta preview of fuzzing support in Go which gave me some new ideas on how to improve my development testing results. There are at least two classes of bugs which I think should surface as a result of applying fuzzing libraries:
- panics as a result of:
nilvariable usage- 3rd party library usage
- timeouts
I’ll probably find other issues as well but making panic surface would save me a lot of time
that I spend on debugging random JSON/REST payload mutations and interpretation of missing or empty JSON fields.
First off I need some code to test. I’ve decided to code a small piece of validation logic with a ton
of bugs and then attempt to find at least some of them with fuzzing. Note that this chunk of code even once it’ll be
executable without panic -ing is still worthless (you have been warned):
package main
import (
"encoding/json"
"log"
)
type Credentials struct {
Username *string `json:"username"`
Password *string `json:"password"`
}
// IsValid returns true if credentials are valid, false otherwise
func (c Credentials) IsValid() bool {
for _, c := range []byte(*c.Username) {
if c == byte('@') {
return true
}
}
if c.Password != nil && *c.Password == "" {
// TODO: autogenerate password .)
log.Fatal("User supplied empty password")
return false
}
return true
}
type UserCredentials []*Credentials
// FilterValid filters invalid credentials out
func (uc UserCredentials) FilterValid() (UserCredentials, error) {
validCredentials := UserCredentials{}
for _, credentials := range uc {
if *credentials.Username == "[email protected]" {
validCredentials = append(validCredentials, credentials)
}
if credentials.IsValid() {
validCredentials = append(validCredentials, credentials)
}
}
return validCredentials, nil
}
It’s relatively simple on the eyes and full of obvious bugs. Now in order to make those bugs trigger I need to write a special kind of fuzzing test. Let’s walk through first attempt slowly.
// +build gofuzzbeta
First off this uses beta functionality that isn’t part of current stable Go release. In order for this to build properly the file needs to start with a build constraint to make this be included in build only if it’s supported.
package main
import (
"encoding/json"
"github.com/google/gofuzz"
"github.com/stretchr/testify/assert"
"testing"
)
Using just gofuzzbeta quickly turns to not be enough as I need to be able to generate input samples
which is what github.com/google/gofuzz is for.
func prettyPrint(creds UserCredentials) string {
s, _ := json.MarshalIndent(creds, "", "\t")
return string(s)
}
prettyPrint is used to output samples for which fuzzing checks fail.
func FuzzValidate(f *testing.F) {
FuzzValidate is picked up when running all fuzzy tests the same way go test finds functions with Test prefix.
gof := fuzz.New().NilChance(0.2).NumElements(1, 5)
To generate test cases I make use of gofuzz library. NilChance makes it
generate nil values with a 20% probability and NumElements makes it generate lists of size in given range.
for i := 0; i < 100; i++ {
creds := UserCredentials{}
gof.Fuzz(&creds)
res, err := json.Marshal(creds)
assert.Nil(f, err)
f.Add(res)
}
Next I generate a 100 credentials lists and use f.Add to add it to current fuzzing test samples.
This will be consumed by f.Fuzz where I check if execution was successful and if not print out the offending input samples.
f.Fuzz(func(t *testing.T, rawCredentials []byte) {
var credentials UserCredentials
err := json.Unmarshal(rawCredentials, &credentials)
if err != nil {
t.Fatalf("Failed unmarshalling input %v: %v", prettyPrint(credentials), err)
}
creds, err := credentials.FilterValid()
if err != nil {
t.Fatalf("Validate failed to execute for %v: %v", prettyPrint(creds), err)
}
if len(credentials) != len(creds) {
t.Fatalf("Some credentials are not valid %v: %v", prettyPrint(creds), err)
}
})
}
f.Fuzz takes a function that is executed on all f.Add-ed samples. I passed in []byte because it didn’t
want to consume my custom type. Hopefully this is something that can be improved on in the future.
If I run this now I get my first panic! (due to verbosity I’ll just paste the relevant part)
--- FAIL: FuzzValidate (0.00s)
--- FAIL: FuzzValidate/seed#0 (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 20 [running]:
...
panic({0x5ee2c0, 0x7c1760})
/home/simonm/sdk/gotip/src/runtime/panic.go:1038 +0x215
github.com/crnkofe/blog/fuzzin.UserCredentials.FilterValid({0xc00009fb00, 0x5, 0xc0000528e0})
/home/simonm/github/blog/fuzzin/bad_code.go:35 +0x77
...
This happens on the if *credentials.Username == "[email protected]" { which (besides having a hardcoded admin)
fails to check for Username field. I decide to add a unit test for nil case and fix the issue.
nil Username bug also pops up in IsValid.
func TestValidateNilUsername(t *testing.T) {
naiveInput := UserCredentials{
{
Username: nil,
},
}
validCredentials, err := UserCredentials(naiveInput).FilterValid()
assert.Nil(t, err)
assert.Equal(t, 0, len(validCredentials))
}
Thinking if I should also exclude this input from fuzzer sample generator I think there’s actually no benefit
since the issue was panicking code which after fixed should no longer panic.
--- FAIL: FuzzValidate (0.00s)
--- FAIL: FuzzValidate/seed#1 (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 22 [running]:
...
panic({0x5fbaa0, 0x7d8830})
/home/simonm/sdk/gotip/src/runtime/panic.go:1038 +0x215
github.com/crnkofe/blog/fuzzin.UserCredentials.FilterValid({0xc00009fd70, 0x5, 0x20})
/home/simonm/github/blog/fuzzin/bad_code.go:39 +0x77
...
This one is basically on the same line as first Username checker. It crashes due to me allowing
nil in the UserCredentials list. A smart thing at at this point might have been to simply make the list
non-nilable but I decide to go against my intuition since I can get more bugs that way.
func TestValidateNilCredentials(t *testing.T) { │
naiveInput := UserCredentials{ │
nil, │
} │
validCredentials, err := UserCredentials(naiveInput).FilterValid() │
assert.Nil(t, err) │
assert.Equal(t, 0, len(validCredentials)) │
}
Back to fuzzing! Next error I get something interesting:
gotip test .
2021/06/13 20:34:33 User supplied empty password
FAIL github.com/crnkofe/blog/fuzzin 0.008s
FAIL
The log.Fatal simply kills the fuzzer which is kind of expected but also annoying. As a dev I want a list of
things to fix and not be locked into an endless try-fix routine. According to some discussions on this subject
capturing log.Fatal would require me to do some mocking but in this particular case I’m not getting payed to do this
so I skip the part but write a unit test nontheless.
func TestValidateEmptyPassword(t *testing.T) {
emptyUsername := ""
emptyPassword := ""
naiveInput := UserCredentials{
{
Username: &emptyUsername,
Password: &emptyPassword,
},
}
validCredentials, err := UserCredentials(naiveInput).FilterValid()
assert.Nil(t, err)
assert.Equal(t, 0, len(validCredentials))
}
I decide to quietly remove the part with the log.Fatal and TODO. Rerunning the fuzzer
oddly enough gives me no panic but I now do get some strange unexpected things:
--- FAIL: FuzzValidate/seed#91 (0.00s)
bad_code_fuzz_combo_test.go:39: Some credentials are not valid []: <nil>
It turns out this is due to fuzzer generating samples with empty string Username or Password.
I make a few quick hacks to simply eliminate such input from being sent into fuzzer:
sanitizedUserCredentials := UserCredentials{}
// skip empty input
for _, el := range creds {
if el == nil || (*el == Credentials{}) {
continue
}
if el.Username == nil || el.Password == nil {
continue
}
if *el.Username == "" || *el.Password == "" {
continue
}
sanitizedUserCredentials = append(sanitizedUserCredentials, el)
}
And I’m done!
ok github.com/crnkofe/blog/fuzzin 0.408s
Or am I?
Well to be honest I’d never use that code in production and I wasn’t expecting the admin hack to get caught by the fuzzer. It’d have to be extremely lucky to generate exactly that particular username as input which would cause not just to have a valid case but also a duplicated valid case. Email and password validation are also really bad but don’t cause visible failures during fuzzing.
Overall I’m somewhat satisfied with the ability to fuzz inputs. There are plenty of cases where I can see it
produce bugs faster than I can reverse engineer, especially for untested legacy code.
It’d be nice if fuzzing was supported by IDEs (GoLand if possible please) and if I could somehow workaround
the need to do de/serialization when executing f.Fuzz.