`io`, files and `bufio` · Go: Zero to Production

Go models all input and output as two interfaces you already met: io.Reader and io.Writer. Files, network connections, HTTP bodies, compressors, hashes and in-memory buffers all implement them, which means one piece of code works with all of them.

The two interfaces

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Read fills the slice you give it and returns how many bytes it wrote into it. When the source is exhausted it returns io.EOF — an error value that means "nothing wrong, just finished".

You rarely call Read directly. The helpers below do it properly, including the tricky partial-read cases.

Readers you can make out of thin air

package main

import (
    "bytes"
    "fmt"
    "io"
    "strings"
)

func countBytes(r io.Reader) (int, error) {
    data, err := io.ReadAll(r)
    if err != nil {
        return 0, err
    }
    return len(data), nil
}

func main() {
    n, _ := countBytes(strings.NewReader("hello from a string"))
    fmt.Println("string reader:", n)

    n, _ = countBytes(bytes.NewReader([]byte{1, 2, 3, 4, 5}))
    fmt.Println("bytes reader: ", n)

    var buf bytes.Buffer
    buf.WriteString("buffers are both readers and writers")
    n, _ = countBytes(&buf)
    fmt.Println("buffer:       ", n)
}

strings.NewReader and bytes.NewReader turn data you already have into an io.Reader. That's how you test a function that "reads a file" without touching the filesystem — pass it a string reader instead.

io.ReadAll reads until EOF and hands you a []byte. Fine for small inputs; for anything large, stream it (below) rather than loading it all.

Writers, including the one that discards

package main

import (
    "bytes"
    "fmt"
    "io"
    "os"
    "strings"
)

func writeReport(w io.Writer, lines []string) (int, error) {
    total := 0
    for i, line := range lines {
        n, err := fmt.Fprintf(w, "%d. %s\n", i+1, line)
        if err != nil {
            return total, err
        }
        total += n
    }
    return total, nil
}

func main() {
    lines := []string{"first", "second", "third"}

    writeReport(os.Stdout, lines)

    var buf bytes.Buffer
    n, _ := writeReport(&buf, lines)
    fmt.Printf("wrote %d bytes to memory, upper-cased:\n%s", n, strings.ToUpper(buf.String()))

    writeReport(io.Discard, lines)
    fmt.Println("and", len(lines), "lines to io.Discard, which keeps nothing")
}

One function, three destinations. io.Discard is a writer that swallows everything — useful for benchmarks and for silencing output you don't want.

Copying between them

package main

import (
    "bytes"
    "fmt"
    "io"
    "strings"
)

func main() {
    src := strings.NewReader("copy me from a reader to a writer")
    var dst bytes.Buffer

    n, err := io.Copy(&dst, src)
    fmt.Println(n, "bytes copied, err:", err)
    fmt.Println(dst.String())

    var limited bytes.Buffer
    io.Copy(&limited, io.LimitReader(strings.NewReader("only the first ten"), 10))
    fmt.Printf("%q\n", limited.String())

    var a, b bytes.Buffer
    tee := io.TeeReader(strings.NewReader("written to both"), &a)
    io.Copy(&b, tee)
    fmt.Println(a.String() == b.String())
}

io.Copy streams from a reader to a writer with a small fixed buffer — it never loads the whole thing into memory, so it copies a 10 GB file in constant space.

io.LimitReader caps how much can be read (essential when the source is untrusted — that's how you avoid a malicious upload eating all your RAM), and io.TeeReader duplicates everything read into a second writer, for hashing or logging in passing.

`bufio.Scanner` for line-by-line

The tool for reading text:

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    input := `alpha
beta

gamma`

    scanner := bufio.NewScanner(strings.NewReader(input))
    line := 0
    for scanner.Scan() {
        line++
        text := scanner.Text()
        if text == "" {
            fmt.Printf("%d: (blank)\n", line)
            continue
        }
        fmt.Printf("%d: %s\n", line, text)
    }
    if err := scanner.Err(); err != nil {
        fmt.Println("scan error:", err)
    }

    words := bufio.NewScanner(strings.NewReader("count these words please"))
    words.Split(bufio.ScanWords)
    n := 0
    for words.Scan() {
        n++
    }
    fmt.Println("words:", n)
}

The pattern is always: for scanner.Scan() { ... scanner.Text() ... }, then check scanner.Err() afterwards. Scan returns false both at the end of the input and on a read error, and only Err() tells you which.

Split changes the token: ScanLines (default), ScanWords, ScanRunes, or your own function.

One limit worth knowing: Scanner has a maximum token size (64 KB by default) and returns an error on longer lines. For very long lines use bufio.Reader.ReadString('\n') instead.

Files

Everything above applies unchanged to files, because *os.File is both a Reader and a Writer:

// read a whole small file
data, err := os.ReadFile("config.json")
if err != nil {
    return fmt.Errorf("reading config: %w", err)
}

// write a whole file (0644 = owner read/write, everyone else read)
err = os.WriteFile("out.txt", []byte("hello\n"), 0644)

// open and stream a large file
f, err := os.Open("huge.log")
if err != nil {
    return err
}
defer f.Close()

scanner := bufio.NewScanner(f)
for scanner.Scan() {
    process(scanner.Text())
}
if err := scanner.Err(); err != nil {
    return err
}

// create for writing, buffered
out, err := os.Create("report.txt")
if err != nil {
    return err
}
defer out.Close()

w := bufio.NewWriter(out)
defer w.Flush()          // buffered writes MUST be flushed
fmt.Fprintln(w, "line one")

(These boxes are reference-only — the browser has no filesystem. Everything in them runs unchanged in a compiled Go program.)

The rules that matter:

defer f.Close() immediately after checking the open error. Not before — closing a nil file panics.
bufio.NewWriter needs a Flush. Buffered bytes that never get flushed are silently lost. defer w.Flush() right after creating it.
os.ReadFile for small files, a Scanner for large ones. Reading a 2 GB log into memory is a real outage, and a very easy one to cause.

Checking what went wrong

if _, err := os.Open("missing.txt"); err != nil {
    if errors.Is(err, os.ErrNotExist) {
        // create it, use a default, whatever fits
    }
    return fmt.Errorf("opening data file: %w", err)
}

os.ErrNotExist and os.ErrPermission are sentinel errors — exactly the errors.Is pattern from module 7, now on a package you didn't write.

Implementing a Reader

Since it's one method, you can write your own source:

package main

import (
    "fmt"
    "io"
    "strings"
)

type repeatReader struct {
    text  string
    times int
    done  int
}

func (r *repeatReader) Read(p []byte) (int, error) {
    if r.done >= r.times {
        return 0, io.EOF
    }
    n := copy(p, r.text)
    r.done++
    return n, nil
}

func main() {
    r := &repeatReader{text: "ha", times: 3}

    data, err := io.ReadAll(r)
    fmt.Printf("%q err=%v\n", string(data), err)

    up := strings.ToUpper(string(data))
    fmt.Println(up)
}

Return 0, io.EOF when there's nothing left, and n, nil when you've filled some of p. That contract is all io.ReadAll, io.Copy and bufio.Scanner need in order to work with your type.

Your turn

Count the non-blank lines in the input using a bufio.Scanner:

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    input := "alpha\n\nbeta\n\ngamma\n"
    // count non-blank lines with a bufio.Scanner
}

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    input := "alpha\n\nbeta\n\ngamma\n"

    scanner := bufio.NewScanner(strings.NewReader(input))
    count := 0
    for scanner.Scan() {
        if strings.TrimSpace(scanner.Text()) != "" {
            count++
        }
    }
    fmt.Println(count)
}

Next: pattern matching with regexp.

57. `io`, files and `bufio`