Go for Statistical Programming

NY Open Statistical Programming Meetup: Knewton HQ

19 August 2013

Personal Background

• Studied computer science and statistics at Columbia
• Worked on the server team at foursquare (Explore recommendation engine)
• Previously on the OkCupid data team (conducted research for OkTrends reports)
• Most recently, Hacker-in-Residence at Quotidian Ventures
• hackNY fellow 2011

Overview of Go History

Birth

• Go was first announced in 2009
• Go 1.0 (first stable release) was released in 2012
• Current version: 1.1

Systems programming

• Developed by Google to address systems programming challenges
• http://dl.google.com was recently rewritten from C++ to Go

Design goals

• Make managing concurrent/distributed systems easy
• Simplify collaboration between multiple developers
• Facilitate evolving codebases (refactoring, etc.)
• Easy to build & deploy

OkCupid Data Workflow (circa 2009)

• 80% munging (Python), 20% statistical analysis (R)

Python in 2009

• In 2009, Python was useable for data science, but not as convenient as today
• Tools like Pandas either didn't exist yet or were not stable/widely adopted
• RPy existed (since 2002)
• Numpy existed (since 2005, previously "Numeric")
• Scipy existed, though with far fewer components and features
• All the munging tools were in place, but numerical tools were lagging somewhat

Go in 2013

• Lots of munging tools available
• Numerical tools exist, but are still WIPs
• go-python provides CPython bindings
• go-matrix provides some vector/matrix operations
• No clear equivalent to Pandas yet

Advantages of Go (for statisticians)

• Very fast (runtime)
• Very fast (compilation)
• Native concurrency
• Easy parallelization (R is single threaded, Python has GIL)
• Portable (cross-compilation)
• Can call other languages from within Go
• Type system: safety of static typing, with the flexibility of dynamic (via interfaces)

Disadvantages of Go

• Relatively new language (but not that new)!
• Academia tends to be slower to pick up new tools

Speed

• Go tends to be >10x faster than Python (and >50x faster than R)
• Go compiles as fast as the Python interpreter takes to start up

Type system

• Go is statically typed
• Go's type inference makes static typing painless

func addOne(a int) int {
    return a + 1
}
func main(){
    a := 5
    b := addOne(a) //Assigned type is inferred
}

• Go's interfaces provide some of the benefits of dynamic typing
• Refactoring and debugging are easier because the compiler is strict

Getting started with Go

Go for Python programmers

• Go fits the Zen of Python very well
• You will love gofmt
• You will love the native concurrency
• You will miss duck typing at first, until you learn to use interfaces

See also: Go for Pythonists

Go for R programmers

• Go's structs & methods are far simpler than S3/S4 methods
• You will love the speed and reliability
• You will love the native concurrency
• You will miss the 'lapply', etc. paradigms at first.
• You will miss the diverse selection of advanced statistical packages

Tips for R programmers

• Go is more imperative than R - get used to explicit iterations
• Structs are very lightweight - don't be afraid to use them
• Let the compiler guide you

Function declaration

func doSomething(){
    var foo int
    foo = 6
    bar := 8
}

Zero values

• By default, an uninitialized value carries the zero value for that type
• Encourages you to design your application in such a way that the default values are meaningful

For loops/If statements

func PrintAllNames(names []string){
    for _, name := range names {
        if name != "" {
            fmt.Println(name)
        }
    }
}

Error Handling

• Explicit is better than implicit
• Instead of raising exceptions, return errors
• Errors are more visible in the code

Structs

• All fields are defined at compiletime
• No inheritance

type CheckingAccount struct {
    Balance int
    superSecretId int64
}

• Embedded structs are permitted

type Person struct {
    MainAccount CheckingAccount
}

type Bank struct {
    Accounts []CheckingAccount
    SavingsAccounts []struct{
        Balance int
        InterestRate float64
    }
}

Interfaces in Go

• If any type provides the methods of an interface, it belongs to the interface
• Like dynamic languages with "duck" typing, you simply use the struct and it "just works"
• Unlike dynamic languages, safety is checked at compile-time

Using Interfaces

type Account interface {
    Deposit(int) error
}

type CheckingAccount struct {
    Balance int 
        superSecretId int64
}

func (destination CheckingAccount) Deposit(amount int) error {
    destination.Balance += amount
    return nil
}

Concurrency in Go

Go's Concurrency Model:

• "Do not communicate by sharing memory; instead, share memory by communicating.""
• Make writing concurrent, asynchronous code as easy as writing synchronous code

Concurrency in Go: Goroutines

• Goroutines are not threads
• Goroutines are not coroutines
• Goroutines are not processes

So what is a goroutine?

From the docs:
- A goroutine is "a function executing concurrently with other goroutines in the same address space"
- Goroutines are light
- Goroutines are multiplexed onto multiple OS threads
- Goroutines are similar to the & operator in bash/sh

Synchronicity by Example

func Greet(name string) {
    log.Printf("Greetings, %s!", name)
    time.Sleep(3 * time.Second)
}

func main() {
    Greet("Alice")
    Greet("Bob")
}

Goroutines by Example

func Greet(name string) {
    log.Printf("Greetings, %s!", name)
    time.Sleep(3 * time.Second)
}

func main() {
    go Greet("Alice")
    go Greet("Bob")

    time.Sleep(5 * time.Second)
}

Yes, it really is just that easy.

Goroutines vs. Callbacks

• Goroutines resemble synchronous code
• Callbacks resemble synchronous code turned "inside out"
• It is easy to turn a function call into a goroutine
• It is difficult to refactor synchronous code to use callbacks

Concurrency in Go: Channels

• Channels provide a convenient way to share information between goroutines
• Channels serve as a "synchronization point" between otherwise independent, concurrent goroutines

Channels by Example:

func Greet(name string, response_chan chan string) {
    greeting := fmt.Sprintf("Greetings, %s!", name)
    response_chan <- greeting
}

func main() {

    cs := make(chan string)
    go Greet("Alice", cs)
    greeting := <-cs
    log.Print(greeting)
}

Let's take a closer look at that:

func Greet(name string, response_chan chan string) {
    time.Sleep(3 * time.Second)
    greeting := fmt.Sprintf("Greetings, %s!", name)
    response_chan <- greeting
    log.Print("Greeting function is tired - sleeping for a bit")
    time.Sleep(3 * time.Second)
    log.Print("now the greeting function is done sleeping - terminating")
}

func main() {

    cs := make(chan string)
    go Greet("Alice", cs)
    log.Print("Continuing execution while we wait for greeter to respond")
    greeting := <-cs
    log.Print(greeting)
    time.Sleep(10 * time.Second)
}

Go makes concurrency easy

• Goroutines are easy to reason about
• Channels make sharing information between goroutines easy
• Go includes a built-in race detector

Data processing in Go

Go Standard Library

Some tools included for free in the standard library:

• Image processing: GIF/JPEG/PNG decoder & encoder, and image composition
• Web scraping: xml parser
• Email: parse and send emails
• Numerical tools: Arithmetic on large numbers and complex/imaginary numbers

Data scraping in Go

Example: Rate-Limiting & Scraping

• (For the full project, see goangel on Github)

//Execute a query that will automatically be throttled
func throttledQuery(queryQueue chan queryChan) {
    for q := range queryQueue {

        endpoint_path := q.endpoint_path
        method := q.method
        keyVals := q.keyVals
        response_ch := q.response_ch
        result, err := Query(endpoint_path, method, keyVals)
        response_ch <- struct {
            result []byte
                err    error
        }{result, err}

        time.Sleep(SECONDS_PER_QUERY)
    }
}

• This lets us decouple the asynchronous scraping from the (synchronous) rate-limiting

Channels for parallel computation

• If a computation can be split into independent components, it can be parallelized
• Channels can be used to signal completion of a portion of the work
• Channels can be used to select the fastest algorithm

Selecting Channels

• "select" will read from the first channel that sends a value

select {
    case <-algorithm1:
        // a read from ch has occurred
    case <-algorithm2:
        // the read from ch has timed out
}

• This can also be used to initiate timeouts

select {
    case <-ch:
        // a read from ch has occurred
    case <-timeout:
        // the read from ch has timed out
}

Example: Data Collection with Background Computation

Writing extensions in C

• This is relatively easy to do via cgo
• While helpful, it is necessary far less often than you might think

Other useful munging tools

go-json - automatically generate static struct definitions for JSON unmarshalling
mgo - a MongoDB client library for Go
redigo - A Redis client library in Go

Conclusion

• Go is already an excellent tool for data processing, scraping, munging, and scripting
• Go is becoming an excellent tool for numerical work and scientific computation

Thank you