Integrating a C library (zlib)

This is a guide to integrating C libraries in Acton code. We will use the zlib compression library, written in C, to build an Acton module that supports zlib compression and decompression.

We will only focus on the inflate and deflate functions in zlib. They are pure functions (meaning they only take some input and return some output, they do not have any side effects like writing to some shared state), that makes them easier to integrate than anything that does I/O. While zlib does expose functions to interact with files, we don't want to reimplement file related functionality since we already have this supported by the Acton stdlib.

Create new project

Let's start by making a new Acton project, let's call it acton-zlib. New projects are created with an example "Hello world" app. Let's remove it and start from scratch.

acton new acton-zlib
cd acton-zlib
rm src/*

Acton's low level build system - the Zig build system

The Acton compiler parses .act source code, runs through all its compilation passes with type checking, CPS conversion, lambda lifting etc and finally produces C code. Internally, Acton then uses the Zig build system to compile the generated C code to libraries and finally binary executables.

To add a C library dependency, it first needs to be buildable using the Zig build system, which means that it needs a build.zig file, the config file for the Zig build, somewhat similar to the CMakeLists.txt of CMake. Some projects have already adopted a build.zig in the upstream repo, like PCRE2 and the Boehm-Demers-Weiser GC (both of which are used by Acton). In some cases, there are forks of projects with build.zig added. Otherwise you will need to write one for yourself, which is usually simpler than it might first seem.

Add the zlib C library as a Zig dependency

In the case of zlib, there is already a repo available with a build.zig for zlib. Navigate to the Tags page, find 1.3.1 and the link to the source files, i.e. https://github.com/allyourcodebase/zlib/archive/refs/tags/1.3.1.tar.gz.

Add it to our acton-zlib project:

acton zig-pkg add https://github.com/allyourcodebase/zlib/archive/refs/tags/1.3.1.tar.gz zlib --artifact z

Note the --artifact z which is provided to instruct which library to link with. Headers from the zlib library, like zlib.h, will now become visible to C files in our project and the z library will be linked in with our executables. The easiest way to discover what the artifacts are called is by inspecting the build.zig file of the package. This particular zlib build.zig starts like this:

const std = @import("std");

pub fn build(b: *std.Build) void {
    const upstream = b.dependency("zlib", .{});
    const lib = b.addStaticLibrary(.{
        .name = "z",
        .target = b.standardTargetOptions(.{}),
        .optimize = b.standardOptimizeOption(.{}),
    });
    lib.linkLibC();
    lib.addCSourceFiles(.{
        .root = upstream.path(""),
        .files = &.{
            "adler32.c",
            "crc32.c",
...

It is the .name argument to addStaticLibrary that tells us the name of the artifact. Zig packages might expose multiple such artifacts, as is the case for mbedtls.

acton zig-pkg add will fetch the package from the provided URL and save the hash sum to Build.act, resulting in:

dependencies = {}

zig_dependencies = {
  "zlib": (
        url="https://github.com/allyourcodebase/zlib/archive/refs/tags/1.3.1.tar.gz",
        hash="122034ab2a12adf8016ffa76e48b4be3245ffd305193edba4d83058adbcfa749c107",
        artifacts=["z"]
    ),
}

Create zlib.act Acton module

Next up we need to create the Acton zlib module. Open src/zlib.act and add a compress and decompress function:

pure def compress(data: bytes) -> bytes:
    NotImplemented

pure def decompress(data: bytes) -> bytes:
    NotImplemented

The NotImplemented statement tells the compiler that the implementation is not written in Acton but rather external. When there is a .ext.c file, the compiler expects it to contain the implementations for the NotImplemented functions. Also note the explicit types. Normally the Acton compiler can infer types, but since there is no Acton code here, only C code, there is nothing to infer from.

Now create src/zlib.ext.c, which is where we will write the actual implementation of these functions. The file must always define an __ext_init__ function. The module's generated initialization function calls it once, when the module is loaded.

__ext_init__ is for setting up the C state that the module's C functions rely on, for example initializing a C library (libxml2 needs xmlInitParser()) or seeding a random number generator. It must not depend on Acton code: do not call functions or methods written in Acton, from this module or from any other module. __ext_init__ runs at the very start of the module's initialization, before the module initializes its imports, its classes and its module-level values. Acton code called from it could therefore find any of these not yet initialized. The rule applies only to __ext_init__. The C functions that implement the NotImplemented declarations are called from Acton code and always run after it.

There is nothing to set up for zlib, so let's just create an empty function, like so:

void zlibQ___ext_init__() {}

Next, we need to fill in the C functions that map to the Acton functions compress and decompress. By invoking acton build we can get the compiler to generate a skeleton for these. We will also get a large error message, since there is no actual implementation:

user@host$ acton build
... some large error message

Ignore the error and instead check the content of out/types/zlib.c and we will find the C functions we need, commented out:

#include "rts/common.h"
#include "out/types/zlib.h"
#include "src/zlib.ext.c"
B_bytes zlibQ_compress (B_bytes data);
/*
B_bytes zlibQ_compress (B_bytes data) {
    // NotImplemented
}
*/
B_bytes zlibQ_decompress (B_bytes data);
/*
B_bytes zlibQ_decompress (B_bytes data) {
    // NotImplemented
}
*/
int zlibQ_done$ = 0;
void zlibQ___init__ () {
    if (zlibQ_done$) return;
    zlibQ_done$ = 1;
    zlibQ___ext_init__ ();
}

Copy the commented-out skeleton into our own src/zlib.ext.c. Just in order to get something that compiles, let's just quickly let the functions return the input data. Since both input and output are bytes, this should now compile (and work at run time).

B_bytes zlibQ_compress (B_bytes data) {
    return data;
}
B_bytes zlibQ_decompress (B_bytes data) {
    return data;
}

user@host:~/acton-zlib$ acton build
Building project in /Users/user/acton-zlib
  Compiling zlib.act for release
   Finished compilation in   0.005 s
  Compiling test_zlib.act for release
   Finished compilation in   0.019 s
  Final compilation step
user@host:~/acton-zlib$

Add a test module

Before we implement the body of the compress and decompress functions, we can write a small test module which will tell us when we've succeeded. We use some pre-known test data (which we could get from another language implementation):

import testing
import zlib

def _test_roundtrip():
    for x in range(100):
        i = "hello".encode()
        c = zlib.compress(i)
        d = zlib.decompress(c)
        testing.assertEqual(i, d)

def _test_compress():
    for x in range(100):
        i = "hello".encode()
        c = zlib.compress(i)
        testing.assertEqual(c, b'x\x9c\xcbH\xcd\xc9\xc9\x07')

def _test_decompress():
    for x in range(1000):
        c = b'x\x9c\xcbH\xcd\xc9\xc9\x07'
        d = zlib.decompress(c)
        testing.assertEqual(d, b'hello')

Note how we run a few test iterations to get slightly better timing measurements for performance testing. Run the test with acton test:

user@host:~/acton-zlib$ acton test

Tests - module test_zlib:
  decompress:            FAIL:  195 runs in 50.728ms
    testing.NotEqualError: Expected equal values but they are non-equal. A: b'x\x9c\xcbH\xcd\xc9\xc9\x07' B: b'hello'
  compress:              FAIL:  197 runs in 50.886ms
    testing.NotEqualError: Expected equal values but they are non-equal. A: b'hello' B: b'x\x9c\xcbH\xcd\xc9\xc9\x07'
  roundtrip:             OK:  226 runs in 50.890ms

2 out of 3 tests failed (26.354s)

user@host:~/acton-zlib$ 

As expected, the roundtrip test goes through, since we just return the input data while the compress and decompress tests fail.

Implement the compress function

Now let's fill in the rest of the owl. Below is the body of the zlibQ_compress function. The bulk of this code is not particularly interesting to this guide as it has more to do with standard C usage of zlib, but a few things are worth noting.

B_bytes zlibQ_compress(B_bytes data) {
    if (data->nbytes == 0) {
        return data;
    }

    // Prepare the zlib stream
    int ret;
    z_stream stream;
    memset(&stream, 0, sizeof(stream));
    ret = deflateInit(&stream, Z_DEFAULT_COMPRESSION);
    if (ret != Z_OK) {
        $RAISE((B_BaseException)$NEW(B_ValueError, $FORMAT("Unable to compress data, init error: %d", ret)));
    }

    // Set the input data
    stream.avail_in = data->nbytes;
    stream.next_in = (Bytef*)data->str;

    // Allocate the output buffer using Acton's malloc
    size_t output_size = deflateBound(&stream, data->nbytes);
    Bytef* output_buffer = (Bytef*)acton_malloc_atomic(output_size);
    stream.avail_out = output_size;
    stream.next_out = output_buffer;

    // Perform the deflate operation
    ret = deflate(&stream, Z_FINISH);
    if (ret != Z_STREAM_END) {
        $RAISE((B_BaseException)$NEW(B_ValueError, $FORMAT("Unable to compress data, error: %d", ret)));
    }

    // Clean up
    deflateEnd(&stream);

    return actBytesFromCStringNoCopy(output_buffer);
}

Memory management is always top of mind when writing C, as it the case here. We can allocate memory via the Acton GC-heap malloc or just plain malloc() (the non-GC heap, to be explicit). Since zlibQ_compress is pure, we have no state leaking out of the function other than via its return value. All return values must be allocated on the Acton GC heap, so we know we must use acton_malloc for any value that we return. Any other local variables within the function can use classic malloc, as long as we make sure to explicitly free it up. For class or actor methods, any allocation for class or actor attributes must be performed using the Acton GC malloc, since there is no destructor or similar where a free can be inserted, so using classic malloc would be bound to leak. Also note that in this particular case, we know that the returned bytes value itself is not going to contain any pointers, so by using acton_malloc_atomic we can get a chunk of memory that will not be internally scanned by the GC, which saves a bit of time and thus improves GC performance. If we allocate structs that do carry pointers, they must use the normal acton_malloc(). Storing pointers into such memory later needs a write barrier, see Storing pointers into GC memory.

Acton's own boxed scalars (int, float, complex and the fixed-width integer types) are allocated pointer-free in the same way, as are ranges and the digits of bigint values. So a pointer converted to an integer and stored in a boxed int does not keep the object it points to alive. Keep such a pointer in a field of a struct allocated with acton_malloc() instead.

actBytesFromCStringNoCopy(output_buffer) takes the buffer (already allocated via acton_malloc_atomic()) and wraps it up as a boxed value of the type B_bytes that we return.

Also note how we convert Zlib errors to Acton exceptions where necessary.

Running the test, the compress test now passes while roundtrip has stopped working (since decompress is not implemented yet):

user@host:~/acton-zlib$ acton test
Tests - module test_zlib:
  decompress:            FAIL:  158 runs in 50.175ms
    testing.NotEqualError: Expected equal values but they are non-equal. A: b'x\x9c\xcbH\xcd\xc9\xc9\x07' B: b'hello'
  compress:              OK:  167 runs in 50.225ms
  roundtrip:             FAIL:  147 runs in 50.266ms
    testing.NotEqualError: Expected equal values but they are non-equal. A: b'hello' B: b'x\x9c\xcbH\xcd\xc9\xc9\x07'

2 out of 3 tests failed (0.941s)

user@host:~/acton-zlib$ 

Implement the decompress function

Much like the compress function, the decompress function mostly relates to how zlib itself and its interface works. We use the same wrappers and transform errors to exceptions.

B_bytes zlibQ_decompress(B_bytes data) {
    if (data->nbytes == 0) {
        return data;
    }

    // Prepare the zlib stream
    int ret;
    z_stream stream;
    memset(&stream, 0, sizeof(stream));

    ret = inflateInit(&stream);
    if (ret != Z_OK) {
        $RAISE((B_BaseException)$NEW(B_ValueError, $FORMAT("Unable to decompress data, init error: %d", ret)));
    }

    // Set the input data
    stream.avail_in = data->nbytes;
    stream.next_in = (Bytef*)data->str;

    // Allocate the output buffer using Acton's malloc
    size_t output_size = 2 * data->nbytes; // Initial output buffer size
    Bytef* output_buffer = (Bytef*)acton_malloc_atomic(output_size);
    memset(output_buffer, 0, output_size);
    stream.avail_out = output_size;
    stream.next_out = output_buffer;

    // Perform the inflate operation, increasing the output buffer size if needed
    do {
        ret = inflate(&stream, Z_NO_FLUSH);
        if (ret == Z_BUF_ERROR) {
            // Increase the output buffer size and continue decompressing
            size_t new_output_size = output_size * 2;
            output_buffer = (Bytef*)acton_realloc(output_buffer, new_output_size);
            stream.avail_out = new_output_size - stream.total_out;
            stream.next_out = output_buffer + stream.total_out;
        } else if (ret != Z_OK) {
            $RAISE((B_BaseException)$NEW(B_ValueError, $FORMAT("Unable to decompress data, error: %d", ret)));
        }
    } while (ret == Z_BUF_ERROR);

    // Clean up
    inflateEnd(&stream);

    return actBytesFromCStringNoCopy(output_buffer);
}

Final test

user@host:~/acton-zlib$ acton test

Tests - module test_zlib:
  decompress:            OK:   42 runs in 51.065ms
  compress:              OK:   25 runs in 50.032ms
  roundtrip:             OK:   24 runs in 50.053ms

All 3 tests passed (0.738s)

user@host:~/acton-zlib$ 

And with that, we're done! A simple wrapper around zlib, which is also available on GitHub if you want to study it further.

Storing pointers into GC memory

In generational mode (ACTON_GC_MODE=generational, see build options) a minor collection does not scan the whole heap. It rescans only the marked objects of the heap blocks that the program recorded as written since the previous collection, through a write barrier. Generated Acton code has the barrier built in. C code that stores a pointer to GC memory into GC memory must add it itself; a missing barrier lets the collector free an object that is still in use. The macros are in rts/wbar.h, which rts/common.h includes:

1. After storing a GC pointer v into a field of a GC object obj, call ACTON_WBAR(obj, &obj->field, v), or write the store as ACTON_STORE(obj, obj->field, v). obj is the start of the object that holds the field; pass NULL if the code only has a pointer to a struct inside it.
2. Before copying, moving or sorting pointers in bulk into a GC object (memcpy, memmove, a copy loop, reversing or sorting an array in place), call ACTON_WBAR_RANGE(obj, dst, nbytes) over the destination.
3. No barrier is needed for stores into an object that the same function allocated and has held in a local variable since, for stores of NULL, None, integers or static objects, and for stores into stack variables, C globals, malloc memory or memory from acton_malloc_atomic. An object reached through a new object is not new itself: filling the array of a list returned by B_listD_new needs barriers. A barrier is always safe.
4. The result of acton_realloc is not a new object: it may be the old one.
5. A GC pointer stored in an int field (a handle kept as an integer) needs ACTON_WBAR too, since the generated code sees only an integer.
6. Memory that a C library writes pointers into by itself, such as libuv handles and requests, comes from acton_malloc_rescanned(), which every collection rescans. Threads that the collector does not stop, like libuv's thread pool, must not write GC pointers into GC memory at all.
7. Test with acton test --gc-check (or ACTON_GC_CHECK=1), which runs the collector's write-barrier checker at every collection and exits with status 70 when it finds a store without its barrier.

B_NoneType fooQ_ThingD_set_name(fooQ_Thing self, B_str name) {
    ACTON_STORE(self, self->name, name);
    return B_None;
}

Converting C strings to Acton strings

The string constructors validate UTF-8 and count Unicode characters. Use an explicit byte length when the input can contain NUL characters:

Shared input must remain immutable and alive for as long as the returned string exists. String literals and completed buffers allocated with acton_malloc_atomic() are suitable. Use Copy for stack buffers, buffers that will be modified or freed, and memory owned by a C library that may release it. The length excludes the final terminator; NUL bytes within that length remain part of the Acton string. Empty strings and one-byte ASCII strings may reuse immutable singleton values.

For bytes, actBytesFromCStringCopy(s) and actBytesFromCStringLengthCopy(s, n) explicitly copy the input. Existing actBytesFromCString and actBytesFromCStringLength still copy during the API migration; their NoCopy variants share the input. The old string names to$str, to_str_len, and to_str_noc also retain their original behaviour. Use the explicit Copy names wherever independent storage is required, so a future change to the bytes defaults does not change the caller's ownership contract.