N5 attribute paths

%mavenRepo scijava.public https://maven.scijava.org/content/groups/public
%maven org.scijava:scijava-common:2.97.0
%maven net.imglib2:imglib2:6.2.0
%maven org.janelia.saalfeldlab:n5:3.1.2
%maven org.janelia.saalfeldlab:n5-imglib2:7.0.0
%maven org.janelia.saalfeldlab:n5-universe:1.3.1
    
import java.nio.file.*;
import java.util.*;
import java.util.stream.*;
import java.util.concurrent.*;

import com.google.gson.*;

import net.imglib2.*;
import net.imglib2.img.array.*;
import net.imglib2.type.numeric.real.*;
import net.imglib2.view.*;
import net.imglib2.util.*;

import org.janelia.saalfeldlab.n5.*;
import org.janelia.saalfeldlab.n5.imglib2.*;
import org.janelia.saalfeldlab.n5.universe.*;

public static void pathInfo(Path p) {
    try {
        System.out.println(String.format("%s is %d bytes", p, Files.size(p))); 
    } catch(IOException e ){}
}

public static void printBlocks(String path) throws IOException {

    try (Stream<Path> stream = Files.walk(Paths.get(path))) {
        stream.filter(Files::isRegularFile)
            .filter( p -> p.getFileName().toString().matches("[0-9]"))
                .forEach( x -> { pathInfo(x); });
    }
}

Recall that structured metadata attributes can be written to a container using

N5Writer.setAttribute(String group, String key, Object value)

and read using

N5Reader.getAttribute(String group, String key, Class class).

These basics are described in the N5 API Basics Tutorial. In this tutorial, we will show that methods accept more sophisticated expressions for the key that we call “attribute paths.” These enable you to set and access any part of the attribute hierarchy.

Arrays

We’ll start by discussing array attribute indexing. First make some array attribute:

var n5 = new N5Factory().openWriter("attribute-demo.n5");
var group = "arrayDemo";
n5.createGroup(group);

n5.setAttribute(group, "array", new double[]{ 5, 6, 7, 8 });
Arrays.toString(n5.getAttribute(group, "array", double[].class));

[5.0, 6.0, 7.0, 8.0]

Individual elements of the array can be retrieved by adding [i] after the key, where i is an integer (zero-based indexing). N5 will return null for indexes outside the bounds of the array, including for negative values.

n5.getAttribute(group, "array[0]", double.class);  // returns 5.0
n5.getAttribute(group, "array[2]", double.class);  // returns 7.0
n5.getAttribute(group, "array[9]", double.class);  // returns null
n5.getAttribute(group, "array[-1]", double.class); // returns null

This syntax lets you set individual array elements as well:

n5.setAttribute(group, "array[1]", 0.6);
Arrays.toString(n5.getAttribute(group, "array", double[].class));

[5.0, 0.6, 7.0, 8.0]

The array will grow if we set a value outside the range of an array. The array will be filled with zeros if the array is numeric.

n5.setAttribute(group, "array[6]", 99.99);
Arrays.toString(n5.getAttribute(group, "array", double[].class));

[5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99]

n5.setAttribute(group, "array[-5]", -5); // does nothing
Arrays.toString(n5.getAttribute(group, "array", double[].class));

[5.0, 0.6, 7.0, 8.0, 0.0, 0.0, 99.99]

IntStream.range(0, 4).forEach( i -> {
    n5.setAttribute(group, "matrix["+i+"][3]", 0);
    n5.setAttribute(group, "matrix["+i+"]["+i+"]", 2*(i+1));
});

n5.getAttribute(group, "matrix", JsonElement.class);

[[2,0,0,0],[0,4,0,0],[0,0,6,0],[0,0,0,8]]

An array that is not numeric that needs to grown will be filled with nulls.

n5.setAttribute(group, "stringArray", new String[]{"a", "b"});
n5.setAttribute(group, "stringArray[6]", "g");

Arrays.toString(n5.getAttribute(group, "stringArray", String[].class));

[a, b, null, null, null, null, g]

N5’s setAttribute will always do what is requested when possible, even if it will overwrite data. If safety is necessary, developers should manually check if an attribute key is present. Use of the type JsonElement type is the most safe, because a non-null JsonElement will be returned if data of any type is present at the requested key.

// overwrite the previous array
n5.setAttribute(group, "array", new String[]{"destroy"});  // array is now [ "destroy" ]

Arrays.toString(n5.getAttribute(group, "array", String[].class));

[destroy]

if( n5.getAttribute( group, "array", JsonElement.class ) == null )
    n5.setAttribute(group, "array", new String[]{});   // array is still [ "destroy" ]

Arrays.toString(n5.getAttribute(group, "array", String[].class));

[destroy]

Objects

JSON objects are structures with “fields” that can be referenced by their String name. One way to set objects is by using a Map.

var group = "objectDemo";
n5.createGroup(group);

var a = Collections.singletonMap("a", "A");

n5.setAttribute(group, "obj", a ); 
n5.getAttribute(group, "obj", Map.class);

{a=A}

The value for an object’s field can be any type, even another object. Individual fields for an object can be accessed by appending /<field-name> to the attribute name. For example:

var b = Collections.singletonMap("b", "B");

// set the value of obj/a to {b=B}
n5.setAttribute(group, "obj/a", b);
n5.getAttribute(group, "obj", Map.class);

{a={b=B}}

n5.getAttribute(group, "obj/a", Map.class);

{b=B}

Notice that it is possible to repeatedly access subfields of nested objects. In fact, the set of all attributes in an N5 group is usually itself an object! We call it the “root object” and access it with the the key "/"

n5.getAttribute(group, "/", Map.class); 

{obj={a={b=B}}}

For the following examples, we’ll use the class Pet defined here:

class Pet {
    String name;
    int age;

    public Pet(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String toString() {
        return String.format("pet %s is %d", name, age);
    }
}

n5.setAttribute(group, "pet", new Pet("Pluto", 93));
Pet pet = n5.getAttribute(group, "pet", Pet.class);
pet

pet Pluto is 93

n5.getAttribute(group, "pet", Map.class);

{name=Pluto, age=93.0}

n5.setAttribute(group, "pet/likes", new String[]{"Micky"});
n5.getAttribute(group, "pet", Map.class);

{name=Pluto, age=93.0, likes=[Micky]}

Mixing object and array indexing

This example sets the value of an integer inside several nested arrays and objects.

Note: When indexing an array, the path separators / before and after the index operator [ ] are optional

// remove all attributes 
n5.removeAttribute(group, "/");

n5.setAttribute(group, "one/[2]/three/[4]", 5);
n5.setAttribute(group, "one[2]three[0]", 12);
n5.getAttribute(group, "/", JsonElement.class);

{"one":[null,null,{"three":[12,0,0,0,5]}]}

Removing attributes and dealing with nulls

We saw about that removeAttribute can be used to remove attributes. The first variant takes the group and attribute key as arguments, and returns nothing after removal. The second variant also takes a Class argument and will return the removed object of type T if possible. If the value of the attribute cannot be parsed into the requested type, the attribute will not be removed, even if the key exists.

var group = "animals";
n5.createGroup(group);

n5.setAttribute(group, "cow", "moo");
n5.setAttribute(group, "dog", "woof");
n5.setAttribute(group, "sheep", "baa");

n5.getAttribute(group, "/", JsonElement.class);

{"cow":"moo","dog":"woof","sheep":"baa"}

n5.removeAttribute(group, "cow"); // void method
n5.getAttribute(group, "/", JsonElement.class);

{"dog":"woof","sheep":"baa"}

System.out.println( "The doggie says: " + 
    n5.removeAttribute(group, "dog", String.class)
);
n5.getAttribute(group, "/", JsonElement.class);

The doggie says: woof

{"sheep":"baa"}

// throws an exception because the value of "sheep" is not an int
try {
    n5.removeAttribute(group, "sheep", int.class);
}catch(N5Exception e ){
    System.err.println("An exception was thrown");
}

// observe that the attribute was not removed
n5.getAttribute(group, "/", JsonElement.class);

An exception was thrown

{"sheep":"baa"}

Warning

In the default implemenation, setting the value of an attribute to null will remove that attribute (i.e. the attribute’s key will be removed).

However, we strongly recommended using the removeAttribute methods when removing attributes, since setting an attribute to null can lead to inconsistent behaviour, depending on how the N5Writer was created (see below).

Setting an attribute path to null can even result in creating attributes along the path, regardless of whether serializeNulls is enabled or not.

In cases where it is useful to write the value null into the attributes, you must create an N5Writer using a GsonBuilder with serializeNulls enabled. This example writes a null value to the key "attr".

var n5WithNulls = new N5Factory()
    .gsonBuilder(new GsonBuilder().serializeNulls())
    .openWriter("attribute-demo.n5");

n5WithNulls.setAttribute(group, "attr", null);
n5WithNulls.getAttribute(group, "/", JsonElement.class);

{"sheep":"baa","attr":null}

Keys are paths

Think about keys as paths into a hierarchy, where / separates levels of the hierarchy. Attribute methods support relative paths, where . refers to “this” path, and .. refers to the parent path.

var group = "details";
n5.createGroup(group);

n5.setAttribute(group, "a/b/c", "tutorial");
n5.getAttribute(group, "/", JsonElement.class);

{"a":{"b":{"c":"tutorial"}}}

The key a/. is equivalent to a

n5.getAttribute(group, "a/.", JsonElement.class);

{"b":{"c":"tutorial"}}

n5.getAttribute(group, "a/..", JsonElement.class);

{"a":{"b":{"c":"tutorial"}}}

The parent of an array element refers to the array:

n5.setAttribute(group, "array", new String[]{"Alice", "Bob"});
n5.getAttribute(group, "array[0]/..", JsonElement.class);

["Alice","Bob"]

Getting the parent attribute relative to the root will return null

n5.getAttribute(group, "..", JsonElement.class) == null;

true

Warnings and caveats

Warning

We strongly recommend against using / or \ in key names. Similarly, . or .. should not be used in between forward slashes, i.e. avoid (/../ or /./ in key names).

While we recommend against it, is it possible to use forward slash (/) or backslash (\) as field names for attributes. Since / is reserved to refer to the root attribute, it must be escaped with a backslash to refer to the literal string "/".

The code below is not suitable for children, or anyone.

var group = "warnings";
n5.createGroup(group);

n5.setAttribute(group, "\\/", "Please don't do this");
n5.setAttribute(group, "\\", "UGH");
n5.setAttribute(group, ".", "what does this mean!?");
n5.setAttribute(group, "..\\/.", "...pain...");

n5.getAttribute(group, "/", JsonElement.class);

{"/":"Please don't do this","\\":"UGH",".":"what does this mean!?","../.":"...pain..."}