Singularity Global Client: Google Storage

These sections will detail use of the Google Storage client for sregistry, which is a connection to a Google Storage cloud bucket. If you are interested in using Google Build and then saving to storage, see the google-build client.


Getting Started

If you are using the sregistry image, the client is likely already installed. If you want to install this natively (or build a custom container) the command to install the module extras is:

pip install sregistry[google-storage]

# or locally
git clone
cd sregistry-cli
pip install -e .[google-storage]

The next steps we will take are to first set up authentication, and then define your Storage Bucket (and other settings) via environment variables.


Singularity Registry Global Client works by way of obtaining information from the environment, which are cached when appropriate for future use. For Google Storage, you will first need to set up authentication by following those steps. It comes down to creating a file and saving it on your system with the variable name GOOGLE_APPLICATION_CREDENTIALS. This variable will be found and used every time you use the storage Client, without needing to save anything to the secrets.

Thus, the complete list of this required variable (and other options, with defaults shown) are the following:

Notice that the first variable is not prefixed with SREGISTRY_ and this is because it is already defined for the Google namespace, and use by sregistry.

For a detailed list of other (default) environment variables and settings that you can configure, see the getting started pages. For the globally shared commands (e.g., “add”, “get”, “inspect,” “images,” and any others that are defined for all clients) see the commands documentation. Here we will review the set of commands that are specific to the Google Storage client:

For all of the examples below, we will export our client preference to be google-storage


but note that you could just as easily define the variable for one command:

SREGISTRY_CLIENT=google-storage sregistry shell

or do away the need to export this environment variable by simply activating the client:

$ sregistry backend activate google-storage
[activate] google-storage
$ sregistry backend status
[backend status]
There are 9 clients found in secrets.
active: google-storage


After we have exported SREGISTRY_CLIENT above, if you are looking to interact with a shell for the google-storage sregistry client, just ask for it:

sregistry shell
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:53:06) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 

Here we see straight away that we are using the default bucket name (sregistry-vanessa) and the google-storage client. The printing of the bucket on the first line indicates we successfully connected to it.


If you don’t have any images in your bucket, that is probably a good start to add some. In this case we will add an image sitting in our present working directory.

sregistry push --name vsoch/hello-world:latest vsoch-hello-world-master-latest.simg
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
[container][update] vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f

You will see again the connection to the bucket, and then a progress bar that shows the status of the upload, and the progress bar is replaced by the final container url.


Of course you can inspect an image (here we will inspect the image we just pushed above), and you will see a ton of goodness:

$ sregistry inspect vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
    "client": "google-storage",
    "collection": "vsoch",
    "collection_id": 1,
    "created_at": "2017-12-29 00:05:09",
    "id": 1,
    "image": null,
    "metrics": {
        "attributes": {
            "deffile": "Bootstrap: docker\nFrom: ubuntu:14.04\n\n%labels\nMAINTAINER vanessasaur\nWHATAMI dinosaur\n\n%environment\nDINOSAUR=vanessasaurus\nexport DINOSAUR\n\n%files\ /\n\n%runscript\nexec /bin/bash /\n",
            "environment": "# Custom environment shell code should follow\n\nDINOSAUR=vanessasaurus\nexport DINOSAUR\n\n",
            "help": null,
            "labels": {
                "MAINTAINER": "vanessasaur",
                "WHATAMI": "dinosaur",
                "": "2017-10-15T12:52:56+00:00",
                "": "333MB",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.usage.singularity.deffile": "Singularity",
                "org.label-schema.usage.singularity.deffile.bootstrap": "docker",
                "org.label-schema.usage.singularity.deffile.from": "ubuntu:14.04",
                "org.label-schema.usage.singularity.version": "2.4-feature-squashbuild-secbuild.g780c84d"
            "runscript": "#!/bin/sh \n\nexec /bin/bash /\n",
            "test": null
        "branch": "master",
        "bucket": "sregistry-vanessa",
        "collection": "vsoch",
        "commit": "e279432e6d3962777bb7b5e8d54f30f4347d867e",
        "contentType": "application/octet-stream",
        "crc32c": "1FCMGQ==",
        "etag": "CIHy5PnTstgCEAE=",
        "generation": "1514668522289409",
        "id": "sregistry-vanessa/vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f.simg/1514668522289409",
        "image": "hello-world",
        "kind": "storage#object",
        "md5Hash": "7ZdVoIcfBNs+FJcb7FajPw==",
        "mediaLink": "",
        "metageneration": "1",
        "name": "vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f.simg",
        "selfLink": "",
        "size": "65347615",
        "size_mb": "333",
        "storage": "vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f.simg",
        "storageClass": "STANDARD",
        "tag": "latest",
        "timeCreated": "2017-12-30T21:15:22.270Z",
        "timeStorageClassUpdated": "2017-12-30T21:15:22.270Z",
        "type": "container",
        "updated": "2017-12-30T21:15:22.270Z",
        "uri": "vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f",
        "version": "ed9755a0871f04db3e14971bec56a33f"
    "name": "hello-world",
    "tag": "latest",
    "uri": "vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f",
    "url": "",
    "version": "ed9755a0871f04db3e14971bec56a33f"

Now let’s say that we pushed an image a while back to Google Storage, we have the record locally, and we want to get it. We could use “get” to get the url and then plug it into our special download logic, or we could instead want to pull the image to our local sregistry database. This would mean that the “remote” record gets updated to “local” because we actually have the image! How do we do that? We will go through two scenarios. In the first, we’ve totally forgotten about our local database (or it blew up) and we need to search the remote endpoint. In the second, we have our local database and just want to get one (pull).

A search without any parameters will essentially list all containers in the configured storage bucket. But how do we know what is a container?

a container is defined by having the metadata key “type” with value “container” and this is set by the upload (push) client.

Thus, if you do some fancy operation outside of using the client to upload containers to storage, make sure that you add this metadata value, otherwise they will not be found. Let’s do a quick search to get our list in Google Storage. This action has no dependency on a local storage or database.

$ sregistry search
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
[gs://sregistry-vanessa] Containers
1      174 MB	/home/vanessa/desktop/expfactory:latest
2       62 MB	vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f

Then to look at details for a more specific search, let’s try searching for “vsoch”

$ sregistry search vsoch
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
[gs://sregistry-vanessa] Found 1 containers
id:      sregistry-vanessa/vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f.simg/1514668522289409
uri:     vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f
updated: 2017-12-30 21:15:24.132000+00:00
size:    62 MB
md5:     7ZdVoIcfBNs+FJcb7FajPw==


With pull, we might have done a search to find a container that we liked, as shown above, and we want to retrieve it. In this case, instead of inspect or get, we just use pull.

$ sregistry pull vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f
[client|google-storage] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
Searching for vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f in gs://sregistry-vanessa
Progress |===================================| 100.0% 
[container][update] vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f
Success! /home/vanessa/.singularity/shub/vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f.simg

and if we do a get, we get the path to the file:

$ sregistry get vsoch/hello-world:latest@ed9755a0871f04db3e14971bec56a33f

You can also see in the pull output that on the backend of pull is the same search as you did before. This means that if you want to be precise, you should ask for the complete uri (version included). If you aren’t precise, it will do a search across name fields and give you the first match. Be careful, my linux penguins.

For debugging scripts, see Google Cloud Debugging.