Singularity Hub 2.1 Release

The release of Singularity Hub v2.1 coincides with Singularity v2.5 release, and Singularity Python v2.5. Importantly, we have done the following:

• you will need to update your builders for old collections, under “Settings” in your collection view.
• the builder is updated to give you a larger instance by default (100MB) and more cores (2), the n1-standard-2 instance up from n1-standard-1
• the interface “snack box” and plots in the container view have been removed.

Singularity Hub v2.0

This release of Singularity Hub v2.0 will follow the official release of Singularity 2.4. Thank you to the entire community for ample help and feedback for the new platform. Changes include the following, and we encourage you to report issues when found.

Container Collections

Singularity recipes As was requested by many, this new version will support multiple container recipes in a repository. The location of the recipes is not important, but rather, the naming of the files. All files that start with Singularity will be found recursively. See Singularity Recipes for more details.

Collection Settings We have a new, consolidated settings page that includes builder settings, branch settings, and collection administration. It’s a much more user friendly, and comfortable layout that declutters the collection view and makes things easier to find. See Build Settings for more details.

Branches Branches are no longer the way to specify a tag. The recommended usage for a branch is to distinguish between development commits and production commits, meaning that Singularity Hub is only instructed to build for your production branch. For example, if I tell Singularity Hub to only build from master (typically production), my pushes to feature-1 won’t be built until merged into master. So as before, you can add and remove branches from building in your container collection, and retrospectively you can easily find the branch that was used with the final container metadata. However, the branch is not relevant for the container’s unique resource identifier. In other words, don’t expect to serve two equivalently named images from different branches. To avoid confusion, we recommend that you build from fewer branches. Ideally, just build from a single branch: either master or a production branch.

Tags The extension of the Singularity Recipe file (e.g., Singularity.tag) corresponds now to the tag. A Singularity file without a tag is considered latest. For example, given a repository named hello-world under user vsoch, all of these “different” files will be considered to specify building the container with uri shub://vsoch/hello-world:latest:

hello-world/
    Singularity
    Singularity.latest
    subfolder/
        Singularity
        Singularity.latest

If you have conflicts in terms of names in the same repository, the file that is newest is used to build the image Thus, it is suggested best practice to not have repeated files.

Container Freezes The first version of Singularity Hub would build a new container for every commit, and save it regardless of your good (or poor) container hygiene. To add a level of explicitness to container creation, the default for a container is now to be ephemeral - meaning if I push the same URI twice, the new push will over-ride the first container. However, this does not impact your ability to build and maintain as many versions are important. For reproducibility you have the ability to “FREEZE” individual containers from your container collection view. Given a frozen container, a subsequent build will not override the container, but will create a new container with the updated commit. The frozen container is then best referenced with it’s complete address, eg:

shub://vsoch/hello-world:tag@commit

Deployment Options

With version 1.0, you had one option - to build on each commit or disable entirely. This new version provides several options:

Automated on Commit: is the default and previously used option. All commits are assessed for new recipes, and then a new container is built for the commit. Previous containers with the same unique resource identifier (e.g. vsoch/hello-world:pasta) that are not frozen are re-created. New containers are created given a found frozen container. Conflicts with recipe names in the same repository are resolved by the date of the most recent file. You have the ability to re-trigger any build given that a build parameter has been changed (e.g., debug mode on).

Automated on Deployment: If you use continuous integration to test and then issue a deployment commit, Singularity Hub (when using this option) will only build containers that have the deployment status. All other commits are ignored.

Manual Trigger: If you preference is to connect the repository and manually trigger builds, this is now possible. To deploy manually, the automatic building is disabled.

Cloud Space

We are proud to announce support from Google for a new Singularity Hub cloud space! This will hopefully mean better integration with tools to do cool things with your containers. It also means a substantial amount of work with regard to the infrastructure, broadly including the following:

migration of databases and images in storage the new Singularity Hub will build more secure, smaller squashfs images, and so we are encouraging users to build images afresh without migrating over old images from storage. For those users that wish to keep collections and old images, a form will be sent out to specify this preference, and the migration done carefully (manually) by @vsoch. Containers that are migrated will have a status of Frozen.

Tools and Metadata

Usage and Favorites Metrics are important! To better track container usage, a complete log of downloads via the API is kept, so you can easily see how many requests have been issued to use your containers. To assess favorites, we now give each collection a star, and as a user you can star your favorite.

Labels The labels that are defined for your container, along with the environment, are extracted at build time and added to the Singularity Hub database. This means that you can compare labels across different container collections, along with search for terms of interest.

[[https://github.com/singularityhub/singularityhub.github.io/blob/master/img/labels.png]]

Apps With Singularity 2.4 we introduced Standard Container Integration Format apps, or another level of modularity internal to the container to allow for multiple entrypoints and modular environments. Apps are now fully integrated into Singularity Hub, so if you build a container with them, they will be discovered.

Container Size Treemap With Singularity Hub 2.0 we are adding a view to show a robust treemap to compare relative size of container collections across the Hub. This is marginally for fun, and importantly is a reminder to developers that the size of the container is an important (and commonly overlooked) factor.

Container Queue

You are still limited to one running build for your collection queue, and five pending and or running jobs. This fits most of our user’s use cases, and if you have a need to build at scale we recommend you check out deploying a Singularity Registry.

Migrations

All previous images and metadata are safe, and will be available until the end of the calendar year. The subdomain is quite a fish and you can download and use your previous containers by simply appending the registry (doc.fish) first.

shub://<registry>/<namespace>/<repo>

For example, the image bauerm97/shubtest exists on the legacy Singularity Hub, but not the new version. To reference it, I simply add the registry:

`singularity pull shub://doc.fish/bauerm97/shubtest
Progress |===================================| 100.0% 
Done. Container is at: bauerm97-shubtest-master.simg

If you are using an older version of Singularity to pull, you might have an issue with the format of the manifest used to name the image. For this you can try any of the following:

singularity pull --name baumerma.simg shub://doc.fish/bauerm97/shubtest
singularity pull --commit shub://doc.fish/bauerm97/shubtest
singularity pull --hash shub://doc.fish/bauerm97/shubtest

These images will be usable until the end of the calendar year. If you have an image you want transferred to the new Singularity Hub, please let us know. Legacy images migrated to the new site will maintain their previous Singularity versions, etc.

Summary and Minor Details

In summary:

• build logs are now download-able (thanks @yarikoptic)
• a single repository hosts multiple container builds, without needing to touch branches (@markeganfuller)
• container usage instructions are easier to find, with the collection (thanks @HenrikBengtsson)
• container builds have a maximum build time of 2 hours, and then are killed (@cclerget).
• unfrozen containers can be rebuilt (@anderbubble) with a trigger (@TormodLandet)
• containers can be frozen for controlled release tagging (@sven1103)
• Github permissions only ask for what is needed, writable webhooks (thanks @olifre). For those that want private repos, this extra level can be selected at login time.
• improved search interface to search over collections, container tags, labels, and apps
• supplementary material for collections can include asciinema, youtube, or external documentation links

Please post an issue if you have questions or concerns! I’ve (@vsoch) put a lot of heart into making this a useful tool for the Singularity community, and want to hear your feedback. Now what are you waiting for? Go check it out!