1.1   Glossary¶

• JupyterLab is an extension of the Jupyter notebook platform that provides an extensible workspace in which users build custom interactive science workflow environments.
• JupyterHub is a server for spawning Jupyter notebooks and now JupyterLab environments.
• Kubernetes is a container orchestration platform.
• GKE (Google Container Engine) is a cloud-hosted kubernetes service.
• Story Point: A DM unit of work; there are 7.5 SPs in a typical developer week.

1.2   SQuaRE Technical Status¶

At the most recent point in the timeline above, we have:

• An experimental sandbox at jupyterlabdemo.lsst.codes (not a stable service; frequently redeployed with no notice; functionality subject to change).
• A Dockerfile for installing JupyterHub and JupyterLab components onto our LSST stack container and exposing the stack kernel to Jupyter.
• A Kubernetes configuration to deploy this service on the GKE cluster.
• An environment-driven or interactive process for deployment of a JupyterLab demo cluster.
• Our deployment involves JupyterHub spawning JupyterLabs in individual pods based on a SQuaRE’s CI-built Docker container distribution of the stack.
• Authentication to this service via SQuaRE’s usual “shim” authentication mechanism that uses GitHub organization membership for OAuth authentication
• Authentication via OAuth2 to NCSA via the CILogon service.
• Consistent Unix uid/gid mapping for users using GitHub as a shim (your user id is your github id, your groups are the orgs you belong to).
• We have a modest amount of persistent storage via an NFS pod to allow alpha-testers to keep notebooks around in between redeployments.
• A tool to do one-command deployment of a cluster; this tool allows customization either through environment variables, command-line flags, or by generating a set of Kubernetes configurations which are then available for manual editing before deployment.

1.3   Timeline Overview¶

2017-03:

In the S17-B cycle (March-May 2017) We allocated a timebox of 30 story points (4 person-weeks) to get SQuaRE familiar with the JupyterLab environment. JupyterLab is still in alpha but after talking extensively to one of the leaders of the project (Brian Granger) in February we felt the platform was mature enough to try prototyping with and the seemingly best techological target for the Science Platform. This work is covered under the SQuaRE epic DM-9984.

2017-04-13:

At this point (2017-04-13) Adam Thornton, who is SQuaRE’s point engineer for Jupyter activities, has spent 19 SPs on this epic. Our technical target was to produce a JupyterLab environment provisioned by the automatic weekly build of a Docker container distribution of the LSST stack (lsst_distrib). Allowances were made regarding JupyterLab’s alpha status. Our overall goal was to give ourselves a JupyterLab environment where we could prototype Science Platform user services (e.g. Notebooks as documentation) and verification activities (Notebook verification reports).

2017-04-27:

We are closing the first DM-9984 after 30 SPs and opening DM-10387 to continue. We now have a workable deployment for alpha-testers (only lacking persistant storage) for (relative) stability and an architecture that includes a kubernetes deployment of Hub spinning off Labs in individual pods (for scaling), shim OAuth, shim Unix uid/gid, and a kernel based on a Docker container automatically produced by CI. Oh and this technote. KSK out of sheer enthusiasm makes a set of killer demo videos based on a stack notebook and DS9 visualisation.

2017-04-28:

We are now producing Docker Python 3 containers from CI and deploying them to our Lab. The KSK notebook ran great. Investigating a bug in the auth context for spun pods.

2017-05-02:

The “missing icons” and “missing Hub menu” issue turned out to be an issue mixing-and-matching node.js modules across packages (which is a known issue upstream and the plugin system is being revisited as a result).

2017-05-09:

We have added a modest amount of persistent storage (NSF pod) to avoid inconveniencing alpha-testers with perennially re-uploading their notebooks and data sets in between redeploys. We have also added a log out button and improved server shutdown.

2017-05-09:

The demo instance is up to date with the features in the sandbox instance.

2017-06-15:

We can provide a multiple choice of containers to users (eg. with different versions of the stack). This meant shifting the kernel selection step up into Hub (before it was at the Lab pod level).

2017-06-30:

We turned on autoscaling as a test. Everything looks good. As threatened we removed the Python 2 containers now that we have multiple stack versions to test the selector with.

2017-07-07:

We have upstreamed our first patch into JupyterLab :-) It’s great to be giving back. Keeping up with the alpha updates is a bit of work right now, but well worth it.

2017-08-16:

Another pull request, this time for OAuth2 and GitHub, to allow token caching, is approved and waiting for merge. We now no longer need any repositories other than lsst-sqre/jupyterlabdemo and jupyterlab-savequit–our authenticator, spawner, and login handler subclasses are handled in JupyterHub configuration.

2017-12-13:

Much work has happened since the last update. A prepuller DaemonSet has been added to keep downloaded container images up-to-date, and the list presented at login time is automatically updated from the repository container set. We have integration with NCSA OAuth2 via CILogon (although there is coordination required for UID/GID determination). There is now an automated deployment process that can use either interactive prompts, a YAML document, or environment variables to drive the deployment. A basic directory layout for LSST-provided notebooks has been agreed to. A repository containing standard notebooks has been defined, and it is populated/refreshed for each container at login time. Workshops and similar things will, for the time being, use GitHub organizations to access control. All currently appropriate patches have been accepted upstream.

2018-02-26:

The prepuller has been rewritten to not require mapping the Docker socket, which makes it able to run without privilege and to be independent of the underlying host OS. Systems Engineering and EPO are now running minor variants of the Demo environment for their own needs. JupyterLab Beta has been released and we have adopted it. Bokeh accepted our PR to jupyterlab_bokeh, so that’s another upstream project we are now contributing to. We are on track to close about another 40 story points on this epic at the end of the month.

1.4   Coming Soon-ish¶

• Chain together OAuth providers, so that we can use NCSA for go/no-go decisions, but still consult GitHub to get a token for magic HTTPS pushes and git configuration.
• Investigate addressing usability concerns (github-based workflows)
• Migrate our configuration to a Helm chart. We may at that point attempt to more closely track what Zero To JupyterHub does.

1.5   Repositories¶

Code repos for system:

https://github.com/lsst-sqre/jupyterlabdemo :

(JupyterLab container provisioning and Kubernetes cofig)

https://github.com/lsst-sqre/jupyterlab-savequit :

(JupyterLab Save-and-Exit menu)

1.6   Related¶

https://github.com/lsst-dm/sciplat-examples :

(Some example demos contributed by DM science staff)

https://github.com/lsst-dm/tutorial-lsst2017 :

(Large tutorial example from LSST All-Hands 2017)

https://github.com/lsst-sqre/notebook-demo :

(Placeholder for automatically updated notebooks for Lab environment)

2.1   Major Issues¶

• Interface to the EFD

  It is a primary requirement of the commissioning team to cross-correlate image data with readouts stored in the EFD. These are later ingested in the main database and can be accessible via DAX. However we strongly feel that for usability reasons (and managing the limited resources of the commissioning cluster) the same notebook should be able to run at the datacenter Jupyter deployment that can run on the commissioning cluster and vice versa. The two obvious ways of achieving this are:

  • request a generalised python interface to both;
  • have the EFD data be availabe through the DAX in timescales short enough to satisfy the commissioning team and only support one way of accessing EFD data.

  We strongly favour the latter approach but it has implications on other parts of the DM Subsystem that need to be discussed.

• Intereface to the batch system: Right now the baseline is that some kind of user intervention will have to happen to go from a satisfactory notebook to running the same code over large datasets through the batch system. This presents significant usability challenges. If the workflow system would present an interface that allows optimized idempotent execution of notebooks (“you just asked me to do a job with this configuration and these inputs; I recognise that I have already executed such a job so I will return the results to you right away as a no-op”) the usability will be vastly improved. We do not know whether such an interface can be provided at this stage.

2.2   Deployment and Scaling¶

• We need to settle on a system for managing our kubernetes applications. This is likely to be Terraform for virtual machine and external DNS provisioning, and Helm for kubernetes configuration. Some custom scheduling logic will be needed, because in configuration of the cluster, some later steps depend on values not generated until earlier steps have complted.
• Integration with datacenter-side persistent storage (GPFS?). This currently seems to be more likely to be GPFS-exported-as-NFS.
• Integration with datacenter-side auth
  • map of GitHub ID to NCSA ID (identity mgt)
    • Ideally we just chain OAuth2 providers and pass GitHub token information along with the CILogon-provided UID/GID data.

3.1   The JupyterLab Platform and Verification¶

• “New face of SQuaSH” interface: Following the adoption of the JupyterLab Platform and the involvement of SQuaRE’s WBS, we need to consolidate the functionality of the front end that is currently being served by the Django portal into the JupyterLab platform as much as possible as we don’t have sufficient effort to maintain two different user interfaces, and the JupyterLab one is likely to be superior in functionality. However we have not yet investigated dashboarding under JupyterLab and might revise this plan.

  We have demonstrated that SQuaSH can run successfully in a kubernetes cluster, just as the JupyterLab platform can. Some work has been done to make Bokeh widgets compatible with JupyterLab, but much remains.

• Telemetry Gateway: while this is not currently an interface to JupyterLab (but rather to SQuaSH), in the event that notebook execution is used to compute metrics that are needed at the summit, the same mechanism that is used for SQuaSH may be required here. Potentially this uncovers an interface to the Telemetry Gateway but we are not certain at this point.

3.2   Integration with Developer/User Services¶

• Verification report generation/publication harness - (LtD support for notebooks)
• Production hardening: During commissioning rapid partial or whole re-deployment of assets is likely to be needed frequently and/or at short notice. While we are designing with this in mind, we have a target date for demonstrating this capability and improving on any bottlenecks (which may be in other components, in particular the CI chain).

3.3   Milestones¶