I’ve had a bit of a slump the last few months, working through what feels like deep mental fog for totally unclear reasons. The lack of clarity on causation is one of the most frustrating parts of this, and one of the most familiar too - this isn’t the first time I’ve felt like this, and I know (rationally at least, if not emotionally) it won’t be the last. Probably a combination of intensely avoiding making some decision that seems too big to be made by me without actually knowing what it is and fun brain chemicals stuff. Who knows, I’ve given up on trying to figure out causality for these things for the most part.

One of the effects has been a distinct lack of drive, at least to the extent I am used to. Another is that it is difficult to remember being in a different emotional state in the past, and difficult to imagine being in a different emotional state in the future. Time dissolves, and I believe how I feel now is how I have always felt, and how I will always feel.

Hence, am trying to make external records any time I feel or encounter something that seems to give me a good boost, and a sense of ‘wanting to do it again’. That in a loop feels like drive to me - A feeling induced in you by external stimuli, and then you wanna find things to do to feel that again.

I felt some of that a few days ago! I built something, put out a quick message about what it is that I had built on a slack. Someone responded with this message:

And that felt really good! I could see how responses like this have driven most of my software development projects, for good and ill. It has been a driver in picking which communities I can be a part of, what I find interesting, and what feels like a slog. I joked to a friend that my ‘anxiety response’ is to start (or get deeply involved in) a new open source projects, but the more I think about it the more it seems to be not a joke at all.

Not unique to myself in any way or form I would assume. But feels great to know this is true for me, and I have come to value explanations for my emotions quite a bit of late :)

Anyway, great to remember that I can feel this!

I walked into a local hardware store, with the following thought in my mind:

I need a 5mm metric hex key

And I wandered around aimlessly as I do in stores, almost bought some double sided tape for a different project (decided against it), and eventually made it to the place with all the sockets, hex keys, screwdrivers, etc.

Words that previously meant nothing to me (Torx, SAE, metric) made sense. Felt a little like the very first time I felt power learning to write code (the strongest memory is that of changing color of a button in VB6 with one line of code). What was noise before had meaning now.

I got myself a little folding metric key set with hex keys going from 1mm to 6mm - I knew I had some new windscreen clip-ons that needed a 4mm key, and am sure I’ll find a 6mm somewhere.

I pick up something that costs 13$, and looks ok. I look at a different brand, and decide I like that better because of the way it folds. “Hey, this means when I try to turn it, I’ll get better leverage!”. I actually know what that sentence means, and I choose one brand over another based on actual knowledge rather than a guess. If my programming experience is anything to go by, I’m completely wrong in ways I won’t even understand for a long time.

But I buy it, leave the store, and try using it for intended purpose on my motorcycle - to remove the bolts holding my bar end weights in. Earlier that week, I had stripped these by attempting to use a friend’s impact driver with a torx bit (it sort of fit, and I had no mental model about what I was doing!). The head stripped, and the thread stripped, and I took it to Performance Motorcycles in Berkeley who fixed it up for me (although the bolt itself did break and they had to put a new one in).

I had since beat myself up about not knowing enough about hardware to not strip the head, been comforted by a friend, realized that I had set weird standards for myself where somehow I must already know all of this having never put in any effort to learn, and then being delighted that ‘oh wait this is just another skillset I can pick up’. It will probably take years, but that’s ok!

Earlier that day, I had went back to the mechanic and asked him exactly what screw he had used for the bolt. He told me it was a 5mm hex, and I should use an exact one or I will strip.

I extend the 5mm key, and fit it in. It seems to be a perfect fit! I try twisting left, repeating internally to myself “Lefty loosy”. It doesn’t budge, and I’m slightly anxious. But I trust that it is not super likely to strip at this point - I’m not a very strong person, and this I fully believed was the correct size. So I put more energy into it.

AND THE BOLT TURNS! I FEEL ELATED! AMAZING JOY! It is a bit like seeing your code run for the first time, AND IT FEELS AWESOME!

So excited for this future :)

If you are running a JupyterHub on Kubernetes and use NFS for home directory storage (a common occurance), I highly recommend the following settings:

singleuser:
  extraEnv:
    # notebook server writes secure files that don't need to survive a
    # restart here. Writing 'secure' files on some file systems (like
    # Azure Files with SMB or NFS) seems buggy, so we just put runtime dir on
    # /tmp. This is ok in our case, since no two users are on the same
    # container.
    JUPYTER_RUNTIME_DIR: /tmp/.jupyter-runtime
  extraFiles:
    ipython_kernel_config.json:
      mountPath: /usr/local/etc/ipython/ipython_kernel_config.json
      data:
        # This keeps a history of all executed code under $HOME, which is almost always on
        # NFS. This file is kept as a sqlite file, and sqlite and NFS do not go together very
        # well! Disable this to save ourselves from debugging random NFS oddities that are caused
        # by this unholy sqlite + NFS mixture.
        HistoryManager:
          enabled: false

These aren’t specific to kubernetes, but to NFS (or AzureFile, or any other shared filesystem). sqlite and NFS do not mix, and you’ll run into weird errors you will not be able to really debug. Save yourself the pain :) Let your users know too, and tell them to not put sqlite databases on NFS.

Alex Merose asked me on Mastodon how to best set up a conda environment on GitHub Actions. I thought I’d write a short blog post about it!

The short version is, use the conda-incubator/setup-miniconda GitHub action instead of the official setup-python action. Specifically, try out these options for size:

  steps:
    - uses: conda-incubator/setup-miniconda@v2
      with:
        # Specify python version your environment will have. Remember to quote this, or
        # YAML will think you want python 3.1 not 3.10
        python-version: "3.10"
        # This uses *miniforge*, rather than *minicond*. The primary difference is that
        # the defaults channel is not enabled at all
        miniforge-version: latest
        # These properties enable the use of mamba, which is much faster and far less error
        # prone than conda while being completely compatible with the conda CLI
        use-mamba: true
        mamba-version: "*"

This should give you a python environment named test with python 3.10 (or whatever version you specify) and nothing much else. You can then use mamba to install packages from conda-forge to your heart’s content in future steps like this:

    - name: Install some packages
      # The `-l` is needed so conda environment activation works correctly
      shell: bash -l {0}
      run: |
                mamba install -c conda-forge numpy scipy matplotlib

My recommendation is to continue using the default setup-python action for almost all your python needs in GitHub actions, and use conda-incubator/setup-miniconda@v2 only if you explicitly need to use mamba (or conda) for something.

I am not smart enough to consistently write and debug shell scripts that use any conditional or looping constructs. So as soon as I’m writing something in bash that requires use of those, I switch to python. This works fine when writing scripts, but what to do when writing GitHub Actions workflows? You can only write bash in run: stanzas in your steps, right?

Not at all! You can set the shell parameter to anything you want, and the contents of run will be passed to the shell in the form of a file. This allows you to use not just Python, but basically any langauge when writing your GitHub actions workflows.

Here is an example step that used python.

  steps:
    - name: Something in python!
      # The -u means 'unbuffered', so print() statements in your python code are output correctly
      # otherwise, they might be out of order with stdout from commands your code calls
      # {0} is replaced with the name of the temporary file GitHub Actions creates with
      # the contents of the run:
      shell: python -u {0}
      run: |
        import sys
        import subprocess

        print("Hello, I am python")

        # We have to use string substitution for getting inputs, which is bad and ugly
        # However, it isn't as bad me trying to write conditionals in bash.
        # It might be possible to use environment values here, I haven't explored.
        variable = '${{ inputs.some_input }}'

        # Use subprocess.check_call to call out to external process. stdout is
        # handled correctly
        subprocess.check_call([
          sys.executable,
          '-m',
          'pip', 'install', 'django'
        ])        

If you instead use mamba or conda to set up your Python, perhaps with the setup-miniconda action, you need to set shell: bash -l -c "python -u {0}". The bash -l makes sure that the appropriate conda environment is loaded, and then passes off to the ‘correct’ python.

I decided to write publicly and publish my motorcycle riding trip notes now.

Montarey Trip notes

Need to learn to plan when to pee. This is important to do! Can’t just get off any exit, as that often doesn’t actually lead to wilderness but to more concrete jungles.

Need to learn more about lane splitting and how and when to do it safely. I did it for a little bit while stuck on CA-1S, but was avoiding it for most of the ride. Feels like it’s only safe to do when traffic is almost at a standstill, and I can go at least 5mph faster than traffic. Also, I don’t wanna get hit by other motorcyclists lanesplitting. More research needed.

I need to adjust my levers to see if it helps with my skin near thumb getting sore.

Get a different seat maybe for more comfort? My butt does hurt and is sore.

I definitely need to feel much more confident in curves. Taking the TC class is a good next step.

DO NOT BELIEVE GOOGLE MAPS TIME ESTIMATES!! I am much slower than that haha.

I need to work on my core strenght, back to the Gym. I think without that, long rides are going to be painful.

Need to plan meals better too. I’m eating ’lunch’ at like 4:30, after having only eaten 3/4 cup of oats and 1 butter croissant all day. That isn’t enough. Partially this is because I couldn’t leave on time either - as usual. Still meant I left at 12 rather than 2 - but could’ve left at 10!

Post trip notes

This is at night. The ride back was all interstate, and quite different!

1. Food makes a huge difference. I think I need to eat more than I think too. Just everything was better after food. I should just always eat, and eat on time.

2. I am getting more confident about turns at speed. I think by default I end up slowing down for each turn from fear. That is pretty dangerous! I need to have a better understanding of how much traction is available, and just turn. Which way to lean is also important - but I think my current intuition (counter lean or neutral) is good enough for me. I saw some guy shred the twisties when I was up in the berkeley hills, and he had to lean quite a bit. Not what I’m doing though.

   I should take more classes and read the total control book properly. Towards the end I was feeling pretty confident, and that was quite nice!

3. The way back was north, with the sun directly into my eyes for almost an hour. Was intense, I could barely see! I had sunvisor down and transition glasses and that wasn’t enough at all. I was in a trance. Looking back, how did I trust myself enough to go through that?! Would I have been able to see hazards on the road? Stopped cars? I was able to control speed pretty well. I was a bit tired too. But I did Montarey to San Jose in one go, and stopped - my body basically forced me to stop and rest. I don’t entirely know how I feel about this. I think it was mostly safe, as it was just on the interstate and I was on the left-ish lane. I also think I could actually mostly see - what I see in my memory now, hours after the fact, doesn’t seem quite right - feels filtered. It really was just an intense trance. I don’t mind interstate riding.

4. I should strap my bag down and not carry it! I think that will definitely help my back hurt a lot less. I had 2 bottles of water, laptop and ipad - I think that’s a lot. I should consider a smaller bag too.

5. A more comfortable seat would also help. My butt was pretty sore.

6. Physical strength is an important component of safety, and I must get back to the gym. Particularly, while turning I could feel my core was not strong enough several times, and I would like more trust in my muscle strenghts.

7. I lane-split a little! Just for parts of CA-17S, when in otherwise almost stopped traffic. I need to learn way more about it.

8. I should understand trail braking better. I think my understanding of it right now is dangerously flawed, as it contributes to me slowing down excessively in turns. Could get rear ended.

9. Really need a windshield. I can hardly even look at my mirrors in the interstate, because I can’t bring my torso up at all.

10. I drifted into next lane almost in one turn I think because I wasn’t braking and I was tired. I was trying to ‘push through’ and at some point even didn’t want to break in San Jose. WTF was that, compromising safety for challenge! No way.

11. I need to leave earlier so I can have more time at my target spaces. In fact, I should look for nice target spaces first and then travel - flip my currnet model a little. I don’t think I need to just haul miles anymore. Past that stage now.

12. The bananas in my backpack were smushed to death and the inside of my backpack is a banana smoothie. Would not recommend

TODOs

Concrete action items!

1. Always eat before a ride
2. Get a better helmet. I’ve spent months looking for the right one, time to look again :(
3. Start going through Champ U that I already bought
4. Look for another intermediate mototrcycling course I can take.
5. Plan time at conclusion of journey better.
6. Get a windshield for the bike.
7. Experiment with getting luggage off my back

I wanted a single keyboard shortcut that would let me switch between the built-in terminal and the editor in vscode. I couldn’t find any, so I made a short macro using the VSCode Macros Extension

First, you define your macros in settings.json. You can open up vscode settings with Cmd+,, search for macros, and edit this under Macros: List.

    "macros.list": {
        "showTerminal": [
            "workbench.action.terminal.toggleTerminal",
            "workbench.action.terminal.focus",
            "workbench.action.toggleMaximizedPanel"
        ],
        "hideTerminal": [
            "workbench.action.toggleMaximizedPanel",
            "workbench.action.terminal.toggleTerminal"
        ]
    }

Then you can bind a keyboard shortcut to it. Open Preferences: Open Keyboard Shortcuts (JSON) from the command palette, and add this:

    {
        "key": "cmd+t",
        "command": "macros.showTerminal",
        "when": "!terminalFocus"
    },
    {
        "key": "cmd+t",
        "command": "macros.hideTerminal",
        "when": "terminalFocus"
    },

showTerminal gets called when you are focused anywhere in the editor except your terminal, and hideTerminal gets called when you are focused on your terminal. This gives me the ’toggle’ behavior I want.

Sometimes you want to temporarily disable a JupyterLab extension on a JupyterHub by default, without having to rebuild your docker image. This can be very easily done with z2jh’s singleuser.extraFiles, and JupyterLab’s page_config.json

JupyterLab’s page_config.json lets you set page configuration by dropping JSON files under a labconfig directory inside any of the directories listed when you run jupyter --paths. We just use singleuser.extraFiles to provide this file!

singleuser:
  extraFiles:
    lab-config:
      mountPath: /etc/jupyter/labconfig/page_config.json
      data:
        disabledExtensions:
          jupyterlab-link-share: true

This will disable the link-share labextension, both in JupyterLab and RetroLab. You can find the name of the extension, as well as its current status, with jupyter labextension list.

jovyan@jupyter-yuvipanda:~$ jupyter labextension list
JupyterLab v3.2.4
/opt/conda/share/jupyter/labextensions
        jupyterlab-plotly v5.4.0 enabled OK
        jupyter-matplotlib v0.9.0 enabled OK
        jupyterlab-link-share v0.2.4 disabled OK (python, jupyterlab-link-share)
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)
        @jupyter-server/resource-usage v0.6.0 enabled OK (python, jupyter-resource-usage)
        @retrolab/lab-extension v0.3.13 enabled OK

This is extremely helpful if the same image is being shared across hubs, and you want some of the hubs to have some of the extensions.

singleuser.extraFiles can be used like this for any jupyter config, or generally any config file anywhere. For example, here’s some config that culls idle running kernels, and shuts down notebooks after 60m of inactivity:

singleuser:
  extraFiles:
    culling-config:
      mountPath: /etc/jupyter/jupyter_notebook_config.json
      data:
        NotebookApp:
          # shutdown the server after no 30 mins of no activity
          shutdown_no_activity_timeout: 1800

        # if a user leaves a notebook with a running kernel,
        # the effective idle timeout will typically be CULL_TIMEOUT + CULL_KERNEL_TIMEOUT
        # as culling the kernel will register activity,
        # resetting the no_activity timer for the server as a whole
        MappingKernelManager:
          # shutdown kernels after 30 mins of no activity
          cull_idle_timeout: 1800
          # check for idle kernels this often
          cull_interval: 60
          # a kernel with open connections but no activity still counts as idle
          # this is what allows us to shutdown servers
          # when people leave a notebook open and wander off
          cull_connected: true

This post is inspired by a slack conversation with Ryan Abernathey and Sarah Gibson.

It is very important to me that we don’t exist in a world where the Internet is just centralized and controlled by a few huge, hypercapitalist players. I’ve written about ways the open source community can help here (by not tying ourselves to a single provider), and helped draft the right to replicate policy to put it into effect at 2i2c. This topic is important to me - I don’t want a future where step 1 of learning to code is something extremely vendor specific.

Theoretically, this is very well within the scope of the ‘web3’ movement (quite different from the web 3.0 movement which has mostly died out). I would love for that to be the case, but unfortunately I’ve found almost nothing about the ‘crypto’ or ‘web3’ communities attractive. I’m going to try write down why in this post.

Most of this isn’t unique thought - others have written about these with much more citations, and deeper analysis than I have. I encourate you to seek those out and read them. This is also not an exhaustive list, but a bare minimum.

Environmental impact of proof of work

Difficult for me to get past the environmental impact of most proof of work systems. Any renewable energy they use is energy that could be used to take more fossil fuels off the grid. I don’t want proof of work systems to switch to renewable energy - I want them to stop existing.

The rich get richer

In the economic models of almost all ‘crypto’, growing inequality is a feature, not a bug. This is outright formalized in Proof of Stake systems, and is de-facto true in most Proof of Work systems. I don’t really want to live in a world that is even more unequal than the one I live in now. Fixes to this would be fundamental to any possible positive outcome from the ‘crypto’ community.

“Distributed Ledger” is not a solution to many problems

The buzzwordiness of ‘blockchain’, ‘crypto’ and ‘web3’ mean they’re now being forcefit into problems they are very poor solutions for. Every time I see the word ‘blockchain’, I mentally replace it with ‘distributed ledger’, and see if it can still be a good fit for the given problem. Most often, it does not. Problems often have to be reframed in terms of ’transactions’ to have a blockchain solve them, and I think this reframing is often fundamentally problematic - life isn’t transactional. When all you have is a distributed ledger, everything looks like a problem of incentives.

So any time something purports to solve a problem that is not use as a currency using a blockchain, consider carefully the problem it is solving, and seeing if the solutions it is proposing actually solve the problem in a way that is generally usable given latency and throughput consideration.

Let’s evaluate the tech on a case-by-case basis

The fediverse is quite cool. I think of IPFS as the spiritual successor to bittorrent in many ways (yes, I know they consider themselves to be web3). There’s a lot of really cool conceptual work happening in removing ourselves from being beholden to a couple of major corporations that doesn’t necessarily require buying into a speculative, environmentally disastrous property bubble. I want to evaluate each tech that might be solving a problem I find important, but if it has brought in strongly to the ‘crypto’ or ‘web3’ hype, it just has a much bigger bar to clear to prove that it just isn’t a speculative ponzi scheme.

I can’t wait for the huge amounts of money, wasteful proof of work and the hype train to die out, so we can pick up the useful tech that comes out of this.

Sharing notebooks is harder than it should be.

You are working on a notebook (in Jupyter, RStudio, Visual Studio Code, whatever), and want to share it quickly with someone. Maybe you want some feedback, or you’re demonstrating a technique, or there is a cool result you want to quickly show someone. A million reasons to want to quickly share a notebook, but unfortunately there isn’t a quick enough and easy enough solution right now. That’s why I built notebooksharing.space, focused specifically on solving the problem of - “I have a notebook, I want to quickly share it with someone else”.

Ryan Abernathey captures the current frustration, and lays out a possible glorious future for what a ‘share a notebook I am working on’ workflow might look like. NotebookSharing.space is a start in tackling part of this. I highly recommend reading this thread.

Someone on Twitter asked me to share a notebook. I'm frustrated that in late 2021 we still don't have a simple, open-science-approved way to do this. A 🧵 about some of the options (cc @choldgraf, @yuvipanda) ... /1 https://t.co/mnd3K68JgV

— Ryan Abernathey (@rabernat) October 8, 2021

Just upload, no signup necessary

As the goal is to have the fastest way to upload your notebook and share it with someone, there is no signup or user accounts necessary. Just upload your notebook, get the link, and share it however you want. Notebook links are permalinks - once uploaded, a notebook can not be changed. You can only upload a new notebook and get a new link.

The upload is a bit slow in the video demos here because I’m sitting on a hammock in a remote beach, but should be much faster for you.

You can upload your notebook easily via the web interface at notebooksharing.space:

Once uploaded, the web interface will just redirect you to the beautifully rendered notebook, and you can copy the link to the page and share it!

Or you can directly use the nbss-upload commandline tool:

On Macs, you can pipe the output of nbss-upload to pbcopy to automatically put the link in your clipboard. Here is the example notebook that I uploaded, so you can check out how it looks.

A jupyterlab extension to streamline this process is currently being worked on, and I’d appreciate any help. I’d also love to have extensions for classic Jupyter Notebook, RStudio (via an addin), Visual Studio Code, and other platforms.

Provide feedback with collaborative annotations

When uploading, you can opt-in to have collaborative annotations enabled on your notebook via the open source, web standards based hypothes.is service. You can thus directly annotate the notebook, instead of having to email back and forth about ’that cell where you are importing matplotlib’ or ’that graph with the blue border’. This is one of the coolest features of notebooksharing.space.

You can annotate the notebook demoed in the video here. Note that you need a free hypothes.is account to annotate, but not to read.

Annotations are opt-in to limit unintended abuse. Enabling an unrestricted comments section on every notebook you upload is probably a terrible idea.

Private by default

By default, search engines do not index your notebooks - you have to opt-in to making them discoverable while you are uploading the notebook. This way, only those you share the link to the notebook with can view it. But be careful about putting notebooks with secret keys or sensitive data up here, as anyone with a link can still view it - just won’t be able to discover it with search engines.

RMarkdown and other formats are supported

It has always been important to me that the R community is treated as a first class citizen in all the tools I build. Naturally, NotebookSharing.space supports class support for R Notebooks as well as R Markdown files experimentally. R Notebooks produced by RStudio are HTML files, and will be rendered appropriately, including outputs (see example). RMarkdown (.rmd) files only contain code and markdown, and will be rendered appropriately (see example) ). If you enable annotations, they will work here too! I would love for more feedback from the R community on how to make this work better for you. In particular, an RStudio Addin that lets you share with a single shortcut key from RStudio would be an amazing project to build.

If you work primarily in the R community, I’d love to work with you to improve support here. I know there are bugs and rough edges, and would love for them to get better.

Internally, the wonderful jupytext project is used to read various notebook formats, so anything it supports is supported on NotebookSharing.space. This means .py files and .md files produced by Jupytext or Visual Studio code will also be rendered correctly, albeit without outputs as those are not stored in these files.

Next steps?

I want extensions that let you publish straight from wherever you are working - JupyterLab, classic Jupyter Notebook, RStudio and VS Code. That should speed up how fast you can share considerably. Any help here is most welcome!

I also want users to interactively run the notebooks they find here easily. This involves some integration with mybinder.org most likely, where you (or the notebook author) can specify which environment to launch the notebook into. Wouldn’t it be wonderful to have a ‘Make interactive’ button or similar that can immediately put the notebook back into an interactive mode?

Tweets from here on in Ryan’s twitter thread sell this vision well.

But neither of these options has a link to execution.

However, I made the notebook on a cloud-based @ProjectJupyter Hub run by @2i2c_org, so I know the exact environment it was run in. The environment was built automatically by https://t.co/vf7xPeQwqv 4/

— Ryan Abernathey (@rabernat) October 8, 2021

Feedback

I love feedback! That’s why I spend my time building these. Open an issue or tweet at me. As with all my projects, this is community built and run - so please come join me :)

Inspiration

Growing up on IRC, pastebin services are part of life. In 2018, GitHub stopped supporting anonymous gists

• so sharing a notebook with someone became a lot more work. NotebookSharing.space hopefully plugs that gap. The excellent rpubs.com is also an inspiration!