Julia pre-compiles packages on first load, allowing them to deeply optimize the generated code for the particular CPU architecture the user is running on for maximum performance. However, it does take some time, so there’s a startup latency penalty here. If you make docker images with precompiled Julia packages, you can pay this pre-compilation penalty at image build time, rather than at image startup time. Much better for your users!

If you look at the tutorial for pre-compilation, it may sound like all you need to do is call Pkg.precompile. And if you actually build and test your docker image, it will work fine - your precompiled libraries are loaded, and startup is fast. Yay!

But then you push your built image to a registry, and someone else pulls it and runs it - and the pre-compilation has no effect! Packages are pre-compiled again on first run, and sometimes this may take multiple minutes, causing weird to debug timeouts. But then you can’t reproduce it, because it runs fine on your computer! What’s going on?

What is going on is an abstraction leak. We think of computers as ‘computers’, and often it’s easy to forget that there’s just a wide wide variety of them. After a few decades of Intel’s (or should I say AMD’s) amd64 architecture hegemony, M1 Macs (and Raspberry Pi) have at least forced us to consider that ARM64 exists as well. But those are still just broad classes of architectures, and each generation of CPUs within them also have differences, with specific instructions available in some that aren’t in others.

So turns out Julia’s pre-compilation is CPU dependent (or more specifically, ISA dependent). So if I build my docker image on one type of CPU, the pre-compiled files will only be used when running on the same type of CPU!

The limits of ‘works on my machine’ :)

This manifested as a ‘Pluto notebook is not starting when run on JupyterHub’ issue. It was fun debugging this, eventually leading me to discover this bit about Julia pre-compilation. We eventually landed a fix that cross compiles to a number of different ISAs, producing a ‘fat binary pre-compilation’ that starts fast on a number of different CPUs!

I’m also personally happy with the amount of time this took - I’m trying to learn to wean myself off high stress ‘gotta fix it now’ open source maintenance, and a slower paced but more consistent approach.

Anyway, if you’re building Docker images with Julia in them, and want to pre-compile any packages you install in the image (as you should), I recommend the following bash snippet before you call Pkg.precompile

if [ "$(uname -m)" == "x86_64" ]; then
    # See https://github.com/JuliaCI/julia-buildkite/blob/70bde73f6cb17d4381b62236fc2d96b1c7acbba7/utilities/build_envs.sh#L24
    # for an explanation of these options
    export JULIA_CPU_TARGET="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
elif [ "$(uname -m)" == "aarch64" ]; then
    # See https://github.com/JuliaCI/julia-buildkite/blob/70bde73f6cb17d4381b62236fc2d96b1c7acbba7/utilities/build_envs.sh#L54
    # for an explanation of these options
    export JULIA_CPU_TARGET="generic;cortex-a57;thunderx2t99;carmel"
fi

This is what the official Julia binary is built with (and you should be using that, not your distro provided binary). It may need updating as new ISAs come out, but otherwise should help.

Thanks to mathbunnyru and benz0li for helping with this.

P.S. Did you know that Python also does pre-compilation, and that can also have a massive effect on python startup time? I did a similar debugging session on that a while back.

Why write this?

This August / September, I had a lot of life changes in one go:

1. Moving from an apartment i disliked to one I absolutely love, and maybe even feel at home in, for the first time in my life
2. Officially switch jobs, becoming an employee of a non-profit I had co-founded
3. Catch COVID for the first time and recover from it, with appropriate MH breakdowns from the isolation
4. Get on Guanfacine as a non-stimulant medication for ADHD

While I can not fully separate the effects of these changes, the overall effect has been very good for me. The ADHD meds form an important and specific component, and I think I am primarily writing this to understand what that is. Many years ago, I had started on bipolar 2 meds and anti anxiety meds at the same time, and attributed some positive changes to the bipolar 2 meds that were not true - it was almost exclusively the anti anxiety meds. This contributed to me living with a misdiagnosis for many years, and taking meds I just didn’t need. So I don’t want to unconditionally say all the good positives in my life are from this, but recognize that some are, especially this time we did not mix it with another pharmacological intervention. Writing it down now should also allow me to come back to it a year or two later to see how that has evolved

Dosage & Side Effects

I currently take 2mg a day in the morning, after starting at 1mg. No plans to increase dose further. I had some dry mouth for a week or so when starting, but it has mostly subsided.

I had been on Ritalin for a few months when I was first diagnosed (happened when I was 20). It was an important marker in my life (my first thought when it kicked in was “Wow, is this how normal people feel?”). I did not enjoy the “crash” in the evening, but now i suspect increased dosage or an extended release would have helped fix that. But I was still in Chennai at that time, and due to various socioeconomic factors + how hard it was to access, I gave up after 2 months. While I got back on other psych meds later, due to various other mental health comorbidities, I’ve been wary of trying stimulants for myself. I’m extremely happy that there are now non-stimulant options for ADHD, and one of them is working for me.

Productive capacity

The first, and most noticeable difference is that doing “work” (broadly defined as ’thing that pays my salary’) is easier. Specifically,

1. I can focus on a single problem for a longer stretch before I lose energy and hence focus. My work sessions now seem to be closer to an hour before a break rather than 20minutes before a break. This also means the total amount of time I can bring to bear on a problem per day has increased, as much as by 50%.
2. I seem to have more control over what I am actually interested in. I know I do great work when I focus on something, but I never felt fully in control over what things I could choose to put my attention on. We have to evaluate this more long term, but so far, this atrophied muscle seems to be slowly growing.
3. I am able to actually plan my day and do the things I have planned, and say ’not now’ to other things I may want to do during the middle of the day. I have never managed to stick with any kind of personal planning system. What I had planned and what I would actually do would often not match up, causing some amount of emotional pain (“Why am I so incompetent?” usually). After a while, I’ll give up said system because the pain is just too much, instead chasing the ephemeral joy of just doing things.

So my overall capacity has increased (1), and so has my ability to apply it in a particular direction. This is awesome. It’s now up to me to work on building systems that allow using these gains, building organizational muscle that was just not possible before (3). I’m excited to build this muscle, re-trying things that I had tried before and failed at. One specific hope I have is that it’ll help me go from someone who gets something to 80% completion to someone who gets things to 100% completion more times.

A side-note here here about my ADHD Tax, which has been primarily paid as shame and emotional turmoil (ask anyone who has been around me whenever I have had to apply for a Visa), although of course there has been a financial and opportunity cost. This has gotten so much better. I just got a ton of pending paperwork finished, without the usual attached emotional turmoil. For the first time in my life, I think I will be able to get on top of this and stay on top of it.

Emotional capacity

Between the stimulus and response, there is a space. And in that space lies our freedom and power to choose our responses.

This quote from Victor Frankl has always stuck with me, and I think Guanfacine has just increased the amount of this space I have to do this.

I have been in therapy for a long time, and will probably continue to be forever. I have learnt many skills I can bring to bear, particularly at times of extreme (and often disproportionate) emotional distress (something that seems to happen far too frequently for my taste). However, often when you are emotionally distressed, you just don’t have the capacity to use those tools! The best you can do is to avoid causing harm, to yourself or others, and retreat for your body to recover.

Guanfacine has definitely made this situation better. I have more capacity now to actually use the tools I have learnt, and iterate on them. Insight is often useless without action, and Guanfacine has given me far more scope for action than I had before. This isn’t something that ‘heals’ everything (no such thing, unfortunately) - but it has vastly increased my ability to ‘do’ rather than just ‘know’ or ‘feel’. Very grateful to have it.

What now?

My overall goal has been to find ways to heal and lead a less miserable, more joyful life than I have in the past. I understand this is probably a years, decades-long process, and I’m much closer to the start than to the middle Guanfacine feels like a significant boost along the way here. I’ll check back-in in a year and see how the things in this post have aged :)

If you think you may have ADHD, consider getting evaluated and trying some medication. This may be a fucked up process that you may not be able to afford for economic, social or even just ADHD reasons (may need a lot of follow through ugh). But, I think it is absolutely worth it, so if it is available to you and you have been putting it off, consider this a sign to re-prioritize that.

You can reach out to me by emailing yuvipanda@gmail.com if you wanna say hi.

Trying to start these back up, as a list of the kind of ‘work’ I have done the last few weeks. Let’s see if this works!

Last 2 weeks

1. Spent a good chunk of time investigating and fixing issues with AzureAD auth for the UToronto hub. “Fixing” it meant moving off AzureAD, so we only have CILogon & GitHub in the 2i2c infrastructure as authentication providers! This is now explicitly mentioned in our docs. An incident report still needs to be written.
2. Upgraded Azure AKS for UToronto, leading to a pretty gnarly outage. Incident report needs writing. issue.
3. Finished setting up the ClimateMatch hub. Part of this involved adding terraform support for proper, single-hub nodepools in GCP.
4. With help from @sean-morris, all Cloudbank Hubs now use a data8 style image. This has reduced the number of non-staging hubs using the old ‘default’ 2i2c image to 5! When unspecified, the default image in use now is jupyter/scipy-notebook. These communities should be contacted and a different image be used for them, so we do not have to maintain this ‘default’ image anymore.
5. With the goal of removing special cases from the 2i2c infrastructure, I am working on removing the manually maintained NFS servers from our infrastructure. I had thought only the 2i2c shared cluster had it, turns out the cloudbank one does too! The 2i2c shared cluster has been migrated off, the cloudbank one will be soon. With that, home directories are unified for all clusters, and no longer require specialized knowledge of filesystems, etc to manage.
6. Did a bunch of cleanup work on our terraform code.
   • Remove unused AzureFile SMB support, as we use NFS everywhere. Yay consistency. Pull
   • Fix terraform linting errors and fix the job that was doing the lint to actually do the lint. Pull.
   • Use optional fields in our terraform variables, for cleaner .tfvars files. Pull

Upcoming week

• Help with stabilizing quarterly goal for next quarter for 2i2c.
• Finish two incident reports for UToronto so we can close that out:
  • https://github.com/2i2c-org/infrastructure/issues/2628
  • https://github.com/2i2c-org/infrastructure/issues/2594
• See this PR through to completion: https://github.com/2i2c-org/infrastructure/pull/2406. Mostly documentation.
• See this PR through to completion: https://github.com/2i2c-org/infrastructure/pull/2621. Mostly upstream documentation, started in https://github.com/jupyterhub/grafana-dashboards/pull/84.
• Finish moving cloudbank hubs to filestore, decomission old NFS VMs. https://github.com/2i2c-org/infrastructure/issues/1105

I’ve had a bit of a slump the last few months, working through what feels like deep mental fog for totally unclear reasons. The lack of clarity on causation is one of the most frustrating parts of this, and one of the most familiar too - this isn’t the first time I’ve felt like this, and I know (rationally at least, if not emotionally) it won’t be the last. Probably a combination of intensely avoiding making some decision that seems too big to be made by me without actually knowing what it is and fun brain chemicals stuff. Who knows, I’ve given up on trying to figure out causality for these things for the most part.

One of the effects has been a distinct lack of drive, at least to the extent I am used to. Another is that it is difficult to remember being in a different emotional state in the past, and difficult to imagine being in a different emotional state in the future. Time dissolves, and I believe how I feel now is how I have always felt, and how I will always feel.

Hence, am trying to make external records any time I feel or encounter something that seems to give me a good boost, and a sense of ‘wanting to do it again’. That in a loop feels like drive to me - A feeling induced in you by external stimuli, and then you wanna find things to do to feel that again.

I felt some of that a few days ago! I built something, put out a quick message about what it is that I had built on a slack. Someone responded with this message:

And that felt really good! I could see how responses like this have driven most of my software development projects, for good and ill. It has been a driver in picking which communities I can be a part of, what I find interesting, and what feels like a slog. I definitely find communities that are at intersections of various fields, and I try to find them early - so things I could do that would feel ’normal’ in one of those fields feels mind blowing to the other. I joked to a friend that my ‘anxiety response’ is to start (or get deeply involved in) a new open source projects, but the more I think about it the more it seems to be not a joke at all.

Not unique to myself in any way or form I would assume. But feels great to know this is true for me, and I have come to value explanations for my emotions quite a bit of late :)

Anyway, great to remember that I can feel this!

I walked into a local hardware store, with the following thought in my mind:

I need a 5mm metric hex key

And I wandered around aimlessly as I do in stores, almost bought some double sided tape for a different project (decided against it), and eventually made it to the place with all the sockets, hex keys, screwdrivers, etc.

Words that previously meant nothing to me (Torx, SAE, metric) made sense. Felt a little like the very first time I felt power learning to write code (the strongest memory is that of changing color of a button in VB6 with one line of code). What was noise before had meaning now.

I got myself a little folding metric key set with hex keys going from 1mm to 6mm - I knew I had some new windscreen clip-ons that needed a 4mm key, and am sure I’ll find a 6mm somewhere.

I pick up something that costs 13$, and looks ok. I look at a different brand, and decide I like that better because of the way it folds. “Hey, this means when I try to turn it, I’ll get better leverage!”. I actually know what that sentence means, and I choose one brand over another based on actual knowledge rather than a guess. If my programming experience is anything to go by, I’m completely wrong in ways I won’t even understand for a long time.

But I buy it, leave the store, and try using it for intended purpose on my motorcycle - to remove the bolts holding my bar end weights in. Earlier that week, I had stripped these by attempting to use a friend’s impact driver with a torx bit (it sort of fit, and I had no mental model about what I was doing!). The head stripped, and the thread stripped, and I took it to Performance Motorcycles in Berkeley who fixed it up for me (although the bolt itself did break and they had to put a new one in).

I had since beat myself up about not knowing enough about hardware to not strip the head, been comforted by a friend, realized that I had set weird standards for myself where somehow I must already know all of this having never put in any effort to learn, and then being delighted that ‘oh wait this is just another skillset I can pick up’. It will probably take years, but that’s ok!

Earlier that day, I had went back to the mechanic and asked him exactly what screw he had used for the bolt. He told me it was a 5mm hex, and I should use an exact one or I will strip.

I extend the 5mm key, and fit it in. It seems to be a perfect fit! I try twisting left, repeating internally to myself “Lefty loosy”. It doesn’t budge, and I’m slightly anxious. But I trust that it is not super likely to strip at this point - I’m not a very strong person, and this I fully believed was the correct size. So I put more energy into it.

AND THE BOLT TURNS! I FEEL ELATED! AMAZING JOY! It is a bit like seeing your code run for the first time, AND IT FEELS AWESOME!

So excited for this future :)

If you are running a JupyterHub on Kubernetes and use NFS for home directory storage (a common occurance), I highly recommend the following settings:

singleuser:
  extraEnv:
    # notebook server writes secure files that don't need to survive a
    # restart here. Writing 'secure' files on some file systems (like
    # Azure Files with SMB or NFS) seems buggy, so we just put runtime dir on
    # /tmp. This is ok in our case, since no two users are on the same
    # container.
    JUPYTER_RUNTIME_DIR: /tmp/.jupyter-runtime
  extraFiles:
    ipython_kernel_config.json:
      mountPath: /usr/local/etc/ipython/ipython_kernel_config.json
      data:
        # This keeps a history of all executed code under $HOME, which is almost always on
        # NFS. This file is kept as a sqlite file, and sqlite and NFS do not go together very
        # well! Disable this to save ourselves from debugging random NFS oddities that are caused
        # by this unholy sqlite + NFS mixture.
        HistoryManager:
          enabled: false

These aren’t specific to kubernetes, but to NFS (or AzureFile, or any other shared filesystem). sqlite and NFS do not mix, and you’ll run into weird errors you will not be able to really debug. Save yourself the pain :) Let your users know too, and tell them to not put sqlite databases on NFS.

Alex Merose asked me on Mastodon how to best set up a conda environment on GitHub Actions. I thought I’d write a short blog post about it!

The short version is, use the conda-incubator/setup-miniconda GitHub action instead of the official setup-python action. Specifically, try out these options for size:

  steps:
    - uses: conda-incubator/setup-miniconda@v2
      with:
        # Specify python version your environment will have. Remember to quote this, or
        # YAML will think you want python 3.1 not 3.10
        python-version: "3.10"
        # This uses *miniforge*, rather than *minicond*. The primary difference is that
        # the defaults channel is not enabled at all
        miniforge-version: latest
        # These properties enable the use of mamba, which is much faster and far less error
        # prone than conda while being completely compatible with the conda CLI
        use-mamba: true
        mamba-version: "*"

This should give you a python environment named test with python 3.10 (or whatever version you specify) and nothing much else. You can then use mamba to install packages from conda-forge to your heart’s content in future steps like this:

    - name: Install some packages
      # The `-l` is needed so conda environment activation works correctly
      shell: bash -l {0}
      run: |
        mamba install -c conda-forge numpy scipy matplotlib        

My recommendation is to continue using the default setup-python action for almost all your python needs in GitHub actions, and use conda-incubator/setup-miniconda@v2 only if you explicitly need to use mamba (or conda) for something.

I am not smart enough to consistently write and debug shell scripts that use any conditional or looping constructs. So as soon as I’m writing something in bash that requires use of those, I switch to python. This works fine when writing scripts, but what to do when writing GitHub Actions workflows? You can only write bash in run: stanzas in your steps, right?

Not at all! You can set the shell parameter to anything you want, and the contents of run will be passed to the shell in the form of a file. This allows you to use not just Python, but basically any langauge when writing your GitHub actions workflows.

Here is an example step that used python.

  steps:
    - name: Something in python!
      # The -u means 'unbuffered', so print() statements in your python code are output correctly
      # otherwise, they might be out of order with stdout from commands your code calls
      # {0} is replaced with the name of the temporary file GitHub Actions creates with
      # the contents of the run:
      shell: python -u {0}
      run: |
        import sys
        import subprocess

        print("Hello, I am python")

        # We have to use string substitution for getting inputs, which is bad and ugly
        # However, it isn't as bad me trying to write conditionals in bash.
        # It might be possible to use environment values here, I haven't explored.
        variable = '${{ inputs.some_input }}'

        # Use subprocess.check_call to call out to external process. stdout is
        # handled correctly
        subprocess.check_call([
          sys.executable,
          '-m',
          'pip', 'install', 'django'
        ])        

If you instead use mamba or conda to set up your Python, perhaps with the setup-miniconda action, you need to set shell: bash -l -c "python -u {0}". The bash -l makes sure that the appropriate conda environment is loaded, and then passes off to the ‘correct’ python.

I decided to write publicly and publish my motorcycle riding trip notes now.

Montarey Trip notes

Need to learn to plan when to pee. This is important to do! Can’t just get off any exit, as that often doesn’t actually lead to wilderness but to more concrete jungles.

Need to learn more about lane splitting and how and when to do it safely. I did it for a little bit while stuck on CA-1S, but was avoiding it for most of the ride. Feels like it’s only safe to do when traffic is almost at a standstill, and I can go at least 5mph faster than traffic. Also, I don’t wanna get hit by other motorcyclists lanesplitting. More research needed.

I need to adjust my levers to see if it helps with my skin near thumb getting sore.

Get a different seat maybe for more comfort? My butt does hurt and is sore.

I definitely need to feel much more confident in curves. Taking the TC class is a good next step.

DO NOT BELIEVE GOOGLE MAPS TIME ESTIMATES!! I am much slower than that haha.

I need to work on my core strenght, back to the Gym. I think without that, long rides are going to be painful.

Need to plan meals better too. I’m eating ’lunch’ at like 4:30, after having only eaten 3/4 cup of oats and 1 butter croissant all day. That isn’t enough. Partially this is because I couldn’t leave on time either - as usual. Still meant I left at 12 rather than 2 - but could’ve left at 10!

Post trip notes

This is at night. The ride back was all interstate, and quite different!

1. Food makes a huge difference. I think I need to eat more than I think too. Just everything was better after food. I should just always eat, and eat on time.

2. I am getting more confident about turns at speed. I think by default I end up slowing down for each turn from fear. That is pretty dangerous! I need to have a better understanding of how much traction is available, and just turn. Which way to lean is also important - but I think my current intuition (counter lean or neutral) is good enough for me. I saw some guy shred the twisties when I was up in the berkeley hills, and he had to lean quite a bit. Not what I’m doing though.

   I should take more classes and read the total control book properly. Towards the end I was feeling pretty confident, and that was quite nice!

3. The way back was north, with the sun directly into my eyes for almost an hour. Was intense, I could barely see! I had sunvisor down and transition glasses and that wasn’t enough at all. I was in a trance. Looking back, how did I trust myself enough to go through that?! Would I have been able to see hazards on the road? Stopped cars? I was able to control speed pretty well. I was a bit tired too. But I did Montarey to San Jose in one go, and stopped - my body basically forced me to stop and rest. I don’t entirely know how I feel about this. I think it was mostly safe, as it was just on the interstate and I was on the left-ish lane. I also think I could actually mostly see - what I see in my memory now, hours after the fact, doesn’t seem quite right - feels filtered. It really was just an intense trance. I don’t mind interstate riding.

4. I should strap my bag down and not carry it! I think that will definitely help my back hurt a lot less. I had 2 bottles of water, laptop and ipad - I think that’s a lot. I should consider a smaller bag too.

5. A more comfortable seat would also help. My butt was pretty sore.

6. Physical strength is an important component of safety, and I must get back to the gym. Particularly, while turning I could feel my core was not strong enough several times, and I would like more trust in my muscle strenghts.

7. I lane-split a little! Just for parts of CA-17S, when in otherwise almost stopped traffic. I need to learn way more about it.

8. I should understand trail braking better. I think my understanding of it right now is dangerously flawed, as it contributes to me slowing down excessively in turns. Could get rear ended.

9. Really need a windshield. I can hardly even look at my mirrors in the interstate, because I can’t bring my torso up at all.

10. I drifted into next lane almost in one turn I think because I wasn’t braking and I was tired. I was trying to ‘push through’ and at some point even didn’t want to break in San Jose. WTF was that, compromising safety for challenge! No way.

11. I need to leave earlier so I can have more time at my target spaces. In fact, I should look for nice target spaces first and then travel - flip my currnet model a little. I don’t think I need to just haul miles anymore. Past that stage now.

12. The bananas in my backpack were smushed to death and the inside of my backpack is a banana smoothie. Would not recommend

TODOs

Concrete action items!

1. Always eat before a ride
2. Get a better helmet. I’ve spent months looking for the right one, time to look again :(
3. Start going through Champ U that I already bought
4. Look for another intermediate mototrcycling course I can take.
5. Plan time at conclusion of journey better.
6. Get a windshield for the bike.
7. Experiment with getting luggage off my back

I wanted a single keyboard shortcut that would let me switch between the built-in terminal and the editor in vscode. I couldn’t find any, so I made a short macro using the VSCode Macros Extension

First, you define your macros in settings.json. You can open up vscode settings with Cmd+,, search for macros, and edit this under Macros: List.

    "macros.list": {
        "showTerminal": [
            "workbench.action.terminal.toggleTerminal",
            "workbench.action.terminal.focus",
            "workbench.action.toggleMaximizedPanel"
        ],
        "hideTerminal": [
            "workbench.action.toggleMaximizedPanel",
            "workbench.action.terminal.toggleTerminal"
        ]
    }

Then you can bind a keyboard shortcut to it. Open Preferences: Open Keyboard Shortcuts (JSON) from the command palette, and add this:

    {
        "key": "cmd+t",
        "command": "macros.showTerminal",
        "when": "!terminalFocus"
    },
    {
        "key": "cmd+t",
        "command": "macros.hideTerminal",
        "when": "terminalFocus"
    },

showTerminal gets called when you are focused anywhere in the editor except your terminal, and hideTerminal gets called when you are focused on your terminal. This gives me the ’toggle’ behavior I want.