Nothing material has changed. Get them to do the job they will be doing, with you. Or, if that would involve proprietary data an exercise that’s as close as possible. They drive, you’re in the background. Some people get nervous, so try to be helpful if you see them blanking on something they seem to know, for example. Do it open-book, and give them access to the tools they’d have on the job - IDEs, google, LLMs. At the end of an hour you should have a clear idea of whether they could do the thing you want them to do.
Things that don’t work… Talking to candidates about the job won’t do this - a lot of people can talk plausibly about things that they can’t do. Whiteboard exercises won’t do this - they can be trained to without a real-world ability to deliver.
FWIW, I had to do this remotely when I gave interviews and didn’t want candidates to access my corporate desktop, so I built a disposable remote environment we could both log into for this sort of test. https://www.edparcell.com/remote-desktop.html
I used to build quant investment notebooks that had to be deployed in production. Lots of problems with that. Mine were: Notebook cells run out of order, so you often have something that works in a session, but not in a fresh run. Developing against limited datasets, so you fail against things you didn’t know to test for. Small adaptions that have to be made every time the notebook is translated into a code file.
We streamlined it by making a graph-structured Computation a first class object that tracked staleness as code or data was updated. Then that class could be directly published, and when failures happened in production, the graph could be serialized with the inputs and intermediate calculation data that caused failure, for investigation in a notebook.
When I was hiring at my old place we set up disposable cloud VMs with a shared screen we could both remote into for interviews. No need for the candidate to install anything, and more responsive and secure than sharing my corporate desktop. Obviously the best thing is in-person at a shared screen (and the human interaction goes better too). This seemed like the best and most respectful compromise. https://www.edparcell.com/remote-desktop.html
I got frustrated with sites that should just be documents bolting on a pile of javascript libraries, third-party requests etc so I made a standard for sites that are just documents, and wrote a checker for document-only pages. For SSL, the thing that actually changed the web was browsers flagging non-SSL - probably similar is needed here. https://certifiedweb10.org/ if you're interested.
It is simpler if you use the notation [x]_a^b (i.e. with a subscript and a superscript b) to mean x, clipped to the range a to b, and skip writing +/- infinity if you don't intend clipping on one side.
Then you get a bunch of obvious identities like [x]^b = min(x, b) = [b]^x (x capped by b is the same as the smaller of x and b which is the same as b capped by x), [x]_a^b = [b]_a^x, and [x]_a^b = [[x]_a]^b. Putting these together you get [x]_a^b = [[x]_a]^b = min(max(x, a), b). But honestly it's just easier to stick to the notation most of the time.
100% agree. People now use “innovate” and “invent” interchangeably. Typically they use the fancier sounding one because they want to impress people with their long words. They are not interchangeable though. Invention is the initial spark to the first version. Innovation is the polishing process of the next n versions. The iPhone 1 is an invention, and every iPhone after that is an innovation.
Now, the iPhone 1 didn’t do very much, and often there is far more value in the innovation than there was in the original invention. But you don’t get the innovation without first inventing something that didn’t previously exist.
Sadly, using words incorrectly swaps into thoughts, and affects reasoning. Because these words have been conflated, organizations are typically no longer able to reason about invention and innovation correctly, and are uninterested in inventing as a result. I would argue we see this in the lack of new underlying technological inventions after the 90s. It is like we have eaten our own seed corn. Very sad.
Loman author here. Thank you very much for the mention. Amazed that I never heard of Athena or pixie graphs. Our intention with Loman was to create a library scoped for a single process - we looked at the possibility of creating a system responsible for executing much larger graphs on a real-time ongoing basis, but it felt like a larger project than we'd be able to execute well. It sounds like Athena was that, and it worked well, subject to being a culture shock for people coming into it?
A similar library from another asset manager - https://github.com/man-group/mdf. Although MDF seemed to work at the level of timeseries instead of scalar values.
I'm a big fan of Graphviz. My old team created a library called Loman, which we open-sourced, which uses DAGs to represent calculations. Each node represents a part of the calculation and contains a value, similar to a cell in an spreadsheet, and Loman tracks what is stale as you update inputs. Loman includes built in support for creating diagrams using Graphviz. In our quant research we have found that invaluable when revisiting old code, as it allows you to quickly see the structure and meaning of graphs with hundreds of nodes, containing thousands of lines of code.
We've found it quite useful for quant research, and in production it works nicely because you can serialize entire computation graph which gives an easy way to diagnose what failed and why in hundreds of interdependent computations. It's also useful for real-time displays, where you can bind market and UI inputs to nodes and calculated nodes back to the UI - some things you want to recalculate frequently, whereas some are slow and need to happen infrequently in the background.
My team has a similar library called Loman, which we open-sourced. Instead of nodes representing tasks, they represent data, and the library keeps track of which nodes are up-to-date or stale as you provide new inputs or change how nodes are computed. Each node is either an input node with a provided value, or a computed node with a function to calculate its value. Think of it as a grown-up Excel calculation tree. We've found it quite useful for quant research, and in production it works nicely because you can serialize entire computation graph which gives an easy way to diagnose what failed and why in hundreds of interdependent computations. It's also useful for real-time displays, where you can bind market and UI inputs to nodes and calculated nodes back to the UI - some things you want to recalculate frequently, whereas some are slow and need to happen infrequently in the background.
The ease of the calculation tree in Excel versus having to keep track of what cells in a notebook you have updated was a large part of why we built and open-sourced Loman [1]. It's a computation graph that keeps track of state as you update data or computation functions for nodes. It also ends up being useful for real-time interfaces, where you can just drop what you need at the top of a computation graph and recalculate what needs updating, and also for batch processes where you can serialize the entire graph for easy debugging of failures (there are always eventually failures). We also put together some examples relevant to finance [2]
Things that don’t work… Talking to candidates about the job won’t do this - a lot of people can talk plausibly about things that they can’t do. Whiteboard exercises won’t do this - they can be trained to without a real-world ability to deliver.
FWIW, I had to do this remotely when I gave interviews and didn’t want candidates to access my corporate desktop, so I built a disposable remote environment we could both log into for this sort of test. https://www.edparcell.com/remote-desktop.html
reply