Rant: Jupyter notebooks are trash.

Question

They should only be used for experimentation and sharing information. Please don’t pass them off as finished products. When data engineers are creating inference pipelines based on the models the data scientists create they shouldn’t have to reverse engineer your feeble code. I am going nuts trying to understand what the nested for loops are trying to accomplish. Just tell me what I need to do to the data and I will do it :)

I love scrolling through a notebook and looking at the visualizations and pretty pictures though when I’m trying make use of the code in the notebook it is turning the rest of my hair grey.

Thank you.

Answer 1

>I am going nuts trying to understand what the nested for loops are trying to accomplish.

same without jupyter

Answer 2

Step 1: I'll just prototype things in jupyter then switch to good code later on once I've got the core functionality written, no problem

Step 2: Okay, it works, should probably refactor this now and do it the right way but I'm on a roll, let's knock out those next couple requested features in the notebook too

Step 3: Uhh if I'm going to stay on track with this timeline there's no way I'm going to spend days rewriting everything, I'm just going to send this enormous notebook to the team and call the problem solved, they can probably figure out which cells are important and how it all fits together

Please don't fall into this trap, lol

Answer 3

Received a notebook once that looped over a 1.5 million rows pandas df multiple times. Only comment it had was at the very end of it: "may need to run this in the cloud since I have a MacBook air".

Answer 4

Feeble, undocumented code and nested for loops is just bad code, it's not inherent to Jupyter notebooks.

What is it that actually makes Jupyter notebooks bad? I have never understood.

Answer 5

ChatGPT: generate a rant about Jupyter notebooks.

Answer 6

Who tf deploys pipelines in Jupyter notebooks?

Jupyter notebooks are for writing the chapters of your code and then when you finish writing and testing and everything is okay, you copy paste the code and you put it in your pipeline software ex. Airflow.

Answer 7

It feels a bit like you're mixing a whole bunch of problems together and labelling them jupyter notebook problems.

- If you want DSs to prototype data pipelines any way they fancy but to then just describe the pipeline to you so you can implement it from scratch, that's a way of working you need to discuss with your team.

- If you are bothered by the low standard of coding with the DSs, that is worth a rant but has nothing to do with notebooks. It sucks for you that you're in a place where DSs don't at least make well organized code to pass to engineers.

- At the end of the day your job is to take wonky prototypes written by people who are worse than you with code but better with data and turn it into a properly engineered code base. If you don't enjoy that, maybe you want to be a traditional software engineer?

Answer 8

Have you ever tried nbconvert? It's super easy to convert .ipynb to .py... Rather than copy/paste.

That being said, it sounds like most of your frustration with notebooks is not the format itself, but that the ones you receive are poorly commented and/or full of erroneous/unnecessary steps for the purpose of production inference. Maybe talk to your coworkers?

Answer 9

I don't think anyone is suggesting they be used for anything other than experiments and collaborating.

Answer 10

Jupyter notebook can also be good to walk someone through an analysis for reproducibility of the results. It can be good to demonstrate how to use an API since you can easily weave in documentation with runnable chunks of code.