Hi, I work with
kedro at work. Kedro is a framework to build DS/ML pipelines, I don't love it but it's what my company uses and I think we can make it work, it's pretty customisable at least.
Kedro is very geared towards jupyter notebook users. They rely on magic command to load special kedro variables such as the catalog, conf file, etc. I use to love magic when I got to know jupyter but nowadays I really don't like anymore.
The point is, this magic makes some variables available in the global namespace so that the user doesn't have to worry about loading the right config/catalog. This seems like a patchy fix, because to be honest, they don't have a robust solution to load these files from a simple python script or for example marimo notebook. They also have special commands
kedro jupyter lab
and
kedro jupyter notebook
that "inject" these variables for the user before hand. See a thread of the issues with Kedro configs + marimo here:
https://kedro-org.slack.com/archives/C03RKP2LW64/p1736872610928369I wonder if it'd be possible to have a similar
kedro marimo edit
command to integrate marimo into this framework.
PS: With all the work you've done on data flows and DAGs I think you actually are in a prime position to create a clean ML pipeline framework, would love to hear your thoughts on that. @Akshay @Myles Scolnick