What's New in Kedro 0.16.6

PublishedOctober 26, 2020

•3 min read•View as Markdown

Waylon Walker

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Nvim Remaps Like a Boss

I've slowly adding more and more lua functions into my neovim configuration, and recently I noticed a pattern for a class of functions that reach out to run shell commands that can be abstracted away. https://youtu.be/8m5ipBuopPU Telegraph.nvim Check...

Oct 5, 20212 min read33

waylonwalker.com/kedro-silence

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome

May 21, 20213 min read247

How I navigate tmux in 2021

In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into

May 12, 20219 min read159

Stand With Your Team

People who are quick to toss team members under a bus are not well trusted or highly thought of and it will lead to some toxic team dynamics. Building Steam While collaborating on any project there are going to be decisions made that aren't necessari...

Jan 22, 20213 min read109

Blogging For Me

I create this blog with one person in mind, me. There are others like me This is not completely selfish, as there are likely many others out there that think similarly to me. Everyone comes from different backgrounds and varying levels of experience...

Jan 22, 20214 min read73

Waylon Walker

30 posts

Deployment Docs

This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible.

Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies.

Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language.

Batch uses the AWS cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization.

I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get this working in my production workflow.

Spaceflight starter

They have officially added the spaceflights pipeline as a starter. I have not yet had a chance to try this out, but I anticipate this will be a great pipeline to teach from as it is a bit more complex than the iris pipeline.

I tell folks all the time the best way to learn something new like kedro is to practice, practice, practice. Having this at their fingertips will give an easy way to fire up a pipeline that is ok to break try out some new ideas and leave it. I see this helping me testing out plugins on more complex pipelines and writing blog posts with examples that readers can more easily follow along with.

practice

testing

blog posts

tutorials

Better Error Messages

Shameless pug, my PR landed in kedro, but fell off of the release.md in a merge issue. I was frustrated working on large projects when the wrong arguments were passed into a node and it did not give you any information in the Error to figure out where the error was thrown. Now it will at least give the name of the function that caused the error. This was a simple fix as all of the information was already there.

What's New in Kedro 0.16.6

Comments

More from this blog

Nvim Remaps Like a Boss

waylonwalker.com/kedro-silence

How I navigate tmux in 2021

Stand With Your Team

Blogging For Me

Deployment Docs

Spaceflight starter

Better Error Messages

Overall

Thanks for supporting contributions

Command Palette

Comments

More from this blog

Deployment Docs

Spaceflight starter

Better Error Messages

Overall

Thanks for supporting contributions