๐ Kedro - My Data Is Not A Table

Search for a command to run...

No comments yet. Be the first to comment.
I've slowly adding more and more lua functions into my neovim configuration, and recently I noticed a pattern for a class of functions that reach out to run shell commands that can be abstracted away. https://youtu.be/8m5ipBuopPU Telegraph.nvim Check...

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome

In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into
People who are quick to toss team members under a bus are not well trusted or highly thought of and it will lead to some toxic team dynamics. Building Steam While collaborating on any project there are going to be decisions made that aren't necessari...

I create this blog with one person in mind, me. There are others like me This is not completely selfish, as there are likely many others out there that think similarly to me. Everyone comes from different backgrounds and varying levels of experience...

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.
These containers for data contain many convenient methods to manipulate table-like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types.
{% post waylonwalker/what-is-kedro-lob %}
unfamiliar with kedro, check out this post
There are times when our data doesn't fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk. Beware that pickle files coming from an unknown source can run malicious code and are considered unsafe. For the most part though when you read and write your own pickle files they are a good tool to consider.
See more about pickle from python.org.
I may have a dictionary that describes some cars.
{
'truck-012-abc': {
'type': 'truck'
'sales': [12, 2, 3, 4, 8]
'weight': 9024,
'accesories': ['leather', 'audio-1']
}
In the catalog, we will simply set the type as pickle.PickleDataSet and give it a filepath.
cars:
filepath: data/cars.pkl
type: pickle.PickleDataSet
This
filepathdoes not have to be on the local filesystem it can be on the cloud thanks to how kedro utilizesfsspecfor each of its datasets.
The benefit of cataloging this dataset compared to leaving it as a MemoryDataSet is that you can easily load this data back into memory for further development or debugging without running any of the pipelines.
catalog.load('cars')