Quickstart

Loading Data

At its most basic level, fyda is a way to interface with your data. The objective is to make loading and saving your data files easy and intuitive. fyda achieves this with two simple concepts:

  1. A file called .fydarc that has manually configured values.
  2. “Shortcuts” that uniquely map to your data files.

Wherever your data is stored, fyda will assume that it all lies under some root directory, which should be the first thing specified in .fydarc:

[directories]
root = ~/myproject/data

Let’s suppose that the myproject folder has the following structure:

.
├── data
│  ├── arrays
│  │   ├── X.npy
│  │   └── y.npy
│  └── raw
│      ├── transactions.pickle
│      ├── consumers.csv
│      └── external.xlsx
└ .fydarc

and that the contents of .fydarc are as above. Normally when loading the data files into python, you need to open the correct reader and point to the raw string location for that file. fyda takes care of this with a recursive search through the root folder and automatic shortcut naming and reader assignment. Using fyda from a python shell running in ~/myproject would look like this:

>>> import fyda
>>> transactions = fyda.load('transactions')
>>> X = fyda.load('X')
>>> y = fyda.load('y')
...
>>> external = fyda.load('external')

And that’s it! By removing the raw string references, the code becomes more extensible and scalable. The recommended way to use fyda with your projects is for each system to have its own .fydarc located at the directory where python is executed from. This .fydarc should not be pushed to the master branch of the repository, but instead maintained locally. You can also manually tell your script where the .fydarc it should use is located by setting fyda.options.CONFIG_LOCATION option to the full filepath of .fydarc.

You can also manually specify shortcuts in .fydarc by creating a section called [data] organized as:

[data]
<shortcut 1> = <relative file location to root>
...
<shortcut n> = <relative file location to root>

This is particularly useful when the original file names are very long and the default shortcut assignment would be cumbersome to use, or when solving duplicate file name issues (more on this below).

Using fyda to get file paths

fyda makes it easy to retrieve the absolute path to data files in a platform-independent way. Using the example data we added earlier, it would look like this:

>>> fyda.data_path('example')
'/home/username/myproject/input/example.csv'

If you use the same code on a Windows machine, it would instead return:

>>> fyda.data_path('example')
'C:\\Users\\username\\myproject\\input\\example.csv'

This is useful when collaborating on a project or when you need platform independence for your code.

How fyda handles duplicate names

There are some situations that can cause namespace issues with the shortcut method above:

  1. A folder contains two files with the same prefix, but different file extensions.
  2. Two or more folders have unique file names (prefix + extension) within themselves, but file names are not pairwise disjoint across combinations of folders.
  3. A combination of both situations above.

fyda’s method of solving this is by first adding on file extensions to the shortcuts of the duplicate offenders and seeing if that fixes the issue. If not, it will start adding on the containing folders’ names to the shortcuts until uniqueness is achieved. As an example, consider the following situation:

.
├── data
│  ├── arrays
│  │   ├── X.npy
│  │   └── y.npy
│  └── raw
│      ├── X.csv
│      └── y.pickle
└ .fydarc

In this case, the prefix for X and y is not unique across folders, so fyda needs to resolve the duplicate name issue. fyda first tries adding the file extensions, and since this leads to a unique namespace, it stops there:

>>> X_arr = fyda.load('X.npy')
>>> y_arr = fyda.load('y.npy')
>>> X_raw = fyda.load('X.csv')
>>> y_raw = fyda.load('y.pickle')

Keep in mind that you can always specify specific shortcut assignments in the [data] section of your .fydarc; for example, if your fydarc looks like this:

[directories]
root = ~/myproject/data

[data]
X = raw/X.csv
y = raw/y.pickle

Then fyda will keep the shortcuts you assigned, but reassign any others that it finds:

>>> X_arr = fyda.load('X.npy')
>>> y_arr = fyda.load('y.npy')
>>> X_raw = fyda.load('X')
>>> y_raw = fyda.load('y')

To complicate matters further, what if we now had a file structure like this:

.
├── data
│  ├── arrays
│  │   ├── X.csv
│  │   └── y.npy
│  └── raw
│      ├── X.csv
│      └── y.pickle
└ .fydarc

Now the X data file has the same name in both folders, so just adding the extension won’t do. Now, fyda has to add the containing folders’ names to gain uniqueness:

>>> X_arr = fyda.load('arrays/X.csv')
>>> y_arr = fyda.load('y.npy')
>>> X_raw = fyda.load('raw/X.csv')
>>> y_raw = fyda.load('y.pickle')

Note that the y files are still only separated by file extensions.

Static Configuration with .fydarc

It almost never happens that our data files have a nice, neat name like X.csv, but instead have some god-awful collection of names, words, random capitalization, a mix of underscores and spaces, dates and versions … you get the idea. Let’s say I receive data from a client in the form of the DATA_for-project_v100_20190130.xlsx.

This would quickly get tiring if we were to try and use it in the shortcut system above. Fortunately, fyda has a way to set static shortcuts, that can be managed in a single place, called your .fydarc. This is a project-specific file, and the path to it can be dynamically set using fyda.options.CONFIG_LOCATION = path/to/my/.fydarc. The fydarc paradigm uses YAML as its markup language, making it very easy for even non-programmers to use. To get started with the poorly named file from the client, we need to tell fyda two things in the fydarc, namely where the data root should be anchored, and where the data is relative to this root. For now, let’s assume we have the following structure in the same working directory as earlier:

.
├── data
│  ├── preprocessed
│  └── raw
│      └── the DATA_for-project_v100_20190130.xlsx
└ .fydarc

Ultimately, your pipeline will probably do some transformations and data cleaning on the raw data and place it in the preprocessed folder, at which point you can name it anything you like, but since this file is from the client directly, it’s probably best not to rename it for consistency’s sake. We can, however, make the shortcut to this file with fyda whatever we like. In our .fydarc, we simply do the following:

directories:
  root: ./data

data:
  client: raw/the DATA_for-project_v100_20190130.xlsx

Now when fyda is booted up in python, the shortcut "client" will be available for loading the client data. If in the future the client sends a new file called the DATA_for-project_v101_20190229.xlsx, then adjusting the code to load this new file instead of the old one is as simple as changing the single configuration value in .fydarc.

Peeking under the hood: the DataBank

< Still under construction >

Connecting to and using fyda with Amazon S3

< Still under construction >