Dootaframe#
Dootaframe is a purePython dataframe library. It is designed to be simple to write, understand and extend. Performance is not the main focus of this project.
This is the worst thing since sliced bread.
Overview of concepts#
Similar to Pandas, we have two main concepts, Series
and DataFrame
.
Series#
A Series
is a onedimensional container of items. While it can hold any type of Python object, it is particulary good at holding numerical data.
If you are familiar with database systems or spreadsheet applications, you might be inclined to store each row of your data inside a Series
, similar to this.
>>> john = Series(["John", "Doe", 26])
>>> jane = Series(["Jane", "Doe", 25])
>>> jack = Series(["Jack", "Daniels", 38])
This would technically work, but it is the wrong approach with this library. What we want to do is to flip this around, so each column is stored in its own Series
, like this.
>>> fnames = Series(["John", "Jane", "Jack"])
>>> lnames = Series(["Doe", "Doe", "Daniels"])
>>> ages = Series([26, 25, 38])
At first glance, this seems a little incovenient.
>>> ages = Series([26, 25, 38])
>>> ages.min
25
>>> ages.max
38
Series Reference#
We can now take a more detailed look at the Series class. As we discussed above, a Series
is a onedimensional collection of elements. When you provide this collection to the Series
class, it adds a bunch of convenient utilities to it.
Series protocol#
As we want to be flexible, there are no checks in dootaframe for particular collections. Instead, we rely on something we call “the Series protocol”. That’s a really fancy name for a trivial concept.
Basically all we require from an input to Series
is that it implements the __len__ and __getitem__ methods. This is a very lax requirement, and it includes basically any collection you might want to use. That includes tuples, lists, numpy arrays etc.
>>> Series((1, 2, 3))
Series(1, 2, 3)
>>> Series([1, 2, 3])
Series(1, 2, 3)
>>> import numpy as np
>>> Series(np.array([1, 2, 3]))
Series(1, 2, 3)
>>> d = {"name": "Leo", "age": 24}
>>> Series(d)
Series('name', 'age')
>>> Series(list(d)).apply(lambda x: d[x])
Series('Leo', 24)
Laziness#
There is no rule against more dynamic structures either. Series
does not load the entire dataset into memory, so you can even use some onthefly collections.
>>> class PowersOfTwo:
... def __len__(self):
... return 9999999999
...
... def __getitem__(self, index):
... return 2 ** index
...
>>> s1 = Series(PowersOfTwo())
>>> s2 = s1.apply(lambda x: x + 123)
>>> s2[50]
1125899906842747
Now, we definitely did not calculate 9999999999 powers of two and then add 123 to them. In dootaframe, a lot of the operations on Series
are lazy. That means until we need the result, or use an operation that requires looking at the whole collection, computation will only happen on the rows that are requested.
Optimized storage#
As we discussed above, the Series
API is very flexible and accepts pretty much anything you give to it. For most use cases, Series
contain a large number of the same data type.
>>> s = Series([1, 2, 3, 4, 5])
>>> s.underlying_storage
[1, 2, 3, 4, 5]
In the normal case, we can see that the input we gave is being stored as a Python list. This is okay for doing small explorations, but it’s not very efficient for storing a lot of data.
In cases like this, we can ask dootaframe to optimize the backing storage of a Series
.
>>> s = Series([1, 2, 3, 4, 5]).optimize_storage
>>> s.underlying_storage
b'\x01\x02\x03\x04\x05'
Instead of a dynamic Python list that has one item for each number, dootaframe was able to convert the backing storage into an immutable byte array. This is okay because it satisfies both the __len__ and the __getitem__ parts of the Series protocol.
That’s not the only storage optimization either. We have the same thing for collections of byte arrays, collections of unsigned 16bit integers and more.
Optimization of byte arrays#
>>> s = Series([b"He", b"ll", b"o wor", b"l", b"d"])
>>> s.underlying_storage
[b'He', b'll', b'o wor', b'l', b'd']
>>> s = s.optimize_storage
>>> s.underlying_storage
<BytesSeriesStorage with 5 items and 11byte buffer>
>>> s.underlying_storage.buf
b'Hello world'
>>> s.underlying_storage.begins
Series(0, 2, 4, 9, 10)
>>> s.underlying_storage.lens
Series(2, 2, 5, 1, 1)
>>> s
Series(b'He', b'll', b'o wor', b'l', b'd')
Instead of a Python list that has 5 buffers in it, dootaframe compacts everything into a single buffer and stores the spans of the individual chunks. The end result is the same though.
API Documentation#
Below is some autogenerated documentation.
Simple dataframe library in Pure Python.
 class dootaframe.Series(s)#

Bases:
object
Onedimensional container of values.
This class includes whatevers.
 apply(func: Callable) dootaframe.Series #

Apply a function to each member of the series.
 Parameters

 funcCallable

The function whose output will be used as the new value.
 Returns

 Series

The new series.
Examples
>>> s = Series([1, 2, 3, 4]) >>> s.apply(lambda x: x + 1) Series(2, 3, 4, 5)
 property as_list: List#

Turn the Series into a Python list.
Examples
>>> s = Series([1, 2, 3, 4]) >>> s.as_list [1, 2, 3, 4]
 property as_numpy#
 property asc: dootaframe.Series#

Sort the items in this series from lowesttohighest.
 concat(other: dootaframe.Series) dootaframe.Series #

Append another series to the end of this one.
 Parameters

 otherSeries

The other series to append to this one.
 Returns

 Series

A new series where other is appended to this one.
 property desc#
 property enumerate#
 property floats#
 property ints#

Convert every item of the Series into an integer.
This uses the native int() method to convert.
 property length: int#

The number of items in this series.
 property max#

The maximum value contained within this Series.
 property mean#
 property median#
 property min#

The minimum value contained within this Series.
 property optimize_storage: dootaframe.Series#

Create an storageoptimized version of this series.
A Series needs to accommodate storing arbitrary data, including data of different types. While this is very flexible, it means that all the bookkeeping data can take a significant amount of memory.
This method tries to optimize certain kinds of storage patterns, and makes them use a lot less memory.
 order_by(fn)#
 order_by_desc(func: Callable)#
 property solidify: dootaframe.Series#
 property sorted#
 property strings#

Turn every item of the Series into a string.
 Returns

 Series

The same series, but every item is converted into a string.
 property sum#
 property underlying_storage#
 property uniq#
 property uniq_count#