Data-Science-For-Beginners/2-Working-With-Data/07-python/notebook.ipynb

Basic Pandas Examples

This notebook will walk you through some very basic Pandas concepts. We will start with importing typical data science libraries:

In [36]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

Series

Series is like a list or 1D-array, but with index. All operations are index-aligned.

In [37]:

a = pd.Series(range(1,10))
b = pd.Series(["I","like","to","use","Python","and","Pandas","very","much"],index=range(0,9))
print(a,b)

0    1
1    2
2    3
3    4
4    5
5    6
6    7
7    8
8    9
dtype: int64 0         I
1      like
2        to
3       use
4    Python
5       and
6    Pandas
7      very
8      much
dtype: object

One of the frequent usage of series is time series. In time series, index has a special structure - typically a range of dates or datetimes. We can create such an index with pd.date_range.

Suppose we have a series that shows the amount of product bought every day, and we know that every sunday we also need to take one item for ourselves. Here is how to model that using series:

In [38]:

start_date = "Jan 1, 2020"
end_date = "Dec 31, 2020"
idx = pd.date_range(start_date,end_date)
print(f"Length of index is {len(idx)}")
items_sold = pd.Series(np.random.randint(25,50,size=len(idx)),index=idx)
items_sold.plot(figsize=(10,3))
plt.show()

Length of index is 366

In [39]:

additional_items = pd.Series(10,index=pd.date_range(start_date,end_date,freq="W"))
print(f"Additional items (10 item each week):\n{additional_items}")
total_items = items_sold+additional_items
print(f"Total items (sum of two series):\n{total_items}")

Additional items (10 item each week):
2020-01-05    10
2020-01-12    10
2020-01-19    10
2020-01-26    10
2020-02-02    10
2020-02-09    10
2020-02-16    10
2020-02-23    10
2020-03-01    10
2020-03-08    10
2020-03-15    10
2020-03-22    10
2020-03-29    10
2020-04-05    10
2020-04-12    10
2020-04-19    10
2020-04-26    10
2020-05-03    10
2020-05-10    10
2020-05-17    10
2020-05-24    10
2020-05-31    10
2020-06-07    10
2020-06-14    10
2020-06-21    10
2020-06-28    10
2020-07-05    10
2020-07-12    10
2020-07-19    10
2020-07-26    10
2020-08-02    10
2020-08-09    10
2020-08-16    10
2020-08-23    10
2020-08-30    10
2020-09-06    10
2020-09-13    10
2020-09-20    10
2020-09-27    10
2020-10-04    10
2020-10-11    10
2020-10-18    10
2020-10-25    10
2020-11-01    10
2020-11-08    10
2020-11-15    10
2020-11-22    10
2020-11-29    10
2020-12-06    10
2020-12-13    10
2020-12-20    10
2020-12-27    10
Freq: W-SUN, dtype: int64
Total items (sum of two series):
2020-01-01     NaN
2020-01-02     NaN
2020-01-03     NaN
2020-01-04     NaN
2020-01-05    54.0
              ... 
2020-12-27    43.0
2020-12-28     NaN
2020-12-29     NaN
2020-12-30     NaN
2020-12-31     NaN
Length: 366, dtype: float64

As you can see, we are having problems here, because in the weekly series non-mentioned days are considered to be missing (NaN), and adding NaN to a number gives us NaN. In order to get correct result, we need to specify fill_value when adding series:

In [40]:

total_items = items_sold.add(additional_items,fill_value=0)
print(total_items)
total_items.plot(figsize=(10,3))
plt.show()

2020-01-01    26.0
2020-01-02    25.0
2020-01-03    37.0
2020-01-04    30.0
2020-01-05    54.0
              ... 
2020-12-27    43.0
2020-12-28    44.0
2020-12-29    36.0
2020-12-30    38.0
2020-12-31    34.0
Length: 366, dtype: float64

In [41]:

monthly = total_items.resample("1M").mean()
ax = monthly.plot(kind='bar',figsize=(10,3))
ax.set_xticklabels([x.strftime("%b-%Y") for x in monthly.index], rotation=45)
plt.show()

DataFrame

A dataframe is essentially a collection of series with the same index. We can combine several series together into a dataframe. Given a and b series defined above:

In [42]:

df = pd.DataFrame([a,b])
df

Out [42]:

	0	1	2	3	4	5	6	7	8
0	1	2	3	4	5	6	7	8	9
1	I	like	to	use	Python	and	Pandas	very	much

We can also use Series as columns, and specify column names using dictionary:

In [43]:

df = pd.DataFrame({ 'A' : a, 'B' : b })
df

Out [43]:

	A	B
0	1	I
1	2	like
2	3	to
3	4	use
4	5	Python
5	6	and
6	7	Pandas
7	8	very
8	9	much

The same result can be achieved by transposing (and then renaming columns, to match the previous example):

In [44]:

pd.DataFrame([a,b]).T.rename(columns={ 0 : 'A', 1 : 'B' })

Out [44]:

	A	B
0	1	I
1	2	like
2	3	to
3	4	use
4	5	Python
5	6	and
6	7	Pandas
7	8	very
8	9	much

Selecting columns from DataFrame can be done like this:

In [45]:

print(f"Column A (series):\n{df['A']}")
print(f"Columns B and A (DataFrame):\n{df[['B','A']]}")

Column A (series):
0    1
1    2
2    3
3    4
4    5
5    6
6    7
7    8
8    9
Name: A, dtype: int64
Columns B and A (DataFrame):
        B  A
0       I  1
1    like  2
2      to  3
3     use  4
4  Python  5
5     and  6
6  Pandas  7
7    very  8
8    much  9

Selecting rows based on filter expression:

In [46]:

df[df['A']<5]

Out [46]:

	A	B
0	1	I
1	2	like
2	3	to
3	4	use

The way it works is that expression df['A']<5 returns a boolean series, which indicates whether expression is True or False for each elements of the series. When series is used as an index, it returns subset of rows in the DataFrame. Thus it is not possible to use arbitrary Python boolean expression, for example, writing df[df['A']>5 and df['A']<7] would be wrong. Instead, you should use special & operation on boolean series:

In [47]:

df[(df['A']>5) & (df['A']<7)]

Out [47]:

	A	B
5	6	and

Creating new computable columns. We can easily create new computable columns for our DataFrame by using intuitive expressions. The code below calculates divergence of A from its mean value.

In [48]:

df['DivA'] = df['A']-df['A'].mean()
df

Out [48]:

	A	B	DivA
0	1	I	-4.0
1	2	like	-3.0
2	3	to	-2.0
3	4	use	-1.0
4	5	Python	0.0
5	6	and	1.0
6	7	Pandas	2.0
7	8	very	3.0
8	9	much	4.0

What actually happens is we are computing a series, and then assigning this series to the left-hand-side, creating another column.

In [49]:

# WRONG: df['ADescr'] = "Low" if df['A'] < 5 else "Hi"
df['LenB'] = len(df['B']) # Wrong result

In [50]:

df['LenB'] = df['B'].apply(lambda x: len(x))
# or
df['LenB'] = df['B'].apply(len)
df

Out [50]:

	A	B	DivA	LenB
0	1	I	-4.0	1
1	2	like	-3.0	4
2	3	to	-2.0	2
3	4	use	-1.0	3
4	5	Python	0.0	6
5	6	and	1.0	3
6	7	Pandas	2.0	6
7	8	very	3.0	4
8	9	much	4.0	4

Selecting rows based on numbers can be done using iloc construct. For example, to select first 5 rows from the DataFrame:

In [52]:

df.iloc[:5]

Out [52]:

	A	B	DivA	LenB
0	1	I	-4.0	1
1	2	like	-3.0	4
2	3	to	-2.0	2
3	4	use	-1.0	3
4	5	Python	0.0	6

Grouping is often used to get a result similar to pivot tables in Excel. Suppose that we want to compute mean value of column A for each given number of LenB. Then we can group our DataFrame by LenB, and call mean:

In [53]:

df.groupby(by='LenB').mean()

Out [53]:

	A	DivA
LenB
1	1.000000	-4.000000
2	3.000000	-2.000000
3	5.000000	0.000000
4	6.333333	1.333333
6	6.000000	1.000000

If we need to compute mean and the number of elements in the group, then we can use more complex aggregate function:

In [58]:

df.groupby(by='LenB') \
 .aggregate({ 'DivA' : len, 'A' : lambda x: x.mean() }) \
 .rename(columns={ 'DivA' : 'Count', 'A' : 'Mean'})

Out [58]:

	Count	Mean
LenB
1	1	1.000000
2	1	3.000000
3	2	5.000000
4	3	6.333333
6	2	6.000000

Printing and Plotting

Data Scientist often has to explore the data, thus it is important to be able to visualize it. When DataFrame is big, many times we want just to make sure we are doing everything correctly by printing out the first few rows. This can be done by calling df.head(). If you are running it from Jupyter Notebook, it will print out the DataFrame in a nice tabular form.

In [59]:

df.head()

Out [59]:

	A	B	DivA	LenB
0	1	I	-4.0	1
1	2	like	-3.0	4
2	3	to	-2.0	2
3	4	use	-1.0	3
4	5	Python	0.0	6

We have also seen the usage of plot function to visualize some columns. While plot is very useful for many tasks, and supports many different graph types via kind= parameter, you can always use raw matplotlib library to plot something more complex. We will cover data visualization in detail in separate course lessons.

In [61]:

df['A'].plot()
plt.show()

In [63]:

df['A'].plot(kind='bar')
plt.show()

This overview covers most important concepts of Pandas, however, the library is very rich, and there is no limit to what you can do with it! Let's now apply this knowledge for solving specific problem.