Data 3-1 Pandas Basics

Data 3-1 Pandas Basics#

series, data frame, columns, index, numpy types, Nan, slicing

What is Pandas?#

Pandas is a Python library for working with tabular data. Pandas is short for PANeled DAta.

Pandas is like a programmable spreadheet. It is used by programmers to wrangle data (sort, filter, clean, enhance, etc.)

Pandas Series and DataFrame#

The two fundamental compoents of Pandas are the Series and DataFrame

a Series is a list of values with labels. This creates a column of data
a DataFrame is a collection of series. This creates a table of data

Null / No Value#

The constant np.nan is used to represent “no value”

import pandas as pd
import numpy as np

Series#

A Series is a named list of values.

The series has an index, too to reference each value. The default index is a zero based, similar to a python list.

grades = pd.Series(data=[100,80,90,np.nan,100], name="Midterm Grades")
grades

  100.0
   80.0
   90.0
    NaN
  100.0
Name: Midterm Grades, dtype: float64

# The the value at index 2
grades[2]

90.0

The index can be anyting . Here’s the same grades with student names as the index.

grades2 = pd.Series( data=[100,80,90,np.nan,100], 
                    name="Midterm Grades",
                    index=["Alice", "Bob", "Charlie", "David", "Eve"])
grades2

Alice      100.0
Bob         80.0
Charlie     90.0
David        NaN
Eve        100.0
Name: Midterm Grades, dtype: float64

# Get Charlie's grade
grades2["Charlie"]

90.0

Series Aggregate Functions#

The beauty of Pandas is that for most operations, you won’t even need a loop! For example, we derive the min(), max(), mean(), sum() and count() of non np.nan values in a series without a for loop!!!

We do this by calling those method functions on the series itself, for example:

print("Highest grade:", grades.max())
print("Average grade:", grades.mean())
print("lowest grade:", grades.min())
print("Sum of grades:", grades.sum())
print("Count of grades", grades.count())

Highest grade: 100.0
Average grade: 92.5
lowest grade: 80.0
Sum of grades: 370.0
Count of grades 4

Other Series Functions#

We use the unique() method function to return only the non-duplicate values from the series.

The value_counts() method function adds up values, creating a new series where the index is the value and the value is the count.

For example consider the following series:

votes = pd.Series(data=[ 'y','y','y','n','y',np.nan,'n','n','y'], name="Vote")
print("deduplicate the votes:", votes.unique())
print("counts by value:", votes.value_counts())

deduplicate the votes: ['y' 'n' nan]
counts by value: Vote
y    5
n    3
Name: count, dtype: int64

DataFrame#

The pandas DataFrame is a table representation of data. It is the primary use case for pandas itself. A dataframe is simply a collection of Series that share a common Index. I like to think of the DataFrame as a programmable spreadsheet. It has rows and columns which can be accessed and manipulated with Python.

names = pd.Series( data = ['Allen','Bob','Chris','Dave','Ed','Frank','Gus'])
gpas = pd.Series( data = [4.0, np.nan, 3.4, 2.8, 2.5, 3.8, 3.0])
years = pd.Series( data = ['So', 'Fr', 'Fr', 'Jr', 'Sr', 'Sr', 'Fr'])
series_dict = { 'Name':  names, 'GPA': gpas, 'Year' : years }  # dict of Series, keys are the series names
students = pd.DataFrame( series_dict )
students

	Name	GPA	Year
0	Allen	4.0	So
1	Bob	NaN	Fr
2	Chris	3.4	Fr
3	Dave	2.8	Jr
4	Ed	2.5	Sr
5	Frank	3.8	Sr
6	Gus	3.0	Fr

Other Ways to create dataframes:#

https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html#dataframe

Lists of lists: pd.DataFrame([['Tom', 7], ['Mike', 15], ['Tiffany', 3]])

Dictionary pd.DataFrame({"Name": ['Tom', 'Mike', 'Tiffany'], "Number": [7, 15, 3]})

Series pd.DataFrame({"Name": pd.Series(['Tom', 'Mike', 'Tiffany']), "Number": pd.Series([7, 15, 3])})

DataFrames share the index#

The dataframe is stitched together from values macthing on their index. For example:

gpas = pd.Series(data=[4.0, np.nan, 3.4, 2.8, 2.5 ], index=['Allen','Bob','Chris','Ed', 'Frank'])
yrs = pd.Series(data=['So', 'Fr', 'Jr', 'Sr'], index=['Allen','Bob','Dave', 'Frank'])
studentsn = pd.DataFrame( {'GPA': gpas, 'Year': yrs})
studentsn

	GPA	Year
Allen	4.0	So
Bob	NaN	Fr
Chris	3.4	NaN
Dave	NaN	Jr
Ed	2.8	NaN
Frank	2.5	Sr

Accessing elements with loc and iloc#

The loc[index, col] and iloc[row_pos, col_pos] properties allow you to slice the dataframe. loc uses the index and column names, while iloc uses ordinal positions starting at zero.

Here are some examples, using studentsn

# Examples using loc
print("loc: Get the Chris' GPA: ", students.loc['Chris', 'GPA'])
print("loc: Get the Year of the last student (Frank): ", students.loc['Frank', 'Year'])

# Same examples using iloc
print("iloc: Get the GPA of the student at row 2 (Chris): ", students.iloc[2, 0])
print("iloc: Get the Year of the last student (Frank): ", students.iloc[-1, 1])

loc: Get the Chris' GPA:  3.4
loc: Get the Year of the last student (Frank):  Sr
iloc: Get the GPA of the student at row 2 (Chris):  3.4
iloc: Get the Year of the last student (Frank):  Sr

# You can also slice using loc and iloc
print("loc: last two rows:\n", students.loc['Ed':, 'GPA':'Year'])
print()
print("iloc: last two rows:\n", students.iloc[-2:, 0:2])

loc: last two rows:
        GPA Year
Ed     2.8  NaN
Frank  2.5   Sr

iloc: last two rows:
        GPA Year
Ed     2.8  NaN
Frank  2.5   Sr

Null Checks#

use isna() to check for np.nan.

students[students.GPA.isna()]

	GPA	Year
Bob	NaN	Fr
Dave	NaN	Jr

Challenge 3-1-1#

Create this dataframe:

   s1   s2 s3
a   1  2.2  q
b   2  NaN  q
c   3  3.0  z
d   4  1.5  z

Use series to create it to make sure the index is correct. print() the dataframe

Then print() the first 2 rows and columns using loc or iloc

Basic Dataframe operations#

info() provide names of columns, counts of non-null values in each columns, and data types.
describe() for each numerical column provide some basic statistics (min, max, mean, and quartiles).
head(n=5) view the FIRST n rows in the dataframe (defaults to 5)
tail(n=5) view the LAST n rows in the dataframe (defaults to 5)
sample(n=1) view a random n rows from the dataframe (defautls to 1)
.columns retrieve a list of columns in the dataframe

customers = pd.read_csv('https://raw.githubusercontent.com/mafudge/datasets/master/customers/customers.csv')
len(customers)

Display the dataframe in Streamlit#

You can use the st.dataframe() function to display a dataframe in Streamlit.

Here is an example: 3-1-streamlit-dataframe.py

Challenge 3-1-2#

Similar to the previous example, load this file into a customers dataframe:

https://raw.githubusercontent.com/mafudge/datasets/master/customers/customers.csv

Then create a radio widget to allow the user to select Head or Tail and a number input widget to enter a number of lines

output the head or tail of the dataframe and only show the number of lines input

Selecting Rows and Columns#

We can pair down the output of a dataframe by using:

a list of column names to select columns.
a boolean index to select matching rows.

data_dict = { 
    'Name':  ['Allen','Bob','Chris','Dave','Ed','Frank','Gus'], 
    'GPA': [4.0, np.nan, 3.4, 2.8, 2.5, 3.8, 3.0], 
    'Year' : ['So', 'Fr', 'Fr', 'Jr', 'Sr', 'Sr', 'Fr'] } 
students = pd.DataFrame( data_dict )
students

	Name	GPA	Year
0	Allen	4.0	So
1	Bob	NaN	Fr
2	Chris	3.4	Fr
3	Dave	2.8	Jr
4	Ed	2.5	Sr
5	Frank	3.8	Sr
6	Gus	3.0	Fr

Selecting Columns#

This example just gets the name and GPA columns

columns_to_show = ['Name', 'GPA']
students[columns_to_show]

	Name	GPA
0	Allen	4.0
1	Bob	NaN
2	Chris	3.4
3	Dave	2.8
4	Ed	2.5
5	Frank	3.8
6	Gus	3.0

Getting the freshmen using a boolean index#

consider the following:

students['Year'] == 'Fr'

  False
   True
   True
  False
  False
  False
   True
Name: Year, dtype: bool

This it called a boolean index. The boolean expression is evaluted for each index in the dataframe.

When we apply the boolean index to the dataframe, only the rows where the index == True are returned.

students[students['Year'] == 'Fr'] 

	Name	GPA	Year
1	Bob	NaN	Fr
2	Chris	3.4	Fr
6	Gus	3.0	Fr

Likewise we can assign these variables for clarity

only_freshmen_index = students['Year'] == 'Fr'
only_freshmen = students[only_freshmen_index]
only_freshmen

	Name	GPA	Year
1	Bob	NaN	Fr
2	Chris	3.4	Fr
6	Gus	3.0	Fr

And Or and Not with Boolean indexes#

What if we want freshmen or seniors? We cannot use or in this case, instead we must use the python bitwise or operator. This is because the series contains multiple values.

Bitwise Operators

and &
or |
not ~

Note: () are required between each bitwise operator.

# freshmen and seniors
only_freshmen_seniors = (students['Year'] == 'Fr') | (students['Year'] == 'Sr')
students[only_freshmen_seniors]

	Name	GPA	Year
1	Bob	NaN	Fr
2	Chris	3.4	Fr
4	Ed	2.5	Sr
5	Frank	3.8	Sr
6	Gus	3.0	Fr

Putting it Together#

Get the name and GPA of onlt freshmen with a GPA

cols = ['Name', 'GPA']
fr_with_gpa = (students['Year'] == 'Fr') & (students['GPA'].notna())
students[fr_with_gpa][cols]

	Name	GPA
2	Chris	3.4
6	Gus	3.0

Challenge 3-1-3#