2.1.1. Missing Data#

Missing data occurs commonly in many data analysis applications. One of the goals of pandas is to make working with missing data as painless as possible. For example, all of the descriptive statistics on pandas objects exclude missing data by default.

The way that missing data is represented in pandas objects is somewhat imperfect, but it is sufficient for most real-world use. For data with float64 dtype, pandas uses the floating-point value NaN (Not a Number) to represent missing data.

We call this a sentinel value: when present, it indicates a missing (or null) value.

The isna method gives us a Boolean Series with True where values are null:

data.isna().sum()

id            0
zip           1
phone        27
lat          30
lng          30
type          0
score        15
risk         17
violation    17
dtype: int64

In pandas, missing data is also refered to as NA, which stands for Not Available. In statistics applications, NA data may either be data that does not exist or that exists but was not observed (through problems with data collection, for example). When cleaning up data for analysis, it is often important to do analysis on the missing data itself to identify data collection problems or potential biases in the data caused by missing data.

The built-in Python None value is also treated as NA.

2.1.2. Numerical Features#

Numeric data is data that can be represented as numbers. These variables generally describe some numeric quantity or amount and are also sometimes referred to as “quantitative” variables.

Since numerical features are already represented as numbers, they are already ready to be used in machine learning models and there is no need to encode them.

In the example above, numerical features include zip, phone, lat, lng, score.

data[['zip', 'phone', 'lat', 'lng', 'score']].head()

data[['zip', 'phone', 'score']].head()

data[['lat', 'lng']].head()

2.1.3. Categorical Features#

Categorical data is data that is not numeric. It is often represented as text or a set of text values. These variables generally describe some characteristic or quality of a data unit, and are also sometimes referred to as “qualitative” variables.

data[['type', 'risk', 'violation']].head()

data[['risk']].head()

data['risk_enc'] = data['risk'].replace({'Low Risk': 0, 'Moderate Risk': 1, 'High Risk': 2})
data.head()

2.1.3.2. Nominal Features#

Nominal data is data that is not ordered in any way. For example, the color of a car is nominal data. There is no order to the colors. The colors are not more or less than each other. They are just different.

data[['type', 'violation']].head()

	type	violation
0	Routine - Unscheduled	High risk food holding temperature
1	Complaint	NaN
2	Routine - Unscheduled	Inadequate warewashing facilities or equipment
3	Routine - Unscheduled	Improper food storage
4	Routine - Unscheduled	Unapproved or unmaintained equipment or utensils

2.1.3.2.1. Encoding Nominal Features#

Since nominal features don’t have any order, encoding them requires some creativity. The most common way to encode nominal features is to use a technique called one-hot encoding. One-hot encoding creates a new column for each unique value in the nominal feature. Each new column is a binary feature that indicates whether or not the original observation had that value.

The figure below illustrates how one-hot encoding for a “day” (of the week) column:

pandas has a built-in .get_dummies function for doing this:

pd.get_dummies(data['type']).head()

	Complaint	New Construction	New Ownership	Reinspection/Followup	Routine - Unscheduled
0	0	0	0	0	1
1	0	0	0	0	1
2	1	0	0	0	0
3	0	0	0	0	1
4	0	0	0	0	1