数据科学家发展前景_如何成为数据科学家

前面有文章数据分析之探索性数据分析，里面详细阐述了何为EDA，以及一些常用的分析方法，感兴趣的小伙伴们可以参考下。

本文分析数据科学家更换工作情况数据集，运用常见EDA方法分析每个特征情况及他们与目标变量之间的关系。使用seaborn进行数据可视化辅助分析数据科学家们更换工作都有哪些特征。

因篇幅过长，将分为上下两篇展开此次探索性数据分析案例精讲。

导包

import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns plt.rcParams['font.sans-serif'] = ['SimHei'] plt.rcParams['axes.unicode_minus'] = False  

数据导入与查看

path_train = '../input/hr-analytics-job-change-of-data-scientists/aug_train.csv' path_test = '../input/hr-analytics-job-change-of-data-scientists/aug_test.csv' path_submission = '../input/hr-analytics-job-change-of-data-scientists/sample_submission.csv' 

train = pd.read_csv(path_train) test = pd.read_csv(path_test) 

train.head() 

test.sample(5) # 表示返回随机5行数据。 

探索性数据分析

一般性分析

train.shape 

(19158, 14) 

test.shape 

(2129, 13) 

train.dtypes 

enrollee_id int64 city object city_development_index float64 gender object relevent_experience object enrolled_university object education_level object major_discipline object experience object company_size object company_type object last_new_job object training_hours int64 target float64 dtype: object 

test.info() 

<class 'pandas.core.frame.DataFrame'> RangeIndex: 2129 entries, 0 to 2128 Data columns (total 13 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 enrollee_id 2129 non-null int64 1 city 2129 non-null object 2 city_development_index 2129 non-null float64 3 gender 1621 non-null object 4 relevent_experience 2129 non-null object 5 enrolled_university 2098 non-null object 6 education_level 2077 non-null object 7 major_discipline 1817 non-null object 8 experience 2124 non-null object 9 company_size 1507 non-null object 10 company_type 1495 non-null object 11 last_new_job 2089 non-null object 12 training_hours 2129 non-null int64 dtypes: float64(1), int64(2), object(10) memory usage: 216.4+ KB 

list(train.columns) 

['enrollee_id', 'city', 'city_development_index', 'gender', 'relevent_experience', 'enrolled_university', 'education_level', 'major_discipline', 'experience', 'company_size', 'company_type', 'last_new_job', 'training_hours', 'target'] 

describe方法用于生成DataFramem描述统计信息。可以很方便的查看数据集的分布情况。注意，这里的统计分布包含NaN值。

• count表示计数。

• mean表示平均值。

• std表示标准差。

• min表示最小值。

• 25%表示四分之一分位数

• 50%表示二分之一分位数。

• 75%表示四分之三分位数。

• max表示最大值。

train.describe(include='all').T 

缺失值分析

def percentage_nulls(df):     number_nulls = pd.DataFrame(df.isnull().sum(),columns=['Total'])     number_nulls['% nulls'] = round((number_nulls['Total'] / df.shape[0])*100,1)          return number_nulls 

训练集的缺失值。

nulls_train = percentage_nulls(train) nulls_train 

	Total	% nulls
enrollee_id	0	0.0
city	0	0.0
city_development_index	0	0.0
gender	4508	23.5
relevent_experience	0	0.0
enrolled_university	386	2.0
education_level	460	2.4
major_discipline	2813	14.7
experience	65	0.3
company_size	5938	31.0
company_type	6140	32.0
last_new_job	423	2.2
training_hours	0	0.0
target	0	0.0

# 选择缺失数量最大的5行数据。 nulls_train.nlargest(5, 'Total') 

	Total	% nulls
company_type	6140	32.0
company_size	5938	31.0
gender	4508	23.5
major_discipline	2813	14.7
education_level	460	2.4

percentage_nulls(test) 

	Total	% nulls
enrollee_id	0	0.0
city	0	0.0
city_development_index	0	0.0
gender	508	23.9
relevent_experience	0	0.0
enrolled_university	31	1.5
education_level	52	2.4
major_discipline	312	14.7
experience	5	0.2
company_size	622	29.2
company_type	634	29.8
last_new_job	40	1.9
training_hours	0	0.0

缺失值的可视化

import missingno as msno 

msno.matrix(train) 

测试集的缺失值。

msno.matrix(test) 

为了后面分析方便，将缺失值用字符串'Unknown'填充。

df_train = train.fillna("Unknown") # 填充完后检查下是否处理完毕。 df_train.isnull().sum() 

enrollee_id 0 city 0 city_development_index 0 gender 0 relevent_experience 0 enrolled_university 0 education_level 0 major_discipline 0 experience 0 company_size 0 company_type 0 last_new_job 0 training_hours 0 target 0 dtype: int64 缺失值分析可以参见缺失值处理，你真的会了吗？ 

分析每个特征

特征 'City' -- 城市

# 切片索引出'City' city_train = df_train['city'] city_train.value_counts() 

city_103 4355 city_21 2702 city_16 1533 city_114 1336 city_160 845 ... city_129 3 city_111 3 city_121 3 city_140 1 city_171 1 Name: city, Length: 123, dtype: int64 

city_train.value_counts().plot() 

city_test = df_test['city'] city_test.value_counts() 

city_103 473 city_21 318 city_16 168 city_114 155 city_160 113 ... city_84 1 city_171 1 city_25 1 city_93 1 city_141 1 Name: city, Length: 108, dtype: int64 

观察到，城市是用数字编码的，训练集用123个不同的城市，测试集用108个不同的城市。

'city_ development _index' 城市发展指数

可以查看维基百科中的名词解释 https://en.wikipedia.org/wiki/City_development_index

# 查看特征分布状况 sns.displot(data=df_train,              x='city_development_index',             height=6,             aspect = 2,                         color = 'lightblue') sns.set_context(font_scale=1)  

with sns.axes_style():     g = sns.displot(data=df_test,              x='city_development_index',             height=5,             aspect = 2,              color = 'coral')     g.set_ylabels(fontsize=15)     g.set_xlabels(fontsize=15)     g.set_xticklabels(fontsize=15)     g.set_yticklabels(fontsize=15) 

注意这种写法，设置x,y轴标签字体大小。 

'gender'性别

性别及后面几个特征为分类型变量，因此分别统计每个变量的总数及占比。

gender_train = df_train['gender'] gender_test = df_test['gender'] 

后面需要多次用到缺失值统计，因此将其定义为函数。

def percent_nan(df):     number = pd.DataFrame(df.value_counts())     number.columns = ['Total']     number['%'] = round((number['Total'] / df.notnull().sum())*100,1)          return number 

percent_nan(gender_train) 

	Total	%
Male	13221	69.0
Unknown	4508	23.5
Female	1238	6.5
Other	191	1.0

percent_nan(gender_test) 

	Total	%
Male	1460	68.6
Unknown	508	23.9
Female	137	6.4
Other	24	1.1

两个数据集的缺失值差不多，因此对两者可视化，以更加方便看出其差异。

同样，为后续分析方便，将其定义为函数。

def draw_countplot(feature,palette,order=None):     fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 5))     sns.countplot(x=feature,data=df_train,palette=palette,ax=ax[0],order=order).set_title('Train')     sns.countplot(x=feature,data=df_test,palette=palette,ax=ax[1],order=order).set_title('Test')     fig.tight_layout() 

draw_countplot("gender","Set1") 

'relevent_experience' 相关经验

relevent_experience_train = df_train['relevent_experience'] relevent_experience_test = df_test['relevent_experience'] 

percent_nan(relevent_experience_train) 

	Total	%
Has relevent experience	13792	72.0
No relevent experience	5366	28.0

percent_nan(relevent_experience_test) 

	Total	%
Has relevent experience	1524	71.6
No relevent experience	605	28.4

draw_countplot("relevent_experience","Set2") 

'enrolled_university' 专业课

已注册的大学课程类型(如有)。

enrolled_university_train = df_train['enrolled_university'] enrolled_university_test = df_test['enrolled_university'] 

percent_nan(enrolled_university_train) 

	Total	%
no_enrollment	13817	72.1
Full time course	3757	19.6
Part time course	1198	6.3
Unknown	386	2.0

percent_nan(enrolled_university_test) 

	Total	%
no_enrollment	1519	71.3
Full time course	435	20.4
Part time course	144	6.8
Unknown	31	1.5

order_enrolled_university = percent_nan(enrolled_university_train).index order_enrolled_university 

Index(['no_enrollment', 'Full time course', 'Part time course', 'Unknown'], dtype='object') 

draw_countplot('enrolled_university',"Set3",order_enrolled_university) 

'education_level' 学历

education_level_train = df_train['education_level'] education_level_test = df_test['education_level'] 

percent_nan(education_level_train) 

	Total	%
Graduate	11598	60.5
Masters	4361	22.8
High School	2017	10.5
Unknown	460	2.4
Phd	414	2.2
Primary School	308	1.6

percent_nan(education_level_test) 

	Total	%
Graduate	1269	59.6
Masters	496	23.3
High School	222	10.4
Phd	54	2.5
Unknown	52	2.4
Primary School	36	1.7

order_education_level = percent_nan(education_level_train).index 

draw_countplot('education_level',"Set1",order_education_level) 

'major_discipline' 专业学科

major_discipline_train = df_train['major_discipline'] major_discipline_test = df_test['major_discipline'] 

percent_nan(major_discipline_train) 

	Total	%
STEM	14492	75.6
Unknown	2813	14.7
Humanities	669	3.5
Other	381	2.0
Business Degree	327	1.7
Arts	253	1.3
No Major	223	1.2

percent_nan(major_discipline_test) 

	Total	%
STEM	1621	76.1
Unknown	312	14.7
Humanities	80	3.8
Other	40	1.9
Business Degree	37	1.7
No Major	22	1.0
Arts	17	0.8

order_major_discipline = percent_nan(major_discipline_train).index 

draw_countplot('major_discipline',"Set2",order_major_discipline) 

'experience' 工作年限

experience_train = df_train['experience'] experience_test = df_test['experience'] 

percent_nan(experience_train) 

	Total	%
>20	3286	17.2
5	1430	7.5
4	1403	7.3
3	1354	7.1
6	1216	6.3
2	1127	5.9
7	1028	5.4
10	985	5.1
9	980	5.1
8	802	4.2
15	686	3.6
11	664	3.5
14	586	3.1
1	549	2.9
<1	522	2.7
16	508	2.7
12	494	2.6
13	399	2.1
17	342	1.8
19	304	1.6
18	280	1.5
20	148	0.8
Unknown	65	0.3

percent_nan(experience_test) 

	Total	%
>20	383	18.0
5	163	7.7
3	154	7.2
4	145	6.8
6	130	6.1
2	128	6.0
7	116	5.4
9	113	5.3
10	96	4.5
11	86	4.0
8	82	3.9
<1	74	3.5
16	68	3.2
15	59	2.8
1	56	2.6
14	55	2.6
13	54	2.5
12	52	2.4
17	36	1.7
19	29	1.4
18	26	1.2
20	19	0.9
Unknown	5	0.2

order_experience = percent_nan(experience_train).index 

draw_countplot('experience',"Set3",order_experience) 

'company_size'公司规模

company_size_train = df_train['company_size'] company_size_test = df_test['company_size'] 

percent_nan(company_size_train) 

	Total	%
Unknown	5938	31.0
50-99	3083	16.1
100-500	2571	13.4
10000+	2019	10.5
10/49	1471	7.7
1000-4999	1328	6.9
<10	1308	6.8
500-999	877	4.6
5000-9999	563	2.9

percent_nan(company_size_test) 

	Total	%
Unknown	622	29.2
50-99	338	15.9
100-500	318	14.9
10000+	217	10.2
10/49	172	8.1
<10	163	7.7
1000-4999	143	6.7
500-999	88	4.1
5000-9999	68	3.2

order_company_size = percent_nan(company_size_train).index 

draw_countplot('company_size',"Set1",order_company_size) 

'company_type'公司类型

company_type_train = df_train['company_type'] company_type_test = df_test['company_type'] 

percent_nan(company_type_train) 

	Total	%
Pvt Ltd	9817	51.2
Unknown	6140	32.0
Funded Startup	1001	5.2
Public Sector	955	5.0
Early Stage Startup	603	3.1
NGO	521	2.7
Other	121	0.6

percent_nan(company_type_test) 

	Total	%
Pvt Ltd	1141	53.6
Unknown	634	29.8
Public Sector	127	6.0
Funded Startup	97	4.6
Early Stage Startup	65	3.1
NGO	53	2.5
Other	12	0.6

order_company_type = percent_nan(company_type_train).index 

draw_countplot('company_type',"Set2",order_company_type) 

'lastnewjob' 以前的工作与现在的工作的年差

last_new_job_train = df_train['last_new_job'] last_new_job_test = df_test['last_new_job'] 

percent_nan(last_new_job_train) 

	Total	%
1	8040	42.0
>4	3290	17.2
2	2900	15.1
never	2452	12.8
4	1029	5.4
3	1024	5.3
Unknown	423	2.2

percent_nan(last_new_job_test) 

	Total	%
1	884	41.5
>4	353	16.6
2	342	16.1
never	258	12.1
3	133	6.2
4	119	5.6
Unknown	40	1.9

order_last_new_job = percent_nan(last_new_job_train).index 

draw_countplot('last_new_job',"Set3",order_last_new_job) 

'training_hours'已完成的培训时长

sns.displot(data=df_train,              x='training_hours',             height=5,             aspect=1.5,             color = 'lightblue') 

sns.displot(data=df_test,              x='training_hours',             height=5,             aspect=1.5,             color = 'coral') 

'Target'

0 - Not looking for job change
1 - Looking for a job change

target = df_train['target'] percent_nan(target) 

	Total	%
0.0	14381	75.1
1.0	4777	24.9

sns.countplot(x='target',               data=df_train,               palette="Set1").set_title('Train') 

本文数据使用 kaggle数据集，关注公众号，并回复【 hr 】获取数据。

推荐阅读

Python数据分析之探索性数据分析(EDA)

缺失值处理，你真的会了吗？

-- 数据STUDIO --