• ??絕大多數(shù)數(shù)據(jù)科學(xué)技術(shù)并不喜歡清理和整理數(shù)據(jù)；
• ??只有20%的時(shí)間是在做有用的分析；
• ??數(shù)據(jù)質(zhì)量問題如果不盡早處理，將會(huì)產(chǎn)生級(jí)聯(lián)現(xiàn)象并影響后續(xù)工作。

1. ??加載一個(gè)混亂的數(shù)據(jù)集；
2. ??分析數(shù)據(jù)質(zhì)量問題；
3. ??進(jìn)一步挖掘警告信息；
4. ??應(yīng)用策略來減輕這些問題；
5. ??檢查在半清洗過后的數(shù)據(jù)的最終質(zhì)量分析報(bào)告。

pip install?ydata-quality

from?ydata_quality import?DataQualityimport pandas as?pddf = pd.read_csv('../datasets/transformed/census_10k.csv')

# create the main class that holds all quality modulesdq = DataQuality(df=df)

# run the testsresults = dq.evaluate()

Warnings: TOTAL: 5 warning(s) Priority 1: 1 warning(s) Priority 2: 4 warning(s)

Priority 1?- heavy impact expected: * [DUPLICATES - DUPLICATE COLUMNS] Found 1?columns with exactly the same feature values?as?other columns.Priority 2?- usage allowed, limited human intelligibility: * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3?numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make?model explainability harder and?potentially give way to?issues like overfitting. Depending on?your end goal you might want to?remove?the highest VIF variables. * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1960?ED values?in the dataset. * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 10?categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and?sorted descending according to?propensity. These will make?model explainability harder and?potentially give way to?issues like overfitting. Depending on?your end goal you might want to?remove?variables following the provided order. * [DUPLICATES - EXACT DUPLICATES] Found 3?instances with exact duplicate feature values.

• 警告（Warning）：其中包括數(shù)據(jù)質(zhì)量分析過程中檢測到的問題細(xì)節(jié)。

• 優(yōu)先級(jí)（Priority）：對(duì)每一個(gè)檢測到的問題，基于該問題預(yù)期的影響來分配一個(gè)優(yōu)先級(jí)（越低的值表明越高的優(yōu)先性）。

• 模塊（Modules）：每個(gè)檢測到的問題與某一個(gè)模塊（例如：數(shù)據(jù)關(guān)系，重復(fù)值，等）執(zhí)行的數(shù)據(jù)質(zhì)量檢驗(yàn)相關(guān)聯(lián)。

dq.get_warnings(test="DuplicateColumns")

[QualityWarning(category='Duplicates', test='Duplicate Columns', description='Found 1 columns with exactly the same feature values as other columns.', priority=, data={'workclass': ['workclass2']})]

from ydata_quality.bias_fairness import BiasFairness

#create the main class that holds all quality modulesbf = BiasFairness(df=df, sensitive_features=['race', 'sex'], label='income')

#?run the testsbf_results = bf.evaluate()

Warnings: TOTAL: 2 warning(s) Priority 2: 2 warning(s)

Priority 2?- usage allowed, limited human intelligibility: * [BIAS&FAIRNESS - PROXY IDENTIFICATION] Found 1?feature pairs of?correlation to sensitive attributes with?values higher than defined threshold (0.5). * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2?values of?'race'?sensitive attribute with?low representativity in?the dataset (below 1.00%).

bf.get_warnings(test='Proxy Identification')

[QualityWarning(category='Bias&Fairness', test='Proxy Identification', description='Found 1?feature pairs of correlation to sensitive attributes with values higher than defined threshold (0.5).', priority=2>, data=features

relationship_sex 0.650656

Name: association, dtype: float64)]

def?improve_quality(df: pd.DataFrame):????"""Clean the data based on the Data Quality issues found previously."""????# Bias & Fairness df = df.replace({'relationship': {'Husband': 'Married', 'Wife': 'Married'}}) # Substitute gender-based 'Husband'/'Wife' for generic 'Married'

???# Duplicates df = df.drop(columns=['workclass2']) # Remove the duplicated column df = df.drop_duplicates() # Remove exact feature value duplicates

???return?df

clean_df = improve_quality(df.copy())

*DataQuality Engine Report:*

Warnings:??TOTAL: 3?warning(s) Priority 2: 3?warning(s)


Priority 2?- usage allowed, limited human intelligibility: * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1360?ED values?in the dataset. * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3?numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make?model explainability harder and?potentially give way to?issues like overfitting. Depending on?your end goal you might want to?remove?the highest VIF variables. * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 9?categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and?sorted descending according to?propensity. These will make?model explainability harder and?potentially give way to?issues like overfitting. Depending on?your end goal you might want to?remove?variables following the provided order.

*Bias & Fairness Report:*

Warnings:??TOTAL: 1?warning(s) Priority 2: 1?warning(s)


Priority 2?- usage allowed, limited human intelligibility: * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2?values?of 'race'?sensitive attribute with low representativity in the dataset (below 1.00%).